MQTT Client Plugin

Not sure I understood what you meant[/quote]
As you talked about Node-RED, I thought that you had tested MQTT with it.

Well I do, that’s pretty much the only reason why I’ve made this plugin :slight_smile:

All you need to do is to monitor the topic you want and react accordingly

Nice plugin, thanks. Exactly what i was looking for to connect different home automation boxes and send infos from my Veralite which is running UI5 :frowning:

I was able to load the plugin and it is connecting to the broker correctly but I am unable to save watchers and alias probably because it’s using a method only available in UI7.

Could you just tell me in the advanced settings of the plugin, how the variables are set? I have empty values like this (not saved):

mqttWatches {}
mqttAlias {}

Thanks for you help

[quote=“djassa1, post:43, topic:190658”]Nice plugin, thanks. Exactly what i was looking for to connect different home automation boxes and send infos from my Veralite which is running UI5 :frowning:

I was able to load the plugin and it is connecting to the broker correctly but I am unable to save watchers and alias probably because it’s using a method only available in UI7.

Could you just tell me in the advanced settings of the plugin, how the variables are set? I have empty values like this (not saved):

mqttWatches {}
mqttAlias {}

Thanks for you help[/quote]

Actually I’ve installed it on my Vera Lite UI5 and noticed the same problem.
Looks like UI5 gets all messed up when displaying a JSON variable, no idea why and won’t bother with it for now.
I’ve made a few tweaks that should make it work in UI5 (even though Vera won’t display the values correctly) - files attached.

Note: don’t forget to clear browser cache and restart LUA after uploading new files.

I’ve also added a new variable mqttVeraIdentifier that acts as an unique identifier on the MQTT server.
It will also change the Topic where the message is posted to mqttVeraIdentifier/events/device ID or alias

[quote=“SchattenMann, post:44, topic:190658”][quote=“djassa1, post:43, topic:190658”]Nice plugin, thanks. Exactly what i was looking for to connect different home automation boxes and send infos from my Veralite which is running UI5 :frowning:

I was able to load the plugin and it is connecting to the broker correctly but I am unable to save watchers and alias probably because it’s using a method only available in UI7.

Could you just tell me in the advanced settings of the plugin, how the variables are set? I have empty values like this (not saved):

mqttWatches {}
mqttAlias {}

Thanks for you help[/quote]

Actually I’ve installed it on my Vera Lite UI5 and noticed the same problem.
Looks like UI5 gets all messed up when displaying a JSON variable, no idea why and won’t bother with it for now.
I’ve made a few tweaks that should make it work in UI5 (even though Vera won’t display the values correctly) - files attached.

Note: don’t forget to clear browser cache and restart LUA after uploading new files.[/quote]

Your modified plugin works for me, many thanks!
By the way the mqttwatches is looking now like this:

{“urn:micasaverde-com:serviceId:DoorLock1”:{“Status”:“Status”},“urn:upnp-org:serviceId:TemperatureSensor1”:{“CurrentTemperature”:“CurrentTemperature”}}

Because it’s a json not on one line, UI5 is not able to display it. AltUI is able to read it.

Actually mqttVeraIdentifier is defaulted to vera.

@SchattenMann - thanks for the awesome MQTT plugin… I made several changes to better meet my use case. First, I want to dump data from multiple residences into an InfluxDB time series database and then plot the data using Grafana. So, each residence has a local MQTT broker running on an RPI that is securely bridged to a broker in my main home. Each Vera reports to their local broker that then forwards the messages securely to the main broker where the data is collected and loaded into an InfluxDB instance.

Here are the changes that I have made:

[ul][li]Added DeviceName, DeviceType, RoomId and RoomName to the mqtt payload[/li]
[li]Upgraded lua mqtt client from 0.2 to the Eclipse Paho mqtt client 0.3[/li]
[li]Added support for mqtt broker authentication using a username and password[/li][/ul]

Here is an example of the output with the aforementioned changes:

Vera/Westshore/Events/138 {"DeviceId":138,"DeviceName":"Highland Lake","DeviceType":"urn:demo-micasaverde-com:device:weather:1","OldWindDirection":"SE","RoomId":0,"RoomName":"No Room","Time":1472319031,"WindDirection":"ESE"}
Vera/SeaSpray/Events/Heliotrope {"AzimuthRounded":"117.3","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAzimuthRounded":"117.2","RoomId":0,"RoomName":"No Room","Time":1472319039}
Vera/Westshore/Events/Cabinet Lights {"DeviceId":112,"DeviceName":"Cabinet Lights","DeviceType":"urn:schemas-upnp-org:device:DimmableRGBLight:2","OldWatts":"3.8","RoomId":3,"RoomName":"Kitchen","Time":1472319041,"Watts":"4.0"}
Vera/SeaSpray/Events/Heliotrope {"AzimuthRounded":"117.4","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAzimuthRounded":"117.3","RoomId":0,"RoomName":"No Room","Time":1472319054}
Vera/SeaSpray/Events/Heliotrope {"AltitudeRounded":"44.7","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAltitudeRounded":"44.6","RoomId":0,"RoomName":"No Room","Time":1472319069}
Vera/SeaSpray/Events/Heliotrope {"AzimuthRounded":"117.5","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAzimuthRounded":"117.4","RoomId":0,"RoomName":"No Room","Time":1472319084}
Vera/SeaSpray/Events/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"517.604","RoomId":2,"RoomName":"Living Room","Time":1472319085,"Watts":"612.301"}
Vera/SeaSpray/Events/17 {"DeviceId":17,"DeviceName":"Refrigerator Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"209.655","RoomId":3,"RoomName":"Kitchen","Time":1472319089,"Watts":"204.669"}
Vera/SeaSpray/Events/Heliotrope {"AltitudeRounded":"44.8","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAltitudeRounded":"44.7","RoomId":0,"RoomName":"No Room","Time":1472319099}
Vera/SeaSpray/Events/Heliotrope {"AzimuthRounded":"117.6","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAzimuthRounded":"117.5","RoomId":0,"RoomName":"No Room","Time":1472319114}
Vera/SeaSpray/Events/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"612.301","RoomId":2,"RoomName":"Living Room","Time":1472319114,"Watts":"451.097"}
Vera/SeaSpray/Events/23 {"DeviceId":23,"DeviceName":"IT Closet Energy MTR","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"144.253","RoomId":4,"RoomName":"Vacation Support","Time":1472319119,"Watts":"149.503"}
Vera/SeaSpray/Events/23 {"DeviceId":23,"DeviceName":"IT Closet Energy MTR","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"149.503","RoomId":4,"RoomName":"Vacation Support","Time":1472319121,"Watts":"148.678"}
Vera/SeaSpray/Events/Heliotrope {"AzimuthRounded":"117.7","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAzimuthRounded":"117.6","RoomId":0,"RoomName":"No Room","Time":1472319129}
Vera/SeaSpray/Events/Heliotrope {"AltitudeRounded":"44.9","DeviceId":62,"DeviceName":"Heliotrope","DeviceType":"urn:schemas-futzle-com:device:Heliotrope:1","OldAltitudeRounded":"44.8","RoomId":0,"RoomName":"No Room","Time":1472319129}
Vera/Westshore/Events/Cabinet Lights {"DeviceId":112,"DeviceName":"Cabinet Lights","DeviceType":"urn:schemas-upnp-org:device:DimmableRGBLight:2","OldWatts":"4.0","RoomId":3,"RoomName":"Kitchen","Time":1472319131,"Watts":"3.8"}
Vera/SeaSpray/Events/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"451.097","RoomId":2,"RoomName":"Living Room","Time":1472319145,"Watts":"538.100"}

Finally, I created a git repo from the plugin directory that tracks the evolution of the plugin in this thread. If you don’t know what git is, don’t worry about it but if you do, when you download and unzip this archive, you can use git to review the changes and make controlled changes. @SchatterMan, do you want to push this to GitHub?

Here is the git log for the plugin directory

bbl:vera-mqtt blacey$ git oneline
 2017855  Added support for MQTT broker authentication with username and password by Bruce B. Lacey 13 minutes ago
 5d3c7df  Upgraded lua mqtt client from 0.2 to the Eclipse Paho client 0.3 - https://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/paho by Bruce B. Lacey 40 minutes ago
 3d1379b  Added DeviceName, DeviceType, RoomId and RoomName to mqtt payload by Bruce B. Lacey 67 minutes ago
 c82c83a  Added mqttVeraIdentifier and added support for UI5 - by SchattenMan by Bruce B. Lacey 3 hours ago
 3589578  Original plugin written by SchattenMann on forum.micasaverde.com by Bruce B. Lacey 3 hours ago

Ok, I have made a couple more tweaks to improve downstream consumption of the messages.

  1. I have changed the mqttVeraIdentifier to accept a pattern of context variables that will be substituted and used for the mqtt topic whenever a payload is published. This allows each user to tailor the mqtt topic to meet their needs while maintaining backward compatibility with the original mqttVeraIdentifer functionality that @SchattenMann implemented. For example, to mimic the current mqttVeraIdentifier semantics, one would use the pattern “Vera/Event/(Alias)”.

For my installation, I am using the mqttVeraIdentifier pattern “Vera/(SerialNumber)/(ServiceName)/(DeviceId)”. This topic name-spacing allows downstream mqtt subscribers to listen for all EnergeMetering1 variable change events by using the subscription “Vera/+/EnergyMetering1/#” or perhaps you want to subscribe to all messages from a specific device, you could use “Vera/+/+/+/42”. Maybe you want to route all messages from a specific vera from a remote broker to a central broker, you would use “Vera/5000797/#” for the topic specifier. The following context variables are available for pattern substitution:

– (SerialNumber) = Vera serial number as shown on home.getvera.com portal.
– (City) = the city defined in the controller location tab
– (ServiceId) = the service identifier
– (ServiceName) = the name of the service
– (DeviceId) = the device id
– (DeviceName) = the device name / description
– (Alias) = the device alias (legacy)
– (Variable) = the variable changed under service for device

– Legacy pattern = Vera/Event/(Alias)
– Recommended = Vera/(SerialNumber)/(ServiceName)/(DeviceId)

  1. I also added “Variable” to the payload so a downstream consumer can easily determine the name to use to retrieve the current value and old value from the JSON object.

  2. Finally, DeviceName in the payload now uses the alias if defined on the “Alias” tab, otherwise it uses the description from luup.devices[].description which is equivalent to what is shown in the UI for a given device.

Here is a sample of output for two Veras reporting in from different locations:

Vera/50000797/EnergyMetering1/104 {"DeviceId":104,"DeviceName":"RPI 3 [ITS 1]","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"4.165","RoomId":4,"RoomName":"Supporting Elements","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352076,"Variable":"Watts","Watts":"3.682"}
Vera/50000796/EnergyMetering1/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"430.191","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352101,"Variable":"Watts","Watts":"397.174"}
Vera/50000796/TemperatureSensor1/33 {"CurrentTemperature":"75","DeviceId":33,"DeviceName":"Thermostat","DeviceType":"urn:schemas-upnp-org:device:HVAC_ZoneThermostat:1","OldCurrentTemperature":"74","RoomId":1,"RoomName":"Hallway","ServiceId":"urn:upnp-org:serviceId:TemperatureSensor1","Time":1472352150,"Variable":"CurrentTemperature"}
Vera/50000796/EnergyMetering1/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"352.119","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352208,"Variable":"Watts","Watts":"462.765"}
Vera/50000797/TemperatureSensor1/139 {"CurrentTemperature":"66.9","DeviceId":139,"DeviceName":"Outside Temperature","DeviceType":"urn:schemas-micasaverde-com:device:TemperatureSensor:1","OldCurrentTemperature":"67.1","RoomId":0,"RoomName":"No Room","ServiceId":"urn:upnp-org:serviceId:TemperatureSensor1","Time":1472352215,"Variable":"CurrentTemperature"}
Vera/50000797/TemperatureSensor1/140 {"CurrentTemperature":"59","DeviceId":140,"DeviceName":"Low Temperature","DeviceType":"urn:schemas-micasaverde-com:device:TemperatureSensor:1","OldCurrentTemperature":"58","RoomId":0,"RoomName":"No Room","ServiceId":"urn:upnp-org:serviceId:TemperatureSensor1","Time":1472352215,"Variable":"CurrentTemperature"}
Vera/50000796/EnergyMetering1/17 {"DeviceId":17,"DeviceName":"Refrigerator Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"205.402","RoomId":3,"RoomName":"Kitchen","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352222,"Variable":"Watts","Watts":"3.792"}
Vera/50000796/SecuritySensor1/55 {"DeviceId":55,"DeviceName":"_Motion Sensor","DeviceType":"urn:schemas-micasaverde-com:device:MotionSensor:1","OldTripped":"1","RoomId":1,"RoomName":"Hallway","ServiceId":"urn:micasaverde-com:serviceId:SecuritySensor1","Time":1472352230,"Tripped":"0","Variable":"Tripped"}
Vera/50000796/EnergyMetering1/82 {"DeviceId":82,"DeviceName":"iMac","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"6.178","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352293,"Variable":"Watts","Watts":"63.585"}
Vera/50000796/EnergyMetering1/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"547.773","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352298,"Variable":"Watts","Watts":"432.081"}
Vera/50000796/SecuritySensor1/55 {"DeviceId":55,"DeviceName":"_Motion Sensor","DeviceType":"urn:schemas-micasaverde-com:device:MotionSensor:1","OldTripped":"0","RoomId":1,"RoomName":"Hallway","ServiceId":"urn:micasaverde-com:serviceId:SecuritySensor1","Time":1472352313,"Tripped":"1","Variable":"Tripped"}
Vera/50000796/EnergyMetering1/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"432.081","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352328,"Variable":"Watts","Watts":"343.394"}
Vera/50000796/TemperatureSensor1/84 {"CurrentTemperature":"63.4","DeviceId":84,"DeviceName":"Temperature","DeviceType":"urn:schemas-micasaverde-com:device:TemperatureSensor:1","OldCurrentTemperature":"64.6","RoomId":0,"RoomName":"No Room","ServiceId":"urn:upnp-org:serviceId:TemperatureSensor1","Time":1472352367,"Variable":"CurrentTemperature"}
Vera/50000796/HumiditySensor1/87 {"CurrentLevel":"72","DeviceId":87,"DeviceName":"Humidity","DeviceType":"urn:schemas-micasaverde-com:device:HumiditySensor:1","OldCurrentLevel":"70","RoomId":0,"RoomName":"No Room","ServiceId":"urn:micasaverde-com:serviceId:HumiditySensor1","Time":1472352367,"Variable":"CurrentLevel"}
Vera/50000796/EnergyMetering1/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"343.394","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352388,"Variable":"Watts","Watts":"541.739"}
Vera/50000796/EnergyMetering1/23 {"DeviceId":23,"DeviceName":"IT Closet Energy MTR","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"146.663","RoomId":4,"RoomName":"Vacation Support","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352411,"Variable":"Watts","Watts":"151.761"}
Vera/50000796/EnergyMetering1/23 {"DeviceId":23,"DeviceName":"IT Closet Energy MTR","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"151.761","RoomId":4,"RoomName":"Vacation Support","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352413,"Variable":"Watts","Watts":"151.536"}
Vera/50000796/EnergyMetering1/41 {"DeviceId":41,"DeviceName":"Home Theater Energy","DeviceType":"urn:schemas-upnp-org:device:BinaryLight:1","OldWatts":"541.739","RoomId":2,"RoomName":"Living Room","ServiceId":"urn:micasaverde-com:serviceId:EnergyMetering1","Time":1472352448,"Variable":"Watts","Watts":"481.798"}

And here is the git change log for the two additional for the plugin repo attached:

git log --oneline
8bb08e2 DeviceName in payload is now set to an alias if defined, otherwise the Vera Device description from luup.devices[].description
af51f9f Extended mqttVeraIdentifer to be expressed as a pattern of context variables for more flexible topic definition

At this point, I am ready to start developing the downstream consumer to pump the Vera event data into InfluxDB (as time permits) and might return here if I find that I need to make additional changes to @SchattenMann’s great prior art. I hope others find these few additions useful.

Thanks for the authentication part! Much appreciated.

Hey blacey,

Fantastic work, congrats!

I’d say the number one request has been to make Vera subscribe topics so that one can control Vera events via MQTT.
Not something I need myself to be honest, but though you should know

Have you had a chance to test it? I’m using unauthenticated broker connections internally over port 1883 with client-cert base authentication over TLS/SSL port 8883 for the remote bridged brokers so the Vera clients don’t require a username and password. If you find that it works, please post your results here.

[quote=“SchattenMann, post:50, topic:190658”]I’d say the number one request has been to make Vera subscribe topics so that one can control Vera events via MQTT.
Not something I need myself to be honest, but though you should know[/quote]

Right now, I just want to collect sensor data from multiple sites so I, not unlike you, will probably leave that to next guy/gal who stops by and takes the bull by the horns. Even though I don’t need authentication either, I added it for others because it was pretty straight-forward to do by upgrading to the newer lua mqqt client library. Thanks again for doing the heavy-lifting to produce the mqtt plugin for Vera. Do you want to push the source up to GitHub?

here it is - GitHub - jonferreira/vera-mqtt

thanks for the help!

Authentication with username/password works over port 1883 unencrypted.

[quote=“SchattenMann, post:53, topic:190658”]here it is - GitHub - jonferreira/vera-mqtt

thanks for the help![/quote]

Excellent! I had to travel today but I have another small improvement (two line change) that I will push to Github. The change is to accurately represent the changed variable’s type information in the JSON representation. Luup apparently returns floats as strings (e.g. Watts) so this change ensures that all changed-variables that contain numeric representations in the string will be represented as ints or floats. The impetus is a more accurate representation of the type information when persisting to a database such as InfluxDB.

Excellent - thanks for verifying! I just removed the untested qualifier from the earlier post that cited the authentication enhancement. :wink: The paho client isn’t capable of establishing a TLS/SSL connection.

Ok, so here are some initial results that leverage the MQTT Client.

Setup:

[ul][li]Two locations (Westshore and Seaspray)[/li]
[li]One Vera Plus controller running the MQTT Client reporting power and climate events on topic Vera/(Serial)/(ServiceName)/(DeviceId) into a local MQTT broker running on an RPI at each location[/li]
[li]The brokers are bridged over the Internet using TSL/SSL[/li]
[li]Node-red running on an RPI subscribes to the Vera events, extracts the measurements, transforms them and persists them into a central InfluxDB instance.[/li]
[li]Grafana is used to visualize the results on dashboards.[/li][/ul]

Everything seems to be working well. The following is the node-red flow and a couple of sample dashboards.

For anyone who is currently using the plugin, there is a new version on GitHub that significantly improves reliability. Here is a list of the changes.

commit 89d0dac17e610ac2c1babf01ed9a309782888670
Author: Bruce Lacey
Date:   Sun Sep 4 08:11:21 2016 -0400

    Refactored to improve connection state management and logging consistency.
    
      - Simplified connection state-machine to greatly improve connection management
        robustness with extensibility for handling subscriptions in the future
      - MQTT client connection resources are ?cleanly? reclaimed if a broker-initiated
        disconnect occurrs
      - Fixed issue that prevented mqtt broker pings after reconnect (pings need to resume too)
      - MQTT client KEEP_ALIVE set properly to ensure MQTT pings are issued within
        the 1.5 MQTT 3.1 time_interval requirement to reduce broker-initiated disconnects
      - Enable multiple vera connections to a single broker by qualifying the connection
        id with the controller serial number
      - All logging is performed either via local debug() and log() functions
      - Last message no longer displays extraneous topic pattern variables
      - Set ?debug? to false to reduce logging for normal users (perhaps this should
        be exposed to the UI)?
      - General code cleanup/reformatting
    
    Fixes #3, #4

commit 37b9c319a728b23b5d41022a97f8f7587a6957a9
Author: Bruce Lacey
Date:   Sun Sep 4 08:08:36 2016 -0400

    Upgraded JSON library from version 20131118.9 to 20160728.17

I tested this extensively by stopping/starting the broker, etc. and the client will now always reliably connect and resume sending variable updates to the broker.

The client does not connect anymore (disconnected) with the last version. This was not the case before.

Which broker are you connecting to?

Please post the SensorMqtt1: log messages here so I can help diagnose/troubleshoot. When you reviewing the log messages, we need to confirm the connection parameters (i.e. server ip, server port, username).