MQTT Client implementation

4integration · November 7, 2013, 9:43pm

I am trying to load a lua module but fails…all files uploaded

In “I_SensorMqtt1.xml” I have the following code but it fails at

u = require(“utility”)

Do you have any hints what’s wrong?

[code]<?xml version="1.0"?>

local p
local u
local m

    function startup(lul_device)
        luup.log("SensorMqtt plugin: loading library L_SensorMqtt ...")
        if (package.path:find ("/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua", 1, true) == nil) then
            package.path = package.path .. ";/etc/cmh-ludl/?.lua;/etc/cmh-lu/?.lua"
        end
        
        package.loaded.L_SensorMqtt = nil
        p = require("L_SensorMqtt")
        if (package.loaded.L_SensorMqtt == nil)
        then
            luup.log("SensorMqtt plugin: plugin is not installed correctly. Library L_SensorMqtt cannot be loaded.", 1)
            luup.task("Plugin not correctly installed", 2, "SensorMqtt plugin", -1)
            return false
        end
        luup.log("SensorMqtt plugin: library L_SensorMqtt loaded")

        luup.log("SensorMqtt plugin: loading library utility ...")
        package.loaded.utility = nil
        u = require("utility")
        if (package.loaded.utility == nil)
        then
            luup.log("SensorMqtt plugin: plugin is not installed correctly. Library utility cannot be loaded.", 1)
            luup.task("utility not correctly installed", 2, "SensorMqtt plugin", -1)
            return false
        end
        luup.log("SensorMqtt plugin: library utility loaded")

        package.loaded.mqtt_library = nil
        m = require("mqtt_library")
        if (package.loaded.mqtt_library == nil)
        then
            luup.log("SensorMqtt plugin: plugin is not installed correctly. Library mqtt_library cannot be loaded.", 1)
            luup.task("mqtt_library not correctly installed", 2, "SensorMqtt plugin", -1)
            return false
        end
        luup.log("SensorMqtt plugin: library mqtt_library loaded")
        
        return p.startup(lul_device)
    end
</functions>
<startup>startup</startup>

[/code]

watou · November 7, 2013, 9:50pm

Loading would fail if there were a syntax error in utility.lua .

Also, if your module is encrypted (file name extension .luaenc) then it has to explicitly name itself in the first line like this:

module("utility",package.seeall)

If you tried to instead use:

module("...",package.seeall)

Then MIOS’s custom module loader for encrypted modules will fail (probably a minor bug in their loader, at least as of 1.5.408).

watou

4integration · November 8, 2013, 11:09pm

Thanks watou, tried that but no success. Moved the other loading into the lua code and that works but now I have another issue.

The MQTT Lua modules is here: http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/lua

Do you see anything strange? (I know it’s a bit out of scope for MCV but since you are skilled Lua guys/girls)

Error:

01 11/09/13 0:00:36.101 luup_log:62: SensorMqtt plugin: Enter sendToMqtt <0x2d773680> 01 11/09/13 0:00:36.101 luup_log:62: SensorMqtt plugin: Create MQTT Client <0x2d773680> 01 11/09/13 0:00:36.102 luup_log:62: SensorMqtt plugin: Connect to 192.168.1.21 and port 1883 <0x2d773680> 01 11/09/13 0:00:36.102 LuaInterface::CallFunction_Timer-5 function sendToMqtt failed [string "module("L_SensorMqtt", package.seeall)..."]:47: attempt to index field 'client' (a nil value) <0x2d773680>

The code:

[code]module(“L_SensorMqtt”, package.seeall)

– Service ID strings used by this device.
SERVICE_ID = “urn:upnp-sensor-mqtt-se:serviceId:SensorMqtt1”

local mqttServerIp = nil
local mqttServerPort = 0

local MQTT = require(“mqtt_library”)
local SENSOR_MQTT_LOG_NAME = "SensorMqtt plugin: "

local function log(text, level)
luup.log("SensorMqtt plugin: " … text, (level or 50))
end

function startup(lul_device)
_G[“sendToMqtt”] = sendToMqtt
– Help prevent race condition ???
– luup.io.intercept()

SENSOR_MQTT_DEVICE = lul_device
log("Initialising SensorMqtt", 1)

--Reading variables
mqttServerIp = luup.variable_get(SERVICE_ID, "mqttServerIp", SENSOR_MQTT_DEVICE)
if(mqttServerIp == nil) then
	mqttServerIp = "0.0.0.0"
	luup.variable_set(SERVICE_ID, "mqttServerIp", mqttServerIp, SENSOR_MQTT_DEVICE)
end

mqttServerPort = luup.variable_get(SERVICE_ID, "mqttServerPort", SENSOR_MQTT_DEVICE)
if(mqttServerPort == nil) then
	mqttServerPort = "0"
	luup.variable_set(SERVICE_ID, "mqttServerPort", mqttServerPort, SENSOR_MQTT_DEVICE)
end
log("Start - Preparing worker thread", 1)
-- Prepare the worker "thread"
luup.call_delay("sendToMqtt", 20, "")
log("Done - Preparing worker thread", 1)

end

function sendToMqtt(luup_data)
luup.log(SENSOR_MQTT_LOG_NAME … “Enter sendToMqtt”, 1)
luup.log(SENSOR_MQTT_LOG_NAME … “Create MQTT Client”, 1)
luup.log(SENSOR_MQTT_LOG_NAME … "Connect to " … mqttServerIp … " and port " … mqttServerPort, 1)
mqttServerPort = tonumber(mqttServerPort)
local mqtt_client = MQTT.client.create(mqttServerIp, mqttServerPort)

luup.log(SENSOR_MQTT_LOG_NAME .. "Connect MQTT Client", 1)
mqtt_client:connect("Vera")

luup.log(SENSOR_MQTT_LOG_NAME .. "Publish MQTT message", 1)
mqtt_client:publish("Vera/Test", "*** Vera test start ***")

--mqtt_client:subscribe({ args.topic_s })

luup.log(SENSOR_MQTT_LOG_NAME .. "Destroy MQTT Client", 1)
mqtt_client:destroy()

luup.log(SENSOR_MQTT_LOG_NAME .. "Done with MQTT", 1)

luup.call_delay("sendToMqtt", 10, "")
luup.log(SENSOR_MQTT_LOG_NAME .. "Leaving sendToMqtt", 1)

end

[/code]

watou · November 8, 2013, 11:22pm

I think your MQTT variable needs global scope so it can be seen when luup.call_delay calls your sendToMqtt function. Since it’s not visible in scope MQTT.client has a nil value, since it’s not the MQTT you think it is.

I think that’s it, anyway.

watou

4integration · November 9, 2013, 1:07pm

Thanks for you idea, tested with both removing “local” as well as using _G …

MQTT = require("mqtt_library") _G["MQTT"] = MQTT

To test more simple code I removed “luup.call_delay(…)” and called MQTT in the startup method, but got the same result

watou · November 9, 2013, 1:19pm

[quote=“4integration, post:5, topic:177705”]Thanks for you idea, tested with both removing “local” as well as using _G …

MQTT = require("mqtt_library") _G["MQTT"] = MQTT

To test more simple code I removed “luup.call_delay(…)” and called MQTT in the startup method, but got the same result :([/quote]

I just noticed that the mqtt_library.lua at the URL you provided ([url=http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/lua/mqtt_library.lua]http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.lua.git/tree/lua/mqtt_library.lua[/url]) does not have a module statement at its top. I don’t know why that is or what the ramifications are of that.

If you add this:

module("mqtt_library", package.seeall)

to the top of that mqtt_lbibrary.lua, does it help? If this is also a dead-end, I’m probably out of advice (even bad advice ).

But I’m very interested to see the results of your labor with MQTT. I’m trying to get into some Arduino tinkering, and I think this is a good middleware to start with.

Regards,
watou

4integration · November 10, 2013, 10:08am

Hey,

I got it working!

But I have to admit I am on deep water and no fully satisfied

I read up a little on Lua modules and checked the Vera-style and MQTT-style. It seems that Vera implements the the old style (and deprecated) way for Lua 5.0 and early 5.1 and MQTT uses the new for Lua 5.1 and 5.2. Then suspected them somewhat being incompatible.

MQTT and Utility uses “objects” (or “classes”), not sure about the correct terminology since I new at Lua).
So last night I hacked mqtt_library and utility and removed these objects and made some other small changes required since I introduced some conflicts that way.

I have attached the hacked MQTT source so you can diff in full but to mention some changes

Added:

module("mqtt_library", package.seeall)

Removed:

local MQTT = {}

and all references to MQTT in mqtt_library e.g.:

MQTT.Utility = require("utility") ... function MQTT.client.create(
to

Utility = require("utility") ... function client.create(

The variable conflict solved from:

MQTT.message = {} MQTT.message.TYPE_RESERVED = 0x00 ...etc
to

messageType = {} messageType.TYPE_RESERVED = 0x00 ...etc

For Utility I had to remove the “object” and it functions and make all local functions global instead of local.

Even if I got it working I don’t like this solution since now I have my own fork of MQTT Lua libraries.

So some questions to you guys:

Is it a fact that modules implemented with different Lua versions are incompatible?
Can this be solved without touching the MQTT library?
Is it possible to make use of the new module style in Vera? I made a quick test to use the new module style in Vera but seems to fail even if it’s 5.1 (maybe it’s the “early 5.1” as mentioned in the link above

Thiemen · November 10, 2013, 10:32am

I hope some other people can assist you, since i think this can be an interesting add on to vera, especially with the “internet of things” coming.
For now i will follow this tread…

Maybe you should also rename the topic of this tread, so other people recognise it is about MQTT?

averell · May 19, 2014, 10:33am

I managed to get this MQTT integration working, based on the code you posted here. My idea is to try to build a plugin that allows me to control devices on the vera by sending messages on MQTT, and the other way atround, to offer a generic way of creating devices in vera that communicate via MQTT to the raspberry, some arduino’s etc. that do the real work. This way I can use the vera sometimes as a master (I love my homewave app!), but sometimes as a slave when I want to control zwave devices from my other solutions.
Anybody else out there already playing with this? My next step is going to be to handle incoming messages.

Averell

hek · May 19, 2014, 2:30pm

Looking forward to see your results.

aPoundOfLaughter · May 20, 2014, 12:19pm

As a side-note I’m building a logging module in node for mqtt messages with storage in memory, memcache, redis or mysql.

averell · May 21, 2014, 12:59pm

So the good thing is that I can indeed react to incoming messages. The bad thing is, it’s not working based on a push, but I need to pull MQTT with the mqtt_client:handler(“”) function from the MQTT library. I’m used to working with node.js where the callback function get’s invoked whenever a messages comes in, here I need to tell the MQTT client to go check, only then it calls the callback function.
For building devices that just need to be updated with some information like temperature etc. it is quite ok, I can call the client:handler every few seconds or so, and update the device status. For turning on devices, ideally I would see my very react a bit faster, but I wonder about the overhead of calling this function to often.
I will play a bit with this to see how fast I can make this work without causing too much problems, every second or so might work out ok, or I might go for a solution where I limit the direct MQTT connectivity to updates to devices (incoming) and switching external stuff on (outgoing), and directly swithing on devices on the vera via http, letting my raspbberry translate mqtt commands to HTTP. Unsatisfactory though it may be…

Av.

XAPPO · May 23, 2014, 10:37am

Watching this topic with great interest. MQTT support would be a really useful addition to Vera.

4integration · May 24, 2014, 9:06am

I have a MQTT plugin running that monitor variables and fire off MQTT messages. The configuration is pretty “raw” though but doesn’t change so often

averell · May 25, 2014, 8:37am

Very interested to see what you did 4integration, as part of what I want to do, I also want to publish out all variables. I didn’t get far due to travel, but I got child devices to send messages, and have temperature sensors publish messages over MQTT that set the device current temperature on the vera. Now thinking about the architecture, naming, making it hardcoded or flexible, etc. I will probably end up with a table with some hardcoded stuff that suits my needs. Some of the knowledge I will keep outside of the vera anyway, the whole idea is to keep some of the rules etc. away from the vera.

XAPPO · May 25, 2014, 11:06am

Hi Averell,

Do you think the poll (cf event) limitation is something that can be worked around as the code matures or is it always going to be required ? Polling is so wasteful so it would be really nice to eliminate if possible.

Will you (and 4Integration) be releasing some usable Vera LUA code for this , albeit with the hardcoded bits that suit your setup ? Or should I just dive in from the code snippets above ?

K

akbooer · May 25, 2014, 11:20am

I don’t know if this helps, but the scheme I used for naming in the DataYours app was:

[tt]vera.device.service.variable[/tt] (for example: [tt]Vera-12345678.043.urn:micasaverde-com:serviceId:HumiditySensor1.CurrentLevel[/tt])

This gives a unique path to every variable on multiple Veras (the Vera number comes from the unique machine number found in the variable [tt]luup.pk_accesspoint[/tt]) and contains all the information that you need to get or set the variable. For use on a file system, I’ve had to substitute the colon ‘:’ in serviceIds with a caret ‘^’ to ensure compatibility with all file systems that it’s likely to come across.

4integration · May 25, 2014, 6:45pm

@averell, @XAPPO
I am not very proud of the code though…beginner at Lua and have not had available time to get it nice.
Upload the files and create the device with “D_SensorMqtt1.xml”, then define IP/Port for your MQTT server on the device (I use ActiveMQ on my server@home).
To monitor variables add a list of devices and variable at “monitorConfig” with the format
:|:|:|…etc

variable is from

local DATATYPE_TEMPERATURE = 1 local DATATYPE_HUMIDITY = 2 local DATATYPE_ENERGY_WATTS = 3 local DATATYPE_ENERGY_KWH = 4 local DATATYPE_ENERGY_UWATTS = 5
So I monitor energy-watts with config “12:3|14:3|102:3|102:4|89:3”

4integration · May 25, 2014, 6:49pm

Noted I didn’t post the dependencies…MQTT libs and JSON I use in the code.
I have those files in “usr\lib\lua”

averell · May 25, 2014, 9:27pm

4integration,

Thanks, I will reuse some of the stuff you did, I’m also a lua newbie, not aiming for the cleanest code, just for working code.
@xappo: I do plan to publish what I make in a short time, for now I’m still figuring out too much, and all still is too messy with hardcoded stuff, lot’s of logs etc. once I know the layout, and clean up the code, I will put it all here. I will try to get the event stuff worked out, the code examples suggest it is not a lua limitation as such, but I still have to learn a bit more about how lua on the vera works to know how I may fix it. I agree that polling is messy

Av