openLuup - running unmodified plugins on any machine

Plugins & Plugin Development openLuup
#1

This is my 2000th post, so I wanted to make it significant to me, if to nobody else!

I’m aware that this can be a pivotal time for some developers (three years ago @Ap15e made his 2000th post his last: My final post) and it’s sadly no surprise to note the issues which drove him away:

  • “My personal ROI from MCV is just too low”
  • “resources visible to me are just too limited to create a sufficiently stable and bug free and documented product”
  • “There is no offical documentation for developers, and when MCV does document items, there is no guarantee that MiOS works as documented”
  • “Installing UI5 on my Vera V2 left the device in a totally unusable state: severe slowdowns, several lockups per week requiring a hard reboot, several soft reboots per day.”
  • New state of my plugins/scripts: EOL, unsupported."

Nowadays, read ‘UI7’ and ‘Vera3/Lite’. We’ve seen others remove their apps from the forum too. The direction of developer drift now seems to be OpenHAB… but I’m loathe to abandon my investment in MiOS, and Lua, so I agree with this comment:

So what to do? I’m not going to pull my apps from the public domain (EventWatcher, Netatmo, DataYours) in fact I want to continue to develop and support them. But I want to work in a more open and stable environment. I note the following:

  • Vera is a great Z-wave to HTTP bridge
  • The plugins developed here on the forum are great
  • The Luup API and HTTP Requests are great
  • Scenes, in particular the definition of triggers are abysmal
  • Lua is the best language in the world (OK, forgive me)

All would be solved if Luup was open source and could be run on the plethora of cheap and reliable hardware available today. But it’s not. But we could get something like that effect if we engineered a sufficient subset of Luup to run on such a platform. Could it be done? What would we need?

  1. UI
  2. scheduler
  3. web server
  4. Luup compatible API
  5. Device and Implementation xml file reader
  6. Zwave bridge to Vera
  7. runs most plugins without modification

What we wouldn’t need is UPnP.

What have we (nearly) got already?

  • We have, courtesy of @amg0, the most excellent altUI: Alternate UI to UI7, and that, I think, is probably the hardest one to do in the above list.
  • Items 2 - 5, and 7, I’ve prototyped, in pure Lua, and posted elsewhere: DataYours on Raspberry Pi, running selected plugins unmodified, including: DataYours, EventWatcher, Netatmo, RBLuaTest, altUI. See screenshot attached.

Is it worth the effort? Probably not. Will I pursue this quest? Yes, because as @Aaron noted in @Ap15’s final thread:

I plan to post the code on GitHub in due course: and now it is here GitHub - akbooer/openLuup: a pure-Lua open-source emulation of the Vera Luup environment

=============================

October 2015: A summary of the project status…

openLuup is an environment which supports the running of some MiOS (Vera) plugins on generic Unix systems (or, indeed, Windows systems.) Processors such as Raspberry Pi and BeagleBone Black are ideal for running this environment, although it can also run on Apple Mac, Microsoft Windows PCs, anything, in fact, which can run Lua code (most things can - even an Arduino Yun board.) The intention is to offload processing (cpu and memory use) from a running Vera to a remote machine to increase system reliability.

Running on non-specific hardware means that there is no native support for Z-wave, although plugins to handle Z-wave USB sticks may support this. The full range of MySensors (http://www.mysensors.org/) Arduino devices are supported though the Ethernet Bridge plugin available on that site. A plugin to provide a bi-directional ‘bridge’ (monitoring / control) to remote MiOS (Vera) systems is provided in the openLuup installation.

openLuup is extremely fast to start (a few seconds before it starts running any created devices startup code) has very low cpu load, and has a very compact memory footprint. Whereas each plugin on a Vera system might take ~4 Mbytes, it’s far less than this under openLuup, in fact, the whole system can fit into that sort of space. Since the hardware on which it runs is anyway likely to have much more physical memory than current Vera systems, memory is not really an issue.

There is no built-in user interface, but we have, courtesy of @amg0, the most excellent altUI: Alternate UI to UI7 (see the Vera forum board http://forum.micasaverde.com/index.php/board,78.0.html) An automated way of installing and updating the ALTUI environment is now built-in to openLuup. There’s actually no requirement for any user interface if all that’s needed is an environment to run plugins.

Devices, scenes, rooms and attributes are persisted across restarts. The startup initialisation process supports both the option of starting with a ‘factory-reset’ system, or any saved image, or continuing seamlessly with the previously saved environment. A separate utility is provided to transfer a complete set of uncompressed device files and icons from any Vera on your network to the openLuup target machine.

What openLuup does:

  • runs the ALTUI plugin to give a great UI experience
  • runs the MySensors Arduino plugin (ethernet connection to gateway only) which is really the main goal - to have a Vera-like machine built entirely from third-party bits (open source)
  • includes a bridge app to link to remote Veras (which can be running UI5 or UI7 and require no additional software.)
  • runs many plugins unmodified - particularly those which just create virtual devices (eg. Netatmo, …)
  • uses a tiny amount of memory and boots up very quickly (a few seconds)
  • supports scenes with timers and ALTUI-style triggers
  • has its own port 3480 HTTP server supporting multiple asynchronous client requests
  • has a fairly complete implementation of the Luup API and the HTTP requests
  • has a simple to understand log structure - written to LuaUPnP.log in the current directory - most events generate just one entry each in the log.
  • writes variables to a separate log file for ALTUI to display variable and scene changes.

What it doesn’t do:

  • Some less-used HTML requests are not yet implemented, eg. lu_invoke.
  • Doesn’t support the <incoming> or <timeout> action tags in service files, but does support the device-level <incoming> tag (for asynchronous socket I/O.)
  • Doesn’t directly support local serial I/O hardware (there are work-arounds.)
  • Doesn’t run encrypted, or licensed, plugins.
  • Doesn’t use lots of memory.
  • Doesn’t use lots of cpu.
  • Doesn’t constantly reload (like Vera often does, for no apparent reason.)
  • Doesn’t do UPnP (and never will.)

Latest release available now available on GitHub: GitHub - akbooer/openLuup: a pure-Lua open-source emulation of the Vera Luup environment

#2

This is fantastic! I will be watching this project closely :slight_smile:

#3

I wish MiOS would adopt a software model. I just don’t trust them to put enough specs into VeraPlus or VeraSecure. But I would pay good money to run their software on the hardware of my choosing.

Failing that, I will watch this project with interest…

#4

Can’t wait to try this out!

#5

Might be looking for suitable guinea pigs soon! Have you a particular use case (favourite apps, …) you’d want to try first (starting simple) ?

#6

Fantastic 2000 post and significant roadmap of future achievements.

#7

happy to support where I can.

#8

+1 here :slight_smile:

#9

Just an update on progress… I’m afraid it’s been a bit slow since I’ve not had as much time to work continuously on this as I wanted, and anyway, I’m not as quick as you @amg0.

One of the frustrations in reverse engineering some of the functionality has been that it’s obvious that there are some better ways to do things, but you have to stick to the same API and the same format. One example: the crucial internal variable “DataVersion” which keeps track of when anything changes, but has hugely inconsistent naming in HTTP request outputs:

[ul][li]in [tt]/data_request?id=status[/tt], it is actually called “DataVersion”. There is a separate variable called “UserData_DataVersion”[/li]
[li]in [tt]/data_request?id=sdata[/tt], it is called “dataversion”[/li]
[li]and in [tt]/data_request?id=user_data[/tt], the variable called “DataVersion” has the value of “UserData_DataVersion” and the value of “DataVersion” is not shown, but is a critical query parameter in calls to that request.[/li][/ul]

Sigh…

Anyway, some progress is being made. From the outset, I wanted as few external dependencies as possible - everything is in Lua. This means that I’ve had to craft my own HTTP server for the port 3480 access, although I rely on the native web server of the target platform to deliver the port 80 requests. I did toy with the idea of using CGI or FastCGI, but in the end that seemed to make things harder rather than easier. I also have a job scheduler and the complete (well, nearly) Luup API. I’ve had enormous fun writing the timer code - particularly the sunrise/sunset calculations (I see that @futzle went through this ages ago when writing the heliotrope plugin.)

What it doesn’t have, at the moment, is:

[ul][li]Persistence. Everything is created in the startup code. This is not as bizarre as it sounds because: (a) variables will be cloned by bridging to a real Vera or imported from non-Zwave devices; and (b) it’s not supposed to restart as often as Vera (in fact, one version I have has been running without restart for weeks.) I see no real difficulty in using an exact clone of a user_data file for implementing this if necessary.[/li]
[li]Scenes. This is a more serious shortcoming right now. I actually have no intention of recreating Vera’s exact scene functionality, because it is so flawed. Whilst scene actions and timers are fine, the trigger definitions, which are so tied up with UPnP, are seriously lacking. I envisage a HomeSeer-like interface which simply scripts the right Lua code. Hence this solution can run on a native Vera too. Think PLEG, but much simpler. (I have no expectations that PLEG will ever run on openLuup - not sure of the licensing implications anyway.)
[/li]
[li]Z-wave bridge. This should be straight-forward and require no code at all on the real Veras. I know how to do it (really just the same as any remote interface) but just haven’t got around to it.[/li]
[li]Arduino bridge. The Arduino plugin should run, since all the Luup infrastructure is there, but I haven’t yet got any Arduino sensors myself, so would need some help there.[/li]
[li]UPnP. No intention of ever implementing this, but all the UPnP-like HTTP requests are there. Device references are always device numbers, not UDN strings. This means that S_*.xml service files are completely redundant.[/li][/ul]

I just have a bit more work to do on the device loader before making a release.

I’d welcome any comments on the architecture, such as I have described it, or simple test cases to start out with (in particular, then, favourite non-Zwave plugins.)

#10

One of the first plugins I want to try is : NorthQ Watt Calculator

I’m using this plugin to calculate Watts from KwH from several devices (not only the NorthQ).

multiple instances of this plugin are running and they take up a lot of memory on the Vera.

I also have Hue, Netatmo, Foscam and KodiRemote running, Hue and netatmo are already working on openluup.

It would be nice to run the Vera free of (non zwave) plugins and free up some of the very needed ram.
I have an raspberry 2 available for testing.

#11

[quote=“nitehawk, post:10, topic:187412”]One of the first plugins I want to try is : NorthQ Watt Calculator

I’m using this plugin to calculate Watts from KwH from several devices (not only the NorthQ).

multiple instances of this plugin are running and they take up a lot of memory on the Vera.[/quote]
Interesting. I have a stand-alone implementation of this (runs on Vera) which does the calculation for all installed NorthQ devices and puts the Watts result back into their respective devices. Not a plugin, but runs as startup code, taking almost no memory.

I also have Hue, Netatmo, Foscam and KodiRemote running, Hue and netatmo are already working on openluup.
Oh! ...so you've done some hacking already, with Hue running OK? I don't have a Hue - what did you need to change to make this work?
It would be nice to run the Vera free of (non zwave) plugins and free up some of the very needed ram. I have an raspberry 2 available for testing.
One of the things that's missing in the published code so far is implementation of device actions. I've been working on this, and am just about there. Rather depends on how involved you want to be as to when I share this (ie. how much beta testing you really want to do!)
#12

I haven’t have openluup running at the moment. That was my bad english, I thought i read somewhere that you had those running already.

#13

[quote=“nitehawk, post:10, topic:187412”]One of the first plugins I want to try is : NorthQ Watt Calculator

I’m using this plugin to calculate Watts from KwH from several devices (not only the NorthQ).

multiple instances of this plugin are running and they take up a lot of memory on the Vera.[/quote]

Further to this, I’ve posted my NorthQ code here: [url=http://forum.micasaverde.com/index.php/topic,16082.msg237849.html#msg237849]http://forum.micasaverde.com/index.php/topic,16082.msg237849.html#msg237849[/url]. You can remove your multiple NorthQ Watt Calculator plugins entirely. This should save some memory on Vera whilst I keep working on openLuup!

#14

Thank you for the code.

I don’t have multiple northq energy meters, just one, i was using the calculator to calculate watts for my green wave powernodes. http://forum.micasaverde.com/index.php/topic,18747.msg142098.html#msg142098

Can i use your code to see if i can get it to work with these powernodes?

#15

I am not skilled enough to contribute much at all but I wholeheartedly offer my moral support.

I have openHab downloaded (but not running) at the moment. But I just have too much invested in Vera/LUUP and I know how to do what I need to do (various LUUP scripts, nested decisions etc.

But my Vera is just too unstable; perhaps due to resources. The ability to keep Vera for HW gateway and then code separately is very appealing.

#16

Time for an update.

I’m pleased to say that Release 1 of openLuup is on the blocks.

What openLuup Release 1 does:

[ul][li]runs on any Unix machine - I often run it within my development environment on a Mac (makes debugging easy.)[/li]
[li]includes a bridge app to link to remote Veras (which can be running UI5 or UI7 and require no additional software.)[/li]
[li]runs many plugins unmodified - particularly those which just create virtual devices (eg. Netatmo, …)[/li]
[li]runs the ALTUI plugin to give a great UI experience[/li]
[li]uses a tiny amount of memory and boots up very quickly (a few seconds)[/li]
[li]creates all devices on startup (this means they are numbered consecutively for those that worry about these things.)[/li]
[li]has its own port 3480 server which supports multiple asynchronous client requests[/li]
[li]has a fairly complete implementation of the luup.xxx API and the HTTP requests[/li]
[li]has a simple to understand log structure - most events generate just one entry each in the log.[/li][/ul]

As an existence proof, I attach a parent/child diagram, produced by the splendid ALTUI plugin from @amg0. It shows openLuup on a BeagleBone Black (might as well be a Raspberry Pi, could be anything really) running Netatmo and DataYours and also four copies of the VeraBridge plugin which connects (both monitoring and control) to three remote VeraLites running UI5 and one Vera Edge running UI7 - some 228 devices in all. (I posted this on the ALTUI thread too, so apologies if you have seen it there already.)

What it doesn’t do:

[ul][li]Has no persistence - this is a design feature, not a shortcoming. Persistence arises from either the remote Veras to which you bridge, or any database applications you may have (eg. DataYours). Your local device structure is created by the startup code using standard luup calls.[/li]
[li]Doesn’t yet run scenes - I mentioned in an earlier post that I plan to support scene actions and timers (all the infrastructure is already built in) but triggers will be done differently.[/li]
[li]Doesn’t have the luup.io module yet. So whilst a plugin can use the LuaSocket library (many do) the I/O support from that part of luup (specifically the handling of job tags) is not yet supported.[/li]
[li]because of the above, it doesn’t yet run the Arduino plugin, which is really my main goal - to have a Vera-like machine built entirely from third-party (open source) components.
[/li][li]Won’t run encrypted, or licensed, plugins. This probably applies to PLEG, sadly, but I will have to talk to Richard about this.[/li]
[li]Doesn’t use lots of memory. The attached example runs in about 8 Mbytes of Lua address space.[/li]
[li]Doesn’t restart as often as Luup on a Vera - in fact, dare I say it, it doesn’t crash at all.[/li]
[li]Doesn’t do UPnP (and never will.) This actually means that it doesn’t use S_xxx.xml files at all, although for full functionality, ALTUI does use them.[/ul]

I can’t release this on the MiOS App Store, of course, since it doesn’t run on Vera! For release 1 I’ll post a .zip file with the lua code and the VeraBridge plugin (which, itself, probably would run on Vera, since it’s just a normal plugin) and some initial documentation to get you going. Just tidying up a few things before I do that.

#17

Kudos & Respect, thanks for all your work!

#18

Here is Release 1 of openLuup. I have done my best with the documentation (and the code!) but have no doubt that there will be questions. I’m happy to help, but bear in mind that you’re installing a system on hardware that I probably don’t have, running under a version of unix that I don’t have, on a network that’s different from mine, …

Nevertheless, openLuup is pure Lua (5.1) so should run as expected.

I’m very keen to get any feedback on this. I’m now regularly using openLuup running on my BeagleBone Black linked to four Veras and running ALTUI as my main system interface. I have a vested interested in making this work, and making it better. There is more to do, but this is a start.

Good luck.

Edit: My first erratum… you do also need to install the LuaSocket library on your machine - “[tt]apt-get install lua-socket[/tt]” worked for me.

Edit: Release 2 available here: [url=http://forum.micasaverde.com/index.php/topic,32315.msg243104.html#msg243104]http://forum.micasaverde.com/index.php/topic,32315.msg243104.html#msg243104[/url]

#19

awesome, will try this next weekend.

I will let you know the results

#20

Great !

I’ve tried to install it and run with ZeroBrane Studio on Windows 7 64bit :

error loading module 'openLuup.init' from file './openLuup\init.lc':
	./openLuup\init.lc: cannot load incompatible bytecode

It seems that you have compiled with your Mac, perhaps is it the problem ?