The problem arises when more than one plugin decides to modify a library module (eg. string) by adding extra functions with the same name, but possibly different functionality. For example
function string:trim()
return self:match "^%s*(.-)%s*$"
end
Whilst it’s relatively straight-forward to ensure that different plugins using this syntax are isolated from each other…
local paddedString = " a string with extra blanks at the ends "
local trimmed = string.trim (paddedString)
…it’s actually somewhat harder to handle this syntax, which also needs to work…
local trimmed = paddedString: trim()
However, the latest release does handle this.
There is an additional console > Scheduler > Sandboxes page (the scheduler handles both scheduling and context switching) to show which plugins are the offenders:
Here, device #3 is AltUI and #345 is ALTHue. You can see the code fix which @amg0 made to avoid a name conflict with the split function, before this was fixed in openLuup. No such code work-arounds should now be necessary. Both string and table libraries are now sandboxed.
I still believe, however, that it is bad practice to modify system libraries.
add archive_video request (&format=1 for snapshots only)
refactored HTTP and SMTP servers
improved io.server closed socket handling
The big change here is a full implementation POP3 server, along with some permanent mailboxes:
mail@openLuup.local - a normal mailbox from which full messages (with any attachments) may be retrieved
events@openLuup.local - only the subject line of any mail sent here will be retained
Together with the SMTP server, the POP3 server allows a fully internal LAN system to send, store, and retrieve email messages. It can conveniently be accessed by almost any email client. I send trigger messages from my camera to both images@openLuup.local which just stores the attached images in the images/ folder, and also events@openLuup.local which lets me review any trigger times on an old iPod Touch.
Of course, if you want this to work when you are away from your home LAN, then you’d need to forward these via VeraAlerts or some such additional plugin.
Your email client can be used to manage the deletion of files from the mailboxes, or, alternatively, a timed scene using the openLuup SendToTrash and EmptyTrash actions will do the job automatically.
Other applications are up to your own imagination (please share!)
VeraBridge ‘onDashboard’ attribute, to allow Vera favourites (thanks @rafale77)
add Trash to console Files menu
link to images from console Images page
fix LastTrip name in I_openLuupCamera1.xml motion sensor (was LastTripped, wrongly)
The major enhancement here is a callback for incoming UDP datagrams from specified ports.
The luup.register_handler() function has been extended to be able to specify the UDP protocol and a port to listen on. The handler is called with a similar parameter list to regular callback functions, but with slightly different semantics:
port - the incoming port as a string, eg. “2222”
data - a table with {datagram = received_datagram_string, ip = sender_ip}
protocol - “udp”, in this case
So, for example, in Lua Startup, or within a plugin you could write:
function myUDPcallback (port, data, protocol)
print "Incoming UDP!"
print (port)
print ((json.encode(data)))
print (protocol)
end
luup.register_handler ("myUDPcallback", "udp:2222")
The callback handler is called for each incoming datagram.
I don’t use the testing branch very often, but this is a slightly unusual set of circumstances.
To address some fundamental problems of device loading, I have replaced the XML decoder with a DOM-style implementation. This has meant some significant modifications to the loader module, which reads all the device / service / implementation files. Since it’s such a basic change, I thought it wise to test thoroughly. I have done my best to do so, updating the unit tests (now over 200 separate ones are run) and running the loader itself on over 350 UPnP files. In the process I have uncovered some interesting errors and omissions, not only in third-party files, but also in standard Vera/MiOS ones. I’ve let a few other plugin developers know about theirs, but hold out no hope for Vera changing anything.
You may blame @a-lurker for instigating this change:
There is some problem with empty tags or tags that contain no ‘child’ tags being searched for when there is a single ‘parent’ tag. Just white space, or missing child tag or completely empty is allowed between tags but in this case needs to be ignored.
As an additional carrot to try this version out, I’ve taken the opportunity to fix/add a couple of other things:
RunScene - the openLuup plugin now supports this action (within the openLuup service. Thanks to @rafale77 for the suggestion.) It means that this is now available as a menu-selectable scene action. Use with care and try not to create recursive calls to the same scene, or an infinite loop!
SetHouseMode - the VeraBridge plugin now supports this action (within the VeraBridge service.) It means that changing house mode is now accessible as a menu-selectable scene action
category and subcategory - should now be correctly retained across reloads (thanks @rafale77)
scheduled scene time - should now update correctly for scenes which are cancelled by their Lua code (thanks @rafale77)
Although this is in the testing branch, I’m using it on one of my everyday systems. It seems fine, but I hope I haven’t broken too many things for others, you can always revert to the development branch.
Just enter ‘testing’ in the Update box and click the update button to try this out.
I’m going to be away from my development system for the next 10 days, and not checking the forum, as I’m still recuperating, so you’re on your own until then.
PS: Something more about the new XML parser:
Features include:
- ignores processing instructions, comments, and CDATA
- expands self-closing tags
- reads and saves attributes
- element relationship links:
- parentNode, firstChild, lastChild,
- previousSibling, nextSibling
- some navigation routines:
- nextNode(), including optional filter function
- getElementsByTagName()
- basic XPath searching:
- xpath(),
- xpathIterator()
- .documentElement field of the model accesses the root XML document element
In a break from the WWW3 standard, text is NOT stored in a child text element,
but in the .nodeValue attribute of the element itself. It seemed much easier this way.
It’s quite easy to navigate this structure and extract information you might need from any XML file.
This development release is a significant enhancement to openLuup functionality. It includes a ‘Data Historian’ which stores previous device variable values in both in-memory cache and (optionally) on-disk archives. Taking lessons learned from DataYours and EventWatcher, also from AltUI’s Data Storage Providers, it offers an almost configuration-free approach to storing (and retrieving) variable values. In addition, and in contrast to those other options and the dataMine plugin, it provides a natural way of reading historic variable values through the usual Luup interface. Also included is a Grafana Data Source API so that data may be easily visualised if you already have a Grafana installation. A big plus here is that you don’t have to have set up storage for any specific variable, they are (almost) all available.
This development has been long in the making, and indeed the ideas for it come from well before openLuup was even thought of…
I do encourage you to read the (short) Ian Mercer blog page linked to by the above URL.
Some more on the Data Historian:
only numeric variable values are stored
By default, the in-memory cache stores all variable changes since system startup, with a small number of exceptions, retaining the most recent 1000 values (by default)
Variables not cached (by default) include timestamp-like ones, including LastUpdate, Polls, low-level Zwave, and some dates
The on-disk archive is enabled by the single LuaStartup line (path relative to cmh-ludl/ where you want the data to reside.)
By default, only a small subset of cached variables are archived on disk, including security sensors Tripped, temperature, humidity, and generic sensors Current values, energy metering KWH and Watts, battery levels, openLuup system memory and cpu stats, …
The on-disk archives are implemented as Graphite Whisper files (as in DataYours)
different data types have different sample rates and retention policies. For example, temperature data is sampled every 20 minutes, initially, but reduced in a number of stages to once per day after a year, and battery levels are just sampled once per day.
Default total on-disk archive duration is 10 years. Typical file sizes, one per variable (containing multiple resolution data) are about 300Kbyte.
System impact is minimal, with high performance. The additional system load is less than writing to the logs.
Database maintenance effort is zero. The disk archive is of fixed size per variable, and total file space is roughly proportional to the number of archived variables.
the openLuup Console has two pages to view historian activities: openLuup > Historian (for the in-memory cache), and Files > History DB (for the on-disk archives)
the URL for the Grafana Data Source interface is simply your openLuup:3480 port
programmatically, data is retrieved using the luup.variable_get() call
I’ve been running versions of this for a while now. On my ‘production’ system which is linked to 3 Veras, Netatmo, Philips Hue (thank you @amg0!), and some MySensors Arduinos, there are over 4000 device variables. About 300 of those are cached in memory, and their on-disk archives take about 90 Mbyte. There are, on average, 12 updates per minute, each one taking about 1.5 mS on an RPi. (I don’t have any security sensors triggering frequently on this system.)
For plugin developers, reading data from the cache/archive is trivial, using an additional {start,end} time parameter to variable_get():
local v0,t0 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33)
print ("normal current value", v0,t0)
local v2,t2 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33, {os.time()-3600, os.time()})
print ("over the last hour", pretty {value = v2, times=t2})
gives
normal current value 31.3 1530026929
over the last hour {
times = {1530023886,1530023926.5466,1530024526.9677,1530025127.1189,1530026327.7046,1530026929.0275,1530027486},
value = {31.1,31,31.2,31.3,31.2,31.3,31.3}
}
If the data over the requested time range is all in cache, then the time resolution will be better than 1mS. Once on disk, it depends on the retention policy of the particular archive (sensible defaults, but all configurable if necessary) never finer than 1 second, but usually 5-10 minutes. If there is no data within the time range, then arrays are empty, but within a non-empty array there are no nil points, the times simply denoting (approximately) when the variable changed.
There are further developments to do before turning this into a master release:
mirroring on-disk archives to remote Graphite or InfluxDB servers
total replacement for DataYours
Also, I plan to implement the shockingly missing Digest authorization for luup.inet.wget().
Edit: this release also includes the luup.openLuup flag to denote that this is openLuup, not Vera.
I’ve just merged the lastest development branch with the master to roll-up all the various changes and bug fixes which have accumulated.
This is not quite the master release I wanted to make, but it does fix a couple of significant problems, and, of course, include the latest things like Data Historian. However, it doesn’t include digest authorization in luup.inet.wget() or a couple of other things I wanted to add. It seemed necessary, though, with recent interest in openLuup, and in particular fixed a long-standing install error.
I’ve been a bit short of quality development time recently.
Thanks to all who have helped to make improvements over the last few months (and before!)
Fixes synchronisation issue between system Mode attribute and openLuup HouseMode variable if luup.attr_set() was used to change mode. Thanks to @DesT for pointing this one out.
enables historian mirroring to external Graphite and InfluxDB databases via UDP
implements missing “actions” request
In addition to the internal Graphite database used by the Data Historian, all archived variables can be mirrored to an external Graphite and/or InfluxDB database. In order to make this efficient in terms of CPU and network resources, the transaction-free UDP protocol is used, avoiding any possibility of system deadlock.
InfluxDB instructions for configuring a UDP port are here:
DataYours may also be used as a destination for the Graphite historian mirror.
To enable sending to Graphite or InfluxDB, the following openLuup system attributes (for example) with IP and port number should be set in the Lua Startup:
Either of those target databases may then be queried for archived variable data. Grafana queries work well, with auto-population of variable name menus.
Schedule VeraBridge remote actions using <job> rather than <run> invocation.
This addresses a technical issue with remote Veras not being able to keep up with openLuup systems running on faster processors. It mitigates the lack of a properly managed command request queue on Vera and apparently decreases the chance of a Vera restart.
Most users should notice little difference, but, as ever, if you find a problem, don’t hesitate to raise an issue.
This release incorporates a significant addition for plugin developers: CPU time logging.
A number of features and tools support this:
plugin devices now have a new attribute cpu(s) which is constantly updated, with microsecond reolution, and available at a code level using luup.attr_get (“cpu(s)”, devNo)
Console > Scheduler > Jobs page now includes a table column for CPU times, and the table is sortable by clicking on the column headers
Console > Scheduler > Startup Jobs page has now been renamed to Plugins and has an additional table showing the total system CPU load in % (since last Luup restart) and the cumulative CPU time per plugin device
Note that plugin startup jobs generally run for a short time and terminate (although they may schedule later processing using luup.call_delay() or suchlike.) The Startup Jobs table is, therefore, essentially static, and records the exit state and any failure message.
The Plugin device CPU usage table, on the other hand, updates to reflect the same per-device time as the new device attribute cpu(s).
Most jobs are run on behalf of the system (typically HTTP server requests and scene timers) but the job tag of a plugin action runs as part of that device. The Jobs page shows current and recently completed jobs (which remain in the list for three minutes after termination.)
Other internal updates are also included in this release.
This release incorporates a couple of changes discussed on this thread:
Includes:
missing serviceId in device_create() or append() sets attributes, not variable
device category_num set by device type, if otherwise missing
Note that, after further testing, I’ve not made a change to luup.variable_set(), because Vera doesn’t allow a missing serviceId there.
Haven’t yet done anything with subcategories.
Also, the console menu structure has changed a bit, with tabs to connect groups of related pages . It’s also totally configurable by a JSON file, so you can have any set of menus you like. You can revert to the old style if necessary. The home page gives an index allowing quick access to any menu page.
I have finally bitten the bullet and switched to a CSS Framework for the console pages.
I’m using W3.CSS which is very lightweight, fits right in with the openLuup ethos, and claims to be…
smaller and faster than similar CSS frameworks.
When run for the first time it downloads a local copy of the CSS file to the www/ folder in the openLuup tree. It’s only 23 KB. This allows seamless operation in the case of operating offline.
The look & feel has inevitably changed, hopefully for the better, and it makes maintenance and updates much easier. The influence of AltUI is probably plain to see. The menu structure remains fully configurable with an external JSON file, should you really want to do that. There are three pre-defined sets to choose from.
Not everything is quite fully implemented, but there is more functionality on the way.
Happy to receive feedback on this change, as ever.
This release contains some significant updates to a number of modules.
The most obvious changes are updates to the default console L&F. The page and menu layouts are something of an homage to AltUI, without which openLuup would not exist in its current form. This is not to be confused with a replacement for AltUI… it is not. The console has always been about viewing and understanding some of the internals of the system and installed plugins, as an aid to debugging. It is not about presenting your devices in a convenient form for controlling your home, and pages are not dynamically updated as device variables change.
Again, taking another cue from AltUI, the Ace editor is used extensively to view internal structures and files in Lua, XML, and JSON formats. Ace may be dynamically downloaded from a Content Distribution Network (default is Cloudflare,) or loaded into a local folder which may be used offline. If neither is available, the system will default to simple HTML textarea elements.
Internally, the XML module has, once again, been refactored, this time to unify parsing and serialisation of XML, HTML, and SVG. Many device files needed by the basic system are generated directly at startup using the internal DOM (Domain Object Model) or JSON encoder. This sets the scene for a fully stand-alone system (without an attached Vera) which at the moment is only possible after downloading Vera device files.
Session cookies are used to persist a number of openLuup states, such as which room to view and sorting/filtering options. A total of three Lua Test windows allow parallel debugging of different snippets of code. A key point here is that the test code is actually persisted in the user_data.json file, and not the web browser, so different sessions from different browsers will actually share the same code, which is checkpointed with the rest of the user data (every hour, or at system reload.)
The underlying openLuup engine is little changed, apart from some preparatory work to improve the HTTP server, so I’m hoping that there are no bugs introduced which should affect the regular running of the system.