Development Branch: 2018 Release 6.26
This development release is a significant enhancement to openLuup functionality. It includes a ‘Data Historian’ which stores previous device variable values in both in-memory cache and (optionally) on-disk archives. Taking lessons learned from DataYours and EventWatcher, also from AltUI’s Data Storage Providers, it offers an almost configuration-free approach to storing (and retrieving) variable values. In addition, and in contrast to those other options and the dataMine plugin, it provides a natural way of reading historic variable values through the usual Luup interface. Also included is a Grafana Data Source API so that data may be easily visualised if you already have a Grafana installation. A big plus here is that you don’t have to have set up storage for any specific variable, they are (almost) all available.
This development has been long in the making, and indeed the ideas for it come from well before openLuup was even thought of…
The State of Vera: Home Control Vs. Home Automation (August, 2013)
I do encourage you to read the (short) Ian Mercer blog page linked to by the above URL.
Some more on the Data Historian:
- only numeric variable values are stored
- By default, the in-memory cache stores all variable changes since system startup, with a small number of exceptions, retaining the most recent 1000 values (by default)
- Variables not cached (by default) include timestamp-like ones, including LastUpdate, Polls, low-level Zwave, and some dates
- The on-disk archive is enabled by the single LuaStartup line (path relative to cmh-ludl/ where you want the data to reside.)
luup.attr_set ("openLuup.Historian.Directory", "history/")
- By default, only a small subset of cached variables are archived on disk, including security sensors Tripped, temperature, humidity, and generic sensors Current values, energy metering KWH and Watts, battery levels, openLuup system memory and cpu stats, …
- The on-disk archives are implemented as Graphite Whisper files (as in DataYours)
- different data types have different sample rates and retention policies. For example, temperature data is sampled every 20 minutes, initially, but reduced in a number of stages to once per day after a year, and battery levels are just sampled once per day.
- Default total on-disk archive duration is 10 years. Typical file sizes, one per variable (containing multiple resolution data) are about 300Kbyte.
- System impact is minimal, with high performance. The additional system load is less than writing to the logs.
- Database maintenance effort is zero. The disk archive is of fixed size per variable, and total file space is roughly proportional to the number of archived variables.
- the openLuup Console has two pages to view historian activities: openLuup > Historian (for the in-memory cache), and Files > History DB (for the on-disk archives)
- the URL for the Grafana Data Source interface is simply your openLuup:3480 port
- programmatically, data is retrieved using the luup.variable_get() call
I’ve been running versions of this for a while now. On my ‘production’ system which is linked to 3 Veras, Netatmo, Philips Hue (thank you @amg0!), and some MySensors Arduinos, there are over 4000 device variables. About 300 of those are cached in memory, and their on-disk archives take about 90 Mbyte. There are, on average, 12 updates per minute, each one taking about 1.5 mS on an RPi. (I don’t have any security sensors triggering frequently on this system.)
For plugin developers, reading data from the cache/archive is trivial, using an additional {start,end} time parameter to variable_get():
local v0,t0 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33)
print ("normal current value", v0,t0)
local v2,t2 = luup.variable_get ("urn:upnp-org:serviceId:TemperatureSensor1","CurrentTemperature", 33, {os.time()-3600, os.time()})
print ("over the last hour", pretty {value = v2, times=t2})gives
normal current value 31.3 1530026929
over the last hour {
times = {1530023886,1530023926.5466,1530024526.9677,1530025127.1189,1530026327.7046,1530026929.0275,1530027486},
value = {31.1,31,31.2,31.3,31.2,31.3,31.3}
}If the data over the requested time range is all in cache, then the time resolution will be better than 1mS. Once on disk, it depends on the retention policy of the particular archive (sensible defaults, but all configurable if necessary) never finer than 1 second, but usually 5-10 minutes. If there is no data within the time range, then arrays are empty, but within a non-empty array there are no nil points, the times simply denoting (approximately) when the variable changed.
There are further developments to do before turning this into a master release:
- mirroring on-disk archives to remote Graphite or InfluxDB servers
- total replacement for DataYours
Also, I plan to implement the shockingly missing Digest authorization for luup.inet.wget().
Edit: this release also includes the luup.openLuup flag to denote that this is openLuup, not Vera.