I really enjoyed reading this thread - and the blog of Ian Mercer - and it has taken me a while to assimilate it. Thinking more about variables, persistence, and history, it seems to me that, in Vera:
[ol][li]Device variables give us persistence[/li]
[li]dataMine gives us history (alas, from a programming point of view, it is write-only memory)[/li][/ol]
So, actually, when you say:
…you’re right, and it’s already been done. However, for this:
…you may be right, but I think this underestimates what’s possible now, and what might be in the future with forthcoming versions of Vera on faster hardware.
By way of demonstration I offer a light-weight, read-only wrapper for the dataMine database, loosely modelled on the Berkeley DB Core API. (This is an open-source, non-relational database designed for storing and retrieving key/data pairs - just what’s needed here - full documentation: http://www.oracle.com/technetwork/products/berkeleydb/documentation/index.html).
Whilst this may mean little to a non-programmer, what it means is that it is very simple to get variable history back out of the dataMine database for any recorded variable.
dm,status = dmDB.open ()
dmc,status = dm:openCursor {key = {Variable = "CurrentTemperature", Device = 51} }
x,t, status = dmc:getSearchKeyRange {t1 = os.time{year=2013, month=1, day=1}, t2 = os.time{year=2013, month=11, day=11} }
dmc:close ()
dm:close()Line by line, this:
[ol][li]opens the database (with default parameters: location, etc.)[/li]
[li]opens a cursor (which allows random access) onto a specific dataMine channel[/li]
[li]gets all the data/time pairs for t1 <= range < t2[/li]
[li]closes the cursor[/li]
[li]closes the database[/li][/ol]
Simple as that. You can have as many cursors opened on different channels (or, indeed, the same) as you like. Other ‘get’ methods allow you to step forwards or backwards by individual samples (which is only slightly less efficient that retrieving a whole block at once, because each cursor has its own cache of a week’s worth of data.) Cursor keys can include Luup service / variable / device information, as above, or dataMine channel Name or Id. They can be truncated (eg. “Temp” for “Temperature”) but together must identify a unique channel.
This just retrieves raw data. What this is not is a proper Data Historian API layer, which would allow you to do all the things which Ian Mercer blogged: count transitions, calculate max / min / averages, resample, … this will be my next but one project.
It’s a few hundred lines of pure Lua - no Luup calls - so will happily run on a separate machine (I’ve done most of the testing on my development machine with a snapshot of my entire dataMine database, some 90 channels.) Code and documentation attached. I’d be pleased to hear any feedback from interested parties.