openLuup: Data Persistence

[quote=“CudaNet, post:11, topic:190272”][3] Data persistence (I’m looking for a flexible method to store and recall variables); including across restarts.

Data Yours possibly ? I need to read up on this as I’ve never used this plugin but it appears to store to a database.
Use case. Device A stores current value, power outage/restore resets Device A to default. Recall Device A and restore to previous state.[/quote]

For (3), DataYours is undergoing a refresh, and when I’m back in front of my development system I’ll publish it on GitHub (and MiOS App Store.). It has fewer files and doesn’t create child devices for each activated daemon. It’s the basis for new functionality to improve database connectivity, graphics, and the UI.

If I can persuade @amg0 to support it within the AltUI framework (currently using Thingspeak) then I will.

If all you need is a local database, the Whisper module from DataYours may be called directly from scene Lua with a one-liner to store or recall to a multi-resolution round-robin database.

It sounds as if a local database will provide exactly what I need (and possibly others). I use Thingspeak today for very specific purposes to share my data with others (never expecting to recall the data) and I do like the implementation that amg0 has performed for ThingSpeak (select variables, select devices). To be able to store the last state of a variable (for select devices) would be great rather than creating a state variable for each device. SO I’m hoping a toggled solution can be worked through to create a cohesive package to suit the various needs of it’s users.

I’ll keep an eye out for your DY refresh and here’s hoping amg0 see’s the benefit…

[quote=“akbooer, post:1, topic:190444”]For (3), DataYours is undergoing a refresh, and when I’m back in front of my development system I’ll publish it on GitHub (and MiOS App Store.). It has fewer files and doesn’t create child devices for each activated daemon. It’s the basis for new functionality to improve database connectivity, graphics, and the UI.

If I can persuade @amg0 to support it within the AltUI framework (currently using Thingspeak) then I will.

If all you need is a local database, the Whisper module from DataYours may be called directly from scene Lua with a one-liner to store or recall to a multi-resolution round-robin database.[/quote]

Reading this, I’m not so sure what you’re after. openLuup already provides device variable persistence across restarts and also checkpoints everything every six minutes in the event of a system crash (OK, that never happens, so let’s say a power failure.)

So the worst off you can be is six minutes stale. Of course, there are circumstances where this is not good enough and a local database to persist the latest change would be required. Data could be restored in the startup script.

So I must be missing something. You need, perhaps, the previous value of the variable? Or some longer history??

Here’s an example,

My Philips Hue bulbs/bridge have a deficiency whereby if a power outage occurs, once restored - the entire state of the bulb is reset (In my case, this is an entire house of bulbs). This being, Hue, Sat, Brightness and on/off. So basically if the lights were in an off state, every single one of them are reset to an on state at 100% brightness. Imagine a whole house of this in the middle of the night. Even if I were to write Lua to toggle the state based on time of day, this doesn’t allow me to return the bulbs to their individual, previous state (hue, sat, bri etc) when the bulbs are turned back on.

So as you indicated I’m in search of both. That is, being able to query the previous state (variable) for certain devices and the ability to store a state change for things such as my home energy meter, temperature etc. long term (for statistics).

[quote=“akbooer, post:3, topic:190444”]Of course, there are circumstances where this is not good enough and a local database to persist the latest change would be required. Data could be restored in the startup script.

So I must be missing something. You need, perhaps, the previous value of the variable? Or some longer history??[/quote]

Well this seems like s major failure of the Hue… so you obviously DO need a solution. Given that keeping a persistent record of the current state is straight-forward, when would you want to apply the old value after the restart? The startup Lua is run before plugin initialisation, so may subsequently be overwritten by anything the plugin chooses to do on its startup.

Yes, Hue dropped the ball and one would expect the bridge would be self-aware (restore to previous state)… As for when to restore, well that’s a really good question. My current bridge (v1.0) requires some time to boot once power is restored (10-15 seconds). SO inevitably the bulbs will beam down until the bridge is back up and the state restored. So to keep things flexible, yes - either at startup or the ability to simply recall the value in a scene (other use cases).

I’m sure there must be a use other than my use case. Hopefully others will come forward and have a voice…

@Akbooer

I see amg0 released a new build yesterday (super cool)… I’ve loaded it to my test box but I’m holding off on my production system. So should I start exploring DataYours or is something brewing for the next release of openLuup ?

If I can persuade @amg0 to support it within the AltUI framework (currently using Thingspeak) then I will.

Thanks to @amg0’s latest release, openLuup will be able to link to DataYours. I have yet to experiment and follow the supplied instructions. Today was a new Netatmo release, an update to DataYours with some under-the-covers work to help has already been done. Tomorrow perhaps (no, only joking, this will take a bit of work.)

Not enough hours in the day.

Well I for one think that’s awesome ! I’ll keep an eye out for any updates…

[quote=“akbooer, post:8, topic:190444”]Thanks to @amg0’s latest release, openLuup will be able to link to DataYours. I have yet to experiment and follow the supplied instructions. Today was a new Netatmo release, an update to DataYours with some under-the-covers work to help has already been done. Tomorrow perhaps (no, only joking, this will take a bit of work.)

Not enough hours in the day.[/quote]

SO, I was able to get Carbon/Whisper and Data Yours installed. It all appears to be working very well, so are there any quick tips on using Lua to recall the last X data points ?

Excellent news.

so are there any quick tips on using Lua to recall the last X data points ?

Yes. There’s two possible routes:

[ol][li]if you have direct file system access from where you’re running your Lua to the whisper database, then you can use a single function call to recover data,[/li]
[li]if you have IP access to the machine running the DataGraph daemon, you can use HTTP to retrieve it in CSV or JSON formats.[/li][/ol]

If you’re running the basic configuration with DataYours installed on your openLuup machine writing to a local file system then you can do both or either of the above.

[hr]

Using the Whisper database directly:

You just need to know where you told DataYours to put the Whisper database. It’s whatever value the LOCAL_DATA_DIR variable holds, for me it’s “whisper/”.

This little bit of code:

local whisper = require "L_DataWhisper"
local name = "OutdoorTemp.w"
local time = os.time() - 60*60

local tv = whisper.fetch ("whisper/"..name..".wsp", time)

print(pretty (tv))

…will return the last hour’s worth of data from the channel “OutdoorTemp.w”. The print statement shows you the returned data structure:

{
  ipairs = function: 0xa0fd18,
  times = {1458119700,1458123300,300},
  values = {7.5,nil,7.7,nil,7.6,nil,7.7,nil,8.2,nil,8.4,n = 13}
}

which contains:

[ul][li]ipairs - an iterator to walk you through the data points[/li]
[li]times - a three element array with the start, finish, and sample rate times (in seconds)[/li]
[li]values - an array, quite possibly containing some nil values, and a table entry “n” giving the actual number of samples[/li][/ul]

Whilst you can delve directly into the values array and do some math to recover the corresponding sample times, the ipairs iterator will do that for you, returning an index number, the sample value, and the sample time. You use it like this:

for i,v,t in tv:ipairs() do
    print (i,t,v)
end

which would print out the result:

1 	1458119700 	7.5
2 	1458120000 	nil
3 	1458120300 	7.7
4 	1458120600 	nil
5 	1458120900 	7.6
6 	1458121200 	nil
7 	1458121500 	7.7
8 	1458121800 	nil
9 	1458122100 	8.2
10 	1458122400 	nil
11 	1458122700 	8.4
12 	1458123000 	nil
13 	1458123300 	nil

This is probably the easiest way to go. If you want more readability on specifying the times to [tt]whisper.fetch()[/tt] then use [tt]os.time{year=2016, month=3, day=16, hour=10}[/tt] or something like that.

[hr]

Using an HTTP request to retrieve data:

A request of this form:

http://openLuupIP:3480/data_request?id=lr_render&target=OutdoorTemp.w&from=-1h&format=json

will retrieve the latest data in JSON format:

[{
  "target": "OutdoorTemp.w"
,  "datapoints": [
  [7.5, 1458119700],
  [null, 1458120000],
  [7.7, 1458120300],
  [null, 1458120600],
  [7.6, 1458120900],
  [null, 1458121200],
  [7.7, 1458121500],
  [null, 1458121800],
  [8.2, 1458122100],
  [null, 1458122400],
  [8.4, 1458122700],
  [null, 1458123000]
  ]
}]

…which is easily decoded with [tt]json.decode()[/tt], or you can get it in (a rather bizarre) CSV format:

http://openLuupIP:3480/data_request?id=lr_render&target=OutdoorTemp.w&from=-1h&format=csv

like this:

OutdoorTemp.w,2016-03-16 09:50:00,nil
OutdoorTemp.w,2016-03-16 09:55:00,8.2
OutdoorTemp.w,2016-03-16 10:00:00,nil
OutdoorTemp.w,2016-03-16 10:05:00,8.4
OutdoorTemp.w,2016-03-16 10:10:00,nil
OutdoorTemp.w,2016-03-16 10:15:00,8.5
OutdoorTemp.w,2016-03-16 10:20:00,nil
OutdoorTemp.w,2016-03-16 10:25:00,8.3
OutdoorTemp.w,2016-03-16 10:30:00,nil
OutdoorTemp.w,2016-03-16 10:35:00,8.2
OutdoorTemp.w,2016-03-16 10:40:00,nil
OutdoorTemp.w,2016-03-16 10:45:00,8.4
OutdoorTemp.w,2016-03-16 10:50:00,nil

If you leave off the [tt]&format[/tt] parameter, or give it the value of “svg” then you will retrieve an HTML file which, when displayed in a browser will give you a plot of the data. You could issue this request via a [tt]luup.inet.wget “http://…”[/tt] call.

This should help you to get started. Ask again if you need further help.

Thanks AK, amazing post… I’ll start work on this today (direct access to the DB).