Just started playing with it on grafana and I have to say kudos. It is pretty awesome and I had been looking for something like this on and off since I never really implemented DataYours.
…still processing the information you provided…
In the meantime I’ve run into a problem with the built-in openLuup console and Historian. Clicking on one of the metrics, my browser tries to open
http://192.168.0.XXX:3480/render?target=openLuup.2_openLuup.openLuup.MemAvail_Mb&from=2019-02-11T04:06:01Then I get back
./openLuup/graphite_cgi.lua:782: bad argument #3 to 'format' (string expected, got nil)Yet in Grafana the data is plotted just fine. In the openLuup console Historian reads 1024 points for this particular metric. Something to do with parsing the ‘to’ value?
Yea…I tried entering (I manually entered ‘&until=’)
http://192.168.0.XXX:3480/render?target=openLuup.2_openLuup.openLuup.CpuLoad&from=2019-02-10&until=4:26:01And the data shows up. Strange.
Strange indeed. Not specifying ‘until’ has always worked before and, indeed, none of the links defined on the Console Historian pages have an ‘until’ parameter and they work just fine. Need to dig further.
Just tried
http://172.16.42.129:3480/render?target=openLuup.2_openLuup.openLuup.CpuLoad&from=2019-02-10on my machine and this works just fine.
As does:
http://172.16.42.129:3480/render?target=openLuup.2_openLuup.openLuup.MemAvail_Mb&from=2019-02-11T04:06:01Ah, no, I have it!
The error showing on line #782 is from a debug statement that I inserted while trouble-shooting your earlier problem, and didn’t remove!
Delete that line. All will be well. Sorry.
I’ve updated both development and master branches.
About six months ago we had a discussion about this and I supplied a crude workaround, saying:
…so I have finally got around to doing this. The latest development version of openLuup has a file in the CGI folder on GitHub: whisper-edit.lua. If you copy this into your cmh-ludl/cgi/ folder and then visit the Console > Files > History DB page, you should find that the last column of database filenames now has hyperlinks to a Whisper file editor.
Here, you can change the aggregation method type, for the whole file, and also use a data picker to select from/until days to extract data into an editable table using the Read button. Errant values can simply be corrected and the changes committed back to the database, with the Commit button (or abandoned, using Reset.)
Hi akboer,
That’s great news, thank you for digging into this! I’m going to try it and let you know how it goes!
Hi akbooer,
the last version of your wispher editor is compatible with DY files too ? tnks
donato
Oh yes, good point, absolutely.
It is simply a Whisper file editor (for the DataYours text-file format) and not linked to the Historian in any way (in particular, it neither clears the cache which might now be inconsistent with the file, nor does it update any remote databases which are mirroring the Historian.)
You can either invoke it with a browser CGI request:
openLuupIP:3480/cgi/whisper-edit.lua?target=whisper/myWhisperFile.wsp
or change the target on the W-Edit page, once you’re there. Should be a much easier replacement for the rather crude JSON-based editor I supplied previously.
Hi AK,
I came home after a week out of town to find out that my verabridge devices somehow dropped out of the historian database. I just updated to the latest console version you have on github/dev and after full reboot of both the VM running openLuup and the VM running grafana, the dataset seems to be updating again but one odd thing is that under the console, all of the verabridge devices show up under the orphans tab and are displayed twice. One instance with the 0.devnumber and another with the PKid.devnumber. Might be a newb question but is it what is expected?
Hmm…
Anything under the orphans tab is there because openLuup can’t find a device number which matches. Anything with a 0.devnumber… is a local device under openLuup, anything with a PKid… is bridged from another device.
You might look at the dates on the orphaned files and see when they were last updated.
Did anything else change that you noticed?
I haven’t, AFAIK, changed anything in the historian recently (the basics remain unchanged for about a year.)
After looking around, it looks like an issue with my vera which somehow was missing it’s PKid… so it sent itself to open luup as 0. Somehow historian recorded the devices from the vera as if they were openluup devices. After fixing the PKid of the vera, all is back to normal. I just have a bunch of orphan devices which had invalid IDs in historian.
Ah, that’s alright, then. To some extent, you have to work with what Vera tells you (always risky, sometimes it’s wrong, but I’ve not seen this one before.)
There shouldn’t be much data in the orphaned files, although you could check from the historian cache page if you have the Whisper-Edit CGI installed (use the latest from GitHub development branch, it’s not downloaded with the usual openLuup updates.)
Something very strange happened to my vera… seems like something got corrupted very badly to lead to this. It seems like I recovered all of my previous data now and indeed there isn’t a whole lot of files which are orphaned so not a bit concern. Thanks!
Hi akbooer,
I’ve several remote systems all updating a central Rpi with Openluup/DY . At central DY I use the Datauser.lua routine to do some custom processing.
Some remote systems are Vera/DY and some are rpi/openluup/DY.
In this configuration can I migrate all rpi/openluup/DY systems to rpi/openluup/datahistorian ?
Can Vera/DY communicate with central openluup/historian ?
What’s the functional and technological difference between DY and Data Historian included in openluup ?
Tnks for your patience
Donato
Hi Donato
These are really good questions. It does make sense to try and consolidate your configuration.
It’s so long ago (over three years) that I added the Datauser option (I believe it was you who asked for it in the first place) that I had to go and check in the source code.
Data Historian is a light-weight subset of DataYours (itself an implementation of the Graphite / Carbon / Whisper set of technologies) but tightly integrated into openLuup itself. It uses a separate database folder which should not be mixed with DataYours files. The graphics rendering, however, is capable of plotting data from both the Historian and DataYours databases, (and, indeed, dataMine) which can be represented as separate branches on the base target search path. I do this all the time, using Grafana as a plotting package.
Since it is a subset, there are some functionalities missing. At the moment, the Historian does not write to the DataYours folder, so you can’t use it as a destination for DataWatcher messages in the same way that you can do for DataCache. It also does not implement the Datauser.lua routine.
There is, however, no conflict at all in running DataYours and the Historian on the same backend system. My recommendation would be to simply enable just DataWatcher (for your custom processing) and DataCache to receive data from the remote Vera/openLuup systems and write it to the whisper/ folder.
To include the DataYours database in the Historian tree for plotting/rendering you should have this line in your Lua Startup code:
luup.attr_set ("openLuup.Historian.DataYours", "whisper/") -- overriding DY finder
I do plan, sometime, to add the capability for the Historian to write to the DataYours database, but had not intended to include any user-defined processing.
I hope this makes some sense. Feel free to ask for further help. I would appreciate hearing how things all work once you’ve made some changes. As ever, I’d encourage you to do plenty of testing before using this in your full-scale environment.
AK
Hi akbooer,
tnks as usual for your exhaustive reply.
Having developed a front-end web application to display, graph, analyze DY data archived at central system is it correct saying that at the moment there isn’t not very important benefits for me to migrate DY to Historian where it’ s possibile ?
Or do you see any advantages ?
Tnks
Donato
Probably not.
Data Historian is all about providing history for many openLuup variables without needing to do any significant configuration. Out of the box, it provides an in-memory cache of 1024 points per device variable (for numeric values only.) Setting one option allows a permanent on-disc archive (using Whisper database files.) Variable history of any given time range can be recovered in Lua using a single variable_get() call. Optionally, the local database can be mirrored to external Graphite or InfluxDB systems.
If most of your interesting data comes from elsewhere, via UDP datagrams from DataWatchers or anywhere else, and doesn’t even appear as bridged variables under openLuup, then there’s really no need to change anything.
With your support now is all quite clear for me .
Would be wonderful if openluup in the future will include a very simple UI to define/display variables for example as the “Multistring Plugin” that a routine on rpi (or another os system) can update via a programming language (python, lua, php, etc.) , without the more complex, complete ALTUI program . Will be fine where are sensors attached to openluup system NOT zwave.
Donato