Something very strange happened to my vera… seems like something got corrupted very badly to lead to this. It seems like I recovered all of my previous data now and indeed there isn’t a whole lot of files which are orphaned so not a bit concern. Thanks!
Hi akbooer,
I’ve several remote systems all updating a central Rpi with Openluup/DY . At central DY I use the Datauser.lua routine to do some custom processing.
Some remote systems are Vera/DY and some are rpi/openluup/DY.
In this configuration can I migrate all rpi/openluup/DY systems to rpi/openluup/datahistorian ?
Can Vera/DY communicate with central openluup/historian ?
What’s the functional and technological difference between DY and Data Historian included in openluup ?
Tnks for your patience
Donato
Hi Donato
These are really good questions. It does make sense to try and consolidate your configuration.
It’s so long ago (over three years) that I added the Datauser option (I believe it was you who asked for it in the first place) that I had to go and check in the source code.
Data Historian is a light-weight subset of DataYours (itself an implementation of the Graphite / Carbon / Whisper set of technologies) but tightly integrated into openLuup itself. It uses a separate database folder which should not be mixed with DataYours files. The graphics rendering, however, is capable of plotting data from both the Historian and DataYours databases, (and, indeed, dataMine) which can be represented as separate branches on the base target search path. I do this all the time, using Grafana as a plotting package.
Since it is a subset, there are some functionalities missing. At the moment, the Historian does not write to the DataYours folder, so you can’t use it as a destination for DataWatcher messages in the same way that you can do for DataCache. It also does not implement the Datauser.lua routine.
There is, however, no conflict at all in running DataYours and the Historian on the same backend system. My recommendation would be to simply enable just DataWatcher (for your custom processing) and DataCache to receive data from the remote Vera/openLuup systems and write it to the whisper/ folder.
To include the DataYours database in the Historian tree for plotting/rendering you should have this line in your Lua Startup code:
luup.attr_set ("openLuup.Historian.DataYours", "whisper/") -- overriding DY finder
I do plan, sometime, to add the capability for the Historian to write to the DataYours database, but had not intended to include any user-defined processing.
I hope this makes some sense. Feel free to ask for further help. I would appreciate hearing how things all work once you’ve made some changes. As ever, I’d encourage you to do plenty of testing before using this in your full-scale environment.
AK
Hi akbooer,
tnks as usual for your exhaustive reply.
Having developed a front-end web application to display, graph, analyze DY data archived at central system is it correct saying that at the moment there isn’t not very important benefits for me to migrate DY to Historian where it’ s possibile ?
Or do you see any advantages ?
Tnks
Donato
Probably not.
Data Historian is all about providing history for many openLuup variables without needing to do any significant configuration. Out of the box, it provides an in-memory cache of 1024 points per device variable (for numeric values only.) Setting one option allows a permanent on-disc archive (using Whisper database files.) Variable history of any given time range can be recovered in Lua using a single variable_get() call. Optionally, the local database can be mirrored to external Graphite or InfluxDB systems.
If most of your interesting data comes from elsewhere, via UDP datagrams from DataWatchers or anywhere else, and doesn’t even appear as bridged variables under openLuup, then there’s really no need to change anything.
With your support now is all quite clear for me .
Would be wonderful if openluup in the future will include a very simple UI to define/display variables for example as the “Multistring Plugin” that a routine on rpi (or another os system) can update via a programming language (python, lua, php, etc.) , without the more complex, complete ALTUI program . Will be fine where are sensors attached to openluup system NOT zwave.
Donato
Funny you should say that, but it is work in progress (and nearly ready!) The latest update to the Console should be what you need.
1 Like
Incredible !! You are the best !!
Have a nice evening
Hi akbooer,
is there any news about a simple UI interface in openluup enviroement to define/display variables.
tnks
donato
Hi Donato
Well, yes, it’s been there for a while.
The console device pages look like this, for example:
The small blue icons take you to a listing or a plot of the recent variable values.
Clicking on the +Create button takes you to a menu like this:
Did you need anything else?
AK
Hi akbooer,
where can I find this version of openluup software? Is this master or development branch ?
This is in the more recent development branch releases. Simply type development into the openLuup update box on the Plugins page and press the update button.
hi @akbooer,
I have been using grafana with historian for quite some time and just today wanted to add a variable to the archive and grafana plotting. I tried doing it through the console/historian menu but it appears that checking the checkbox under that menu does not take. The graphing of that variable under the console works but I am trying to add it to the grafana dashboard. How do we go about adding a variable to be archived?
Yes. If you are not on a tablet, you should see a popup on that column title (“variable…”) which says:
“note that the checkbox field is currently READONLY”
Sorry about that, it is clearly on the “to do” list.
All you need to do is create the right historian file, and once it’s there, it will be updated. There’s two ways to do this:
- Add the generic variable name to the historian rule base. This is found at the end of the file
servertables.luain theopenLuupfolder. The pattern matching rules are of the form
device_number.short_serviceId.variable_name
and is grouped under different sample rates (“every_5m, every_10m, etc…”). It takes ‘*’ as a wildcard in any of these three fields, or more complex syntax (as used by the historian’s data finder.)
If that’s not clear, then let me know and I can make a specific rule set for you.
- Create the file as a one-off, manually. This would be in the
history/database, or wherever you’ve specified for the historian archives. There are default values for the various archives in the file created, depending on variable type, but again these can be overridden.
Again, you’ll need a bit more detail on this, but I can tell you that if you tell me eactly what you’re trying to archive.
Sorry this is so difficult, I really must get around to making tht checkbox work!
AK
1 Like
Thank you. I have no intent to hardcode it so I will go the file creation route. No rush I can you the console in the meantime. It appears that the free memory on the vera which I was trying to plot may not be needed after all as it seems like I resolved the memory leak I thought I had.
OK, so a one-off file create can be done with code like this run in Lua Test:
local whisper = require "openLuup.whisper"
local filename = "history/0.377.Weather1.TodayHighTemp.wsp"
local archives = "10m:7d,1h:30d,3h:1y,1d:10y"
whisper.create (filename,archives,0)
print((whisper.info (filename)).retentions)
The key parameters you will need to change are filename and archives. This also assumes that your historian folder is history/.
The filename syntax is 0.deviceNumber.shortServiceId.variableName, and the archives are in the usual Whisper syntax – these particular ones are what I use for most measurements, with a 10 minute basic sample rate, but you could, for example, switch that to 5.
If the code is successful, it will echo the archive retentions read from the newly created file, thus:
10m:7d,1h:30d,3h:1y,1d:10y
Hope that makes sense. You’ll have to reload for the historian to start using this new set of archives.
1 Like
Thanks. I had already done it but doing a copy paste of an existing history file and renaming it to the variable I wanted. Seems to be working.
Yup, that works great too, if you want the same archive structure and you don’t mind the old data hanging around in there!
