Have updated openLuup - AltUI/More/plugins - typed development into Update box and hit Update Now. That timeout is indeed for the problem box. It is connected via wifi so I will try to get it wired into the network asap. Will see what happens now and report back.
Hi akboer,
I’ve just installed the latest development release (18.8.10) of openLuup to use Data Historian.
I’ve set the startup attribute to the right directory, and after a reboot it nicely creates all the Whisper files.
I can also see the logging in the console, it shows graphs. However, the Whisper files don’t seem te get updated so it seems it’s only logging to memory. This is my startup log:
2018-08-18 22:45:28.345 openLuup.historian:: starting data historian
2018-08-18 22:45:28.345 openLuup.historian:: using on-disk archive: /etc/cmh-ludl/history/
2018-08-18 22:45:28.346 openLuup.historian:: Graphite schema/aggregation rule sets: 15/3
2018-08-18 22:45:28.346 openLuup.historian:: disk archive storage rule sets: 9
2018-08-18 22:45:28.346 openLuup.historian:: using memory cache size (per-variable): 1024
Anything I can try?
If it’s created the files, then the file directory permissions must be OK.
[ul][li]I assume that this problem arose after restarting the system?[/li]
[li]Is the openLuup.Historian.Directory attribute still defined? (from the log, it looks like it is…)[/li]
[li]Do you have code in Lua Startup which sets it?[/li][/ul]
It would be diagnostic to share the Console > Files > History DB and Console > openLuup > Parameters pages.
Thanks for your reply.
These are the parameters (I’ve used a full path just to be sure):
{
"Backup":{
"Compress":"LZAP",
"Directory":"backup/"
},
"DataStorageProvider":{
"-- Graphite = '127.0.0.1:2003'":" -- EXAMPLE Graphite UDP port",
"-- Influx = '172.16.42.129:8089'":"-- EXAMPLE Influx UDP port"
},
"HTTP":{
"Backlog":2000,
"ChunkedLength":16000,
"CloseIdleSocketAfter":90
},
"Historian":{
"-- Directory = 'history/'":"-- on-disc archive folder",
"CacheSize":1024,
"DataYours":"/etc/cmh-ludl/whisper/",
"Directory":"/etc/cmh-ludl/history/",
"Graphite_UDP":"",
"InfluxDB_UDP":""
},
"Logfile":{
"Incoming":"true",
"Lines":2000,
"Name":"logs/LuaUPnP.log",
"Versions":5
},
"POP3":{
"Backlog":32,
"CloseIdleSocketAfter":600,
"Port":11011
},
"SMTP":{
"Backlog":100,
"CloseIdleSocketAfter":300,
"Port":2525
},
"Scenes":{
"-- set Prolog/Epilog to global function names ":" to run before/after ALL scenes",
"Epilog":"",
"Prolog":""
},
"Status":{
"CpuLoad":"0.9%",
"IP":"172.20.2.10",
"MemAvail":"23.5 Mbyte",
"MemFree":"5.6 Mbyte",
"MemTotal":"245.8 Mbyte",
"Memory":"12.2 Mbyte",
"StartTime":"2018-08-18T22:45:27",
"Uptime":"0.5 days"
},
"UserData":{
"Checkpoint":60,
"Name":"user_data.json"
},
"Version":"v18.8.10",
"Vnumber":180810
}
And a part of my History DB log:
Data Historian Disk Database, 2018-08-19 10:54:30
updates/min: 0.0, time/point: -nan ms (cpu: -nan ms), directory: /etc/cmh-ludl/history/
archives (kB) #updates filename (node.dev.srv.var)
openLuup
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.CpuLoad
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.HouseMode
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.MemAvail_Mb
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.MemFree_Mb
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.MemTotal_Mb
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.Memory_Mb
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.Uptime_Days
5m:7d,1h:30d,3h:1y,1d:10y 335 0.2.openLuup.Vnumber
EventWatcher
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.AppMemoryUsed
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.CacheSize
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.CpuLoad05
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.Debug
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.MemAvail
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.MemFree
5m:7d,1h:30d,3h:1y,1d:10y 335 0.7.EventWatcher1.Uptime
DataYours
5m:7d,1h:30d,3h:1y,1d:10y 335 0.9.DataYours1.AppMemoryUsed
5m:7d,1h:30d,3h:1y,1d:10y 335 0.9.DataYours1.UDP_RECEIVER_PORT
Office Lights
1m:1d,10m:7d,1h:30d,3h:1y,1d:10y 351 45103840.6.SwitchPower1.Status
Office Wall Lights
1m:1d,10m:7d,1h:30d,3h:1y,1d:10y 351 45103840.7.SwitchPower1.Status Also, LuaUPnP.log doesn’t show any errors either.
Hmmm…
Can you just try the relative path instead (just history/) ?
…or did you already?
That’s it!
First I tried history/, which didn’t work, that’s when I changed it to the full path.
Now I’ve changed it back and did a reload, and it’s writing to the files. Also, the DataYours whisper files are exposed through Historian, which wasn’t the case before.
Thank you!
Glad it’s going. It’s designed to be that simple.
Obviously something amiss with an absolute path that I need to investigate, because there shouldn’t be anything fundamental to stop that working.
I think I know what it is.
I’m running openLuup on ESXi (virtualised). It seems that when I reboot the VM, the time is not correct. openLuup starts and uses this incorrect time, thus logging is not working, even though in the meantime the VM syncs it’s time with an NTP server.
When I restart openLuup, everything works correctly because the VM time has been updated using an NTP server in the meantime and openLuup uses this time.
Now I’m going to make sure the time gets synchronised at startup. Does openLuup check the system clock once in a while?
Ah yes! I had the same problem with my main openLuup system on a BeagleBone Black. It’s now on the same UPS as the optic fibre modem, but another way to fix, of course, it to delay the openLuup startup.
Does openLuup check the system clock once in a while?
No, I thought of that but discounted it since so many things are wrong if the time is wrong, or changes. I suppose I could adopt the ‘Vera solution’ and restart…, but one of openLuup’s features is that it does NOT spontaneously restart!
I took the freedom to make two small changes to the default archive and aggregation rules of Historian:
- Instead of only watching the status of a light (on/off) also watch the dimming state
In servertables.lua, archive_rules:
schema = "every_1m",
patterns = {
"*.*.Status", -- on/off status
"*.*.LoadLevelTarget", -- dim status
},- For security sensor that get tripped, it would make more sense (IMHO) to use SUM instead of average so you can see how many times a sensor has been tripped over time.
In virtualfilesystem.lua, storage_aggregation_conf:
[sum]
pattern = [Ttripped]
xFilesFactor = 0
aggregationMethod = sumMaybe it’s possible to add this to a future release as well?
Yes to both of those ideas, I think.
The details of the archives and aggregations certainly need fine tuning, although not everyone will ever be happy. I’ve certainly used sum for security trips previously myself, and that is really the reason that security sensors have an initial 60 second archive, so that multiple trips in under a minute get counted.
I don’t think that your pattern [Ttripped] is quite correct, since this just matches a single initial character from the list. Perhaps you mean [Tt]ripped
Ah thanks your right, I’ve changed it to Tripped.
Also, I had to change the regex for minima to Min, because [Mm]in matched with Dimming1.LoadLevelTarget.
Now the files get created and they have the right aggregation headers.
The console shows me the Dimming level graph, but it doesn’t get written to the Whisper database on disk. All other variables do get written so it’s not the same problem from yesterday.
I’m toying with a manual override method of displaying and selecting on the History Cache menu which variables to archive. See attached.
The console shows me the Dimming level graph, but it doesn't get written to the Whisper database on disk. All other variables do get written so it's not the same problem from yesterday.
Have you done a reload?
That would be a great feature, nice!
After a openLuup reboot, the LoadLevelStatus files get created but not updated (no updates on the History DB page).
Afer a reload, the files do get updated, but with wrong values I think.
If you compare graph from the values in memory (which are correct) to the graph of the values from the DB (incorrect), it gives two separate charts.
Also, I found out that LoadLevelStatus of all dimmers get logged in files, whereas both LoadLevelStatus and LoadLevelTarget (same graphs/values) get logged in memory.
I’ve added the two graphs for reference, the Memory one shows correct values.
The header of the LoadLevelStatus whisper files are:
1, 315360000, 0, 5
228, 60, 1440
52068, 600, 1008
88356, 3600, 720
114276, 10800, 2920
219396, 86400, 3650Which should be correct considering the every_1m scheme and average aggregation.
I don’t believe that these are incorrect values. The schema you are using has a one minute resolution (for the last hour), with as you say average aggregation, and you are only plotting a two minute period in which all the changes happen in the first minute, and nothing changes in the second minute. The 45, or so, that you see from the archives is in fact the last value from the previous minute. The aggregations only take place after the data has gone into the archive, not before. This is standard Whisper database behaviour, which works well for normal timescales. You could, instead, try an initial archive of one minute’s worth of one second sampling to get finer averages, if needed, but I’m not convinced that for this type of data it’s really required. After the first hour they would get further averaged anyway, with the existing schema.
My initial guesses at the archive durations seemed like reasonable defaults, and actually conform to a number of other constraints which make storage and plotting effective and efficient. But they’re not for everyone.
Also, I found out that LoadLevelStatus of all dimmers get logged in files, whereas both LoadLevelStatus and LoadLevelTarget (same graphs/values) get logged in memory.
Everything should be logged in memory, except for those variables explicitly blocked by the cache rules in the servertables file:
local cache_rules = {
nocache = {
dates_and_times = "*.*.{*Date*,*Last*,Poll*,Configured,CommFailure}",
zwave_devices = "*.ZWaveDevice1.*",
},
}(actually, in the latest, I’ve also added Time)
I should point out that the archive rules only apply to the creation of new files. If you happen to have created, for example, a LoadLevelStatus file previously, then the Historian will spot that on disk and write to it, not caring how it was originally created.
So I think that all is well. Anyway, try things out for a bit longer and see how it transpires.
Hello guys!
I’m pretty new to Vera and openLuup. At first I want to thank akbooer for this awesome work!
I bought a Vera Plus some weeks ago and now I’m playing around with it a bit. I was searching for a way to archive data for analysis if scenes would work properly (for example opening my venetian blinds when my anemeter measures strong gusts, etc.).
After walking throgh DataMine and so on. I saw that you added the Historian feature to openLuup, so I setup a Raspi with openLuup and connected the Vera with Verabridge. Historian is working basically and Grafana os a really nice tool to analyse the data.
Some questions:
- Is the Historian the right tool for longterm archiving data? Is it possible to change the aggregation to save the high resolution data say for example 3 months?
- I read that the resolution thing is configured in the header of the Whisper database files, is there manual somewhere? Is it planed to add the possibility to change these configs from the webconsole?
And a short remark to binary data:
I read through the thread and have seen you guys discussing about binary values and how to process them. I’m a plant automator, so I have quite some experience with archiving analog and binary data for analysis of malfunction and things like that.
I found out that archiving of binary data (in plant automating mostly alarms and things like pump running) works best with timestamp based data triggered by a change, without any aggregation.
In most industry automation systems analog and binary signals are used in different ways.
[quote=“streilu, post:76, topic:199464”]Hello guys!
I’m pretty new to Vera and openLuup. At first I want to thank akbooer for this awesome work![/quote]
Hello! You’re very kind, although perhaps you should wait a bit until you discover whether it’s really right for you! Sorry for the delay in replying to your post, but somehow I missed it earlier.
I setup a Raspi with openLuup and connected the Vera with Verabridge. Historian is working basically and Grafana os a really nice tool to analyse the data.
OK, glad you found it. This is only the first release with this functionality so there may be a few rough edges to work on.
Is the Historian the right tool for longterm archiving data? Is it possible to change the aggregation to save the high resolution data say for example 3 months?
The Data historian is actually three tools in one:
[ol][li]a way to watch all the variables[/li]
[li]somewhere to store them[/li]
[li]graph plotting (built-in code uses Google Charts)[/li][/ol]
In order to make it fast, (2) and (3) are all integrated, and designed to work efficiently together. However, other apps/services can be used, as you’ve found, Grafana for #3, for example. What’s not obvious (and, TBH, not fully exposed to the user ATM) is that, for #2, the data can also be mirrored to external databases, in particular, an external Graphite installation and/or InfluxDB. In order to make this happen efficiently, UDP is used as a protocol since it requires no handshake. Both those databases are easily configured to receive UDP datagrams.
Alternatively, as you suggest, the aggregation is easily changed to retain higher resolution data. However, this will clearly entail an increase in archive file size. It’s basically 36 bytes/point, so, e.g., 10 minutes x 3 months = 12960 x 36 = 467 kB, not too shocking (per variable.)
I read that the resolution thing is configured in the header of the Whisper database files, is there manual somewhere? Is it planed to add the possibility to change these configs from the webconsole?
Actually, the resolution, etc., is stored in the file header, but is defined by rules. I’ve described this earlier in the thread, and the Graphite/Whisper rule syntax is defined here: Configuring Graphite: Storage Schemas. If you want to change the defaults, you should create the files storage-schemas.conf and storage-aggregation.conf in the history/ directory. You can copy the default contents of these from the file openLuup/virtualfilesystem.lua. Once defined, and suitably edited, these will override the built-in defaults.
And a short remark to binary data: I read through the thread and have seen you guys discussing about binary values and how to process them. I'm a plant automator, so I have quite some experience with archiving analog and binary data for analysis of malfunction and things like that. I found out that archiving of binary data (in plant automating mostly alarms and things like pump running) works best with timestamp based data triggered by a change, without any aggregation.
Yes, fair point. The Graphite/Whisper database is ideal for analogue data, but it’s a bit of a squeeze to use it for state information. This is one reason I’m allowing mirroring to external databases. However, the in-memory cache stores events to sub-millisecond resolution, so any ‘real-time’ processing that needs it is possible (for the last 1000 or so events per variable.) A huge plus for Whisper is that it requires exactly NO maintenance and its resource utilisation is fixed at file creation. If I could find a suitable candidate of ‘binary/digital’ data, then I’d consider adding it, although a real problem with Vera is that you can’t tell which variables are going to be ‘analog’ or otherwise.
Hope it does what you need, anyway. Don’t hesitate to ask further.
Hi all,
I found this article to be quite useful to understand the actual Whisper DB file format:
http://falcolas.com/2014/12/24/graphite_whisper/
Also, for binary sensors, I have quite some success with the Sum aggregation method which shows me amount of triggers over a period of time.
Thanks for that, I had not seen it. Although there is almost nothing there that is not in Graphite documentation, it’s a better and more concise read.
Regarding the two ‘weaknesses’ it mentions at the end:
[ol][li]this is exactly the issue you ran into, more of a ‘feature’ than a weakness?[/li]
[li]is not strictly correct for a genuine Graphite system, and even less so for my implementation. Whisper has the option to cache headers, so none of the header reads are done except when opening the file for the first time in a session. I’ve gone further and it turns out that the need to read the first archive entry to establish a time baseline is totally unnecessary if you use a different algorithm for where in the circular file buffer to start writing. It also turns out that this can be made compatible with a genuine Graphite Whisper file.[/li][/ol]
So, actually, Whisper is quite efficient, which is what it was designed to be in the first place.
Regarding the two ‘weaknesses’ it mentions at the end:
- this is exactly the issue you ran into, more of a ‘feature’ than a weakness?[/quote]
I think the way Whisper works is very efficient and smart, but it’s totally different (at least for me) from more SQLesque databases which I’m more used to. I really had to dive in but I think I’ve got quite some understanding of how it works now.
2. is not strictly correct for a genuine Graphite system, and even less so for my implementation. Whisper has the option to cache headers, so none of the header reads are done except when opening the file for the first time in a session. I've gone further and it turns out that the need to read the first archive entry to establish a time baseline is totally unnecessary if you use a different algorithm for where in the circular file buffer to start writing. It also turns out that this can be made compatible with a genuine Graphite Whisper file.So, actually, Whisper is quite efficient, which is what it was designed to be in the first place.
Very true, I think caching can go a long way to optimise disk operations.