openLuup: Data Historian

:slight_smile:

Quick update - the CurrentTemperature values are all plotting like a dream. CurrentSetpoint and SwitchpowerStatus only display when I set the timeframe to last 7 days. Part of me was thinking only a change in value is recorded but then SwitchpowerStatus has not changed in value ever so that probably is not it. Console/Files/History DB is still showing ‘On-disk archiving not enabled’ if that makes a difference.

Has your system been up and running continuously for just over 7 days?

Part of me was thinking only a change in value is recorded but then SwitchpowerStatus has not changed in value ever so that probably is not it.

Except for startup, when current values are written, only variable changes are stored in cache (this saves memory space.) However, all variable_set values are written to disk archive. This helps with the application of retention policies between archives.

Console/Files/History DB is still showing 'On-disk archiving not enabled' if that makes a difference.

AHA! A eureka moment.

Am I correct in saying that you have not ever installed DataYours on your openLuup system? This is fine, you don’t need it, but in early implementations I used one of its files and I did have it installed, so I was not getting this message.

I believe that you are only seeing data from the in-memory cache, although it will be archived to disk… you just can’t see it!

All fixed (I hope) in development release v18.7.19

Bingo - as you deduced there is no Datayours installed. History DB is now populated and all is appearing on Grafana. Huge thanks.

Excellent. Thanks for your help… several bugs squashed in the process!

I’m seeing a bug in the latest development version. Setting the startup luup.attr_set (“openLuup.Historian.Directory”,“/media/usb/whisper/”) does not start the archive writing to disk (though the cache is functional and can be accessed from the historian tab on the console.)

If I change the attribute pointer to luup.attr_set (“openLuup.Historian.Directory”,“/media/usb/whisper/data”), the process of writing to disk begins but each whisper file name is prepended with “data” and accordingly, cannot be opened from the historian tab (I get a series of lua errors.)

The three core datayours metrics write to disk regardless of the startup attribute.

Can I see the beginning of the actual log just after startup?

This should look something like:

2018-07-20 23:04:02.901   :: openLuup LOG ROTATION :: (runtime 0.0 days) 
2018-07-20 23:04:02.939   openLuup.init:: init phase completed
2018-07-20 23:04:02.940   openLuup.io.server:: starting HTTP server on port: 3480 tcp{server}: 0x19b7a90
2018-07-20 23:04:02.940   openLuup.io.server:: starting SMTP server on port: 2525 tcp{server}: 0x1967ca8
2018-07-20 23:04:02.940   openLuup.io.server:: starting POP3 server on port: 11011 tcp{server}: 0x745890
2018-07-20 23:04:02.941   openLuup.historian:: starting data historian
2018-07-20 23:04:02.941   openLuup.historian:: using on-disk archive: history/
2018-07-20 23:04:02.941   openLuup.historian:: Graphite schema/aggregation rule sets: 15/3
2018-07-20 23:04:02.941   openLuup.historian:: disk archive storage rule sets: 9
2018-07-20 23:04:02.941   openLuup.historian:: mirroring archives to Graphite at 127.0.0.1:2003
2018-07-20 23:04:02.941   openLuup.historian:: using memory cache size (per-variable): 1024
2018-07-20 23:04:02.942   openLuup.scheduler:: starting
2018-07-20 23:04:02.942   openLuup.scheduler:: [2]     openLuup device startup
2018-07-20 23:04:02.942   luup_log:2: v18.7.19

The directory path should end with ‘/’

With startup luup.attr_set (“openLuup.Historian.Directory”,“/media/usb/whisper/”)

2018-07-20 15:39:36.254 :: openLuup LOG ROTATION :: (runtime 0.0 days) 2018-07-20 15:39:36.255 openLuup.init:: init phase completed 2018-07-20 15:39:36.255 openLuup.io.server:: starting HTTP server on port: 3480 tcp{server}: 0x1e30bc48 2018-07-20 15:39:36.255 openLuup.io.server:: starting SMTP server on port: 2525 tcp{server}: 0x1e688758 2018-07-20 15:39:36.255 openLuup.io.server:: starting POP3 server on port: 11011 tcp{server}: 0x1e115848 2018-07-20 15:39:36.256 openLuup.historian:: starting data historian 2018-07-20 15:39:36.256 openLuup.historian:: using on-disk archive: /media/usb/whisper/ 2018-07-20 15:39:36.258 openLuup.historian:: Graphite schema/aggregation rule sets: 17/16 2018-07-20 15:39:36.258 openLuup.historian:: disk archive storage rule sets: 9 2018-07-20 15:39:36.258 openLuup.historian:: using memory cache size (per-variable): 1024 2018-07-20 15:39:36.258 openLuup.scheduler:: starting 2018-07-20 15:39:36.258 openLuup.scheduler:: [2] openLuup device startup 2018-07-20 15:39:36.258 luup_log:2: v18.7.19 2018-07-20 15:39:36.258 luup_log:2: synch in 23.7 s 2018-07-20 15:39:36.258 luup.variable_watch:: callback=openLuup_watcher, watching=2.openLuup.HouseMode 2018-07-20 15:39:36.258 luup.register_handler:: global_function_name=openLuup_email, request=openLuup@openLuup.local 2018-07-20 15:39:36.258 luup.register_handler:: global_function_name=openLuup_images, request=images@openLuup.local 2018-07-20 15:39:36.258 luup.register_handler:: global_function_name=openLuup_events, request=events@openLuup.local 2018-07-20 15:39:36.259 luup.register_handler:: global_function_name=openLuup_mailbox, request=mail@openLuup.local 2018-07-20 15:39:36.259 luup.chdev.append:: [AltAppStore] Alternate App Store 2018-07-20 15:39:36.259 luup.chdev.sync:: [2] openLuup, syncing children 2018-07-20 15:39:36.259 luup_log:2: 12 Mb, cpu 3%, 0 days 2018-07-20 15:39:36.260 openLuup.scheduler:: [2] openLuup device startup completed: status=true, msg=synch in 23.7 s, name=L_openLuup 2018-07-20 15:39:36.260 openLuup.scheduler:: [3] Alternate UI device startup 2018-07-20 15:39:36.260 luup_log:3: ALTUI: initstatus(3) starting version: v2.29 2018-07-20 15:39:36.260 openLuup.scheduler:: [3] Alternate UI device startup completed: status=nil, msg=nil, name=nil 2018-07-20 15:39:36.260 openLuup.scheduler:: [4] Alternate App Store device startup 2018-07-20 15:39:36.260 luup_log:4: AltAppStore : starting... 2018-07-20 15:39:36.260 luup.variable_set:: 4.urn:upnp-org:serviceId:altui1.DisplayLine1 was: AltAppStore now: AltAppStore #hooks:0 2018-07-20 15:39:36.261 luup.variable_set:: 4.urn:upnp-org:serviceId:altui1.DisplayLine2 was: now: #hooks:0 2018-07-20 15:39:36.261 luup_log:4: AltAppStore : v18.6.28 2018-07-20 15:39:36.261 luup.set_failure:: status = 0

With luup.attr_set (“openLuup.Historian.Directory”,“/media/usb/whisper/data”)

2018-07-20 15:44:20.222 :: openLuup LOG ROTATION :: (runtime 0.0 days) 2018-07-20 15:44:20.223 openLuup.init:: init phase completed 2018-07-20 15:44:20.223 openLuup.io.server:: starting HTTP server on port: 3480 tcp{server}: 0xea07a98 2018-07-20 15:44:20.223 openLuup.io.server:: starting SMTP server on port: 2525 tcp{server}: 0xe1bb5a8 2018-07-20 15:44:20.223 openLuup.io.server:: starting POP3 server on port: 11011 tcp{server}: 0xe6ac1b8 2018-07-20 15:44:20.223 openLuup.historian:: starting data historian 2018-07-20 15:44:20.223 openLuup.historian:: using on-disk archive: /media/usb/whisper/data 2018-07-20 15:44:20.225 openLuup.historian:: Graphite schema/aggregation rule sets: 15/3 2018-07-20 15:44:20.226 openLuup.historian:: disk archive storage rule sets: 9 2018-07-20 15:44:20.226 openLuup.historian:: using memory cache size (per-variable): 1024 2018-07-20 15:44:20.226 openLuup.scheduler:: starting 2018-07-20 15:44:20.226 openLuup.scheduler:: [2] openLuup device startup 2018-07-20 15:44:20.226 luup_log:2: v18.7.19 2018-07-20 15:44:20.226 luup_log:2: synch in 99.8 s 2018-07-20 15:44:20.226 luup.variable_watch:: callback=openLuup_watcher, watching=2.openLuup.HouseMode 2018-07-20 15:44:20.226 luup.register_handler:: global_function_name=openLuup_email, request=openLuup@openLuup.local 2018-07-20 15:44:20.226 luup.register_handler:: global_function_name=openLuup_images, request=images@openLuup.local 2018-07-20 15:44:20.226 luup.register_handler:: global_function_name=openLuup_events, request=events@openLuup.local 2018-07-20 15:44:20.226 luup.register_handler:: global_function_name=openLuup_mailbox, request=mail@openLuup.local 2018-07-20 15:44:20.227 luup.chdev.append:: [AltAppStore] Alternate App Store 2018-07-20 15:44:20.227 luup.chdev.sync:: [2] openLuup, syncing children 2018-07-20 15:44:20.227 luup_log:2: 12 Mb, cpu 3%, 0 days 2018-07-20 15:44:20.228 openLuup.scheduler:: [2] openLuup device startup completed: status=true, msg=synch in 99.8 s, name=L_openLuup 2018-07-20 15:44:20.228 openLuup.scheduler:: [3] Alternate UI device startup 2018-07-20 15:44:20.228 luup_log:3: ALTUI: initstatus(3) starting version: v2.29 2018-07-20 15:44:20.228 openLuup.scheduler:: [3] Alternate UI device startup completed: status=nil, msg=nil, name=nil

And the error message when trying to open a variable. And the directory listing.

Simply rename all the files to start with 0 rather than data0 and all should be well after a restart. I’ll add some checks to catch a path not ending in /

It’s straight-forward to write a Lua script to do the rename if you don’t otherwise know an easy way to do it for all the files.

…or just delete them and start again.

Another problem I note:

…you must NOT mix the historian directory with a regular DataYours Whisper database. They must be kept separate, although the Grafana interface is able to access and display data from both separate directories.

I think the combined folder situation was the culprit. I created a folder explicitly for the historian data and after a startup pointer, the folder immediately populated upon a luup reload.

Hi - am tracking a ‘Tripped’ state which is either 1 or 0. The google graph from console shows a value of 0.5. Not sure how this has happened? Have attached what I believe to be the wsp file …

Yes, this is not a surprise, I know what it is…

The default action for aggregating values from one archive to the next (lower time resolution) one is average. For some things, like temperature and the like, this is exactly what you want. For others, it may not be, this being a case in point.

The choices you have for aggregation functions are:

[ol][li]average[/li]
[li]sum[/li]
[li]last[/li]
[li]max[/li]
[li]min[/li][/ol]

You can, retrospectively, change the aggregation function on an existing archive file, and although this will not change the existing data in the file, it will ensure that this does not happen for new data.

The tricky question is: what aggregation function is correct for your situation? Depending on the sensor type, you could, say, use: average to calculate a room occupancy; sum for the total number of trips in that time interval, last for the most recent state in that interval, … (max/min unlikely to be useful for security-like data)

You can set up rules to automatically configure whichever you want for any new metrics of the same type, but that’s a somewhat separate topic.

There is a Whisper API function to make this change to a file, but it really needs to be wrapped into a simple-to-use way. I have on the ‘to do’ list, a utility to do this, but in the meantime, here’s one rationale for having chosen plain text as the Whisper file format - you can change the aggregation with a normal text editor. The file is fixed-width format, so be careful about keeping the number of characters in a line the same. The aggregation is indicated by a single digit, per the list numbering above, and is the first number in a whisper archive file.

For example, the first few lines of your attached file are:

          1,  315360000,          0,          6
        264,          1,         60
       2424,         60,       1440
      54264,        600,       1008
      90552,       3600,        720
     116472,      10800,       2920
     221592,      86400,       3650
          0,                      0
 1531746841,                      1

Changing it to:

          2,  315360000,          0,          6
        264,          1,         60
       2424,         60,       1440
      54264,        600,       1008
      90552,       3600,        720
     116472,      10800,       2920
     221592,      86400,       3650
          0,                      0
 1531746841,                      1

will switch it to sum aggregation. Be careful that your editor doesn’t mess with line terminators.

You can read more about Whisper archives here: The Whisper Database — Graphite 1.2.0 documentation

Super - I’ll try 3 for last and see how things turn out. Many thanks.

OK. Let us know how it goes.

Actually, max is not such a bad idea either - it will tell you if the sensor was ever tripped at any time in that interval.

Well - gave it a go : ) Edited said file in WinSCP by changing just that number. Hoping I didn’t muck up any formatting. Wierdly some nil files have returned for that vera box? Also, even though there are wsp files for the variables I was tracking the console & grafana seem unable to access them.

No idea why the nil files are back, but you need to delete them and restart. Although, before you do you might check the creation time and see if that happened after your edit. Things will not work until they’re gone.

I don’t like the proliferation of this line in the logs:

2018-07-28 11:33:52.950   openLuup.http:: WGET status: timeout, request: http://192.168.1.170/port_3480/data_request?id=status2&output_format=json&DataVersion=

I’m not able to tell which machine that corresponds to… is it the one with nil node name again? Looks like a network issue, OR a constantly crashing Vera??

You’ve done an openLuup update since we previously fixed that? HOW are you doing that exactly? Once again, it would be very diagnostic to have the startup of the main log directly after a reload.

I’ve added further checks in VeraBridge to flag errors when it can’t access the remote Vera. This should stop the Historian creating archive files with nil node names.

Github development branch commit 2018.07.29