DataYours - a Data Storage Provider for AltUI

Plugins & Plugin Development DataYours
1

DataYours now provides a local, or remote, data storage provider within the AltUI framework, and works on Vera and under openLuup on any machine. It enables the use of the AltUI interface to archive and plot device variable history. The source files are available from GitHub: GitHub - akbooer/DataYours: Pure Lua implementation of the Graphite / Carbon Whisper database system

DataYours will register itself with AltUI, if installed, and be available as a Data Storage Provider under the graphing menu for each variable. A simple naming convention with a one-letter suffix will, by default, configure a data archive for that variable to be:

[ul][li].d - one minute resolution for one day[/li]
[li].w - five minute resolution for one week[/li]
[li].m - twenty minute resolution for one month (30 days)[/li]
[li].q - one hour resolution for one quarter (90 days)[/li]
[li].y - six hour resolution for one year[/li][/ul]

The database can be configured to be local to Vera (or openLuup) or on a remote machine, through the setting of the LOCAL_DATA_DIR parameter.

Attached is an example of system memory usage plotted using the name “memory.d”, although not a full day’s data yet populates the graph.

The mapping from name extension, like .m, to the archive structure is done with a standard DataYours configuration file [tt]storage_schemas.conf[/tt] which should be located in LOCAL_DATA_DIR. In the following code snippet are the schema rules which will implement the above mapping. They can, of course, be changed to suit. These rules are only use in the creation of new files, so it is also possible to create a specific archive structure for a particular metric and just use that name without any extension modifier.

#
# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is read whenever a file create is required.
#
#  [name]  (used in log reporting)
#  pattern = regex  
#  retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...

#  2016.01.24  @akbooer
#  basic patterns for AltUI Data Storage Provider

[day]
pattern = \.d$
retentions = 1m:1d

[week]
pattern = \.w$
retentions = 5m:7d

[month]
pattern = \.m$
retentions = 20m:30d

[quarter]
pattern = \.q$
retentions = 1h:90d

[year]
pattern = \.y$
retentions = 6h:1y

Since DataYours is inherently a modular, distributed system, the data may be replicated on local and remote databases. It is also possible to run without DataYours installed on the local system at all, sending everything to a remote machine, but there needs to be a one-time manual registration of the remote machine as an AltUI Data Storage Provider. The AltUI interface for saving and plotting the data remains, in either case, the same (and is far more straight-forward than the dashboard interface provided by DataYours.)

Example screenshot attached below.

2

A plot of archived memory usage after one day. Since the archive is for one day only, and it is a round-robin database, the file size will remain fixed and data older than one day automatically overwritten. Additional graphic options are available for the plot command, in this case an area fill mode to enhance the plot.

3

Wondering How to get dataYours listed under the graphing menu when using openluup. It is installed and saving data… but for some reason does not show graphing option in ALTUI.

4

Are you using the latest version? It registers with AltUI as a data provider and should appear on the list of one of the storage options.

5

I’m 2 weeks behind the current openluup development branch. Considering the DataYours was updated back in January the current running development version running on the system should not be a problem.

Here are some screenshots of the current DataYours configuration and the ALTUI graphing options listed.

wondering if I could be missing something.

6

I’ve copied the .lzo files from the installed version on the vera controller, uncompress and moved them into the opneluup directory…

I realize I was using a DataYour outdated version after searched for “RegisterDataProvider” and there was nowhere to be found.

Got a fresh copy from github and DataYours is now listed under graphing tools but I’m getting the following error…

./L_DataWhisper.lua:622: unable to open Whisper file ‘/var/lib/graphite/whisper/unknown.wsp’

7

Ah, better.

I was just searching through my log. You should now see at startup, something like:

2016-05-17 21:45:34.178   luup_log:7: DataWatcher: registering with AltUI [3] as Data Storage Provider

The file read error is because the new data file has not yet been written to… it needs a data variable update! You should find that this goes away after a while, or a reload, or you can force it with a manual update, and you’ll start seeing your data.

8

Everything seems to be working as expected @akbooer… Although I’m trying to see if I can use the regular graphite interface to later see if I can get grafana to plot the data being store by DataYours.

DataYours and AltUI can show the .wsp files just fine but graphite can’t process any data stored by dataYours. something that I noticed is that the .wsp files created by dataYours are different than the ones created by graphite or by storing data like “echo “test.count 4 date +%s” | nc -q0 127.0.0.1 2003” to create new .wsp files

dataYours data is in the clear (plain comma delimited text)
and other wsp files are in binary mode.

Wondering if there is a way for graphite to read files generated by dataYours?

9

I previously made some adjustments for other to try Grafana, http://forum.micasaverde.com/index.php/topic,32701.0.html, although I’m unsure as to what degree of success they had. I’m very keen to support that.

dataYours data is in the clear (plain comma delimited text) and other wsp files are in binary mode.

Yes, that’s well documented in the DataYours guide:

…however, I don’t think that it’s necessary to have binary compatibility of .wsp files, because seeing, in the above referenced thread, HTTP requests like:

grafana/metrics/find/?query=*

leads me to believe that it accesses via HTTP rather than direct file access, which indeed is the usual Graphite way of doing things. In fact, there’s a great piece of work here: Graphite-API documentation — Graphite-API 1.1.3 documentation, and here: GitHub - brutasse/graphite-api: Graphite-web, without the interface. Just the rendering HTTP API., which implements these calls and which I have also already translated from Python to Lua. This provide generic ‘storage finders’ which can be customised to access any database and I’ve already written one for dataMine, as well as DataYours.

The problem with doing this on Vera was that you would have to write a separate server to avoid the usual request syntax. With openLuup, however, I can easily arrange that the native-syntax graphite_api requests work correctly (on port 3480.)

Wondering if there is a way for graphite to read files generated by dataYours?

To reiterate, I don’t think that’s what you want if your final goal is Grafana. However, if you do want that, and you like Python, I’d suggest writing your own storage finder using the graphite_api code linked to above.

10

At last, I’ve made some progress with this. A new implementation of the DataYours back-end, utilising graphite_api (translated into Lua) shows promise, as per the attached screen shot. This would mean we can abandon the terrible DataDash dashboard implementation entirely. AltUI already provides the hooks in the form of Data Storage Providers to create and manage which Vera/openLuup variables to record.

This example actually shows data coming from both a Whisper database (in yellow, as used by DataYours) and a dataMine database (in green) since I’ve written storage finders for both. Other data formats could be handled the same way.

At the moment, I’ve implemented this as an openLuup CGI module, which makes the Grafana HTTP calls easy to handle. It’s still work in progress as I seem to be having a bit of a learning curve with Grafana templates.

[hr]

Edit: added screen capture of a complete dashboard.

11

I’ve bumped into 3 issues with DataYours on AltUI using openLuup:

#1 I used a space in the Metric name in the push as in “Energy Usage”. This doesn’t work well. Part of the code thinks that is the name, but the file it tries to open in the whisper directory is “Energy”. I could not get this changed in the GUI and eventually removed it in user_data.json and then it worked.

#2 I am recording data from a smart meter and somehow it got a bad value. This was put into the whisper database as “nan” (not a number?). Unfortunately this causes the graph in AltUI to not be displayed with no error or information on why it is not there. Could the push to DataYours filter these out? Or replace them with 0 or the last value? And could the graphing routing ignore them? Or at least tell you that you have a bad value?

#3 I cannot add any more push variables in the GUI. I can click on the graph button on the variable and select DataYours and set the metric name, but it doesn’t stick. Nothing is sent to openLuup and nothing is saved. I can edit the user_data.json directly and add push variables, but I can’t do this through the GUI. What am I doing wrong? I tab to the next field, I press return - nothing seems to make it stick. There is no ‘save’ button, just a ‘close’ button. Any ideas?

Otherwise this works well. Thanks.

12

Don’t do that. The field is a metric name, not a plot title or free text, or anything like that. One of the reasons this breaks is that the (industry standard) Whisper plaintext format used to send data to the cache back end is of the form “metric value timestamp” (space separated) so it ends up being stored in a Whisper file called Energy.wsp with invalid value and timestamp.

#2 I am recording data from a smart meter and somehow it got a bad value. This was put into the whisper database as "nan" (not a number?). Unfortunately this causes the graph in AltUI to not be displayed with no error or information on why it is not there. Could the push to DataYours filter these out? Or replace them with 0 or the last value? And could the graphing routing ignore them? Or at least tell you that you have a bad value?

Bad values seem to happen fairly frequently to (my) meters. Whether this is a Z-wave transmission error or a device firmware problem, I don’t know. “nan” is a perfectly valid entry in the Whisper database (as is any non-numeric value) and indeed represents NaN, or in fact null. This is not a problem at all - my files are riddled with NaNs - it’s really a consequence of fixed sample rates in each of the archives. The graphing routine does ignore them, or can be configured to repeat the last valid value, or can treat it as zero. All defined by command-line parameters that you can edit in to the plot line in the data watch panel for that variable. I don’t know why you’re not seeing anything. Perhaps there hasn’t been a valid data update to the variable yet?

#3 I cannot add any more push variables in the GUI. I can click on the graph button on the variable and select DataYours and set the metric name, but it doesn't stick. Nothing is sent to openLuup and nothing is saved. I can edit the user_data.json directly and add push variables, but I can't do this through the GUI. What am I doing wrong? I tab to the next field, I press return - nothing seems to make it stick. There is no 'save' button, just a 'close' button. Any ideas?

You’ve done it once at least correctly? Check the AltUI docs on this. You need to make your entries on the page and then press the (red) “graph” icon button at the top (the same one you used to enter the page.) Also, the configuration will not be permanently saved until the next user_data.json checkpoint (might be up to 6 minutes away) or the next Luup reload (whichever the sooner.)

13

Yes, now that I know spaces aren’t good, I won’t use them. It would be nice to get the code to print a warning, or not accept it, or something else, but I’m aware of it so it won’t be a problem for me in the future.

I had plenty of data but the graph was just blank with the ‘nan’. I edited the whisper file to correct this single value and then went back to the browser and then the graph was fine. I’ll check out the graph options. This is the graphing accessed via AltUI - I think you have other methods as well.

Yes, the GUI once before let me put in values to be graphed and saved them. The graph button does change color, I select DataYours. But it seems that this is not saved now. I close the window after setting the metric. After this, I can go right back into it and see that it was not saved. And nothing pops up in the openLuup log. Perhaps I should be more patient? Does it take some time for the AltUI to send this over to openLuup?

14

Any UI change should immediately be reflected in openLuup device variables. If you’re not seeing any variables change in the log when you do this, then it’s not happening.

Sounds like an AltUI-related thing? Maybe some invalid data somewhere in its device variables?

15

A restart of the Chrome browser seemed to clear up my entry issues.

But I still get a blank graph when I have a ‘nan’ in the data. Chrome says:

data_request?id=lr_render&target=EnergyUsage.e&hideLegend=true&height=250&from=-y:2922 Uncaught ReferenceError: nan is not defined(?)

This is on the openLuup system with AltUI. Is there an option I can use to tell it to ignore the 'nan’s?

Also, I tried out the multi-storage and I can see the data getting populated in the .wsp file appropriately, but I’m not sure how to graph that data. Any examples? Or are you viewing this outside of AltUI with grafana? Sorry, I’m not quite up to speed on the graphing and would appreciate some pointers. Thanks.

16

I seem to be having some problems with dropped data points. I have aeotec 6-1 sensor installed on vera lite and AltUI, DataYours and DarkSky installed on BeagleBone. I configured whisper via the dataYours UI web page to record humidity readings from the Aeotec and DarkSky every 30 minutes. Vera is set up to poll humidity every 15 minutes. DarkSky data is mostly there, but the data from Vera is real spotty. DarkSky outages may correspond to times when I was messing with the system. I’m new to the software, so is there something simple I’m missing about how this is supposed to work?

17

I’d like to use the ‘derivative’ function in Grafana to show the change in accumulated energy (kWH) as read from my electric meter (i.e., power over time). But I can’t get this to work. Nothing shows up on the graph. Is Grafana doing this derivative calculation? Or is it requesting that the DataYours provide this?

If I don’t use the derivative function I can see the stored values, but that isn’t what I want to plot. So Grafana is working, just not this function.

18

This is a complex issue because of the relationship between Zwave device settings, Vera polling, your selected archive structure, and the plotting options…

[ul][li]it’s quite likely that the Aeotec sensor only reports values when they change[/li]
[li]Vera polling is not a reliable indication of how often things will actually be sampled[/li]
[li]I use DarkSky with 15 min polling and 10 min archive sampling (for historical reasons), so for me every other sample is null[/li]
[li]plotting can be configures to ignore nulls, in which case the lines are connected[/li][/ul]

Connected lines across intervals with many nulls can give very odd looking results, so a good option here is to use ‘staircase’ plots which hold the previous value until the next updated one.

19

[quote=“jswim788, post:17, topic:190766”]I’d like to use the ‘derivative’ function in Grafana to show the change in accumulated energy (kWH) as read from my electric meter (i.e., power over time). But I can’t get this to work. Nothing shows up on the graph. Is Grafana doing this derivative calculation? Or is it requesting that the DataYours provide this?

If I don’t use the derivative function I can see the stored values, but that isn’t what I want to plot. So Grafana is working, just not this function.[/quote]

Grafana doesn’t compute derivatives, the function is passed to the Graphite system, in this case DataYours. Currently no functions are implemented in DataYours, although I have a new, much more compatible, graphics server in the works which is based on Graphite-API (but translated in Lua) as described here GitHub - brutasse/graphite-api: Graphite-web, without the interface. Just the rendering HTTP API.

This version could, in due course, have functions easily added (even user-definable.) However, this is a long way off release. In the mean time, you could try the solution I use for power meters and published (long ago) here NorthQ Watt Calculator

20

Well that explains my issue. I look forward to the new version when it arrives. I can implement the workaround similar to what you did in your other post. I thought I had an easy way to do with with the function, but the other way can work. Thanks for the quick response.