These questions put you into the “advanced user” league!
[quote=“bruring, post:3, topic:199628”]Would love to migrate to historian!
I see the structure is a bit different from DataYours. Could you point me in the right direction for the following:[/quote]
You mean the metric naming hierarchy? Yes, for good reason. I feel the need to justify this. There are, in fact, three naming conventions that you may run into when using Data Historian:
[ol][li]Standard Luup naming - full serviceId, variable name, LOCAL device number[/li]
[li]Graphite API naming (as seen by Grafana) - nodeName, REMOTE device number/name, short serviceID, variable name[/li]
[li]Whisper archive files - nodeNumber, REMOTE device number, short serviceId, variable name[/li][/ol]
where:
[ul][li]nodeNumber - 0 for openLuup otherwise PK_AccessPoint or remote Vera [/li]
[li]nodeName - the name of the associated VeraBridge, or “openLuup”[/li]
[li]shortSid - final apha-numeric part of the full serviceId[/li]
[li]shortDevName - device name with all non-alphanumerics removed
[/li][/ul]
Convention (1) is inevitable for Luup access, as in luup.variable_…(), (2) seemed to be the most readable and understandable, (3) is locked to the remote machine ID.
This way, even if the order of the bridges is changed in openLuup, the files are the same. If you have to swap out a Vera, then its PK_AccessCode will change and you’ll have to rename files to continue using them, but the finder metrics names will not change (unless you rename the associated VeraBridge.)
Pattern-matches for schemas, finders, and cacheVariables() ALL use the API paths and wildcards described here: HTTP API — Graphite-API 1.1.3 documentation
- I've set up watched variables and retention resolution and aggregation manually per watched variable. Is there a way to customize this in Historian?
Yes, certainly. There’s two levels of definition/indirection, in order to maintain compatibility with DataYours and the original Graphite/Carbon system on which it is based.
[ul][li]There are two files which may be put in the historian/ data directory (whatever you do, don’t alias or combine this with the DataYours directory): storage-schemas.conf, and storage-aggregation.conf. In their absence, virtual defaults are used. You can find them in the openLuup/virtualfilesystem.lua file. These are the absolutely standard Graphite configuration files which define archive structure depending on the written metric filename. However, for Data Historian, these rules are used in a different way…[/li]
[li]…using the archive_rules found in the openLuup/servertables.lua file. These provide the name of which schema to use based on the LOCALdeviceNo.shortServiceId.variable name pattern used internally by Data Historian.[/li][/ul]
These rules are only used to create missing files. As you seem already to have discovered, if you create a correctly-named file with any archive structure you like, then that file will be used for the matching variable.
I’m sorry this is complicated, but it’s logical, and preserves existing conventions.
- I made symlinks in my Whisper DB folder to give everything more human readable names in Grafana (i.e. I've linked Humidity.Living.wsp to Vera-88800000.10391.urn^micasaverde-com^serviceId^HumiditySensor1.CurrentLevel.wsp, Temp.Basement.wsp to Vera-88800000.10461.urn^upnp-org^serviceId^TemperatureSensor1.CurrentTemperature.wsp). So in the Grafana dropdown query menu's, I can choose Temperature, then Basement. This looks a lot nicer in the Legend as well. Not sure if this is the best way to do it though.
This ugly DataYours naming was something I abandoned, and, in fact, if you were to have used the AltUI Data Providers route (don’t do that now) then you could choose arbitrary names according to your fancy.
If you’ve not defined the openLuup.Historian.DataYours attribute to point to your DataYours Whisper directory, then those metrics will all appear at the top level of the Graphite search tree. If you do define it, then they will be moved to a sub-tree called DataYours. (Equally, there is a DataMine attribute which will bring your dataMine database into the tree as well, but perhaps you have abandoned that already.)
The MODERN way to make your metrics legend and labels look nice is to use the additional functions at the trailing end of the Grafana data chooser: alias, aliasByNode, aliasByMetric. Suppose you have a metric:
openLuup.303_OutdoorTemperature.TemperatureSensor1.CurrentTemperature
then following that metric selection with these functions will give:
[ul][li]alias (“Foo”) - “Foo”[/li]
[li]aliasByMetric - “CurrentTemperature”[/li]
[li]aliasByNode (1) - “303_OutdoorTemperature”[/li]
[li]aliasByNode (0,3) - “openLuup.CurrentTemperature”[/li][/ul]
Magic!
Is there a way to migrate the DataYours WSP database files to Historian?
If I’ve managed to enable you to understand any of the above, then the answer is “yes”. You simply rename the old file to the one which Data Historian expects for that variable and move it into the historian/ directory. WHATEVER YOU DO, do NOT do this when the system is running. Whisper file headers are cached and changing the file mid-stream will completely trash it.
- On a side note: am I correct by saying that variables first get logged in memory, and written to disk every x minutes? So the Grafana charts only update every x minutes?
No, not quite. Variables are written synchronously to the disk archive - this is one reason why you should really have it on a SSD and separately back up the database elsewhere from time to time. It’s written to be very fast, but obviously is at the mercy of I/O speed. Grafana can be configured to update at any rate. The memory cache saves variable update times to sub-millisecond precision. Once they get written to disk they are quantised according to the retention rules. If a Grafana graph time interval falls completely within the memory cache then you get times to maximum precision.
Let me know if there are any solutions for above, then I can migrate everything. If not, I would prefer to fix my DataYours whisper database, and migrate slowly to Historian. It's definitely the next step forward!
Just to curiosity, then, can you tell me what the value of the DataYours CONFIG_DIR is, whether there are actually any files there, and what the link from the DataYours device panel to DataWatcher config shows you?
