Just started out with the whole data logging adventure and am looking forward to being able to observe some of this information over the long haul. As a starting point, I’d like get a basic activity log rolling with the status of lights and security sensors as they change throughout the day. EventWatcher does an excellent job of reporting this information with the standard “log” report but ultimately I would like to have this information in my custom dashboard which means I need the data in json or some other standardized format.
Checking into the development branch of DataYours I found the inclusion of the Graphite API which would appear to be what I need to get the data in the format I need. Following the setup documentation of DataYours I went in and “watched” all of the variables (Last Trip, Tripped ← For Security & Status ← For Lighting) to get the ball rolling.
So now… if I do something like this: data_request?id=lr_render&format=json&target=Vera-50002137.018.urn:upnp-org:serviceId:SwitchPower1.Status I’m able to get what appears to be on/off (status) for Device ID: 18 for every second. So… partial success? Of course what I want to do is gather the same data available in the Event Watcher log but am somewhat lost how to get this info and/or if it’s even possible to get from DataYours. So before I go spending hours if not days trying to retrieve this info from DataYours I’m hoping to confirm #1…that this is possible and #2 Any tips on getting this data?
I’ve played around with wildcards etc as suggested in the Graphite API documentation but I’m just not having any luck… Any direction would be greatly appreciated!
Need a bit more on your existing system and planned architecture for your ultimate destination.
[ul][li]are you running on Vera or openLuup?[/li]
[li]do you care what protocol is used to send you the data?[/li]
[li]are you really interested in the real time updates or the database archive?[/li][/ul]
The DataYours front-end (DataWatcher) is very able to capture the basic information as it happens. I’m also using the Whisper database, and these days my dashboards are all in Grafana, which pulls the data through the Graphite_API. I can post examples if you need.
I’m currently running on Vera and logging to a flash drive (although openLuup has always been a bit of a goal).
With regards to the protocol…this is a bit over my head unfortunately so I’m not sure how to answer. I hadn’t really considered the concept of pushing the data to me… In my mind I had anticipated querying DataYours to supply the data every few seconds. This may put unnecessary strain on Vera though if I’m constantly polling it?
Real time updates or at least close to realtime would be ideal. In my perfect world I’d have a JSON string with the last 24 hours of data that would be updating constantly.
Since I’m probably not doing the best job with explaining this on a technical level (My coding experience is limited to web languages) a good example of what I’m trying to achieve is shown in the attached screenshot. It’s effectively the exact same info that EventWatcher provides but I just want to style it up and have it on my own dashboard.
Would love to see any examples you are willing to provide I’m happy to explore all the documentation etc but wanted to confirm that I was at least using the appropriate plugin to do so. Thank you very much for the direction and all of the great work you have put into this and many other plugins!
Well, that is rather inefficient, since Vera doesn’t have a generic lazy-polling mechanism for user-written web servers.
Real time updates or at least close to realtime would be ideal. In my perfect world I'd have a JSON string with the last 24 hours of data that would be updating constantly.
That doesn’t sound very perfect since most of the data will be the same each time? You’re probably going to have to do a bit of work to marshal the incremental changes.
Since I'm probably not doing the best job with explaining this on a technical level (My coding experience is limited to web languages) a good example of what I'm trying to achieve is shown in the attached screenshot. It's effectively the exact same info that EventWatcher provides but I just want to style it up and have it on my own dashboard.
That’s helpful. You’re interested more in EventWatcher-like log data than in raw measurements. Whilst the front end data captures of both EventWatcher and DataYours are similar, the subsequent treatment of data is quite different. EventWatcher adds meta-data like device name, etc. DataYours simply puts this into a database and is more suited to regularly-sampled data (like temperature) than random events (like a door lock) although it can handle either.
Would love to see any examples you are willing to provide :)
Attached in an example of a dashboard rendered with Grafana and pulling data directly out of a DataYours (Whisper) database. In this case, the database is polled every five minutes, so that’s not a significant overhead. The data goes into the database quite asynchronously from that. You can see that this is a rather different concept from where you’re going.
I'm happy to explore all the documentation etc but wanted to confirm that I was at least using the appropriate plugin to do so.
Having responded to the above, I come now to the conclusion that, unless you want to do a lot of work yourself, neither EventWatcher nor DataYours is perhaps the way to go. EventWatcher is closer and can write understandable (ie. ASCII) data to a log file, but more than that, you’re on your own. There may be other plugins or apps which are closer to what you need, but I don’t know, although apps which handle Alerts might be suitable (VeraAlerts?)
Thanks @akbooer! This is really helpful even just ruling out DataYours / Event Watcher so thank you for sharing your wisdom
Really clever idea to use VeraAlerts for this functionality… I hadn’t even considered this but I can see how this might work well now that you mention it!
Off to explore these additional possibilities now!
@akbooer
I am starting logging data from my veraplus to a RP3 with openluup and DataYours
To install openluup and datayours I used the image file from this forum
I also installed Grafana following the instructions from NikV
All appears well but the data will not appear in the metrics tab
I tried both storing the data in cmh-ludl and in smh-ludl/whisper
Do you have an idea where to look?
Regards,
Jacob
@akbooerFound the problem
Installing datayours into openLuup did not create directory /www/
Did it manually and performed an update via More->Plugins
After this the file DataWatcher.conf populated the directory /www/
Now I could select the information in the grafana dashboard
There is an other issue you may have a solution for.
After a reload of the luup-engine the collection of data into the wisper database files is stopped.
I have to restart is by selecting the datapoints in the datayours web interface and select watch
Regards,
Jacob
There is an other issue you may have a solution for.
After a reload of the luup-engine the collection of data into the wisper database files is stopped.
I have to restart is by selecting the datapoints in the datayours web interface and select watch
Regards,
Jacob
this should now be sorted when you create the /www/ directory it needs datawatcher.conf to persist past an openluup reload
I’m happy to explore all the documentation etc but wanted to confirm that I was at least using the appropriate plugin to do so. Thank you very much for the direction and all of the great work you have put into this and many other plugins!