Hey Guys,
Looking for some advice on how to best set up Data(mine/yours/somethingelse) in my home. Trawled the forums and have some general ideas, but I couldn’t find any examples where all these devices were available for use.
I have an Edge (which does all the Z-wave devices and control logic for 60 devices, mysensors and stacks of apps). A Vera 3 to which I have 7 IP Cams connected (another 6 to add) a Vera Lite which does bugger all, a Raspberri Pi running a Bluetooth scanner and Unifi controller, and a QNAP NAS which is recording all the camera video.
I don’t want to add more load to the Edge as it’s liable to implode, but I want to log a lot of activity, power and other data from it.
Given the resources above, how would you go about setting up a data visualisation solution?
I have three VeraLites, one Edge, and three openLuup machines (Arduino Yun running OpenWRT, BeagleBone Black running Debian, RaspberryPi 3 running Raspian.)
All of them have DataYours configured to run the DataWatcher front-end and send the data (via UDP datagrams) for storage to two different locations. Those two locations (one on a VeraLite and one on the Arduino Yun) also have DataCache daemons running as part of their DataYours configuration and have a NAS mounted with CIFS for the database storage.
All the machines run the AltUI interface and several of them also use AltUI’s built-in Data Storage Provider logging to record and display data using DataYours. Aside from that interface, the DataGraph daemon on DataYours renders SVG plots to several custom webpages.
The system has been running reliably in one form or other (originally just on one VeraLite) for around three years. I use the plots every day, mostly for energy use and PV generation monitoring, but also for internal temperatures and humidity, and Vera system health (memory, cpu, reboots.) DataYours also plots dataMine data and graphs which go back several years earlier. The legacy dataMine data has 102 channels and the current DataYours metric count is 189.
Although it sounds complex, it is not difficult to install and configure. DataYours, and the Graphite/Carbon system on which it is based, is designed from the ground up to be a distributed system. The dual storage back-end ensures data integrity in the case of one disk failure, without the need for separate backups.
Edit: I should also add that it can efficiently monitor power usage on each machine, with minimal setup, using (open)Luup’s built-in ‘Live Energy Usage’ http://wiki.micasaverde.com/index.php/Luup_Requests#live_energy_usage
Thanks Akbooer.
Your setup sounds like it has the flexibility I’m looking for, multiple capture points, offload processing and storage somewhere else.
I’ve been looking to play with Openluup and ALTUI, but I’ve been waiting for them to stabilise a bit in terms of updates. By my own admission I’m basically lazy and don’t want to be updating stuff every week. 
Are they at a place where this can all be put together and left to run, or at least are there simple auto update mechanisms?
Yes to both. AltUI and openLuup have been pretty robust from early days. openLuup has supported AltUI auto-updates for a while, initially from the MiOS Trac repository, but now both can update themselves from GitHub (a much better place.)
DataYours has a one-click install under openLuup and can also update from its own GitHub repository (although updates are much less frequent, it has been very stable for quite a while.)
Awesome - sounds like it’s time to fill the fridge full of beer and install OpenLuup on the RPi3.
I remember when my idea of a fun way to spend an evening was very different. 
Hi Akbooer,
Been trying to install the Datamine Plugin on my Vera Lite (1.7.830) to play around with it, before figuring out a deployment scenario. Been following your latest docco on GitHub.
I have a problem in that I’ve installed it from the app stopre, but the there’s no device created on the devices page. I can go into Installed apps and follow the link to the instance it’s created, but there’s no menu options other than rename or change the room. Nothing else to select, so I can’t get into the variables to do the next step.
Any pointers on to what’s gone pearshaped with the plugin install?
You are, in fact, meaning DataYours, not dataMine?
I have a problem in that I've installed it from the app stopre, but the there's no device created on the devices page. I can go into Installed apps and follow the link to the instance it's created, but there's no menu options other than rename or change the room. Nothing else to select, so I can't get into the variables to do the next step.
When you say “app store”, you mean the MiOS one?
Any pointers on to what's gone pearshaped with the plugin install?
Just wondering if it’s not simply the case for needed a browser page refresh? But perhaps not by this time.
Hi akbooer,
Yes, Datayours sorry. Been reading the Datayours documentation for the last hour and I swear it mentions Datamine just as frequently. :-[
MiOS install was the source. I uninstalled, reinstalled, rebooted a few times and finally it came good. Was able to get it running and play around with the Dashboard, and put the watcher on my main Vera. Impressive stuff.
Couple of questions:
- Is it possible to plot multiple series on a single chart?
- Is there a mechanism for converting power monitoring series for Watts into a kWh or Kwh/day type graph? And for bonus points, take a $ figure for power and produce cost charts?
Next I’m going to spin up OpenLuup to run this on my RPi3… and hopefully log to the Qnap.
Yes, if you define a graph. You can simply select multiple metrics to be plotted. However, at the moment, you’re restricted to the same y-axis, so they have to be relatively compatible magnitudes.
I’m working on a significant upgrade to the plotting back end, which will make a lot of improvements to this aspect, but I keep getting waylaid by work on a ZWave plugin and a few other things.
2. Is there a mechanism for converting power monitoring series for Watts into a kWh or Kwh/day type graph?
Sort of… although not really. You can set up a multi-archive file with ‘sum’ aggregation and two archives: the first for for 1 day only at whatever sample rate, and the second one for as long as you like with a 1 day sample rate. So, then, a plot for the last day will show you your Watts sampled at the rate you have chosen, and a plot of more than one day will show you daily totals. But the units will be strange. I have a special case set up for this of Whole House power and PV generation, but I use a scene which captures the data on a one-minute interval and scales it appropriately so that the daily totals come out in kWh.
And for bonus points, take a $ figure for power and produce cost charts?
Not yet. So once again a decent upgrade will fix all this, allowing function calls in the request to do scaling, etc.
Next I'm going to spin up OpenLuup to run this on my RPi3.. and hopefully log to the Qnap.
Ah, be sure to install DataYours from the Plugins page update button. This will configure it to start logging system uptime, cpu and memory usage, having registered with AltUI as a Data Storage Provider. This is by far the easiest way to use it: you don’t have to use the dashboard interface at all, although you can still do so, if you like.