I’ve been thinking of the best way to get DataYours data from Vera to the cloud hosted Azure Database, ideally without a Syslog middle man.
Originally with Event Logger, I was going to attempt to change the Alternate Event Server to small web service running on Azure (Which small web sites running PHP or ASPX are free assuming CPU is minimal). This would easily parts the data from the HTTPS request with the variables.
Do you have any ideas how this may be accomplished in the future with DataWatcher? I’m OK today and the data structure shouldn’t change on the back end database…but it would be a cheaper option whom want unlimited cloud storage of their data.
The only way I know how to do this for scenes is to put Luup code into the triggers and scenes themselves.
PS Up to about 1400 MPH...Lots of fun data that a future, smarter me will be able to analyze :)
Impressive! Interestingly, you were worried about the load on Vera when writing to the Whisper database. Stress testing on a VeraLite system shows that it can sustain a database update rate (to local 'disk' space) of 700 points per second, so 1400 points would take just 2 seconds, making the cpu utilization at your write rate 2 / 3600 - so small as to be negligible. However, those 1400 points would take ~50kb, so a real database has to be on a USB or NAS device.
I run about 40-55% CPU when things are idle. I feel the interrupt contention and IO/disk contention (for DB files)… it’s better to echo it via the network (syslog now, but ideally a http request in the future) which is stored in a small cloud database.
Interesting note, you need to re-rerun the watch command when adding new devices.
My new power arithmetic plugin wasn’t showing even thought I previously ran the watch command for watts (watch=*.urn:micasaverde-com:serviceId:EnergyMetering1.Watts) and kwh. As soon as I re-ran, the data showed up on syslog for the recently added device.
Sorry, I love my Excel for sifting through data. I have an auto updating (refreshes every minute) connection to the cloud SQL back end data source , easily filterable (when debugging) or charting (analyzing).
When my electrician tried to install the whole home energy meter, we could only get it around one of the two incoming power clamps which means I can only measure half my data. Since I do get my date from the power company broken up hourly increments (24), I’m now able (almost) to figure out the deltas for all my devices (based on electric company data, based on switch power, figure out the actual wattage used). I have to wait 48 hours to get the electric company data so still untested.
I’ve updated the Data Warehouse to Azure SQL Server for Data Mining to now support DataYours (Specifically the DataWatcher engine only) as well as enhance functionality for EventWatcher to included room assigned.
In addition to adding support for DataWatcher, it now also adds some data to each column in EventWatcher or Datawatcher based on a device ID lookup to get the Room#, Display Name, and other fairly static entries added row of mined data.
Table EventWatcher
Syslog: Mar 11 18:31:45 Vera-30010099 EventWatcher: K [046] Thermostat, CurrentTemperature = 70
To Columns:
HA Unit (new colum): Vera-30010099
DeviceId: 46
DeviceName: Thermostat
Variable: CurrentTemperature
Value: 70
Class: K
Category: Table Lookup match EventWatcher.Class to CategoryKey.Class
Room: Table Lookup match EventWatcher.Class to DeviceKey.CategoryKey
Table DataWatcher
Syslog: Mar 13 13:30:14 Vera-30010099 DataWatcher: Vera-30010099.449.urn:micasaverde-com:serviceId:EnergyMetering1.Watts 65 1394742614
To Columns:
HA Unit (new colum): Vera-30010099
DeviceID: 449
ServiceURL (new colum): micasaverde-com
ServiceType (new colum): EnergyMetering1
Variable: Watts
Value: 65
Device Name: Table Lookup match DataWatcher.DeviceID to DeviceKey.Device Name
Device Type: Table Lookup match DataWatcher.DeviceID to DeviceKey.Device Name
Room: Lookup match DataWatcher.DeviceID to DeviceKey.Room
I think you’d just have to take a snapshot of the device list every restart and do the comparison yourself. Easy to get most, if not all, of what you need with an HTTP request for ‘sdata’, see [url=http://wiki.micasaverde.com/index.php/Luup_Requests#sdata]http://wiki.micasaverde.com/index.php/Luup_Requests#sdata[/url]
Sadly, my skills are at capturing and parsing the data DataYours generate. I’m newbie at Lua and can’t dedicate the resources to learn it well…in fear the knowledge will usurp some knowledge that pays the bills
Certain variables don’t show up until their set for the first time. So if I run a watch command, and the variable is still empty, it won’t show up until the variable is not null, and I rerun the watch command for all the serviceID’s the specified device.
Similarly, this is also time consuming and error prone process to add new devices, and then find the required serviceID’s to rewatch to match the present configureation.
Could there be an option to “rescan/rewatch” all configured Service ID’s…either instantly or on reboot?
That’s an interesting observation… not entirely sure I fully understand the problem. When you say “the variable is still empty”, I’m confused. Either the variable is not there, because it hasn’t been created yet, or it has a value. Certainly, if you ask to watch a variable that isn’t there, then it won’t be watched when it does appear. Trying to understand the solution that you’re suggesting… you want to have a list of things to watch even if they’re not there? Just trying to envisage how to solve this one.
I already have a GUI ‘dashboard’ which makes this really easy. I could package it for you to try. Or you could help me understand this all a bit more. I really feel I’m missing something basic in what you’re saying. Sorry to be so dim.
Ah, I think that’s because your paradigm is “watch everything.” And I think that’s actually rather unusual. Consider this:
“I have a number of devices of the same sort, say Temperature, but I only want to watch some”
I’ve got two choices in setting up the watched variables:
[ol][li]generate a watch command for each device/variable, or,[/li]
[li]use ‘*’ to watch all existing devices with such variables, and then ‘nowatch’ the ones I’m not interested in.[/li][/ol]
If ‘’ worked the way you’re asking, then every time I add a new device with that variable it would be watched, whether I wanted that or not. But all '’ does at the moment is go through all the devices seeing if that service/variable exists… it can’t know about things that don’t.
That’s not to say that I can’t use a different type of ‘wildcard’ symbol to denote the functionality that you need, or, indeed, change the semantics of ‘*’, but I would have to think quite carefully about how to do that because, at the moment, there’s no way that the daemon knows that you originally set watch on a variable using the wildcard syntax… it only knows that that specific device/variable is on the watch list.
HOWEVER…
I think you may be telling me (and I’m trying to understand) something different? Is that true? Am I already off track?
OK, good. I will have to think very hard about that. Although it seems a trivial change it is far from that.
In your specific example, why do you not simply watch “BatteryLevel” ?
I do that for ALL my (existing) battery powered devices with DataWatcher (example attached, plotted with DataYours). It would be easy for your SQL system to generate an alert on BatteryLevel falling below a threshold?
I do monitor battery level I do plan on updating them to the ‘Device’ table the latest battery level, battery level date, last poll OK time coming in from DataWatcher or Eventwatcher watched variables.
The example above is just an example generate the variable until it exists for the first time. Water leak are notorious for this too.
My biggest issues is needing to add all these variables to watch every time I add a new motion sensor for example.
Fair point! I installed one just the other day… must check that!
(Actually, it sends me an email and a notification to my iPad, and, come to think of it, would you ever install such a device without testing it first anyway?)
My biggest issues is needing to add all these variables to watch every time I add a new motion sensor for example.
