[quote=“Freddan101, post:20, topic:179386”]I would like to create a flexible energy consumption report based on accumulated data per day/week/month/year and device/room/whole house in combination with category (lighting, heating, appliance etc.). Is this something I can already or will be able to do with DataYours or with the help of?
It would be really nice to be able to embed these kind of reports into a customized dashboard/interface. I realize I might be on my own here though… :)[/quote]
There’s an Energy Calculator plugin which does some of this, I believe. Accumulated day/week/month/year totals can be got for ‘free’ through configuring a Whisper file with the right archive retentions. The Graphite webapp (crudely mimicked by DataGraph at this time) could do grouping of results into categories.
Bear in mind that DataYours is, very definitely, a prototype. I just wanted to put something concrete out there to support my initial suggestions of where we might go beyond dataMine (which I’m still using in ‘production’ mode at the moment, although running in parallel with Whisper.) Whilst I’m trying to do a good job, much of the code might be throw-away and many of the features are not yet fully implemented or tested. I’m actually chasing a problem with archive aggregation incorrectly accumulating over null values at the moment. However, I am definitely committed to the Whisper database and the external interfaces, and I’m doing some rigorous unit testing to qualify these to a release level. The idea is that there are lots of dashboard tools out there which do already interface to Graphite / Whisper. I’m not very good at writing user interfaces, so someone else will have to use what’s there or write their own.
Well, I see that it is just about one month since I gave an update on this prototype, so it’s time for another.
The Whisper database is now fully stress-tested, and I found a couple of problems: some from the translation to Lua; some in the original Python code. But all is sorted now and it has been running flawlessly for over a month logging data from three (un-bridged) systems.
What has been taking the time is the user interface (as ever.) I am not a GUI developer, or, for that matter, an HTML one, or a JavaScript one… so progress has been slow.
However, nearly there now in terms of another iteration of the prototype. I attached a snapshot of the database browser showing multiple Vera channels and a plot of one of them. As before, it retrieves data from both dataMine and the Whisper database, but time is moving on so the need to access the dataMine database diminishes for plotting recent data.
I’ve modified the code below to only accept SysLog and not create the local directory/whisper files. I’m LUA naive…so not sure of the next steps.
Upload L_DataDaemon.lus
Run the code below in test LUA code window. It should not store any data locally (all forwarded to Syslog)
------
--
-- DataYours - a prototype next-generation datamine
-- version = 2014.02.08 @akbooer
--
--
-- load and start the three component daemons
require "L_DataWatcher"
local IP = "127.0.0.1" -- this machine
local PORT = "1392" -- unassigned port
local SYSLOG = "xxx.xx.xx.xxx:yyy" -- set syslog
local function DWconfig (x) luup.inet.wget ("http://"..IP..":3480/data_request?id=lr_DataWatcher&"..x, 1) end
-- configure DataWatcher to watch all Termperature, Security, Power and Batteries and send to specific port
DWconfig ("send="..IP..":"..PORT)
DWconfig ("syslog="..SYSLOG)
DWconfig "watch=*.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature"
DWconfig "watch=*.urn:micasaverde-com:serviceId:SecuritySensor1.Tripped"
DWconfig "watch=*.urn:micasaverde-com:serviceId:EnergyMetering1.Watts"
DWconfig "watch=*.urn:micasaverde-com:serviceId:HaDevice1.BatteryLevel"
DWconfig "watch=*.urn:micasaverde-com:serviceId:Volume1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:SwitchPower1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:InputSelection1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:MediaNavigation1.*"
DWconfig "watch=*.urn:upnp-org:serviceId:RenderingControl.*"
DWconfig "watch=*.urn:dcineco-com:serviceId:MSwitch1.*"
DWconfig "watch=*.urn:upnp-org:serviceId:VSwitch1.*"
DWconfig "watch=*.urn:upnp-org:serviceId:HVAC_UserOperatingMode1.*"
DWconfig "watch=*.urn:upnp-org:serviceId:TemperatureSetpoint1_Heat.*"
DWconfig "watch=*.urn:upnp-org:serviceId:TemperatureSetpoint1_Cool.*"
DWconfig "watch=*.urn:upnp-org:serviceId:HouseStatus1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:DoorLock1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:GenericSensor1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:SceneController1.*"
DWconfig "watch=*.urn:upnp-org:serviceId:SwitchPower1.*"
DWconfig "watch=*.urn:micasaverde-com:serviceId:LightSensor1.*"
-- configure DataCache to listen and store the results
DCconfig ("listen="..PORT)
DCconfig ("syslog="..SYSLOG)
--
-- Congratulations! You should now be logging specified device data
--
-- to plot, visit, for example, the URL
-- <yourVeraIP>:3480/data_request?id=lr_render&target=Vera-12345678.321.urn:upnp-org:serviceId:TemperatureSensor1.CurrentTemperature
--
--
Question on above, will the code work above with the “.*” at the end of the service ID (without the variable). I was hoping to catch all variables and filter it down later.
Lastly, is there any problem running DataYours in parallel to Event Watcher (which is great for debugging)?
[quote=“AgileHumor, post:23, topic:179386”]I’ve modified the code below to only accept SysLog and not create the local directory/whisper files. I’m LUA naive…so not sure of the next steps.
[…]
Question on above, will the code work above with the “.*” at the end of the service ID (without the variable). I was hoping to catch all variables and filter it down later.
Lastly, is there any problem running DataYours in parallel to Event Watcher (which is great for debugging)?[/quote]
[ul][li]ignore the [tt]DataYours[/tt] script entirely. Just do manual configuration for a couple of variables to begin with. All you need to do is use a browser to access the URL request.[/li]
[li]No, wildcard variable names will not work, only device numbers - it’s actually very unusual to want to do that. Many devices have tens of variables which are mostly useless for data gathering.[/li]
[li]Should be no conflict at all between [tt]EventWatcher[/tt] and [tt]DataYours[/tt]. In fact, I plan to merge aspects of them ([tt]DataYours[/tt] for data logging, [tt]EventWatcher[/tt] for events, strangely enough.)[/li][/ul]
Just remember to include the [tt]require[/tt] statement for the [tt]DataWatcher[/tt] module (it loads the [tt]DataDaemon[/tt] one itself.)
The two files are tiny - ignore those resources.
If you remove the [tt]require[/tt] statement from startup Lua, neither of them will be loaded into memory.
You will have difficulty detecting any use of resources - memory or cpu - they are incredibly lightweight and do not create a stand-alone device.
I’m assuming I make an HTTP request to set the syslog based on the code below. But will that really only echo the syslog? Is it being stored elsewhere in a log file or database? I’m assuming since I’m not running the other code those files/whisper databases won’t be used.
Just the require statement for L_DataWatcher in startup, then restart.
Yes, set syslog like that, and &watch=… for the variables.
Sorry, have another appointment this evening, so can’t help until much later!
Good luck.
[quote=“AgileHumor, post:33, topic:179386”]Can you expose in the data stream the:
Display Name (in addition to number)
Room Number or Name
Device Category[/quote]
Not directly. This is part of the system which complies with an industry-standard format, the Carbon Whisper plaintext protocol, see: [url=http://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-plaintext-protocol]Feeding In Your Data — Graphite 1.2.0 documentation.
However, you see encoded in it the vera id, device number, serviceId, and variable name - unique coordinates to all the metadata you could wish for, easily extracted from the Luup request for device status information (viz. room, device name, category, …) It would be straightforward to pipe this through another daemon which recovers it and sends it along in a different format (perhaps along the lines of EventWatcher.) Presumably, also, it’s easy to enable your back-end system do the Luup request itself.
I have all the code for this up and running in the [tt]DataDash[/tt] module, a prototype replacement for the dataMine front-end GUI. Just a bit more work before releasing the next version of that, but I fear you will not be interested in it.
Thanks sir, I can easily have this written in the SQL stored procedure to do a lookup table for the devices/room numbers. I can use the EventWatcher devices report and manually populate the SQL lookup table…
My current experience with the different services ID’s below to help save you and others time.
BTW, For the ones that don’t work, do I need to “unregister” them so they don’t consume resources?
I would be fairly sure that ‘does not work’ is due to the devices in question not having the correct json or xml files to enable those services to be watched/triggered. There is absolutely nothing device specific in the DataWatcher code, so little that can be done from my end.
You could ‘nowatch’ those devices, but it really doesn’t matter. If you do, the ‘nowatch’ doesn’t take effect until the next reboot, because there is, simply, no [tt]luup.unwatch_variable[/tt] command. (I could have coded around this, but I didn’t, to keep it short.)
…but delighted that, on the whole, it delivers what you needed.
…just to be clear: there is no ‘default’ list for [tt]DataWatcher[/tt]. These just happen to be the ones I put into the [tt]DataYours[/tt] prototype script. Out of the box, it watches nothing.
EventWatcher uses: [tt]urn:micasaverde-com:serviceId:SceneController1[/tt] and variable [tt]sl_SceneActivated[/tt] to watch scene controllers, so it should work for DataWatcher too. This certainly works for secondary controllers… it may be, though, that Vera scenes are not triggered in this way. I just don’t know.
I do get these sl_SceneActivated…but that is only when a scene runs when manually pressed on a keypad. Another example I would love to see is scenes running from schedules or triggers…like knowing what scene ran when motion tripped a light.The PLEG example shows every time action/condition becomes true.
PS Up to about 1400 MPH…Lots of fun data that a future, smarter me will be able to analyze
I feel like an idiot.