Pulling data from the appliance without plugins.

Developer's Corner General Dev Discussions
1

Hi, all.

I’ve used the forum’s search function, please excuse if I’ve overlooked something.

As background, I’m a scientist and a competent programmer with *nix experience. I’m going to use the Edge for data collection in some biotech experiments. To do so, I need to pull logged sensor data from my Edge at will. I see there was a dataMine plugin which has been superseded by the DataYours(2) plugin. Since I just want the raw data, this is overkill, adds complexity, and removes control.

I’m still waiting for my Edge to arrive, so I haven’t actually logged into MiOS yet. However, I understand the vera* are linux appliances so I expect the native linux tools I’m accustomed to will be available. From the wiki, I think that the data that I want is going to be logged in /var/log/cmh/LuaUPnP.log

So, the two approaches that seem immediately available are:

  1. Run cron jobs on the appliance, pull data from /var/log/cmh/LuaUPnP.log (using grep sed etc), and rsync or scp it somewhere.
  2. Use the Luup HTTP interface, and use python/ruby/perl to remotely poll for data

To you experienced folks, do either of these options sound sensible, and is your intuition that one or the other will be easier to work with?

I’m also curious where I can use the Luup interface to make a device read data back to the controller. In other words, if I have a temperature sensor, can I send an http request that causes the controller to talk to the sensor and log the current temperature?

Thanks all for your thoughts.

2

Option 2 is the way to go.

Essentially you are writing a very simple version of the Vera UI, which is limited to showing only one value from one device, and which has no user interaction. With that in mind, read the Micasaverde wiki page http://wiki.micasaverde.com/index.php/UI_Simple and pay particular attention to the poll loop pseudocode on that page. It’s reasonably efficient because it only reports variables that have updated since the last poll, so you can run the loop as often as once a second if you want.

Your followup question about forcing Vera to query the sensor depends very much on the type of sensor that you have. There is a “poll” command (See http://wiki.micasaverde.com/index.php/Luup_UPnP_Variables_and_Actions#HaDevice1) but it depends on the device whether that command is implemented and (if it is) what “polling” means. Examples: a battery-operated Z-Wave sensor can’t be reliably polled at all, because it is sleeping most of the time to conserve power. A wired sensor can probably be polled most of the time.

3

[quote=“chekoff, post:1, topic:187960”]Hi, all.

I’ve used the forum’s search function, please excuse if I’ve overlooked something.

As background, I’m a scientist and a competent programmer with *nix experience. I’m going to use the Edge for data collection in some biotech experiments. To do so, I need to pull logged sensor data from my Edge at will. I see there was a dataMine plugin which has been superseded by the DataYours(2) plugin. Since I just want the raw data, this is overkill, adds complexity, and removes control.

I’m still waiting for my Edge to arrive, so I haven’t actually logged into MiOS yet. However, I understand the vera* are linux appliances so I expect the native linux tools I’m accustomed to will be available. From the wiki, I think that the data that I want is going to be logged in /var/log/cmh/LuaUPnP.log

So, the two approaches that seem immediately available are:

  1. Run cron jobs on the appliance, pull data from /var/log/cmh/LuaUPnP.log (using grep sed etc), and rsync or scp it somewhere.
  2. Use the Luup HTTP interface, and use python/ruby/perl to remotely poll for data

To you experienced folks, do either of these options sound sensible, and is your intuition that one or the other will be easier to work with?

I’m also curious where I can use the Luup interface to make a device read data back to the controller. In other words, if I have a temperature sensor, can I send an http request that causes the controller to talk to the sensor and log the current temperature?

Thanks all for your thoughts.[/quote]

Try this on your Windows box just to see if you like it. It’s an Excel macro enabled file that uses your password with pre and post filters and various log levels displaying them neatly on the Excel file.

http://forum.micasaverde.com/index.php?topic=30265.0

4

[quote=“tomtcom, post:3, topic:187960”]Try this on your Windows box just to see if you like it. It’s an Excel macro enabled file that uses your password with pre and post filters and various log levels displaying them neatly on the Excel file.

http://forum.micasaverde.com/index.php?topic=30265.0[/quote]

Thanks for pointing me that way. I don’t run windows, but looking through the forum thread, it looks like it uses a secure copy to pull the log and then applies some text filters. Not a bad tool, but won’t quite suit my purposes completely, and it would probably be about as quick to write it as use the Excel file (copying the log and looking for all entries of a particular device could be a short 1-liner from a command shell). Nice to see the successful approaches people are taking. Thanks again.

5

[quote=“futzle, post:2, topic:187960”]Option 2 is the way to go.

Essentially you are writing a very simple version of the Vera UI, which is limited to showing only one value from one device, and which has no user interaction.[/quote]

In some sense! I’m actually going to be accumulating time series data for charting and statistical analysis, but it’s true the main thing I’m doing is pulling down a few values. There won’t be any user interface per se.

This looks extremely helpful! Thank you very much for this. It looks like that polling loop is the tool I need.

I suppose that’s a sensible design. I’ll probably be going plugged in for many of my sensors for reliability and responsiveness. Cheers.

6

Hi. When i poll wit loadtime=xxx=&dataversion=XXX i don get the housemode setting in the root of the XML document.
Is it possible to include thath in the limitet poll request? Or are there a workaround with out polling the Whole document?