Charts

Developer's Corner General Dev Discussions
1

The Dashboard/UI is useful to see what state devices are currently in. However, the UI is no good for debugging/tracing/visualisation of measurements or events over longer time (one might want to see how things develop over time).

I created a few scenes which dump periodically the state of sensors/devices in logfiles. Then I have a lua script in /www/cgi-bin/ which creates a webpage which uses google chart tools. In my web browser I then get something that looks like the attached image. It works, but I have a number of questions and maybe you can answer some of these…

  1. Tracking data and display results with graphs seems to me essential. Will MCV have something along these lines (and better than my DIY thing) in future versions (or is it there and I just could not find it?). It seems on the mios page there are things like it for energy monitoring.

  2. I created a folder /var/log/sensors/ where I dump my data. Something is removing this folder once in a while. Which shell script is that? How can I prevent this?

  3. I would like to change the index.html page to include a link to my charts page. It seems I cant change index.html. Can I just delete the link and place a copy of index.html there and then modify it?

  4. What happens with updates. Will I loose what I put in cgi-bin?

  5. What is included in the backup in UI4 (scenes?, devices?, luup files? )

Thanks,

Utz

2

It’s great !

Can you explain more what script, and how do you make this ? I’m very interesting to have charts like this !

3

@utz,

All of /var lasts only until Vera reboots. It’s essentially a ramdisk. If you want your files to survive a reboot you will have to put them somewhere else. I’d recommend a new directory in /usr. But Vera’s main “disk” is flash with limited write cycles, so I wouldn’t want to be writing to it frequently. Better to use an external USB memory or drive.

For contrast, I have a rival graphing method: I run Nagios, which collects information from my LAN. I’ve taught it how to query Vera through Vera’s HTTP interface. Attached is an example.

Nagios is not for the frail-of-command-line, but if you aren’t daunted after reading about its config file format, ask me for how to have it talk to Vera.

4

The aim is to create a web page with graphs just using the default vera (no additional server, no databases, no software to install on vera, …). So I have to work with what is installed on vera.

There is a directory /www/cgi-bin/ on vera where you can place cgi scripts (You can then execute these by accessing http://XXX.XXX.XXX.XXX/cgi-bin/myscript.sh). The script has then to build a webpage (as large string) that is returned after execution. The script can be written in any language that is available on the vera platform, so I used Lua as it is there.

The script creates a webpage containing javascript for the graphs using the Google Charts API. The google charts API is not perfect to create graphs but I had nothing else (gnuplot installed on vera would be better). The problem is that google charts can only deal with uniform distances on the x-axis. If you have a data point at 10min one at 12min and one at 20min it plots all 3 data points with the same distance on the x-axis (which gives the wrong impression). To compensate this I record data values in uniform time steps (for example every 2 min). The problem is then that you miss changes that have a frequency of less than 4min.

The data for the graph is recorded via scenes. Periodic scenes run and record the data in logfiles (also it limits all files to 250k not to have too much data on vera). That data is put into the webpage when it is built (as javascript lines for the google chart tools). The webpage gets therefore quite large (contains all the data that needs to be plotted).

I can put the lua script and scene scripts here later when I am at home. But it is not nice to look at (and probably not the most efficient coding) as I just made it work quickly. Thats why I was asking, there needs to be a better and integrated way with vera.

5

@futzle

Nagios graph stuff plots nicer. But as I understand you need an additional server (for the Nagios stuff) up and running at home to do this, right? I wanted to avoid having to look after several servers at home and just want to use the little vera box (I have enough computers at work to look after).

The USB memory drive is a good point. Can I use there any USB stick or does it need to be something specific.

6

This depends on how many users request this function. Not many have requested it so far, but it’s doable through a plugin.

The script is called RotateLogs.sh, but it only deletes the files starting with log. However, I would do what futzle said and place the files on an USB memory stick. Any memory stick should work.

Yes.

If you modify an existing script it will be overwritten at upgrade. However, if you create a new file, it should survive the upgrade process.

Everything you listed.

My suggestion is to make a plugin. It is easier to install and, if upgrading with Keep my settings, it will survive the upgrade.

7

Yeah, you do. But I already had the server, and it already runs 24/7, and it was already running Nagios (to monitor the other dozen devices in my Intranet of Things) so it was less of a leap for me. For your needs, I agree that your approach is right.

8

I changed my charting stuff from Google Charts to Open Flash Chart (Open Flash Chart – Teeth Grinder UK). The main reasons are that i) programs nicer ii) i can pot non-uniform distances on the x-axis.

How this works

  • a web page in /www/cmh that contains the layout of the page and the div for the graphs
  • each graph loads the data via dedicated script in /www/cgi-bin

this way I do not have to produce the entire webpage with a cgi script; only data for the graphs comes from cgi scripts.

Still, I have one problem and maybe someone knows what that might be. Sometimes the graphs do not load because the webserver on vera produces a “500 - Internal Server Error”. The error is not deterministic, sometimes works, sometimes not. The cgi-script takes the data out of a file located on a USB stick attached to vera. The error seems more frequent when the data set increases. Seems like a timing issue… Where does the http server on vera writes its logfile?

9

From what I’ve seen the access logging of [tt]lighttpd[/tt] has been switched off, but you could probably re-enable it. It looks like [tt]mod_accesslog[/tt] is present/shipped, so you’d need to add it to [tt]server.modules[/tt] (in [tt]lighttpd.conf[/tt]) and configure it.

Some error logging is configured (by default) to be written to:
[tt] /tmp/log.lighttpd_error[/tt]

… most of which will likely be blown away next time you upgrade (etc) so don’t go too nuts configuring stuff in here…

On the script returning/triggering 500’s, you may need to attach it to get inputs from folks.

10

One of the scripts attached (I changed the ending from .sh to .lua for upload). It creates a json file which is fetched by the webbrowser and then plotted.

@guessed
I will try to get that logging going… Maybe I see something.

11

So it looks like this script is dependant upon the file:
/var/log/cmh/light.txt

If this file doesn’t exist, or has an incorrect format, the the Lua will fail. I wouldn’t be surprised if your file is being “rotated” by the standard CMH Log rotation script, leaving periods of time when this [sh] script errors out with 500’s.

That’s my first guess :wink:

Given what you’re doing, you might find the ‘[tt][[[/tt]’ construct in Lua as a good replacement for all the strcat work you’re doing. You’ll see a nice example of it here, including “formatting” (latter addition of parameter values):

http://code.mios.com/trac/mios_smartphones/browser/L_sPhoneUI.lua#L155

BTW: I added [tt].sh[/tt] as a supported file extension for attachments, so you can use that moving forward.

Q: Does the graphing lib support both spellings of color? (eg [tt]color[/tt] and [tt]colour[/tt])

12

[quote=“guessed, post:11, topic:168751”]So it looks like this script is dependant upon the file:
/var/log/cmh/light.txt

If this file doesn’t exist, or has an incorrect format, the the Lua will fail. I wouldn’t be surprised if your file is being “rotated” by the standard CMH Log rotation script, leaving periods of time when this [sh] script errors out with 500’s.

That’s my first guess ;)[/quote]

The file contains lines of the format

2011-07-24 14:04:17 1311512657 75

I named it from light.log to light.txt so that the logfile cronjob does not touch it (that seems to take things with log in the name).

The problem is that I get the 500 error; then I call the script a few seconds later and it may work. It is not failing all the time (which would indicate the file might be missing or have an error). That is why I suspect it may have to do with the filesystem on the memory stick or something like it (kernel busy and cant handle my request or something crazy).

[quote=“guessed, post:11, topic:168751”]Given what you’re doing, you might find the ‘[tt][[[/tt]’ construct in Lua as a good replacement for all the strcat work you’re doing. You’ll see a nice example of it here, including “formatting” (latter addition of parameter values):

http://code.mios.com/trac/mios_smartphones/browser/L_sPhoneUI.lua#L155[/quote]

Yes looked at that, a bit of a strange language construct but seems helpful for this sort of thing. Will try to change to that … Thanks

Thanks

yes, I tried all versions and the one I use works.

I can also ssh into vera and execute the shell script on the command line. That works always. Only when I call the shellscript via webserver it seems to fail sometimes (but not always)

13

Looking at this again it seems it is the flash loading the graph data. It might be that the flash thing decides it cant load the data if vera is not able to deliver the data quick enough (and vera is not the fastest webserver on the planet). I have to look at this for a bit longer…

14

Ok, tested a bit more

  • I can execute my lua scripts in cgi-bin via commandline
  • I can cat the output of the script in a textfile located in /www/cmh/
  • If I use the textfile instead of the cgi script as graph input everything works
  • I can write a shell script that just echos the content of the textfile and use this shell script in cgi-bin. This fails as well

→ It seems that lighthttpd has a problem processing cgi-bin scripts that produce a large output
→ There is no hard cut off point (the larger the output gets the more likely is that the web server fails with 500)

15

In the original Shell/Lua script, you were buffering up all the output until the very end, where a [tt]print(page)[/tt] command was issued. This provides a neat/complete version of the output at all times, but requires buffering in memory (etc) until it’s complete.

What happens the printing of the results is done incrementally?

ie. one [tt]print(…)[/tt] block for the header, another for each loop iteration, and one for the footer?

This might stream the results better, just in cases there’s a [tt]lighttpd[/tt] <-> CGI Buffer limit that you’re hitting.

It kinda depends upon how you did the “echo” test above (and how big the output was/is), since my suggestion above might be moot.

16

I will try that tomorrow, have to go to bed now…

I thought the web server is not that clever. I would think it executes the cgi script first and then hands the result back over the tcp connection. Not sure if it can execute and serve bits of the result back. I will try.

For the one script the change is easy as it plots only one line. For other graphs with multiple lines that would mean I have to open and read the file several times (not very efficient but if it works …)

17

The problem seems to be that my cgi scripts start, do a lot of stuff and at the very end send the data. Adding a print of just one character at the start of the script solves my problem (it needs more testing). It seems that the webserver terminates the cgi script when it thinks it is not doing anything (printing the character at the start seems to prevent this). An odd one.

18

Interesting finding. Apache does something similar.

Either way, probably better use regular(ish) [tt]print(…)[/tt] commands throughout the code to that the output is more stream like in nature and not bufferred in Vera’s [limited] memory. You’ll be able to generate arbitrarily large output once you have that in place.

19

Howto make graphs of sensor readings using Open Flash Charts (Open Flash Chart 2 – Teeth Grinder UK)

Attached is a picture of the result.

  1. Generating data
    I use a scene for each type of data I want to graph. Below is the Luup code for logging brightness of the one light sensor I have in the house. The scene is triggered every 10 min via a timer. The data is logged in a file at /var/log/cmh/light.txt which is located on a usb memory stick attached to vera. The file generated has the form of:
...
2011-07-26 20:55:48 1311710148 7 
2011-07-26 21:05:48 1311710748 7 
2011-07-26 21:15:48 1311711348 15
2011-07-26 21:25:48 1311711948 15
2011-07-26 21:35:48 1311712548 15
2011-07-26 21:45:48 1311713148 13
...
-- logfile
local logfile = '/var/log/cmh/light.txt'
-- Device IDs to log
local frontroom  = 11

-- empty file if it reaches 250kb
local outf = io.open(logfile , 'a')
local filesize = outf:seek("end")
outf:close()
if (filesize > 250000000) then
   local outf = io.open(logfile , 'w')
   outf:write('')
   outf:close()
end

local outf = io.open(logfile, 'a')
outf:write(os.date('%Y-%m-%d %H:%M:%S '..os.time()..' '))
outf:write(luup.variable_get("urn:micasaverde-com:serviceId:LightSensor1","CurrentLevel",frontroom)..' ')
outf:write('\n')
outf:close()
  1. A webpage displaying the graphs
    I use a page charts.html which is placed at /www/cmh/ on vera. The page contains javascript that is used to load the flash displaying the graph. Below the code for the brightness graph.

I also changed the index.html file in /www/cmh/ to contain a link to load the charts.html. I added a line:

<div class="wizard" title="Graph Wizard" onclick="window.location='charts.html'"></div>

Which gives in the dashboard a second wizard icon that loads the graphs.

<script type="text/javascript" src="js/swfobject.js"></script>
<script type="text/javascript">
		swfobject.embedSWF(
			"open-flash-chart.swf", "light_chart",
			"1200", "200", "9.0.0", "expressInstall.swf",
			{"data-file":"..%2Fcgi-bin%2Flight.lua"} );
</script>

<center>
		<div id="light_chart"></div>
</center>
  1. Vera changes
    I changed /etc/lighttpd.conf so that the webserver understands lua cgi. The two lines need changing:
static-file.exclude-extensions = ( ".lua", ".php", ".pl", ".fcgi", ".sh", ".cgi", ".awx" )

cgi.assign = ( ".lua" => "/usr/bin/lua", ".pl"  => "/usr/bin/perl", ".cgi" => "", ".sh" => "", ".awx" => "/usr/bin/awx" )

Also, it is necessary to install the javascript files and swf file for Open Flash Charts on vera.

  1. Chart data
    When charts.html is loaded, the graphs will load the data via lua cgi scripts. These scripts take data files and convert them in json format which is required by Open Flash Charts. For brightness the file light.txt is converted with the lua cgi script I attached.
20

Cool! If you’re interested, there are mechanisms in Vera that let you subscribe to change events on named serviceId variables on any device. The old Power monitoring service used this to “monitor” (record) every power change event (light switches, dimmers, etc) and “phone home” with the data to the MiOS Servers.

Then, instead of polling on an interval, you can collect data anytime it changes.

See this file for reference:
L_EnergyMonitor_j.lua in energy monitor plugin – Energy Reporting framework

and look at the [tt]luup.variable_watch[/tt], along with the [non-local] callback function and it’s parameters (in this case function [tt]w_watts[/tt]).

@futzle is looking at doing something along the lines of this (inferred from postings) since it could be made into a common plugin that “graphs stuff” - based upon a configurable list of device/serviceId/variable subscriptions.

Oh, did you not like Google’s Flash-based interactive charts? Seems like they could do most of what you’re already doing - albeit with a complex syntax, without needing to install the bits onto Vera itself (although we can package that into a Plugin’s install logic if needed).

They’d also get live/localized CDN delivery so folks would pull them from their regional Google locale (all privacy issues aside)

Just a thought…