Update 6 July 2012 - UI5
dataMine on UI5 should now have a seemless install. I’ve also started an overview document that I’ll try and keep up to date with information rather than expecting everyone to sift through what is now a long thread…
To install -:
[ul][li]Install the app as usual through UI5 the app install tab
[/li][li]Format a USB stick (on Windows works fine for me)[/li]
[li]Plug the USB stick into Vera[/li]
[li]Configure up one variable in the dataMine properties (see below)[/li]
[li]Restart Lua[/li][/ul]
Once you’ve installed the plugin, you need to go to the advanced properties, and set the SetMountPount variable. For Vera3, this is /dev/sda1 for the bottom USB slot, or /dev/sdb1 for the top slot. For Vera2, unfortunately, I’m not 100% sure. I’ve been using /dev/scsi/host0/bus0/target0/lun0/part1 on my Vera2 (top socket I think). For Vera3Lite I’m told it is /dev/sda1. If you are using a USB hub, these may change - refer to the document for a description of how to find the mount point.
Note that this should not be used with Veras USB logging enabled. If you have USB logging enabled, you should use a separate USB stick - don’t be tempted to use the same drive as this can cause problems.
Original Post
I like the Vera UI - it’s clean and simple (although I hope UI5 improves on this). As others have found though, reading logs to get sensor history is not useful. I also am not a fan of the model of hosting data on a separate website - I really like to have my data under my control. This is mainly for security and communication tollerance reasons.
I’ve seen other plugins that allow data to be logged, and then either use a script to produce graphs, or use an external server to process and visualise the data. This is quite possibly a more powerful solution, but I wanted the simplicity of a single unit, and the flexibility of on the fly graphing rather than fixed graphs processed daily in a script.
So… the dataMine plugin is born. This logs all value changes in specified variables to log files on Vera, and provides a GUI graphing interface complete with zoom, scroll etc. The system really must be used with an external USB memory stick since it does produce a relatively large amount of data. I’ve mounted a 4GB flash drive (FAT32) as /dataMine, and all data files are logged here. I estimate that a 4GB stick would support the system for 10 years or more, so memory isn’t really an issue.
In addition to the “raw” data logging, the system also logs hourly min/max, and daily min/max. This data is used to support different zoom levels so that the large raw files don’t need to be heavily processed to see long timeframes. The system does work with large raw data sets, but it starts to slow down (taking maybe 10 seconds or so to return a 1 year history of temperature data). The creation of these files is still work in progress - currently only the hourly log is generated, and then only manually.
Installation
As I’ve made use of a number of javascript libraries (primarily the flotr graphing package) installation isn’t quite as straight forward as a standard plugin (sorry). The usual files (XML, JSON, LUA) are loaded in the normal way through the “developpers” button in the control panel and all files specific to dataMine are placed into a directory /www/cmh/plugins/dataMine - this includes the js files, and a number of icons for buttons. The config file needs to be stored in another directory /dataMine, which is also where all the data files are stored. The ZIP file also includes a set of files with approximately 1 year history of outside temperature at my house to allow you to play with the system - these can of course be deleted. There are basic installation instructions in the readme.txt file. (I had to trim down the data due to ZIP size limitations on the server!).
Use
The dataMine plugin installs as a device, and by clicking on the tools button you are presented with two tabs. The first one is the graph tab. This has a listbox to list all the available variables, and a number of buttons. The buttons allow you to select the last day/week/month, or all data, and to scroll and zoom the data. You can also zoom by selecting an area of the graph with the mouse.
The config tab allows you to select the variables to log. This provides a list of all variables for devices that aren’t hidden. Variables are listed by service, and if you click on the variable name, logging will be enabled. Clicking again will disable logging, but the variable will still be listed in the graph page dropdown. There’s also a little clock button for selected variables. Clicking on this will generate the hourly/daily data for the different zoom levels (currently this is only manual, but it will be made automatic soon).
Limitations
[ul][li]Currently, only 1 channel can be graphed at a time. This is something I may change later - the graph library supports multiple channels, but Vera is likely to get a bit grumpy if we serve up too much data![/li]
[li]If there is no current data, the graph may not generate. This is because the default graph is currently the last day. Clicking on the month or week button should fix this.[/li]
[li]The hourly and daily files are not yet generated automatically, but will be processed once per day (this is next on my todo list). I intend to do it this way rather than an “on the fly” method primarily as I find that Vera isn’t quite as reliable as I would like, and I find it rebooting many times per day. Data statistics would need to be maintained over these reboots, and the easiest way to do this is to write a temporary log and process this data daily. This does however mean that current data may not be displayed on large zoom levels. This shouldn’t be an issue - if you zoom in, you’ll see the most up to date data.[/li]
[li]Data is subsampled. This means that spikes etc may not be displayed on larger zoom levels. I don’t think this is a major issue in most applications.[/li][/ul]
This is still very much work in progress (consider this an alpha version that needs further work). Feel free to give it a go and send any feedback and I’ll try and fix any problems. It’s worth noting that I’ve only had my Vera a couple of weeks, so I’m very new to it, and the Lua/Luup system, so I’ve probably screwed things up, or at least poorly written parts of the code, so go easy on me 
(and maybe a “use at own risk” warning is in force!).
A wiki, and the latest code, is available at [url=http://code.mios.com/trac/mios_datamine]http://code.mios.com/trac/mios_datamine[/url].
Cheers
Chris
