*DataMine Development Stopped* anyone want to take over?

Plugins & Plugin Development Unmaintained
21

Hi @akbooer

As I’ve just had yet another USB key fail, after it was working for months, ???
Maybe providing the choice of internal or external storage would be a great first enhancement?

I’ve had nothing but issues with the USB route, and I know that a number of people who use both USB logging and DataMine seem to have more issues.

I hope your exploration into storage options bears fruit :wink: soon

22

I second this request. I currently don’t use datamine mainly due to the USB limitations. If the data could be written to a network location it would make things much cleaner.

23

So I don’t quite see the problem here. Or, rather, I see the problem, but the solution exists already. Earlier this week I installed CIFS, and today I’ve checked that dataMine uses it to log successfully to a NAS. In my case, that’s an Apple Time Capsule. All seems to work well.

I have to say that this does make disk access slower (actually ‘memory stick access’ in my case, since I simply transferred the stick from Vera to a port on the NAS) but it will mean that other machines can access the data directly, rather than through Vera. It’s just that Vera offers a really simple way to code up a web server, as I have done with the dmDBserver code.

24

A while back, I posted some thoughts on where we might go with dataMine: [url=http://forum.micasaverde.com/index.php/topic,17232.msg143007.html#msg143007]http://forum.micasaverde.com/index.php/topic,17232.msg143007.html#msg143007[/url]

I thought I’d just give an update on how my thinking has evolved after doing some prototyping:

[ul][li]database - I was totally wrong here. Changing from the dataMine database to a Lua implementation of the Python-coded Graphite/Whisper database ([url=https://graphite.readthedocs.org/en/latest/whisper.html]https://graphite.readthedocs.org/en/latest/whisper.html[/url]) offers speed, some statistical functionality, storage schemas for archiving, and fixed storage size (through being a ‘round robin’ database.)[/li]
[li]configuration interface - still a closed book to me, I look to others for work (perhaps already done) in this area, but in prototyping I can configure everything through HTTP URLs. [/li]
[li]variable watch - as easy as expected, but raises the question as to whether watching for changes or polling (or a combination) is the way to go.[/li]
[li]event logging - The Whisper database doesn’t support non-numerical data. Syslog seems the right place for this. For the time being, the functionality offered by EventWatcher (as an Alternate Event server) suffices.[/li]
[li]graphics - Google charts works well with the uniformly-spaced sampling provided by the Whisper database.[/li][/ul]

What this means for my current prototype is:

[ul][li]“improved data-querying capabilities including statistical functions (max/min, mean, running averages, histograms, resampling, ?)”
[list][li]Many of these are offered directly by the Whisper database.[/li][/list][/li]
[li]“improved support for external storage devices (USB, NAS, Cloud, ?)”
[list][li]the CIFS file system seems to be working well for accessing NAS or remote machines, but other protocols (such as UDP as used by syslog) offer a way around this too.[/li][/list][/li]
[li]“the capability to link to, and mine, other databases”[list][li]I have a working database federation layer which accesses dataMine and Whisper simultaneously. If data is not found in Whisper, then it reverts to dataMine.[/li][/list][/li]
[li]“web server interface to allow configuration of new watched variables, meta-data queries (which variables are being logged, over what times are they available, ?)”
[list][li]the configuration of what variables to store is already done. Other metadata not yet implemented.[/li][/list][/li]
[li]“a new GUI, perhaps several tailored to different needs, which can be built almost stand-alone to use the above API.”
[list][li]not done, but a standard set of interfaces offer the option of using third-party applications (eg. Graphite-compatible tools [url=https://graphite.readthedocs.org/en/latest/tools.html]https://graphite.readthedocs.org/en/latest/tools.html[/url], or something like Dashing [url=http://thomasvm.github.io/blog/2012/11/19/bling-dashboards-using-graphite-and-dashing/]http://thomasvm.github.io/blog/2012/11/19/bling-dashboards-using-graphite-and-dashing/[/url] (great demo here: [url=http://dashingdemo.herokuapp.com/sample]http://dashingdemo.herokuapp.com/sample[/url][/li][/list][/li]
[li]“a more modular internal structure … could allow distributed processing, allowing the best use of new Vera hardware and other computing resources.” [list][li]the prototype is totally modular with data collection / storage / database / graphing all capable of running on separate machines and supporting multiple data-collection front-ends and databases. [/li][/list][/li][/ul]

This sounds like a lot, but it’s very compact and is all pure Lua. For a quick peek of what it does see the attached plot which shows a temperature variable (actually in an outdoor building with heating coming on during the day). The plot covers the last week and shows that for the last two days the data has been logged in the Whisper database whereas earlier data is retrieved, quite transparently, from dataMine. (The flat-line in the middle of the plot was due to some system reconfiguration.)

I don’t think this particular thread is the right place to discuss more technical details, but I’d appreciate any thoughts as to whether this is the right way forward. If anyone is interested we can discuss design and implementation elsewhere. I’d like to be able to package the prototype into a simple install for others to try (not quite there yet!)

25

akbooer
Really glad to see you are making nice progress.

26

My new prototype has now been running for over a week (see attached vs. previous post) and I think it’s time to get a critical look from a wider audience.

I’ll post an install tomorrow. It is just a prototype, so no easy GUI, but it watches, stores, and graphs results (including existing dataMine files).

27

The promised next-generation prototype is available here: [url=http://forum.micasaverde.com/index.php/topic,23109.msg156425.html#msg156425]http://forum.micasaverde.com/index.php/topic,23109.msg156425.html#msg156425[/url]

[— Edit: [tt]DataYours[/tt] is now available from the App store —]

28

What “other HA” system is chris working on now? Is there a different HA system that offers more integrated graphing and logging?

29

Chris works on openHAB, you’ll see him on http://community.openhab.org