dataMine 2 documentation

This is the forum for the dataMine2 User Guide is now available for download as a pdf here: http://code.mios.com/trac/mios_datamine_2/raw-attachment/wiki/WikiStart/DataMine2%20Overview%20v1.914.pdf

dataMine 2 is a derivative work that builds on the excellent dataMine system for logging variables. It adds the ability to aggregate data over various time periods, provides access to the system over a remote connection and now has a new data source compatible with Google charts. As this version uses the same web locations as dataMine 1, you will need to uninstall it for version 2 to work - all your data and configuration is shared and will be retained.

On top of the original functionality it has:

[ul][li]the ability to aggregate data by none (as before), hour, day, week and month[/li]
[li]aggregation functions of sample (as before), difference, sum, weighted average, minimum, maximum and on-duration[/li]
[li]ability to plot data as a column chart[/li]
[li]a new API call to provide aggregated data[/li]
[li]a new API call to provide data in a format suitable for Google Charts[/li]
[li]the ability to access web UI remotely (ALTUI plug-in needed for this to work in UI7)[/li]
[li]ability to save and edit aggregation and chart data with a graph (since 1.910)[/li]
[li]ability to create duplicate variables and display multiple times on a single graph (since 1.911)[/li]
[li]ability to customise a graph though themes (since 1.912)[/li]
[li]Ability to time shift data for direct comparison (since 1.913)[/li]
[li]various performance improvements[/li]
[li]improvements to the documentation[/li][/ul]

Version 2.0 is now published in the MiOS Marketplace here ​MiOS Apps

Please only use this forum for comments on the documentation. Comments on the plug-in itself should be directed to http://forum.micasaverde.com/index.php/topic,35592.0.html

update 25 April 2016: updated new features and documentation for version 1.914

Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:

  1. Remote access of dataMine2 under UI7. I run UI5 so have no idea about this and am not even sure if it works.
  2. Configuring dataMine2 to logg to a CIFS device rather than a USB stick. I’ve seen some forum posts on it (e.g. http://forum.micasaverde.com/index.php/topic,16452.0.html) but haven’t tried it myself.

Your help would be much appreciated. Thanks!

[quote=“ConstantSphere, post:2, topic:190542”]Help Needed!
2. Configuring dataMine2 to logg to a CIFS device rather than a USB stick. I’ve seen some forum posts on it (e.g. http://forum.micasaverde.com/index.php/topic,16452.0.html) but haven’t tried it myself.[/quote]

I can confirm dataMine works fine with CIFS by installing the required packets:

cd /tmp/ opkg update wget http://download.mios.com/firmware/openwrt/ramips/packages/kmod-crypto-md5_2.6.37.1-1_ramips.ipk opkg install kmod-crypto-md5_2.6.37.1-1_ramips.ipk wget http://download.mios.com/firmware/openwrt/ramips/packages/kmod-crypto-hmac_2.6.37.1-1_ramips.ipk opkg install kmod-crypto-hmac_2.6.37.1-1_ramips.ipk wget http://download.mios.com/firmware/openwrt/ramips/packages/kmod-fs-cifs_2.6.37.1-1_ramips.ipk opkg install kmod-fs-cifs_2.6.37.1-1_ramips.ipk opkg install cifsmount rm kmod-crypto-md5_2.6.37.1-1_ramips.ipk kmod-fs-cifs_2.6.37.1-1_ramips.ipk kmod-crypto-hmac_2.6.37.1-1_ramips.ipk

Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status)) 
    if ok 
      then log ("mounted OK: "..tostring(ok)) 
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount" 
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene

[quote=“reneboer, post:4, topic:190542”]Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status)) 
    if ok 
      then log ("mounted OK: "..tostring(ok)) 
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount" 
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene[/quote]

I was using the same code for dataYours but you don’t really need it. I just removed that bit from the LUA startup, configured dataMine as above screenshot and that it - dataMine create the cifs mount on its own

[quote=“SchattenMann, post:5, topic:190542”][quote=“reneboer, post:4, topic:190542”]Hi,

I put this in the startup LUA of my Veras as some updates can delete the mount directory and then often also do not have the required CIFS packages anymore. It is an extension of when akbooer wrote to mount after a reboot. In DataMine (1 or 2) you then set it for manual mount.

-- cifs mount for use during startup
-- @akbooer 2014-04-23
--

local function df ()
  local info
  local p = io.popen ("df",'r')
  if p then info = p:read "*a"; p:close () end
  return info
end

function cifsmount (x)
  local function log (msg) luup.log (("cifsmount: '%s' %s"): format (x.device or '?', msg)) end
  local df = df ()
  if df: find (x.device, 1, true) then
    log "already mounted"
  else
    -- First assure we have the mount folder as UI7 updates clear it. When not there also assume the cifs packages are required
    os.execute(("[ ! -d %s ] && opkg update"):format(x.directory))
    os.execute(("[ ! -d %s ] && opkg install --force-depends kmod-fs-cifs"):format(x.directory))
    os.execute(("[ ! -d %s ] && mkdir %s"):format(x.directory, x.directory))
    local cmd = ("mount -t cifs -o user='%s',password='%s',nounix,noserverino %s %s"): format (x.user, x.password, x.device, x.directory)
    local ok, term, status = os.execute (cmd)
    log ("mounted status: "..tostring(status)) 
    if ok 
      then log ("mounted OK: "..tostring(ok)) 
      elseif x.retry then luup.call_delay ("cifsmount", 60 * x.retry)
      else log "failed to mount" 
    end
  end
end

-- Parameters for mount. Note that device must point to the share and not a subfolder in the share.
local device = "//192.168.xxx.xxx/folder"
local directory = "/dataMine"
local user = "your uid"
local password = "yourpwd"

Cheers Rene[/quote]

I was using the same code for dataYours but you don’t really need it. I just removed that bit from the LUA startup, configured dataMine as above screenshot and that it - dataMine create the cifs mount on its own[/quote]
You can, but after several Vera updates I got tiered of having to perform several of these steps manually over and over. It is all about automation isn’t it ;D.

Cheers Rene

I’m guessing this applies to UI7 only?

Hi,
The need to reinstall the CIFS packages? Indeed as it is the Vera updates that can drop them.

Cheers Rene

Hi,
The need to reinstall the CIFS packages? Indeed as it is the Vera updates that can drop them.

Cheers Rene[/quote]

So to check… on UI5 I guess It wont be a problem as there will be no updates anymore?!

[quote=“ConstantSphere, post:2, topic:190542”]Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:

  1. Remote access of dataMine2 under UI7…
    …[/quote]

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: [url=http://forum.micasaverde.com/index.php/topic,24942.0.html]http://forum.micasaverde.com/index.php/topic,24942.0.html[/url]

and user wadeb kindly posted a PHP implementation on GitHub described here: [url=http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484]http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484[/url]

[quote=“Terryleier, post:10, topic:190542”][quote=“ConstantSphere, post:2, topic:190542”]Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:

  1. Remote access of dataMine2 under UI7…
    …[/quote]

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: [url=http://forum.micasaverde.com/index.php/topic,24942.0.html]http://forum.micasaverde.com/index.php/topic,24942.0.html[/url]

and user wadeb kindly posted a PHP implementation on GitHub described here: [url=http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484]http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484[/url][/quote]
Hi,
If you use ALTUI remotely you can fairly simple put the ?id=lr_dmPage where that of ALTUI starts and DataMine works remote. I will be looking in tweaking that into ALTUI for amg0 to include. I do not think anyone managed to get it working from a link on a Vera. I have sure not been able to figure it out.

Cheers Rene

[quote=“reneboer, post:11, topic:190542”][quote=“Terryleier, post:10, topic:190542”][quote=“ConstantSphere, post:2, topic:190542”]Help Needed!
Does anyone fancy writing up a few paragraphs on the following topics, perhaps with a screen shot that I can add to the documentation:

  1. Remote access of dataMine2 under UI7…
    …[/quote]

This requires multiple steps with MMS authentication, see the UI6Connect.txt notes RichardTSchaefer has posted here: [url=http://forum.micasaverde.com/index.php/topic,24942.0.html]http://forum.micasaverde.com/index.php/topic,24942.0.html[/url]

and user wadeb kindly posted a PHP implementation on GitHub described here: [url=http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484]http://forum.micasaverde.com/index.php/topic,30793.msg220484.html#msg220484[/url][/quote]
Hi,
If you use ALTUI remotely you can fairly simple put the ?id=lr_dmPage where that of ALTUI starts and DataMine works remote. I will be looking in tweaking that into ALTUI for amg0 to include. I do not think anyone managed to get it working from a link on a Vera. I have sure not been able to figure it out.

Cheers Rene[/quote]
Hi,
I tweaked the datamine draw routing in ALTUI so it works remotely box UI7 users. You must go via ALTUI though. I will send this to amg0 to put in the standard ALTUI code.

You have to do this in the J_ALTUI_plugins.js code (don’t forget to refresh your browser.

[code] function _drawDataMine( device) {
var html =“”;
var url = “”;
// var ipaddr = MultiBox.getIpAddr(device.altuiid);
// var hostname = (ipaddr==‘’) ? window.location.hostname : ipaddr;
// var url = window.location.protocol+‘//’+hostname+“/dm/index.html”;
if (window.location.protocol==‘https:’) {
var controller = MultiBox.controllerOf(device.altuiid).controller;
var isUI5 = MultiBox.isUI5(controller);
if (isUI5 == false) {
var main_url = window.location.href;
var url_parts = main_url.split(“?”);
url = url_parts[0]+“?id=lr_dmPage”
} else {
url = ‘https://’+hostname+“/port_3480/data_request?id=lr_dmPage”;
}
} else {
var ipaddr = MultiBox.getIpAddr(device.altuiid);
var hostname = (ipaddr==‘’) ? window.location.hostname : ipaddr;
url = ‘http://’+hostname+“/dm/index.html”;
}
html+= (“{1}” .format( device.altuiid,_T(“Open”) )) ;
html += “”;
return html;
};

[/code]
Have fun.

Cheers Rene

@reneboer - very cool 8)

I’ve updated the documentation to cover the new features of the pending 1.910 release, i.e. the ability to save and edit various graph parameters in the Save Graph dialog. I’ve also added a section to clarify connecting to a remote CIFS store and a paragraph on connecting remotely via UI7 - thanks for all those who contributed. Much appreciated :slight_smile:

Currently, opkg commands are failing with this error:

wget: server returned error: HTTP/1.1 404 Not Found

The server that opkg uses is down so any opkg command related to updating or installing will fail. For additional info, see this post (once approved by Mods): http://forum.micasaverde.com/index.php/topic,16452.msg272134.html#msg272134

cb4

dataMine 2 and Linux for dummies like me.

Not being a Linux expert I’ve struggled for a while trying to get dataMine 2 working on my Very Edge UI7. Starting but not limited to the fact that when you plug the first USB drive into a Vera no matter what size the drive is it creates one 500MB partition which isn’t easy to undo.

I’ve even spent time with Vera support asking if the 2nd USB drive had mounted and according to them everything was working as expected, whatever that means. I’m sure had I known Linux this would have been easy but there is so much info out there it’s hard to know what’s current and what’s not.

One big issue for me is the way many people who understand Linux post things. One post showed the output of what was said to be a “file” command. The output was interesting but they didn’t show the command itself so it was of little help.

If the folks at Vera are reading this it would be very helpful to have a tab that gives some basic systems info. Like what drives are mounted, how full are they, IP address, when a sensor was last seen in a human readable form not Epoch time. Epoch time, that’s Linux speak for the number of seconds since midnight Jan 1st 1970, I think that’s right.

Some posts say you can have Vera logging turned on some say no. Anyway I’m still not sure exactly what was needed but I’ll give you what I did and hopefully someone can tell us what wasn’t.

One note to start, I’ll show all Linux and Vera commands in quotes i.e. “lsusb” so you don’t type in the quotes just what’s between them. Also if you don’t know this Linux is case sensitive so be aware type everything in as shown.

The whole DataMine thing may just work for you through the Vera interface but that wasn’t what happened with me.

First thing I would plug in an old 1GB USB drive then turn logging on in the Vera “Settings, Logs” tab. Vera will grab hold of it and hopefully not think the next drive you plug in belongs to it.

Second you’ll need to download a program called PuTTY www.putty.org/, it allows you to “terminal” into the Vera box and run Linux commands in a DOS like window. Unix/Linux was around long before DOS and is a bit of a steep learning curve but to get what you need for dataMine you’ll only need a few commands.
Be forewarned you can KILL your system with the wrong command here so be very careful what you type in.

When you run PuTTY you’ll type in the IP address of you Vera box, you’ll find the IP at the top of your browser page when you’re logged into your Vera. Next make sure the port number is 22 and SSH is selected, then press the Open button.

If all is working a DOS like window will open and after several seconds a username prompt will appear. Type in “root” and press return. Next a password prompt will appear, my password was printed on a label on the bottom of my Vera, yours may vary. Type in your password and hit return. If all is well you’ve just logged into Linux on your Vera machine. I don’t think it drops you into the root directory so to make sure your are type in “cd /”, (cd space forward slash), that should put you at the root directory of your machine.

One problem I found with the version Linux installed on the Vera is that it doesn’t support many of the traditional tools supported on many Linux system. So if you Google for a command to show the UUID of a USB drive most of the commands you’ll find don’t work here. All of the commands I show here worked on my Vera.

First you’ll need to get the UUID of your USB drive. Before you plug in your new dataMine logging USB drive type in the following command “fdisk -l” (lower case L). At the end of the output you’ll see something like this which is my 2GB Vera logging drive with its 500MB partition.

Disk /dev/sdb: 2003 MB, 2003828736 bytes
62 heads, 62 sectors/track, 1018 cylinders
Units = cylinders of 3844 * 512 = 1968128 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 273 524705+ 83 Linux
/dev/sdb2 274 1018 1431890 83 Linux
root@MiOS_45011987:~#

Now plug in your dataMine drive and run the same command “fdisk-l”. You should see the newly mounted drive. Mine is a 16GB USB drive and it looks like this.

Disk /dev/sda: 16.0 GB, 16008609792 bytes
255 heads, 63 sectors/track, 1946 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 1947 15632384 c Win95 FAT32 (LBA)

The important thing to note here is the /dev/sda1 which you’ll type into Vera’s dataMine SetMountPoint. Yours may be different so write it down!

Next type in “lsusb” this will give you your drives UUID and should look something like this. Mine is the Cruzer and its UUID is 0781-5530, write yours down!

root@MiOS_45011987:~# lsusb
Bus 001 Device 002: ID 050d:0237 Belkin Components F5U237 USB 2.0 7-Port Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 003: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 004: ID 0781:5530 SanDisk Corp. Cruzer
Bus 001 Device 005: ID 8644:800a
root@MiOS_45011987:~#

Next install dataMine 2 on your Vera but make sure to leave the PuTTY window open, go to Vera’s “Apps, Install apps” tab then search for dataMine. There will be two pick dataMine 2. After the install is complete go to Vera’s Devices tab and find the dataMine 2 graphing device and open it. Then open the Advanced tab. Near the top of the page you’ll find 4 tabs, Parms, Variables, New service and Commands. Select the Variables tab. Under the Variables tab add the following information. In SetDataDirectory type in “/dataMine/”, after you type that in you have to right click in any of the white space on the page to update the information, strange but it’s the way you have to do it. In SetMountUUID you’ll have to type in your USB drives UUID. Then last but not least in SetMountPoint type in the mount point for your system each time right clicking in the white area in the web page. It will flash on the screen that the data has been updated.

The SetDataDirectory crates a dataMine folder in the root directory. The SetMountUUID and SetMountPoint attaches your USB drive to the dataMine folder. Now go back to the PuTTY window and type in “df -h”. It should show your drive attached to /dataMine directory and the used and available space.

root@MiOS_45011987:/dataMine# df -h
Filesystem Size Used Available Use% Mounted on
rootfs 9.4M 5.3M 4.1M 56% /
/dev/root 9.5M 9.5M 0 100% /rom
tmpfs 61.9M 3.4M 58.5M 5% /tmp
/dev/mtdblock6 9.4M 5.3M 4.1M 56% /overlay
overlayfs:/overlay 9.4M 5.3M 4.1M 56% /
tmpfs 512.0K 0 512.0K 0% /dev
/dev/mtdblock10 64.8M 9.9M 54.8M 15% /storage
/dev/mtdblock10 64.8M 9.9M 54.8M 15% /etc/cmh-firmware
/dev/mtdblock10 64.8M 9.9M 54.8M 15% /etc/cmh-backup
/dev/mtdblock9 7.8M 7.8M 0 100% /mios
/dev/sda1 14.9G 184.0K 14.9G 0% /dataMine

At this point you should be able in Vera to go to the Devices, dataMine 2 graphing, dataMine tab and select the “dataMine graphing page (local access)” and bring up a new tab in your browser called dataMine - graphing. If it asked select the USB Drive with the correct UUID. Under the Configuration tab add the sensors you want to graph. If you want to see where the data files for the sensors get built type in the PuTTY window “cd /dataMine” then “du -h”. That’s all I got hope it helped. Please if I messed something up PLEASE let me know.

root@MiOS_45011987:~# cd /dataMine
root@MiOS_45011987:/dataMine# du -h
16.0K ./database/1/raw
32.0K ./database/1
16.0K ./database/2/raw
32.0K ./database/2
8.0K ./database/3/raw
24.0K ./database/3
96.0K ./database
184.0K .

hi guys,

i have an issue with Datamine 2 resetting its configuration a couple of days ago

Is there any way we can access or download the data from the USB drive?

2 weeks worth of data were logged into the USB drive and we do not knw if there is a way to retrieve this data

help!

Could someone explain the easiest way to access dataMime 2.0 remotely from the VeraEdge on UI7 Im not sure where to place the “?id=lr_dmPage”. Thank you!!

Hi,
You cannot via the Vera portal. You can only do that via ALTUI portal. Details of that you can find in the ALTUI forum articles.

Cheers Rene

Thanks Rene,

I have ALTUI but i cant figure out how to remotely accesss datamine on my Vera Edge. Would you happen to have a line to the article or post? I have been searching for ever. Thank you!!

Lou