dataMine 0.979

ChrisJ · May 27, 2013, 9:04am

This is now on the app store awaiting MCV approval… It doesn’t fix the migration of data issue that people have reported, so you’ll still only have the last week or two of data available. I wanted to get this small update out to fix the directory problem, but I need to work around a file copy issue with Veras “cp” command to make sure I don’t overwrite the new data with the old… I’ll get an update out to finish off the transfer of data in the next few days.

Version 0.979

Fixed problem creating database/ directory
Fixed problem where y-axis lookup may be reset on startup

Cheers
Chris

nanotube · May 29, 2013, 5:07am

Hi, Chris: I just installed 0.979 from app.mios.com and it doesn’t work

Basically upon install finish there is a never-ending message of “Installing update of dataMine web application…” in the notification area in mios. I waited 20 minutes and it’s still up there.

The app did show in the app area and it did install a new device. When open the new device it offers me two links, one to the GUI, one to the debug. The GUI link just shows a never ending scroll wheel with GUI — and the LUA version also —. Clicking on debug just says “No handler”.

I have tried the lua file you provided in this post: http://forum.micasaverde.com/index.php?topic=13517.0
It does get rid of the Installing update message but still the GUI didn’t work. I have yet to see the GUI prompting for the USB drive.

Any suggestions? Thanks.

ChrisJ · May 29, 2013, 3:28pm

Hi,
Sorry you’re having problems. I really need the log to determine what’s going on. Since the plugin isn’t working, the debug link is also not working, so we’ll need to do this manually…

If you have WinSCP, then first restart Luup to make sure we view the log when there’s an error, then in the terminal box in WinSCP type “cat /var/log/cmh/LuaUPnP.log | grep -i DataMine”.

Hopefully this will dump all the dataMine log entries and it should allow me to find the problem.

Cheers
Chris

ChadM · May 29, 2013, 6:02pm

I ran into something similar using a Vera 2. I think it was a mixture of messed up configuration files from prior attempts and the flash drive getting filled up. The latter was a problem a few times. Take a look at “df -h /overlay”. It should have some free space left, hopefully more than a few hundred kB. If it doesn’t, that’s a problem.

To deal with the first one, I deleted everything in the /dataMine directory. Lost a few days worth of data, but that’s fine.

For the second one, I poked around and found the biggest culprits I had for space usage was dataMine, the Denon AVR app, and the Weather Underground app. I uninstalled all of them, then went and deleted the remnants of the app from flash. Not sure when, or even if, those remnants are cleared out since they aren’t removed at application uninstall.

Most of the stuff lives in /overlay/etc/cmh-ludl although dataMine also has stuff in /overlay/www.
dataMine: Two areas to clean up. Both must be cleared.
rm -f /overlay/etc/cmh-ludl/DataMine
rm -rf /overlay/www/dm

Denon AVR app: This one is easy.
rm /overlay/etc/cmh-ludl/Denon

Weather Underground app: I’m going from memory since I haven’t reinstalled the app to ensure I have some free space.
rm /overlay/etc/cmh-ludl/WUI

The worst state I got it in was when the flash was completely full and left dataMine sort of halfway installed. The files in /overlay/www/dm, primarily the 1.2 MB ext-all-datamine.js file, were incomplete. Only by completely clearing it all out did I free up enough space for the installation to proceed properly from beginning to end.

-Chad

DaveL17 · May 30, 2013, 1:17am

Having a little trouble with 0.979 myself. I’m not quite sure how to interpret the debug log but it appears that I have at least two problems. The first is that dataMine can no longer see my USB flash drive. The second seems to be that there’s no room left in /dataMine. I’m tracking 20 variables.

Chris, I’ll PM you a link to my log file now. Thanks for any help that you can provide. I’m missing my dataMine.

nanotube · May 30, 2013, 3:25am

[quote=“Chris, post:3, topic:175538”]If you have WinSCP, then first restart Luup to make sure we view the log when there’s an error, then in the terminal box in WinSCP type “cat /var/log/cmh/LuaUPnP.log | grep -i DataMine”.

Hopefully this will dump all the dataMine log entries and it should allow me to find the problem.

Cheers
Chris[/quote]

Thanks for the quick reply Chris. So I repeated the steps when Datamine refuses to be installed and here is the log. I am running a Vera Lite with UI5 version 1.5.346. I have a freshly formatted FAT32 drive plugged in also. Can you help me with what the error is? Thanks again.

/var/log/cmh$ cat /var/log/cmh/LuaUPnP.log | grep -i DataMine
02	05/29/13 20:17:33.878	e[33;1mJobHandler_LuaUPnP::FetchMissingFiles need to download D_DataMine1.jsone[0m 

<0x2ae35000>
02	05/29/13 20:17:33.878	e[33;1mJobHandler_LuaUPnP::m_sMissingFile_set D_DataMine1.jsone[0m <0x2ae35000>
01	05/29/13 20:17:33.953	e[31;1mDevice_LuaUPnP::LoadDeviceDoc can't load /etc/cmh-lu//D_DataMine1.xmle[0m 

<0x2ae35000>
02	05/29/13 20:17:33.954	e[33;1mJobHandler_LuaUPnP::m_sMissingFile_set /etc/cmh-lu//D_DataMine1.xmle[0m 

<0x2ae35000>
01	05/29/13 20:17:33.954	e[31;1mJobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 47/D_DataMine1.xml 

so device 47 is offlinee[0m <0x2ae35000>
09	05/29/13 20:17:40.140	JobHandler_LuaUPnP::Run device 47  room 0 type urn:schemas-cd-jackson-

com:device:DataMine:1 id  parent 0/0xc0e390 upnp: 0 <0x2ae35000>
03	05/29/13 20:17:40.153	JobHandler_LuaUPnP::Run ready 23 devices installation 35001106 missing /etc/cmh-

lu//D_DataMine1.xml version *1.5.346*  allow_evt: 1 allow_ud: 1 upnp 0 changed: 0 pMem 0xf43000 start 12132352 diff: 

3870720 <0x2ae35000>
09	05/29/13 20:17:52.442	JobHandler_LuaUPnP::Run device 47 dataMine graphing and logging room 0 type 

urn:schemas-cd-jackson-com:device:DataMine:1 id  parent 0/0xcde2d0 upnp: 0 <0x2aeef000>
50	05/29/13 20:18:09.521	luup_log:47: dataMine: Initialising dataMine System (0.979) <0x2b823680>
06	05/29/13 20:18:09.521	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetDataDirectorye[0m was: EMPTY now: /dataMine/ #hooks: 0 upnp: 0 v:

(nil)/NONE duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.522	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetMountUUIDe[0m was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.523	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetMountPointe[0m was: EMPTY now:  #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 

<0x2b823680>
06	05/29/13 20:18:09.523	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetManualMounte[0m was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.524	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetTimeoutPeriode[0m was: EMPTY now: 12 #hooks: 0 upnp: 0 v:(nil)/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.525	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetHistoryEnablee[0m was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.525	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetUseBackupe[0m was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.526	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mSetEventsEnablee[0m was: EMPTY now: 1 #hooks: 0 upnp: 0 v:(nil)/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.527	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mChannelCnte[0m was: EMPTY now: 0 #hooks: 0 upnp: 0 v:0xdf5a90/NONE 

duplicate:0 <0x2b823680>
06	05/29/13 20:18:09.528	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mChannelRece[0m was: EMPTY now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:0 

<0x2b823680>
50	05/29/13 20:18:09.528	luup_log:47: dataMine: Installing update of dataMine web application <0x2b823680>
50	05/29/13 20:18:14.296	luup_log:47: dataMine: ERROR: Startup state is 'uninitialised'! <0x2b823680>
06	05/29/13 20:18:14.297	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mChannelCnte[0m was: 0 now: 0 #hooks: 0 upnp: 0 v:0xdf5a90/NONE duplicate:1 

<0x2b823680>
06	05/29/13 20:18:14.297	Device_Variable::m_szValue_set device: 47 service: urn:cd-jackson-

com:serviceId:DataMine1 variable: e[35;1mChannelRece[0m was: 0 now: 0 #hooks: 0 upnp: 0 v:(nil)/NONE duplicate:1 

<0x2b823680>
50	05/29/13 20:18:14.298	luup_log:47: dataMine: Reinitialising configuration structure <0x2b823680>
01	05/29/13 20:18:14.298	e[31;1mLuaInterface::CallFunction_Startup-1 device 47 function initialise failed [string "-- 

dataMine Plugin for Vera..."]:488: attempt to call field 'sunrise' (a nil value)e[0m <0x2b823680>
01	05/29/13 20:18:14.299	e[31;1mLuImplementation::StartLua running startup code for 47 I_DataMine1.xml failede[0m 

<0x2b823680>

garrettwp · May 30, 2013, 3:44am

I highly recommend that you update your Vera’s firmware.

Garrett

ChrisJ · May 30, 2013, 9:57am

[quote=“DaveL17, post:5, topic:175538”]/dev/scsi/host0/bus0/target0/lun0/part1: LABEL=“MiOS” UUID=“93b23adf-9c65-4e13-8b80-718ebd718515”
/dev/scsi/host1/bus0/target0/lun0/part1: LABEL=“MiOS” UUID=“d66568ba-7da6-45cc-91bb-c658063246cf”[/quote]
This is from your debug log - it looks like Vera has taken over the drive. Further up the debug log, it says it can’t find your USB based on the UUID (which is confirmed by the above which is the output of blkid) - the drive doesn’t appear to exist.

[quote=“DaveL17, post:5, topic:175538”]50 05/29/13 19:48:07.989 luup_log:61: dataMine: Mounting to UUID ‘5C88-A273’ <0x402>
50 05/29/13 19:48:10.508 luup_log:61: dataMine: Error reading tmpfile during UUID check <0x402>
50 05/29/13 19:48:10.557 luup_log:61: dataMine: No UUID mount ‘5C88-A273’ was found! <0x402>[/quote]

However, this is slightly inconsistent with other data -:

[quote=“DaveL17, post:5, topic:175538”]50 05/29/13 20:01:48.303 luup_log:61: dataMine: Watching: D[51] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>
50 05/29/13 20:01:48.308 luup_log:61: dataMine: Watching: D[51] S[urn:upnp-org:serviceId:TemperatureSetpoint1_Heat] V[CurrentSetpoint] <0x402>
50 05/29/13 20:01:48.312 luup_log:61: dataMine: Watching: D[51] S[urn:upnp-org:serviceId:TemperatureSetpoint1_Cool] V[CurrentSetpoint] <0x402>
50 05/29/13 20:01:48.317 luup_log:61: dataMine: Watching: D[55] S[urn:upnp-org:serviceId:TemperatureSensor1] V[CurrentTemperature] <0x402>[/quote]
This indicates that the config file has been read, but it’s not reading it from the USB drive!

So, I suspect that Vera has at some point formatted your USB drive (hence the two MIOS drives), but dataMine is probably putting all its data into the Vera drive, and this is now full.

If this is correct, you should be able to see by running WinSCP, removing the USB sticks, and seeing what’s in the /dataMine directory. If I’m right, then there will be a lot of stuff in there…

If so, then the following should get you up and running again…

Copy all the data from the /dataMine directory to your PC and delete the data. This will free up your memory in Vera.
Remove one of the USB sticks - I don’t know why one the Vera logging is using - you may need to try and work this out first. It may also be random since Vera uses the label to work this out, and given both sticks have the same label, I don’t know what will happen.
Format the stick in Windows
Copy all the data back onto the USB stick
Plug the stick back into Vera and get the dataMine debug. Look for the section above which starts with -2- and lists the UUIDs
In the dataMine plugin, change the SetMountUUID to the UUID from the previous step.

I hope this will get you running again. If you have any problems, let me know and we’ll sort it out. Obviously if you don’t find all the data in your /dataMine directory when you remove the sticks, then my assumption is incorrect and I need to rethink

Cheers
Chris

ChrisJ · May 30, 2013, 6:26pm

[quote=“nanotube, post:6, topic:175538”]01 05/29/13 20:18:14.298 e[31;1mLuaInterface::CallFunction_Startup-1 device 47 function initialise failed [string “-- dataMine Plugin for Vera…”]:488: attempt to call field ‘sunrise’ (a nil value)e[0m <0x2b823680>
01 05/29/13 20:18:14.299 e[31;1mLuImplementation::StartLua running startup code for 47 I_DataMine1.xml failede[0m[/quote]

Garrett gets the prize - you need to update your firmware. The error above is with a call to the function sunrise - the function to get the next sunrise time. The documentation states that the required version for this function is 1.5.353, so this will not work on your Vera.

Cheers
Chris

DaveL17 · May 30, 2013, 10:02pm

Well, that’s alarming.

I’m pretty certain that I didn’t do anything to cause this to happen–initially I had USB logging for Mios on one drive, and dataMine logging on another. Of course, with distinct drive labels. No idea what would’ve caused this behavior.

I had been toying with the idea of using one, large hard drive with Mios and dataMine writing to a single drive (another reason behind why I was considering this is the potential to implement a swap file on said drive). Bad idea?

Regardless, I’ll follow the steps you’ve outlined and will report back with the results.

Thanks Chris, you’re aces.

Dave

nanotube · May 30, 2013, 10:49pm

Thanks a lot, Chris (and garrettwp too)!

Just upgraded firmware to .408 and it installed fine.

garrettwp · May 31, 2013, 12:53am

You might want to upgrade to 1.5.622.

Garrett

DaveL17 · May 31, 2013, 2:49am

Chris, I’ve followed the instructions above (incidentally, there were 127 items in the /dataMine directory on Vera) and named the freshly formatted stick “dataMine”. I should say that, while I typically use OSX to interact with Vera, and use RBrowser (rather than WinSCP), I did format the stick as FAT32 on a Windows XP partition and used Windows File Manager to write the files back to the stick.

I reinserted the stick and my Vera unit became non-responsive. By that, I mean that I couldn’t load UI5. So I power cycled Vera. In the status window at the top, the following message appeared:

dataMine graphing and logging[61] : Installing update of dataMine web application

I gave it some time and refreshed my browser (by that, I mean that I did not reload UI5). I proceeded to the dataMine debug file to find:

/dev/scsi/host0/bus0/target0/lun0/part1: LABEL="MiOS" UUID="f17bd608-2488-43f5-9ccd-f95acae3642e" /dev/scsi/host1/bus0/target0/lun0/part1: LABEL="MiOS" UUID="d66568ba-7da6-45cc-91bb-c658063246cf"

Which appears to show that, as soon as I reinserted the new stick, Vera renamed it. I can load the dataMine web client, but I expect that if I start to use it, it will revert to writing to the /dataMine directory on Vera.

Any ideas why Mios wants my thumb drives so badly? Are there compelling reasons to not just use one large eSATA drive and write everything to one location?

Thanks again for everything, and cheers.
Dave

mfp · May 31, 2013, 6:56am

Hi looks like the Vera captured your second usb stick, while you have logging enabled. I consider this as a bug.

I also use 2 usb sticks and experieneced this a couples of times. Vera sometimes (mostly on boot or during startup) mix up the usb ports. It searches on your datamine stick for the structure needed for logging. If this can’t be found the stick will be reformatted and gets a mios label. You end up with two mios sticks for logging.

I fixed it by moving the usb-sticks over the different usb ports to find out which one gets initialized first by your vera. In this port you want your miosusb stick. The datamine stick can be put in any other port.

The second thing I did was formatting the datamine stick as ext3 instead of Fat32. It looks like vera better recognises linux ext3 partitions.

The third thing I did was using a usb powered hub, to make sure the usb subsystem gets enough power for normal operation.

Since those changes my datamine stick has not be captured by mios again.

DaveL17 · May 31, 2013, 11:52am

Thanks mfp.

Would it be safe to assume that the port that holds the stick (once labelled dataMine, now labelled Mios) to be the one that Vera checks first?

If Vera just randomly changes its mind, though, I might just try going with one large drive after all. Then I’d only have one mount point to worry about.

mfp · May 31, 2013, 12:15pm

I don’t think you can safely assume that, as usb ports seem to be random initialised on vera. For this reason they get mixed up and you end with two mios sticks.

I put my stick for logging directly on the vera and my datamine stick on the usb hub.

I advise you to experiment with these steps:

reformat and/or repartition (ext3) the dataminestick give it a datamine label
then change the location of your sticks
remove the power of the vera
watch what happend, review setting and when needed repeat the steps

ChrisJ · May 31, 2013, 1:21pm

Completely agree with this. We’ve also seen that they can change while Vera is running!

I investigated this a little a while ago, and there’s a post here (http://forum.micasaverde.com/index.php/topic,13634.0.html). I really don’t know what happens if there are two drives with the label MiOS - it’s seemingly undefined.

Chris

DaveL17 · June 4, 2013, 3:05am

Okay, I’ve had a chance to play around with this a little bit. I’ve removed my 2 USB sticks (unfortunately both named Mios) and replaced them with 1 USB hard drive (40GB) that is being driven by both ports (presumably one for data and the second to draw more power). I have enabled USB logging on Vera and all seems to be functioning normally. Vera formatted the drive as ext3 and I can browse to tmp/log/cmh.

Is there any reason why I couldn’t configure dataMine to use this same mount point and have the files stored to tmp/log/cmh/dataMine?

TIA,
Dave

ChrisJ · June 4, 2013, 11:40am

Yes - this should work in the same way as if you had a USB stick configured for logging and dataMine. There’s a section on how to configure this in the manual, so if you follow this, you should be fine.

Please let me know how you go with this. We know USB is a bit poor, so I’d be interested to see if USB HD is any better. We are also testing network drives at the moment and this quite looks promising…

Cheers
Chris

DaveL17 · June 4, 2013, 11:51am

Okay, so far, I can see one drawback to what could have been a straightforward solution. Even though the drive is 40GB, apparently Vera only mounted ~500MB for USB logging. I expect that I would chew this up rather quickly with both Vera and dataMine logging. I’m assuming that OpenWRT can handle much bigger partitions.

mfp suggested formatting the drives in ext3, but I think in another thread Chris, you suggested that it’s probably not a good idea to use Vera to format the drive (although that was in the other thread about having two drives named Mios I believe.)

I don’t have ready access to another Linux box (I should probably do something about that…), so I need to figure something out.