DataYours - App Store Release

Which browser are you using? Have you tried another?

And, how can I delete Graphs?
From page 10 of the User Guide:

"Clear empties the selection list. Saving a graph with nothing in the selection list deletes an existing graph of the same name. "

So far so good on the install. I have some basic graphs up and running and playing around with the options now to figure things out. I also had the missing icon issue, but was able to resolve it by copying over all the Data*.png files as below …

FROM:
/overlay/www/cmh/skins/default/icons/

TO:
/www/cmh/skins/default/img/devices/device_states

Refresh the browser after that and you should be good to go.

Yup, that fixes it - I thought the App store installation dealt with that, but apparently not. (I also thought I had done that manually already, but apparently also not!) I can put a fix in the next update, or indeed, produce a bit of code to run in Test Luup which will do the trick too.

Thanks. Karma to you!

Hi All - I am a newbie to Vera. Had the Data Yours plugins installed from App Store a day ago and uninstalled them. I now want to give it another try but not installing (even after reboot). Install starts from the app store and then I either get an error or nothing appears in the devices or apps list. DO I need to manually clean up the old install before attempting reinstall?

OK, thanks for arriving at the right thread!

[ul][li]Can you confirm that you are on a Vera Edge with the latest firmware 1.7.961 ?[/li]
[li]What did you do to unistall [tt]DataYours[/tt]?[/li][/ul]

There really should be no cleanup required between uninstall and a new install. You should, though, confirm that none of the [tt]DataYours[/tt] devices now appear on your devices list. There’s lots we can do to sort this out, so this is perhaps just the start. I’ve not come across this problem before, but UI7 is relatively new…

First Off… Thank You for all your hard work!! I’m really hoping I can get this working this time.

  1. How do I use a USB stick for the data storage.
  2. How do I “If you have them, copy the files storage-schemas.conf and storage-aggregation. conf, into your target Whisper database directory.”

Please keep in mind, I’m a complete idiot when it comes to these things… :stuck_out_tongue:

Welcome back, and thanks for giving the App Store version a go!

1) How do I use a USB stick for the data storage.
[tt]DataYours[/tt] does not mount any storage devices, it simply uses the given directory path, which may be local, mounted over a network, or on a USB stick. You'd have to ask elsewhere for advice on using USBs, sorry.
2) How do I "If you have them, copy the files storage-schemas.conf and storage-aggregation. conf, into your target Whisper database directory."
They are in a zip attachment to the first post of this thread. I couldn't make them part of the install because I don't know where your database directory is going to be. I suggest you use them - it reduces the pain of setting up individual archive specifications for different measurement types. You can always edit these files later if you want different defaults.
Please keep in mind, I'm a complete idiot when it comes to these things.. :P
Believe me, we ALL are, when things don't go to plan!

“ak”

I’d love to implement this but I’m concerned about load on the Vera. Right now I have two Veras running, one with all the ZWave stuff on it (VERA-1/Vera3+UI5) and another with only my DSC alarm interface and the Blue Iris Unoffical Plug in (VERA-2/Vera3+UI5).

Vera-1 runs LuaUPnp at about 90% memory and bumps 90+% CPU about once a minute (SSH/TOP) … in between it runs about 5-10% and I suspect the once per minute bump is due to PLEG scheduling checking but I can’t prove it.

Vera-2 runs LuaUPnp at 110% memory and bumps about 80% CPU about once per minute (again PLEG?)… in between it also runs about 5-10%.

I’ve had to give up most of the inter-Vera communications (tried using semaphores because the bridging didn’t work well at all) due to the loading causing frequent restarts

So the question is: a) if my Vera can’t handle just ZWave activity how is it going to handle that AND DataYours? and b) what the data store for DataYours? Internal memory? USB?

thanks
John

How do I copy these files and where is the target Whisper database?

See I told you I’m a complete Idiot with this stuff … 8)

Click on the link in the post to download the file. Unzip the archive, this should give you two separate files (or possibly a folder with them in.) Copy the files to where you want your database to be. This would probably be a USB drive or a NAS mounted drive - I can’t give specifics here, because it depends on where you put things. The target Whisper database is where you want your data to be stored and the location set up in the LOCAL_DATA_DIR variable(s) as you configure the different parts of [tt]DataYours[/tt].

Can some here help me use a USB drive for data storage for DataYours? I have search and search and search and just about everything I find is related to datamine’s internal USB mount system. Can’t seem how to do this for use with DataYours.

TIA

I understand your concern, and certainly, adding something extra is not going to take less resource!

Vera-1 runs LuaUPnp at about 90% memory and bumps 90+% CPU about once a minute (SSH/TOP) ... in between it runs about 5-10% and I suspect the once per minute bump is due to PLEG scheduling checking but I can't prove it.

Vera-2 runs LuaUPnp at 110% memory and bumps about 80% CPU about once per minute (again PLEG?)… in between it also runs about 5-10%.


[tt]DataYours[/tt] actually tracks how much memory it is using within whatever allocated address space it has. By default, the UI7 version creates a device in which to run, but it is possible to configure it to run in memory which has already been allocated to something else, specifically the space where scenes and Startup Lua runs. I doubt you could detect the difference between it running and not running in this configuration.

Equally, the CPU is tiny. I typically run at 17% CPU whether or not I have [tt]DataYours[/tt] up and running.

I've had to give up most of the inter-Vera communications (tried using semaphores because the bridging didn't work well at all) due to the loading causing frequent restarts
The UDP datagram protocol is incredibly lightweight. Nothing like doing HTTP or the like.
So the question is: a) if my Vera can't handle just ZWave activity how is it going to handle that AND DataYours? and b) what the data store for DataYours? Internal memory? USB?

All I can say is try it and see what happens. The storage is wherever you like - either local to Vera, USB stick if you have one mounted, or NAS if you have that.

I can tell you how to run without creating a Vera device and save space that way.

ak

I thought I might have figured out how to mount the USB drive… but maybe not…

I get this error when attempting to create a Storage Aggregation…

Error creating: Vera-30010583.140.urn:micasaverde-com:serviceId:EnergyMetering1.KWH Status messsage = [string "module ("L_DataWhisper7", package.seeall)..."]:636: Cannot create file '%s'

That’s certainly an error during file creation, most likely due to not having a valid path.

[ul][li]What have you set as the path?[/li]
[li]Can you access that from Vera, never mind [tt]DataYours[/tt][?/li][/ul]

I understand your concern, and certainly, adding something extra is not going to take less resource!

Vera-1 runs LuaUPnp at about 90% memory and bumps 90+% CPU about once a minute (SSH/TOP) ... in between it runs about 5-10% and I suspect the once per minute bump is due to PLEG scheduling checking but I can't prove it.

Vera-2 runs LuaUPnp at 110% memory and bumps about 80% CPU about once per minute (again PLEG?)… in between it also runs about 5-10%.


[tt]DataYours[/tt] actually tracks how much memory it is using within whatever allocated address space it has. By default, the UI7 version creates a device in which to run, but it is possible to configure it to run in memory which has already been allocated to something else, specifically the space where scenes and Startup Lua runs. I doubt you could detect the difference between it running and not running in this configuration.

Equally, the CPU is tiny. I typically run at 17% CPU whether or not I have [tt]DataYours[/tt] up and running.

I've had to give up most of the inter-Vera communications (tried using semaphores because the bridging didn't work well at all) due to the loading causing frequent restarts
The UDP datagram protocol is incredibly lightweight. Nothing like doing HTTP or the like.
So the question is: a) if my Vera can't handle just ZWave activity how is it going to handle that AND DataYours? and b) what the data store for DataYours? Internal memory? USB?

All I can say is try it and see what happens. The storage is wherever you like - either local to Vera, USB stick if you have one mounted, or NAS if you have that.

I can tell you how to run without creating a Vera device and save space that way.

ak[/quote]

I’d very much like to hear how to implement in UI5 without a device to keep load down as much as possible.

I’d also propose to log to NAS, I currently use a SysLog running on a NAS for logging EventWatcher and Vera Alerts so if logging data to the NAS would work that’s the place that would seem to make sense.

In the same vein, what happens when it comes time to move to UI7? Will all of this migrate cleanly or are we in for major re-installations? :slight_smile: When the time comes to move to UI7 my plan is to migrate the ZWave to a new Vera Edge and put the current Vera3 in applications service, perhaps move the Blue Iris interface there.

You have a number of Veras running. How do they interact if at all? Bridging? Standalone?

John

[quote=“akbooer, post:34, topic:186018”]That’s certainly an error during file creation, most likely due to not having a valid path.

[ul][li]What have you set as the path?[/li]
[li]Can you access that from Vera, never mind [tt]DataYours[/tt][?/li][/li][/ul][/quote]

So this is what I did:

  1. Formatted a USB Drive ext3
  2. Copied folder named “datayours” with the 2 files listed above.
  3. Plugged USB into USB Hub connected to Vera 3
  4. Using Info Viewer’s LuCI Router Interface >> to System >> Mount Points
  5. Clicked on the Add button under Mount Points.
  6. Added /dev/sda1 mount

Then in the datayours children I added : “/dev/sda1/” as the location (I also tried /dev/sda1/datayours/)

First step is to install from the App Store, and then unistall it straight-away. This will retain the files for us to assemble a system by hand.

I'd also propose to log to NAS, I currently use a SysLog running on a NAS for logging EventWatcher and Vera Alerts so if logging data to the NAS would work that's the place that would seem to make sense.
Perfect. So you have that all up and running already. All you need is to create a directory for the database.
In the same vein, what happens when it comes time to move to UI7? Will all of this migrate cleanly or are we in for major re-installations? :) When the time comes to move to UI7 my plan is to migrate the ZWave to a new Vera Edge and put the current Vera3 in applications service, perhaps move the Blue Iris interface there.
Your biggest challenge here, is how you migrate your devices. I'm assuming that this will mean that all your device Ids will change. The important thing to do is to note what the OLD devices are, migrate, and then you simply need to rename the individual database files (one per device variable logged) to the new Ids. You could do that manually, or with a piece of code.
You have a number of Veras running. How do they interact if at all? Bridging? Standalone?
I tried bridging, but it was fairly hopeless. From a control point of view, I use HomeWave which interacts with multiple Veras transparently. From a monitoring point of view, [tt]DataYours[/tt] is designed from the ground up as a multi-processor system. I have four machines, all running [tt]DataWatcher[/tt] to monitor their own variables, but all sending to multiple other machines for database mirroring and/or plotting. You don't need to run the storage ([tt]DataCache[/tt]) on the same machine as you acquire the data. The dashboard application ([tt]DataDash[/tt]) federates multiple machines into one homogeneous interface. I attach a screenshot of my Whisper database showing variables acquired across the four different machines (over 100 channels) and also the Devices display (over 200 devices.)

[quote=“ServiceXp, post:36, topic:186018”]So this is what I did:
[…]

Then in the datayours children I added : “/dev/sda1/” as the location (I also tried /dev/sda1/datayours/)[/quote]

I’m sorry, but where do those images come from? It looks like nothing I’ve ever seen.

The only thing you need to know is that your storage can been seen from Vera. [tt]ssh[/tt] into it and do [tt]df[/tt]:

[ul][li]Do you see your device? [/li]
[li] What is its pathname?[/li][/ul]


[Edit: actually, it looks like the mount point might be /datayours/ ? Try that as LOCAL_DATA_DIR ]

First step is to install from the App Store, and then unistall it straight-away. This will retain the files for us to assemble a system by hand.

I'd also propose to log to NAS, I currently use a SysLog running on a NAS for logging EventWatcher and Vera Alerts so if logging data to the NAS would work that's the place that would seem to make sense.
Perfect. So you have that all up and running already. All you need is to create a directory for the database.
In the same vein, what happens when it comes time to move to UI7? Will all of this migrate cleanly or are we in for major re-installations? :) When the time comes to move to UI7 my plan is to migrate the ZWave to a new Vera Edge and put the current Vera3 in applications service, perhaps move the Blue Iris interface there.
Your biggest challenge here, is how you migrate your devices. I'm assuming that this will mean that all your device Ids will change. The important thing to do is to note what the OLD devices are, migrate, and then you simply need to rename the individual database files (one per device variable logged) to the new Ids. You could do that manually, or with a piece of code.
You have a number of Veras running. How do they interact if at all? Bridging? Standalone?
I tried bridging, but it was fairly hopeless. From a control point of view, I use HomeWave which interacts with multiple Veras transparently. From a monitoring point of view, [tt]DataYours[/tt] is designed from the ground up as a multi-processor system. I have four machines, all running [tt]DataWatcher[/tt] to monitor their own variables, but all sending to multiple other machines for database mirroring and/or plotting. You don't need to run the storage ([tt]DataCache[/tt]) on the same machine as you acquire the data. The dashboard application ([tt]DataDash[/tt]) federates multiple machines into one homogeneous interface. I attach a screenshot of my Whisper database showing variables acquired across the four different machines (over 100 channels) and also the Devices display (over 200 devices.)[/quote]

Very impressive.

Having given up on bridging I’m using ImperiHome as a merged interface.

I’m experimenting again with semaphores using MultiSwitches in pulse mode, each switch set by a Blue Iris camera motion detection alert to give outside-home motion information to the Zwave Vera. Ultimately I want to use this info to activate sets of outside lights in conjunction with recording, notifications, etc. etc.

Anyway, once I install/uninstall everything how do I make use of the residual files then since ther is no device interface to them?

The NAS is local and supports SysLog used by both Veras with EventWatcher/Vera Alerts so I know that part works. I gather I’m going to have to add a CIFS interface to allow DataYours to transfer data to the NAS. I’ve seen that here on the forum so that should be fairly straightforward. As far a reports/graphing then that can run on either of the Veras and report on everything?

Thanks

That, @clippermiami, is the right question to ask! (*see below)

The NAS is local and supports SysLog used by both Veras with EventWatcher/Vera Alerts so I know that part works. I gather I'm going to have to add a CIFS interface to allow DataYours to transfer data to the NAS. I've seen that here on the forum so that should be fairly straightforward.
Yes. As far as I can tell, I'm the only one who hasn't managed to get this going on an Edge!
As far a reports/graphing then that can run on either of the Veras and report on everything?
Yes. The graphing and dashboard components need to run on the same machine and have directory path access to the NAS. The storage (cache) component can run on another machine but, again, needs NAS access. The watcher front-ends can run on any machine and just need to be on the network. The dashboard gleans its metadata from the combination of the database files and the online Veras, the IPs of which it gets explicitly from the VERAS variable setting.

*overview of how to configure without a [tt]DataYours[/tt] device

[ol][li]the configuration for all components is kept in a file [tt]/www/carbon.conf[/tt] (rather than in device variables) (same as Beta test code)[/li]
[li]the variables in the above file are named identically to the device variables used in normal configuration[/li]
[li]the daemons are started in the Startup Lua code[/li]
[li]you can add watched variables through HTTP requests, or by editing the [tt]DataWatcher.conf[/tt] configuration file, or through the dashboard interface[/li][/ol]

You’ll need details of (2) and (3) above separately. It’s not hard.