DataYours Beta Test - VERSION 2

Plugins & Plugin Development Unmaintained
#1

Here is a new version of the DataYours data logging plugin. To avoid confusion, it’s called DataYours2.

The original prototype proved difficult to configure and although the underlying system of data acquisition and storage worked well, the dashboard interface proved unreliable on different Vera configurations and complex to configure Whisper database archives.

Significant changes to address these, and other, issues include:

[ul][li]A more complete reverse-engineering of the Graphite/Carbon system means that system configuration is now largely done through configuration files located on the target Veras and Whisper database (which may be on a remote NAS)
[/li]
[li]Two Carbon configuration files [tt]storage-schemas.conf[/tt] and [tt]storage-aggregation.conf[/tt], placed in the Whisper database directory, work exactly as described in the Graphite documentation, defining default archive retentions and aggregation parameters for common Vera serviceIds and variables.
[/li]
[li]Only those watched variables which do not have matching rules in the above files require manual configuration of Whisper archives.
[/li]
[li]Static system configuration, including UDP port allocation, machine IP addresses, and logging options are now defined in the Graphite/Carbon standard file [tt]carbon.conf[/tt] located in Vera’s [tt]/www[/tt] directory.
[/li]
[li]Only one critical parameter may need changing in the distributed configuration file, that is the location of the Whisper database on your system.
[/li]
[li]The default UDP port allocation is different from that used in the previous version of DataYours and now conforms to the defaults used in Graphite.
[/li]
[li]As a result of the above, the configuration page has been replaced by a read-only comprehensive diagnostic information page which will facilitate further debugging.
[/li]
[li]The capability to plot dataMine and Whisper database channels simultaneously has been removed, since this is probably of minor interest to most future users.
[/li][/ul]

There is a section in the documentation on migrating from DataYours (original version) to DataYours2.

If anyone is interest in trying it, I’d be pleased to get feedback. I’ve completely switched to this version so am fully committed to ensuring that it works.

Edit: Here are posts which update some of the contents of the original .zip files (so should be downloaded to replace originals)

Latest Lua files (except for whisper2): [url=http://forum.micasaverde.com/index.php/topic,25800.msg184489.html#msg184489]http://forum.micasaverde.com/index.php/topic,25800.msg184489.html#msg184489[/url]
Icons: [url=http://forum.micasaverde.com/index.php/topic,25800.msg184080.html#msg184080]http://forum.micasaverde.com/index.php/topic,25800.msg184080.html#msg184080[/url]

Edit: All very out of date now, use the App Store version of DataYours and DataYours-Graphite

#2

Nice work akbooer, as allways!

#3

Regrettably, an early update to DataWatcher2, fixing a bug which had stopped some variables from being set to [tt]watch[/tt] status.

#4

Works for me now (version 1 didn’t)
Excellent!

#5

[quote=“Cosworth32, post:4, topic:181793”]Works for me now (version 1 didn’t)
Excellent![/quote]

Excellent indeed! Thanks very much for persevering. Hope it does what you need, and ask if it doesn’t.

#6

At last had some time to try this out. I’ve read the PDF and installed the files accordingly. Edited the file /www/carbon.conf for the correct dir and reloaded but all I see is “No handler” on the webpage. I must be doing something wrong but at this moment I have no clue. ???
Suggestions are welcome 8)

#7

Have you run the require statements, or put them in Lua Startup?
Are you using the correct URL request (note the “2” at the end)?

#8

I’m having mixed results trying to mass create watch statements. After getting this response

watch=022.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=051.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=053.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=060.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=073.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=074.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=075.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=076.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=077.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=078.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=079.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=080.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=081.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=082.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=083.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=084.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=085.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=086.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=087.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=088.urn:micasaverde-com:serviceId:EnergyMetering1.Watts watch=089.urn:micasaverde-com:serviceId:EnergyMetering1.Watts

I don’t get an archive per line. It actually skips most of them. I have no issue watching one at a time but that is rather tedious to do.

#9

[quote=“akbooer, post:7, topic:181793”]Have you run the require statements, or put them in Lua Startup?
Are you using the correct URL request (note the “2” at the end)?[/quote]
The require lines are in Lua startup and Lua reloaded normal. I copied them from the PDF.
Also the URL I copied from the PDF and put my Vera’s IP number in it. Also tried different browsers.

#10

[quote=“Cosworth32, post:8, topic:181793”]I’m having mixed results trying to mass create watch statements. After getting this response
[…]
I don’t get an archive per line. It actually skips most of them. I have no issue watching one at a time but that is rather tedious to do.[/quote]

[ul][li]How, exactly, are you trying to do that? [/li]
[li]Is this with a URL request direct to DataWatcher2?[/li]
[li]What does the request [tt]:3480/data_request?id=lr_DataCache2&page=config[/tt] tell you? (Any errors in DAEMON section?)[/li][/ul]

#11

[quote=“C4Vette, post:9, topic:181793”]The require lines are in Lua startup and Lua reloaded normal. I copied them from the PDF.
Also the URL I copied from the PDF and put my Vera’s IP number in it. Also tried different browsers.[/quote]

Can you verify that the various daemons have started and are finding the [tt]cabon.conf[/tt] file by getting responses to the following URLs:

[ul][li][tt]:3480/data_request?id=lr_DataWatcher2&page=config[/tt][/li]
[li][tt]:3480/data_request?id=lr_DataCache2&page=config[/tt][/li]
[li][tt]:3480/data_request?id=lr_DataGraph2&page=config[/tt][/li]
[li][tt]:3480/data_request?id=lr_DataDash2&page=config[/tt][/li][/ul]

#12

[quote=“akbooer, post:10, topic:181793”][quote=“Cosworth32, post:8, topic:181793”]I’m having mixed results trying to mass create watch statements. After getting this response
[…]
I don’t get an archive per line. It actually skips most of them. I have no issue watching one at a time but that is rather tedious to do.[/quote]

[ul][li]How, exactly, are you trying to do that? [/li]
[li]Is this with a URL request direct to DataWatcher2?[/li]
[li]What does the request [tt]:3480/data_request?id=lr_DataCache2&page=config[/tt] tell you? (Any errors in DAEMON section?)[/li][/ul][/quote]

Another thought here: there may be nothing wrong at all. If, in fact, you have started the watch with a wildcard command to DataWatcher2 directly, then the files will not be created until the variables update.

#13

[quote=“akbooer, post:11, topic:181793”][quote=“C4Vette, post:9, topic:181793”]The require lines are in Lua startup and Lua reloaded normal. I copied them from the PDF.
Also the URL I copied from the PDF and put my Vera’s IP number in it. Also tried different browsers.[/quote]

Can you verify that the various daemons have started and are finding the [tt]cabon.conf[/tt] file by getting responses to the following URLs:

[ul][li][tt]:3480/data_request?id=lr_DataWatcher2&page=config[/tt][/li]
[li][tt]:3480/data_request?id=lr_DataCache2&page=config[/tt][/li]
[li][tt]:3480/data_request?id=lr_DataGraph2&page=config[/tt][/li]
[li][tt]:3480/data_request?id=lr_DataDash2&page=config[/tt][/li][/ul][/quote]
The first and second dump a kind of text config. The third and fourth give the “No Handler” message.
I uploaded the last two files again but that made no difference.

#14

[quote=“akbooer, post:10, topic:181793”][ul][li]How, exactly, are you trying to do that? [/li]
[li]Is this with a URL request direct to DataWatcher2?[/li]
[li]What does the request [tt]:3480/data_request?id=lr_DataCache2&page=config[/tt] tell you? (Any errors in DAEMON section?)[/li][/ul][/quote]

Request example: http://10.0.0.28:3480/data_request?id=lr_DataWatcher2&watch=*.urn:micasaverde-com:serviceId:EnergyMetering1.Watts

result from config request:

[code]CONFIGURATION at Thu Jul 3 21:52:31 2014

CACHE = {
stats = {
cpu = 1.567,
updates = 160},
version = “2014.06.26 DataCache2 @akbooer”,
whisper = “/VeraNAS/Vera/whisper/”}

DAEMON = {
client = “DataCache2”,
destinations = {},
errors = {count = 0},
ip = “10.0.0.28”,
start_time = “Thu Jul 3 21:45:50 2014”,
udp = {
last_received = {[“127.0.0.1:2003”] = “Vera-35109925.085.urn:micasaverde-com:serviceId:EnergyMetering1.Watts 57 1404417143”},
received = 160,
sent = 0},
version = “2014.06.30 DataDaemon2 @akbooer”}

[aggregator] = {
DESTINATIONS = “127.0.0.1:2004”,
LINE_RECEIVER_INTERFACE = “0.0.0.0”,
LINE_RECEIVER_PORT = 2023,
MAX_AGGREGATION_INTERVALS = 5,
MAX_DATAPOINTS_PER_MESSAGE = 500,
MAX_QUEUE_SIZE = 10000,
REPLICATION_FACTOR = 1,
USE_FLOW_CONTROL = true}

[cache] = {
CACHE_QUERY_INTERFACE = “0.0.0.0”,
CACHE_QUERY_PORT = 7002,
ENABLE_UDP_LISTENER = true,
LINE_RECEIVER_INTERFACE = “0.0.0.0”,
LINE_RECEIVER_PORT = 2003,
LOCAL_DATA_DIR = “/VeraNAS/Vera/whisper/”,
LOG_UPDATES = false,
MAX_CACHE_SIZE = “inf”,
MAX_CREATES_PER_MINUTE = 50,
MAX_UPDATES_PER_SECOND = 500,
UDP_RECEIVER_INTERFACE = “0.0.0.0”,
UDP_RECEIVER_PORT = 2003,
USER = “”,
USE_FLOW_CONTROL = true,
WHISPER_AUTOFLUSH = false}

[relay] = {
DESTINATIONS = “127.0.0.1:2003”,
LINE_RECEIVER_INTERFACE = “0.0.0.0”,
LINE_RECEIVER_PORT = 2013,
MAX_DATAPOINTS_PER_MESSAGE = 500,
MAX_QUEUE_SIZE = 10000,
RELAY_METHOD = “rules”,
REPLICATION_FACTOR = 1,
USE_FLOW_CONTROL = true}[/code]

Looks ok to me but you’re the expert.

#15

Yes, this all looks fine - no errors created by the cache in trying to create files. Did you see my earlier response saying that database files won’t be created until the next variable change? So you should see these files appearing subsequently.

A similar URL request to DataWatcher2, viz:

http://10.0.0.28:3480/data_request?id=lr_DataWatcher2&page=config

should confirm the currently watched variables.

#16

[quote=“C4Vette, post:13, topic:181793”]The first and second dump a kind of text config. The third and fourth give the “No Handler” message.
I uploaded the last two files again but that made no difference.[/quote]

…and you DO have in your Lua Startup (you said so previously, but worth checking):

require "L_DataGraph2"
require "L_DataDash2"

…if not, then that should fix it. If yes, then something is failing before these statements, but you’ve reported that there are no errors on startup, so possibly there is a mis-typing of these commands (although you also said you cut and pasted)? Is the “2” on the end? Is it an “_” rather than a “-” ?

Can you PM me the output of the two configuration pages you have successfully dumped, and possibly your Startup Lua? We must be missing something very basic. Your system isn’t low on memory, is it? I’m struggling to think of why the first two should load, but not the second two. Do you see any related errors in the Lua log file?

#17

Ah yes that was it. Files appeared some time later. “case closed” :slight_smile:

#18

[quote=“akbooer, post:16, topic:181793”][quote=“C4Vette, post:13, topic:181793”]The first and second dump a kind of text config. The third and fourth give the “No Handler” message.
I uploaded the last two files again but that made no difference.[/quote]

…and you DO have in your Lua Startup (you said so previously, but worth checking):

require "L_DataGraph2"
require "L_DataDash2"

…if not, then that should fix it. If yes, then something is failing before these statements, but you’ve reported that there are no errors on startup, so possibly there is a mis-typing of these commands (although you also said you cut and pasted)? Is the “2” on the end? Is it an “_” rather than a “-” ?

Can you PM me the output of the two configuration pages you have successfully dumped, and possibly your Startup Lua? We must be missing something very basic. Your system isn’t low on memory, is it? I’m struggling to think of why the first two should load, but not the second two. Do you see any related errors in the Lua log file?[/quote]

I’ve sent you a PM with the output and de startup LUA. The log shows just one line with the same error:

02      07/05/14 16:55:47.130   JobHandler_LuaUPnP::REQ_Handler no handler for lr_DataGraph2 <0x2e181680>

According to the ‘free’ command there is about 3.6MB free and ‘top’ shows Mem: 59044K used, 3436K free, 0K shrd, 13800K buff, 19316K cached

#19

@akboer

I have after installing the files , including update, and modify the config file the following error on the screen

string “module (“DataDash2”, package.seeall)…”]:633: attempt to call field ‘info’ (a nil value)

Can you give some guidance

Rgds

Huib

Verstuurd van mijn GT-P3110 met Tapatalk

#20

This looks like a problem with the Whisper library. You are sure you have uploaded all the Lua files? … specifically the whisper2.lua one?

It would also help if you can say what you did just prior to this message appearing, and also to send a copy of the configuration page.