hi guys, is there anyway to export the raw data collected?
Yes, there is. What sort of API are you looking to use?
[ul][li]Lua code[/li]
[li]HTTP request[/li][/ul]
Can you say a bit more about your requirements?
UI7 remote access?
Using lastest UI 7 firmware, I am unable to access remotely via the link with in the device control. I also found a url via search which I put my UID,Pass and UnitID and not luck. Do I need to change firewall? Port Forwarding? (I did forward 3480) I read the whole instructions and did not notice anything.
These are the url results:
[ul][li]Via device control page (remote): ERROR:Module not found:port_3480[/li]
[li]Via url found in search: ERROR:Module not found for user[/li][/ul]
Thank you for any advice.
Paul
[quote=“akbooer, post:2, topic:190679”]Yes, there is. What sort of API are you looking to use?
[ul][li]Lua code[/li]
[li]HTTP request[/li][/ul]
Can you say a bit more about your requirements?[/quote]
Hi, I have been trying to find ways to access the log files remotely since I have multiple locations and would like to keep all the data in one place. I currently use WinSCP and port forwarding to access the files but the issue with this is that it relies on a static IP. Could you expand on the two API methods because ideally I would like to send the files (Push) from my VeraEdge rather than logging into it to access them (Pull).
When you say ‘logs’, you mean the dataMine database, yes?
Actually, DataYours and its separate Whisper round-robin database, was built as a distributed multi-processor system to support just this need, but that’s the subject of other threads.
Could you expand on the two API methods because ideally I would like to send the files (Push) from my VeraEdge rather than logging into it to access them (Pull).
A Lua API to the dataMine database was first proposed here: Taking the DataMine logs further, and that thread is still worth a read (although out of date.)
This post: dmDB - a read-only wrapper for the dataMine database provides Lua code and documents how to use it. Again, a bit out of data, but functional.
This particular post has information on an HTTP server which acts as a wrapper to the above Lua API http://forum.micasaverde.com/index.php/topic,18749.msg142518.html#msg142518
The most recent incarnation is the DataYours distribution which includes all of the above functionality, providing:
[ul][li]Lua API to dataMine (readonly)[/li]
[li]HTTP server to retrives data[/li]
[li]HTTP server to render dataMine stored graphs and plots as SVG[/li][/ul]
and you don’t need to install the entire DataYours system just to use the dataMine access bits.
There’s a lot out there. It just depends on exactly what you want to do. Or you could dig in yourself and discover the maze of twisty little passages which is the dataMine file system. Underneath that, though, all the data is in CSV files and all the metadata in JSON.
I’ve created a separate thread for discussions on dataMine2 APIs and remote access as the main forum thread is getting a bit difficult to follow.
@akbooer - would you mind moving the recent posts on APIs and remote access from the DataMine 2.0 topic to this one? Thanks
Edit by moderator: Done!
@smartloft - it is possible to access the dataMine2 data via an API that is mentioned in the documentation which you can download from here: http://forum.micasaverde.com/index.php/topic,35724.0.html. An example of the API call is as follows:
response:
{"series":[{"label":"Heating - Status","Id":36,"data":[[1451569377000,2.1888888888889],[1451606400000,4.4472222222222],
[1451692800000,2.915],[1451779200000,4.0527777777778],
[1451865600000,3.2597222222222],[1451952000000,4.4275],
[1452038400000,5.25],[1452124800000,0.99722222222222]],
"pointsRet":8,"pointsTot":180,"min":0.99722222222222,"max":5.25}],"procTime":0.029999999999973,"min":1451569377,"max":1452174177}
What do you want to do with the data, once you have it? Back it up? Draw a graph in Excel? Merge it with other data?
@paulcusick3 - the remote access feature of dataMine2 was originally written to work on UI5 as that is the only system I own and have available to test on. I was waiting for someone to let me know what happens on UI7 as I wasn’t sure if it would work or not, so thanks for being the guinea pig and testing it out! Unfortunately and perhaps not surprisingly, it doesn’t appear to work :(.
Call for help
It would be great to get remote access working on UI7 but I am going to need some help, so if anyone reading this can provide me with the LUA code required to get a web app working remotely on UI7, I’d be pleased to take a look. Is there some other plug-in out there that already does this? Many thanks!
The developer of AltUI has kindly pointed me in the direction of the code for remote access on UI7 here: Remote – Alternate UI. However, it needs to be run on an externally hosted server which is not something I want to get into for dataMine2 right now.
Therefore, I’m afraid, dataMine2 remote access will have to remain UI5 only for the foreseeable future.
[quote=“ConstantSphere, post:9, topic:190679”]The developer of AltUI has kindly pointed me in the direction of the code for remote access on UI7 here: Remote – Alternate UI. However, it needs to be run on an externally hosted server which is not something I want to get into for dataMine2 right now.
Therefore, I’m afraid, dataMine2 remote access will have to remain UI5 only for the foreseeable future.[/quote]
Hi, see my post in the documentation thread on doing this. You do need ALTUI installed though.
Cheers Rene
hi ConstantSphere,
thanks for replying, will try the APIs
we are looking to collate these data (energy consumption) in csv format for a research project
likely will be using them to plot graphs in Excel and merge them with other data (e.g. temperature + lux)
[quote=“ConstantSphere, post:7, topic:190679”]@smartloft - it is possible to access the dataMine2 data via an API that is mentioned in the documentation which you can download from here: http://forum.micasaverde.com/index.php/topic,35724.0.html. An example of the API call is as follows:
response:
{"series":[{"label":"Heating - Status","Id":36,"data":[[1451569377000,2.1888888888889],[1451606400000,4.4472222222222],
[1451692800000,2.915],[1451779200000,4.0527777777778],
[1451865600000,3.2597222222222],[1451952000000,4.4275],
[1452038400000,5.25],[1452124800000,0.99722222222222]],
"pointsRet":8,"pointsTot":180,"min":0.99722222222222,"max":5.25}],"procTime":0.029999999999973,"min":1451569377,"max":1452174177}
What do you want to do with the data, once you have it? Back it up? Draw a graph in Excel? Merge it with other data?[/quote]
Hi akbooer,
Thank you for your advice it is really appreciated.
Furthering on from what you said please could you point me in the direction of some instructions/resources for setting up a whisper database. When I looked at the Graphite website the configuration was very convoluted and required multiple third party softwares which, I read in another thread, are not required.
Well, the (almost) latest release is here, including full documentation http://forum.micasaverde.com/index.php/topic,30808.msg263795.html#msg263795.
But see also here http://forum.micasaverde.com/index.php/topic,35966.0.html if you’re a user of AltUI, since it makes it SO easy to use. If you’re not an AltUI user, then you are missing a seriously good trick.
Let me know how I can help more.
hi akbooer,
leaning towards the HTTP option to export raw data to csv format
is that possible with your dmDB wrapper?
[quote=“akbooer, post:2, topic:190679”]Yes, there is. What sort of API are you looking to use?
[ul][li]Lua code[/li]
[li]HTTP request[/li][/ul]
Can you say a bit more about your requirements?[/quote]
Reading @ConstantSphere’s documentation on DataMine 2.0 leads me to believe that this functionality is now incorporated into the plugin itself. Whilst you could certainly still install dmDB and dmDBserver, or DataYours, to give you an HTTP interface, I think there already is one. Take a look.
Hi ConstantSphere,
after going through your documentation again, i have a few questions
example, i have a popp meter plug (device #14) measuring energy consumption
to request for raw data output, i will input as such
- unix timestamp for start and end - how is this specified?
- how does one specify a channel number? i do not see any channel number on the web interface for the device under configuration tab
- output format in *csv?
- is it possible with 40-50 popp plugs monitoring energy simultaneously?
much thanks!
[quote=“ConstantSphere, post:7, topic:190679”]@smartloft - it is possible to access the dataMine2 data via an API that is mentioned in the documentation which you can download from here: http://forum.micasaverde.com/index.php/topic,35724.0.html. An example of the API call is as follows:
response:
{"series":[{"label":"Heating - Status","Id":36,"data":[[1451569377000,2.1888888888889],[1451606400000,4.4472222222222],
[1451692800000,2.915],[1451779200000,4.0527777777778],
[1451865600000,3.2597222222222],[1451952000000,4.4275],
[1452038400000,5.25],[1452124800000,0.99722222222222]],
"pointsRet":8,"pointsTot":180,"min":0.99722222222222,"max":5.25}],"procTime":0.029999999999973,"min":1451569377,"max":1452174177}
What do you want to do with the data, once you have it? Back it up? Draw a graph in Excel? Merge it with other data?[/quote]
I have some code, shown below, which I eventually want to use to send the datamine weekly log files from my Vera to my PC(it is only sending a string at the moment). however I do not want to try it remotely because it is not secure. Has anyone got any suggestions on how to make this secure?
Vera:
local host = “192.168.2.13”
local port = 51423
local socket = require(“socket”)
local tcp = assert(socket.tcp())
tcp:connect(host, port)
tcp:send(“Hello world!\n”)
Server:
local socket = require ‘socket’
local server = socket.tcp()
server:bind(‘*’,22)
server:listen(32)
local client = server:accept()
client:settimeout(10)
local line = client:receive(“*l”)
print(line)
1) unix timestamp for start and end - how is this specified? 2) how does one specify a channel number? i do not see any channel number on the web interface for the device under configuration tab 3) output format in *csv? 4) is it possible with 40-50 popp plugs monitoring energy simultaneously?
- the unix timestamp is simply the number of seconds since midnight on 1 January 1970 (known as the epoch) - for examples see http://www.epochconverter.com/
- The channel numbers can be found by using the API call http://192.168.0.252/port_3480/data_request?id=lr_dmList - it’s called the Id in the response.
- the output format is JSON - if you want the raw CSV files then the API approach is not the right one (see below)
- possibly but I suspect you need to try something different.
If you just want the raw csv files for back-up purposes then there is no proper way to do this from within dataMine. However, it should be possible to execute a script to call the MCV forwarding servers the following secure URL (this will only work in UI5) and count up through the channel numbers 1 by 1 loading the current weeks data - the name of the text file is simply the number of weeks since the epoch.
https://active_server/username/password/serialNumber/data_request?id=file¶meters=…/…/dataMine/database/1/raw/2404.txt
where active_server and serialNumber can be found from this request (which must be done locally and the results noted)
https://sta1.mios.com/locator_json.php
This will return the raw csv file which you can save locally. Note that the request must be https (not http) and as such your username and password will be encrypted during transmission. As I understand it dataYours solves this issue rather more elegantly!
@smartloft - If all you want to do is get data into Excel then you need my Excel dataMine importer! It’s simply a VBA script embedded in an Excel file that requests data and decodes the JSON for you. Take a look and let me know if you need any help getting it to work. You will need to ensure you have macros enabled. It currently won’t work remotely, but it could be modified to do so (but only on UI5).
@ConstantSphere,
great work on the importer!
however, i still have a problem
an example is when i specified a period from 01 Jan 16 to 30 Jan 16, aggregate period hour, aggregate type sample, i do not seems to get the right values (the values are all the same)
morever, the full period wasn’t displayed, up to 02 Jan 16 only
my aim is to collect the energy used for the day in either 10/ 20/ 30/ 60 minute blocks 24/7, for a couple of months
reference attached graph, it has all the data points but how do i get these numbers?
much thanks!