Not sure if this is the right part of the forum to ask. I want to extract data from a website, ultimately I am trying to extract a figure (in the red rectangle) to be used to trigger a scene in PLEG. I know how to do that, once I get a value into a variable. But, how can I get that website to feed its value into a variable ? Or is that possible at all ?
If I cannot do it in Vera, I could try to extract the variable from my Android tablet, which could then use tasker to trigger the scene, but ideally I would have it on Vera (and I have no idea how to do it on the Android tablet either).
I guess there might be a http get, but I do not understand it well enough to know how to use it.
[quote=“mikee123, post:1, topic:186888”]Not sure if this is the right part of the forum to ask. I want to extract data from a website, ultimately I am trying to extract a figure (in the red rectangle) to be used to trigger a scene in PLEG. I know how to do that, once I get a value into a variable. But, how can I get that website to feed its value into a variable ? Or is that possible at all ?
If I cannot do it in Vera, I could try to extract the variable from my Android tablet, which could then use tasker to trigger the scene, but ideally I would have it on Vera (and I have no idea how to do it on the Android tablet either).
I guess there might be a http get, but I do not understand it well enough to know how to use it.[/quote]
This information needs to be pulled via HTTP from this site. How that can be done usually comes from and “API” from that system (if one is even available). The API will tell you want commands can and are used to send and receive requests. Alot of API’s are not made public due to security concerns. You need to research more on that solar company and what it shares. It’s also gonna be hard for most people to help as if they don’t have access to that website they are SOL.
Ok so I’ll email them to see if they have a api. I thought (probably being naive and not really understanding it) it might be possible to give something like coordinates on the website and to send that to a variable. You can do something like that on Bloomberg or Reuters, so I thought it might be possible on any website. That does not seem to be the case. So if they do not have a api i’ll probably have to give up on the idea.
From what I see from uncle Google they’ve got API available:
[quote=“ddluk, post:4, topic:186888”]From what I see from uncle Google they’ve got API available:
https://theowl.zendesk.com/hc/en-gb/articles/201284603-Multicast-UDP-API-Information[/quote]
Not sure where you got this, but this appears to be for heating/air condition controls maybe of the same brand/manufacture but I didn’t see anything in that API that would be of use for SOLAR.
From what I see from github there are using exactly this way to grab data from home energy monitor devices developed by this company. I don’t know how this device exactly looks like because I never use them but if this API is not available there is a mobile application available so it’s possible that they’ve got different API for developers and mikee123 just need to ask them for access 
I do not really understand this, but from what I can see it might be possible to do what I want. This website just takes readings from a clamp at the PV panel, and works out what power its generating (I have that data on Vera as I can see it on a device my Rfxtrx has created). But I think what they are saying is that this data can be pushed anywhere (somehow). It is sent to a website and I think it might be possible to grab it somehow. Rfxtrx can see and show the current power generated on a device in my UI5, what is missing is the total of the day which has been generated. So I think the answer is in that Network OWL API.pdf… but getting it out is going to be far beyond my abilities…
But good find, that might be getting me another little step closer to what I am trying to achieve
Looking at the page @ddluk has posted, it seems there is a way of extracting the data I need
The Network OWL sends UDP packets to the following multicast group: Address:
224.192.32.19 Port: 22600
MULTICAST PACKETS
EXAMPLE
PACKET
0.000.00
0.000.00
PARAMETERS
The main tag contains sections for the current solar readings () and the totals for the day (). Both sections have and tags that contain the values for power generated by the PV system and power exported to the electricity grid.
So it looks to me as if the data I want can somehow be accessed
Mike,
You will have to write a little app that run’s somewhere on your LAN (does not have to be Vera) that
listens for the UDP message receives it, extracts the data, and then sends the HTTP request to Vera to set some device variable.
Then this code needs to be launched … to make sure it’s running all the time. It might also need to be throttled … I am not sure how often the UDP packets are broadcast.
This topic should likely be moved under the programming sub. What you are trying to do is extract a “field”(it may not be adequately marked) from a webpage and stuff it into a variable.
The Multicast API information is for receiving data directly from the Owl device, not the website/page. In order to receive the information from the device, you would need to setup a receiver for the multicast packets and then a handler for the data. This is a good candidate for a Raspberry Pi with some Python, but that would be massive overkill to extract a single variable to then pass it on to Vera. It might be possible to setup such a receiver on Vera itself, but it would require a rather advanced plugin to do it and I doubt that this is the way you want to go.
It is likely possible for Lua code to scrape the webpage you posted and extract the desired information. However, it would not be trivial. By this I mean that it is unlikely that anyone here is going to be willing to spend the time writing it for you, especially in the most painful, five forum posts per line of code way. Here is a link to a Lua webpage scraper if you want to see what’s involved and have a sample to start from.
It would be far easier if Owl provided a JSON file with the data on their site, but you’d have to ask if they do that. It’s a possibility. Even with a JSON file, you’ll still be looking at a somewhat complex bit of Lua.
hmmm that would mean having to run a pc all the time, I don’t really want to do that. Could that be an app running on a Android tablet ? I have never written an app, so not sure if its even worth me trying but if it would be possible I would look into it. I have 2 Android tablets wall mounted which are connected to my LAN, and they are running tasker anyway to interface with Vera, Vera Alerts, Autovera and Authomation. I wonder if there is maybe a plugin for tasker I could utilise…
Yes that sounds like another idea (the web scraper) I will look into. - had a look, probably a non starter for me as I do not really understand it
enough…
you would need to setup a receiverNot sure if that's what you mean, but I have a receiver, my Rfxtrx, that already receives the current output and displays it in a device ?
How can I (or a moderator maybe ?) move this thread to programming ? I had a look I don’t think I can move it
[quote=“mikee123, post:12, topic:186888”]Not sure if that’s what you mean, but I have a receiver, my Rfxtrx, that already receives the current output and displays it in a device ?[/quote]I was referring to a process that listens to the network for multicast packets.
But, If you already receive the information in a Vera device, why are you bothering with scraping a webpage? You should be able to get the variable that you want from the Vera device quote easily. It may already be available to PLEG as a property.
How can I (or a moderator maybe ?) move this thread to programming ? I had a look I don't think I can move itA moderator would have to move it. Don't worry about it, if it doesn't happen.
I have the current output of my PV panels in the device, but not any other information. So what I need is missing. But, I could well be that it is received by the Rfxtrx but just not displayed ? I am quite sure that is the case, as its wireless transmitted to a receiver, which in turn sends it to the website. And the Rfxtrx is receiving that information. Maybe there is a way to get that information. It does not necessarily need to be displayed, all I need is the value of the total produced for the day
I think I might post a question about the Rfxtrx in that part of the forum. That might be another avenue worth pursuing.
It does not necessarily need to be displayed, all I need is the value of the total produced for the day
If that’s all you need then you should be able to calculate that inside PLEG. You may need to add a device variable (or two) to the PV where you can save the previous running totals. You can reset your daily total(s) using a PLEG Schedule at, say, 00:01:00.
Thats what im thinking and sounds like what he said that page was doing. He just needs to tally up the totals with the fata hes already getting.
Ok theoretically that sounds easy enough. I am quite comfortable with PLEG, but not sure how I would go about it. That value can change every second (the watts produced figure). I can get that into PLEG as a device property I guess. Setting up a schedule is easy enough. I only need to run it from 6am to lets say 11pm every day. But now the difficult part. I will probably have to take a reading every second. Thats still easy with a schedule I think. But how do I add them up to end with kWh produced ?
Does the device have any other variables apart from instantaneous power? Often they have a running total as well.
If you only have the current power, you could run it every second but is that really necessary? There would be much less load on Vera if you updated it less frequently - maybe every ten seconds? Just how accurate do you need this value?
However often you do it, the calculation is quite simple. KWh = KWh + ((Watts /1000) * period (seconds) / 3600).
It would be possible to do this using a Conditional expression but I would suggest using a device variable and performing the calculation in Lua attached to a PLEG Action.
Edit: 14/04/2015 21:50 Corrected for KWh instead of Wh.
I had a look under advanced, there is a history where I can see the last values, but no total unfortunately. I do not need it that accurate. Its only when its above 10kWh I want to trigger a scene to stop my hot water being heated by gas as its ben heated enough by the solar panels.
Ok next point performing it in Lua. From what I read its the most efficient way. So I would run a action every 10 seconds which calculates the formula and gives me a running total of kWh. How do I do that in Lua ? And then the next morning I have to reset kWh to 0. I could do that with another Lua action ?
To avoid a lot of polling, it may be worth exploiting the fact that PLEG will be notified whenever the power changes. This is untested but similar to something I use.
Schedules
sMidNight On: Weekly, all days, 00:01:00 Off: Interval, 00:00:05
Device Properties
pPower Current power variable in your device
Conditions
chgPower lastPower; pPower
KWh sMidNight ? 0 : (chgPower ? (KWh + ((lastPower/1000) * (#chgPower - #lastPower) / 3600)) : KWh )
lastPower pPower
BTW: When you first create this in PLEG you will get a crazy number for KWh as the timestamp for lastPower will be zero. It should become a real value after the first reset happens. You could preempt that using the Do it now button for sMidNight.
Edit: 14/04/2015 21:51 Corrected for KWh instead of Wh.