Energy/Temperature Publication using SmartEnergyGroups.com (SEG)

Other Devices & Integrations Power Management
#1

I just moved it over to SEG and disabled the Vera side of things. Both my GEM and ECM1240 are running through a RPi with btmon, which in turn is shuffling the data over to SEG. Only took a few hours to do all of it, including a significant GEM SW upgrade.

If I want the data in Vera, I’ll write a SEG hookup to bring it back, or talk to RRD on the RPi. In the yrs that I’ve run this stuff, I’ve only ever done analysis/tuning, not automation/alerting, and SEG is night and day better for analytics.

It’s similar in concept to running a DashBox, just more command line steps to get I setup.

EDIT: Add hyperlinks
EDIT: Originally part of the “Adventures in reducing baseline power …” thread, but broken out for clarity.

#2

Excuse the ignorance… what is SEG?

#3

Oops, sorry about that. I’d linked it in the other thread but not here.

SEG is Smart Energy Groups. There are a number of energy portals, and this one is one of the more flexible ones.

You register for a free account, get a site key, and start pushing RESTful stream of data to them. This contains data for streams of information for one or more devices. information in units like kWh, W, Temp, Volts, Humidity, CO2, Tank Levels, Pulse Counts (etc) all in one or more named data feeds that are using your key.

On the receiving end, you just tell SEG to discover and it tells you what’s being poked at it, and enters the relevant device-stream names into it’s catalog (that can then be further augmented, via UI or API, with the units information)

At that point, it starts collecting the data, and automatically doing rollups for various periods. It’ll provide a set of per stream graphs by default, with “real-time”, day, month yr type graphs, but then you can totally customize the portal with a set of widgets on configurable dashboards that you create.

All of this is done via the UI, but there’s a backing API for doing it programmatically as well (as well as getting data both in and out of the service)

And “btmon.py” is a bridge program that takes feeds in from the Brultech devices, and puts them into one or more configurable targets, either locally (RRD, SQLLite, MySQL) or remotely (SEG, PlotWatt, etc)

#4

Very cool. Thanks for the detailed explanation.
Something else to investigate!

#5

In fact, excellent.

No need for any Vera power-related app, then. I’ll just throw all power data from DataYours to SEG and cross that one off the list.

#6

For those interested in pushing data to SEG for analysis, here’s a snippet of Lua that can be used in your Vera Startup.

It’s not a plugin, and it’s not resilient (if SEG goes down, the data isn’t accumulated at Vera) but it works reasonably well (I use it for supplementary data like Room & Outdoor temperatures)

The code will work for any Energy Data in Vera, not just from Brultech units, as well as for Temperature data.

[code]local http = require(‘socket.http’)
http.TIMEOUT = 5

local SEG_SITE = “”
local SEG_URL = “http://api.smartenergygroups.com/api_sites/stream”
local segLog = function (text) luup.log('SEG Logger: ’ … (text or “empty”)) end

SEG_DEVICES = {
{node=‘vera’, stream=‘t_upstairs’, deviceId=335, serviceId=‘urn:upnp-org:serviceId:TemperatureSensor1’, serviceVar=“CurrentTemperature”},
{node=‘vera’, stream=‘t_downstairs’, deviceId=384, serviceId=‘urn:upnp-org:serviceId:TemperatureSensor1’, serviceVar=“CurrentTemperature”},
{node=‘vera’, stream=‘t_outside’, deviceId=318, serviceId=‘urn:upnp-org:serviceId:TemperatureSensor1’, serviceVar=“CurrentTemperature”}
}

local function findStream(deviceId, serviceId, serviceVar)
– TODO: Change from a simple scan to something that’ll support more entries efficiently.
for k, v in pairs(SEG_DEVICES) do
if (v.deviceId == deviceId and
v.serviceId == serviceId and
v.serviceVar == serviceVar) then
return v.node, v.stream
end
end

return nil, nil
end

local function initWatch()
for k, v in pairs(SEG_DEVICES) do
luup.variable_watch(‘segWatch’, v.serviceId, v.serviceVar, v.deviceId)
end
end

function segWatch(deviceId, serviceId, serviceVar, oldValue, newValue)
local nodeName, streamName = findStream(deviceId, serviceId, serviceVar)

if (nodeName == nil or streamName == nil) then
segLog(string.format(“Node not found for %s,%s,%s”, deviceId, serviceId, serviceVar))
return
end

– TODO: Encoding
segData = string.format(“(site %s (node %s ? (%s %s)))”, SEG_SITE, nodeName, streamName, newValue)
segLog(segData)

result, status = http.request(SEG_URL, segData)
end

initWatch()

[/code]

You need only substitute in your SEG Site Id, and the “table” (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track, and the script will watch them for changes and push the values across to SEG.

In my case, I have a SEG Device registered with the label “[tt]vera[/tt]”, and beneath that I have SEG Streams for “[tt]t_upstairs[/tt]”, “[tt]t_downstairs[/tt]” and “[tt]t_outside[/tt]”. In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data ([tt]urn:upnp-org:serviceId:TemperatureSensor1[/tt], [tt]CurrentTemperature[/tt])

Since the data is only pushed when a value changes, you may have to wait a little while before you see the data in SEG (after loading the code into Vera’s Startup method, and restarting Vera)

NOTE: When naming your streams for SEG, use prefixes of

[ul][li]t_ - temperature data[/li]
[li]e_ - energy data (Wh, kWh, etc)[/li]
[li]p_ - power data (W, kW, etc)[/li][/ul]

When using SEG’s “Discoveries” feature, it automatically decodes the Stream names, and applies the correct typing information if the streams are named using these prefixes.

This tool is on http://smartenergygroups.com/tools
MyThings → tools → Discoveries

When using the Discovery feature, you may need to leave SEG on that page for some time, as I’m only sending data over as it changes.

#7
And here's what that looks like

Ooh, that’s Google Charts, but what produced it? SEG??

#8

[quote=“akbooer, post:7, topic:186392”]

And here’s what that looks like

Ooh, that’s Google Charts, but what produced it? SEG??[/quote]
Straight out of SEG, they have an option/menu they display on each Graph to Save it. Formats include PNG, JPEG, PDF and SVG.

Saves taking screenshots :wink:

#9

oh, and I enabled @watou’s option on the Nest T-Stats to get finer grained data. It was a bit chunky before I did that…

#10

SEG is great stuff, and run by a lad in Sydney. Always happy to promote enterprising companies from home :wink:

I find myself using it for a variety of measurement/graphing/comparison stuff now, esp given how simply it ingests data.

There is another way you could improve your fuel economy, perhaps without changing your driving style ;-) http://www.discovery.com/tv-shows/mythbusters/videos/dimpled-car-minimyth.htm
That's hilarious! I was laughing a ton as I watched, so thankyou!

Those guys are nuts. Every time I see them I’m reminded of this little “mishap” and wish there was video of it 8)
Map Charts Path of MythBusters' Wayward Cannonball | WIRED

On a related note, just noticed that electricity rates have climbed to 33-39c/kWh back home, which is nearly as bad as Hawaii! No wonder Solar PV installs went nuts down there…

#11

Guessed, I am trying to use this script, but I’m pretty weak in this area so I could use some clarification.

You need only substitute in your SEG Site Id

So first I create a device on SEG. That gives me a token number and a stream name which I then use as the site ID and teh stream name in the script? And is that site ID with the < > brackets, or without? I am assuming without.

and the "table" (SEG_DEVICES) of Device/ServiceId/StateVariable that you want to track
In my Vera, these are devices 335, 384 & 318 and they all support the standard Temperature data (urn:upnp-org:serviceId:TemperatureSensor1, CurrentTemperature)

So in the advanced settings tab for my energy monitor I have the deviceId under id, the ServiceId info is under device_type, and the State Variable would be Watts. I suppose I would set another stream up for KWH.

But my serviceId looks like this:

urn:schemas-micasaverde-com:device:PowerMeter:1

I have a script pushing this data to Plotwatt with this serviceId:

urn:micasaverde-com:serviceId:EnergyMetering1

I tried that too but nothing working so far. I’m not getting an error from the script, just nothing on SEG.

#12

For SEG, the Site Id looks like the attached image. Once logged in, it’s under:

My Things → Sites → → [Edit]

The string doesn’t have <> characters around it, use it as displayed when you select the pane above.

This is the serviceId:
[tt] urn:micasaverde-com:serviceId:EnergyMetering1[/tt]

The other string you had was the device type, which can’t be used. If you want to validate, just “hover” the mouse over the Service Variable name/label in the Device’s Advanced tab. It’ll tooltip the ServiceID for the Service Variable.

Once you set the Vera-end of this “running” with the SEG Site ID, you can goto SEG:

My Things → Tools → Discoveries

and then press b[/b]. Give it a few minutes and it’ll create ALL of the relevant Devices for you, which you can then just go and fill in the blanks (like UoM, range, etc)

#13

The proper site ID did it. Thanks!

#14

Yup, same here. site-id is actually the token key that’s generated. The info here made it really easy to set up and get data into SEG. Now to figure out how to forecast my monthly bills.

Thanks for the 'fo!

#15

Hi Guessed and others

I am adding a veralite to my ISY HA setup, in part because of my frustration in trying to get energy data from ISY to SEG or any other good energy monitoring site. In my case I want track circuits AND also individual smart plugs on individual devices.

[ol][li] Overall are you happy with what you describe above to report to SEG for both GEM and other Vera energy reporting, e.g. smart plugs? [/li]
[li] Is SEG “smart enough” to allow the user to avoid double counting the smart plug loads which are also included in the whole-house-meter loads?[/li]
[li]I am not a scripter (but can follow along and fill in basic blanks, etc.) Do you expect this is an appropriate solution or will I be jumping off a clifF?[/li][/ol]

Thanks for posting this info and your work to make it straightforward for others!!

#16

Yup, completely. I look at the data in SEG daily after I make significant changes in the house.

I report data to SEG from 2 different sources:

[ul][li]Weather, House-Temperature & Vera Performance data from my Vera
These streams use the technique above to push that to my SEG Account.[/li]
[li]Brultech GEM and ECM-1240 Energy Data
These streams are published using btmon.py running on a RaspberryPi.[/li][/ul]

I moved from using Vera as the conduit for the Energy streams in order to reduce the load on Vera (I have a lot of stuff on Vera that needed to be more real-time than the energy monitoring stuff)

SEG has no knowledge of the relationship between the [data] Streams you register with it. It sees Devices (think “Main Panel”, “SubPanel”, “Vera”) and the Streams that come from those Devices (think “Upstairs Temperature”, “Grid Energy”, “Solar Energy”, “Office Energy”, “CPU Usage”)

Once you’ve connected the Devices and established the Streams, you use the SEG UI to build Dashboards to display the UI for those Streams (you pick the UI/Graph styles, as well as what Streams to display in those graphs). It’s a fairly advanced graphical Portal builder for “energy” data.

You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system. I do this to “rollup” certain low level stats into more useful stuff.

eg. TotalEnvironmentalEnergy = MainPanel/AC Energy + SubPanel/Furnace Energy

So this is where, indirectly, relationships are being captured… by how you build Graphs, and how you build these Computed Streams. All of this is done in a UI, and is easy to do.

The hardest part of my setup is the bymon.py running on Pi. Since you need to setup the Pi, btmon.py, deal with Serial Ports (for attachment to ECM-1240) and Wifi (for dealing with my GEM) and script it all to start on reboot (etc).

Not impossible, but it would take little searching to pull the parts together.

That said, if you’ve got a Vera, and it’s not doing too much else, then you could just run the Brultech Plugin and it’ll do the work of acquiring the data for you (esp if you’ve got the GEM on WiFi and/or Ethernet). You’d then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG. That’s all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id’s)

#17

I find SEG really confusing. It looks very flexible and powerful and I’m sure I could get it dialed in given enough time with it, but I wouldn’t call it an intuitive UI.

#18

[quote=“guessed, post:16, topic:186392”]…You can then also use the UI to build new, computed, streams from whatever source-streams you have in the system. I do this to “rollup” certain low level stats into more useful stuff.

…That said, if you’ve got a Vera, and it’s not doing too much else, then you could just run the Brultech Plugin and it’ll do the work of acquiring the data for you (esp if you’ve got the GEM on WiFi and/or Ethernet). You’d then just need to tweak the Luup Startup script (I list this above) for the channels that you want to push over to SEG. That’s all very doable, and quite simple script editing (to enter the relevant Device#'s & Service Id’s)[/quote]

Guessed, thanks! Those were the answers I was hoping to receive…

As to calculated streams on SEG: Would you please clarify, can one do subtractions or only additions, e.g can I create a rollup which takes the whole-house meter stream and then subtracts other particular device streams, that might otherwise be double counted?

Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to “AV Stream”, (iii) the AV Stream combined data then merges with other data streams and rolls up into “AV and XYZ Stream”. In thinking about this, I suppose one could just to the upper tier rollup manually from each device, vs as a two-tier roleup.

As to Vera vs PI and complexity: I have an ISY with Zwave module and find it to be VERY reliable/stable. However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN). Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data. Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc. I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-). In that context I am hoping these Vera bells-and-whistles won’t cripple it. (PS: exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Finally, you mention GEM wifi or Ethernet. Are these options in the GEM one buys, or ad-ons/hacks to do after the fact. I have not purchased yet. I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

Thanks again!

#19

Both Add and Subtract, of any Stream with the same Units of measure. In my case, I have some streams that are “rolled up” at the source, based upon where I’ve clamped… sometimes I re-roll them up from the component streams, in SEG, since there might be errors in measurement that I’m interested in seeing.

I’ll post a series of screen-captures shortly to illustrate these bits…

Going one step further, on SEG can one do rollups into rollups, e.g. (i) all AV equipment in the house comes in via smart plug streams (and possibly from dedicated circuits via GEM) (ii) these role up to "AV Stream", (iii) the AV Stream combined data then merges with other data streams and rolls up into "AV and XYZ Stream".

Yep, Calculations are just another Stream, so you can composite new Calculations based upon others. Sometimes this limits the granularity of the result (to 30 minutes) but in general that’s not an issue.

As to Vera vs PI and complexity: I have an ISY with Zwave module and find it to be VERY reliable/stable. However, it does not receive data IN from other sources as well as Very (it communicates OUT very well, but not good at data IN). Thus, I may continue to use ISY as my primary Insteon and Zwave controller and use the ISY plugin for Vera to populate Vera with the ISY data. Then I would use Vera primarily for things like energy reporting, GUI on iOS, Nest Tstats and a few other things it seems to do well via plugins, etc. I am cautious about using Vera for mission-critical items, like turning off my bathroom lights ;-). In that context I am hoping these Vera bells-and-whistles won't cripple it. (PS: exchanging my VeraLite for the new VeraEdge (and preparing to temporarily suffer through firmware v7, which it comes loaded with).

Take a look at the stuff that Teken is doing on the Brultech forums. He has ISY gear. My Vera is a Vera3, so it has more memory than a VeraLite, but the same as the newer VeraEdge.

When running a reasonable load, Vera is good at Lighting/Z-Wave control (etc)… but when it runs out of steam, you’re hosed. They don’t currently have a “bigger” (and, in all honesty, less-buggy) option for power-users so I paired back my Vera3 to running the critical stuff, and have moved everything else to Pi’s (and now, via openHAB, I’m expanding/migrating to my Mac)

Finally, you mention GEM wifi or Ethernet. Are these options in the GEM one buys, or ad-ons/hacks to do after the fact. I have not purchased yet.

You buy them as part of the GEM when you order it. Mine is a 1st Gen GEM, which I bought with ZigBee, since I already had ECM-1240 units with the ZigBee module (and the related ZB <-> Ethernet Bridge). At the time the ZB Modules were socketed, so I replaced mine with a RovingNetworks RN-XV module (now part of MicroChip).

In the latest models, these are simply options that you buy with the GEM. I like WiFi here, since I don’t have cabling for Ethernet anywhere near my GEM (and I don’t want to either, since my GEM is right next to my SubPanel)

In my setup, I have a WiFi rPi which is USB-Serial attached to a ECM-1240, reading the Main-Panel. This is on one side of the garage and it’s using WiFi to get access to the GEM, which is next to the Sub-Panel on the other side of the Garage. The rPi aggregates the data and phones it up to SEG (as well as PVOutput, for the Solar bits)

I understand there is another sister device which helps GEM with reporting and data storage, etc. I forget the name, maybe you know what I mean.

You mean the DashBox. It still appears to be somewhat under development. It’s performing the role of what I’m using a Pi + SEG for, albeit using “locally-stored” data, which will be of interest to some folks. I may do this sometime, to replace the Pi and make my solution more consumer-friendly… handy if you ever want to sell you house with this stuff as “features” :wink:

#20

A few more SEG images, showing the default Stream-list, a default graph for one of the Streams, as well as custom dashboard tabs I’ve built for Energy Overview/Detail and Temperature monitoring.