EventWatcher - simple events, logs, and device status snapshots

Plugins & Plugin Development EventWatcher Events & Logs
#41

I’ve written a ‘user guide’ for EventWatcher, which covers installation issues as well.

I’d be grateful if someone can cast an eye over this to see if it makes sense before I revise the initial post on this thread with both the manual and the latest software.

All comments welcomed.

Thanks in advance.

#42

Hi @akbooer

I’ve scanned the PDF and it looks very good - I need to have time to play with the plugin more - but one thing that stands out for me and might be better placed upfront is the logging option.

From what I can tell do you have 3 different storage options?

  1. written to the internal memory buffer (where they are routinely wiped) or
  2. written to the internal memory buffer (where they are backed up to an external share) or
  3. written only to an external share (where they can be retained long term)

Also as a side question - is it possible to write a customised entry to the a EventWatcher log? For example if I want to store an error message which would normally just be written to/seen in the log - I could write it to thevEventWatcher file ?

#43

Done.

From what I can tell do you have 3 different storage options?
  1. written to the internal memory buffer (where they are routinely wiped) or
  2. written to the internal memory buffer (where they are backed up to an external share) or
  3. written only to an external share (where they can be retained long term)

In fact, only (1) and (2). You can make the internal buffer larger or smaller (arbitrarily with a minimum of 200 and maximum of 2000), but not switch it off altogether.

is it possible to write a customised entry to the a EventWatcher log? For example if I want to store an error message which would normally just be written to/seen in the log - I could write it to thevEventWatcher file ?

Yes, in the latest version, you can. An HTTP request of the form:

http://[your-Vera-IP]:3480/data_request?id=lr_EventWatcher&event=YourMessageHere

will get written to the event log with a tag of ‘User’ and the message. Note that the message has to be URL encoded (no spaces, for example). You can do this from any machine which has access to Vera, including through the MiOS secure servers. If you’re doing this from code on the Vera you log the event to, then you can use the ‘local’ URL 127.0.0.1. Here’s a snippet of Lua code which will deal with the URL encoding for you:

local function myEvent (text)
  local url = require "socket.url"
  luup.inet.wget ("http://127.0.0.1:3480/data_request?id=lr_EventWatcher&event=" .. url.escape(text))
end

So if you invoke it like this:

myEvent "hello world"

…then you get an entry in the event log like this:

2014-01-09 10:41:23.797 E   0 , User = hello world

Hope you enjoy using EventWatcher further.

#44

Thanks for providing the documentation @akbooer. Good job - it motivated me to install your plugin.

One slight problem: the latest L_EventWatcher.lua (2014.01.06) requires L_EventWatcher2.lua. Where do I get that from?

#45

Oops… that’s why I need to update the first post here with the latest.

Both attached (includes the user event logging functionality)

Would very much appreciate constructive criticism from such an expert as you.

#46

Thank you, sir, that’s fixed it.

Would very much appreciate constructive criticism from such an expert as you.
I don't deserve the title [i]expert[/i] but I shall happily comment on how this works for me.

First off, this is an impressive piece of work. You have achieved a lot of capability in a relatively small amount of code.

Response-time for reports is very good - at least while my history is small. :slight_smile:

I really like the Memory report. Given all the issues people have with memory, it is worth installing EventWatcher for this alone - assuming they have enough memory to do so… ;D

I notice that you changed the handler name from lr_eventWatcher to lr_EventWatcher in the latest version. This corresponds to the documentation. I’m only flagging it here to warn users who may have saved bookmarks on the previous version so will get No Handler after upgrading.

#47

You are far too modest.

I really like the Memory report. Given all the issues people have with memory, it is worth installing EventWatcher for this alone - assuming they have enough memory to do so.... ;D

It would be really straightforward to make the memory report server an entirely stand-alone piece of code, not a device but just something initialised in startup code, which would take minimal memory resources itself.

I notice that you changed the handler name from [b]lr_eventWatcher[/b] to [b]lr_EventWatcher [/b]in the latest version. This corresponds to the documentation. I'm only flagging it here to warn users who may have saved bookmarks on the previous version so will get [i]No Handler[/i] after upgrading.

Yes, fair point, and thanks for the ‘heads up’ to others (all three or four of them, I think!)

Thanks also for the other comments - I blush with pride.

#48

Seems that relatively trivial changes (those are the changes the OTHER guy has to make :slight_smile: ) would be required to have this log to an external SYSLOG server running on Windows, Linux, et al.

I have a SYSLOG running to capture other events on tmy LAN such as router events, server events, etc. I know I’d really like to have something like that than digging through VERA all the time :slight_smile:

#49

[quote=“clippermiami, post:48, topic:177299”]Seems that relatively trivial changes (those are the changes the OTHER guy has to make :slight_smile: ) would be required to have this log to an external SYSLOG server running on Windows, Linux, et al.

I have a SYSLOG running to capture other events on tmy LAN such as router events, server events, etc. I know I’d really like to have something like that than digging through VERA all the time :)[/quote]

Is there a standard format/API for SYSLOG servers? (I know next to nothing about Windows or Linux)

#50

[quote=“akbooer, post:49, topic:177299”][quote=“clippermiami, post:48, topic:177299”]Seems that relatively trivial changes (those are the changes the OTHER guy has to make :slight_smile: ) would be required to have this log to an external SYSLOG server running on Windows, Linux, et al.

I have a SYSLOG running to capture other events on tmy LAN such as router events, server events, etc. I know I’d really like to have something like that than digging through VERA all the time :)[/quote]

Is there a standard format/API for SYSLOG servers? (I know next nothing about Windows or Linux)[/quote]

Have a look at the logger command.

http://www.cyberciti.biz/tips/howto-linux-unix-write-to-syslog.html

  • Garrett
#51

There may be more people watching than you think…

#52

I have Store logs on USB device enabled so I’ve set EventWatcher’s LogDirectory to /tmp/log/cmh/ so it places the weekly logs along with Vera’s. Vera makes a 512MB partition on the USB stick and uses less than 10% of it so there is plenty of space.

The advantage of this approach is that it doesn’t require any additional file-system software and is not dependent on the network being up. Vera mounts the USB stick anyway so no special start-up script is required.

There is a risk that Vera could wipe out the files if it decides to reformat the stick - although I haven’t ever seen it do this other than when first enabling USB logging. I have WinSCP run a script periodically (using Task Scheduler) to synchronize all *.txt files from /tmp/log/cmh/ to my NAS. The script file is:

[code]# Synchronize EventWatcher Logs

from Vera3 to NAS

open scp://root:@192.168.1.250
cd /tmp/log/cmh
lcd \NAS\Data\EventWatcher
synchronize local -filemask=“*.txt | */”
close
exit
[/code]

#53

For some reason the log isn’t showing the activity of my lock and my Russound zones. Lights show up fine as well as ping sensors (thought you excluded those? I’m glad they’re there). For WatchCategories I have XYSMDV. Am I missing something here?

#54

Not yet implemented for those devices, I’m afraid. If you look at the category table in the documentation or on the first post here, you’ll see those rows are empty. The reason is simply that I don’t have any of those devices and I don’t know which variable/services to watch.

Very easily fixed, but I need the info. The basic philosophy is just one variable per device type, if at all possible.

#55

I have some suggestions for populating some of the missing service/variables:

W - WindowCovering
Service = urn:upnp-org:serviceId:Dimming1, Variable = LoadLevelStatus

It also has SwitchPower1, Status that indicates open/closed but I suggest Dimming1 as it can show intermediate settings.

K - HVAC
Tricky one as there are three important variables in this single category:
Service = urn:upnp-org:serviceId:HVAC_UserOperatingMode1, Variable = ModeStatus
Service = urn:upnp-org:serviceId:TemperatureSetpoint1_Heat, Variable = CurrentSetpoint
Service = urn:upnp-org:serviceId:TemperatureSetpoint1_Cool, Variable = CurrentSetpoint

? - Virtual Switch
Any chance to add VirtualSwitch? It doesn’t set a Category_Num so it would need to be recognised by device type (urn:schemas-upnp-org:device:VSwitch:1).
Service = urn:upnp-org:serviceId:VSwitch1, Variable = Status

I would really like to be able to keep the cache across restarts. Would you consider making that an option? Even more useful would be the option to keep the memory stats across restarts!

This is working really well. Thank you for making such a useful plugin available.

#56

Hi @akbooer, I have a couple more suggestions (well you did ask :D):

Fibaro switch modules usually have hidden child devices that mirror the state of the parent. These are getting logged along with the parent. This isn’t a major issue but I wondered if you could either ignore devices where the invisible attribute is set ( ~= nil , ~= “”) or provide a field to exclude specified device IDs.

If I want to log my internal TemperatureSensors, I also get masses of logging from external sensors and WeatherUnderground. This can be useful but maybe not all the time. This would be another vote for an ExcludeIDs option.

#57

Most welcome, I will implement the easy ones and think on the multi-variable problem. I did wonder about the wisdom of this constraint, but it was so simplifying, I went with it. These things grow in the telling.

I would really like to be able to keep the cache across restarts. Would you consider making that an option? Even more useful would be the option to keep the memory stats across restarts!

This is called “dataMine”, I think. In fact, I have implemented much of the functionality of the EventWatcher reports and plots in the dmDBserver module I posted here [url=http://forum.micasaverde.com/index.php/topic,18749.msg142518.html#msg142518]http://forum.micasaverde.com/index.php/topic,18749.msg142518.html#msg142518[/url]

This is working really well. Thank you for making such a useful plugin available.

Oh good, you’re welcome. Although I should say this is all taking me away from re-engineering dataMine with a round-robin database (based on a Lua revamp of the Python Graphite/Whisper database)… perhaps I should just merge the two.

#58

Looks like ignoring logging for hidden devices would be the first to try.

If I want to log my internal TemperatureSensors, I also get masses of logging from external sensors and WeatherUnderground. This can be useful but maybe not all the time. This would be another vote for an [b]ExcludeIDs[/b] option.

Yes, I have the same problem. I will search for an elegant solution (no guarantees!)

#59
Quote I would really like to be able to keep the cache across restarts. Would you consider making that an option? Even more useful would be the option to keep the memory stats across restarts!

This is called “dataMine”, I think. In fact, I have implemented much of the functionality of the EventWatcher reports and plots in the dmDBserver module I posted here http://forum.micasaverde.com/index.php/topic,18749.msg142518.html#msg142518


I’ve considering installing dataMine but I was more attracted to the simplicity of EventWatcher. I’ll wait to see how your magic-touch improves dM. :wink:

Thanks for considering my suggestions.

#60

In response to recent requests, here’s an update (consider it Beta):

[ul][li]invisible devices - not included in watched list[/li]
[li]window devices - added[/li]
[li]HVAC services - all three requested variables added. [/li][/ul]

I’ve not been able to test those devices which I do not have (of course.) I tested the ‘multiple variable per device’ capability by adding KWH to the Power Meter device as well as Watts. Ping sensors have been formally re-admitted to watched security devices. Virtual switches and explicitly excluded devices need a bit more thinking about - I may need to change from categories to device types, but this is also somewhat fraught.

Whilst you’re there, check out the new [tt]&report=treemap[/tt] capability. This provides a graphical overview of all your devices, showing which ones are being watched and which not. It’s one of my favourite representations for large, multi-dimensional, datasets and could be extended to capture other device information. It first presents an overview of all your devices grouped into rooms: blue/grey ones are not being watched, yellow/gold ones are. You can click on any room to zoom in on the devices there and a tooltip appears when you mouse over any device with a bit more information (I will extend the content here in due course.) You get back to the top level by right-clicking (or control-click on a Mac.) A couple of snapshots attached.

Enhancements and error reports welcomed.