Event & Device Activity Log - does a plugin exist?

Plugins & Plugin Development General Plugin Discussion
1

I know we can pull Vera logs, but these have a ton of system info in it. I’m curious if there is, or someone would be willing to build, simply a log that records device and event actions.

Example…

15:51:32 Vera.Master Bathroom.Vanity Light.ON.0.0
15:51:34 Vera.Master Bathroom.Vanity Light.ON.31.0
15:51:36 Vera.Master Bathroom.Vanity Light.ON.100.0
15:54:29 Vera.Master Bedroom.Fan Light.OFF.100.0
15:54:31 Vera.Master Bedroom.Fan Light.ON.80.0

… this is actually how the plugin for EventGhost logs activity for the Vera. I’m actually trying to remove dependance on EventGhost now that the awesome PLEG plugin exists (thanks Richard!).

2

Could you use datamine which can record the state changes on/off to achieve the desired outcome.

3

There are a few reasons DM will not work for this…

  1. I’d need to setup ALL possible devices
  2. DM does not log Events
  3. DM has a MAJOR problem - if there are too many log files it stops working. I hit this issue about once a month with only a few items being tracked then I must go in and manually delete files.

I’m looking for a plain/simple text log file of all activity for all Devices & Events.

4

Chris released a new version of DM and he’s been running it for a long time now with many datasets. You might want to contact him regarding your issue.

  • Garrett
5

I think any solution would need to do this. I don’t think there’s a way in Vera to get a notification for every variable without explicitly setting up a callback (I might be wrong on this, but I think this is the case). This could probably be automated, but the result would be the same.

dataMine logs all events into a file called “Notifications” (with a number on the end). The format of this file is a JSON file so it’s pretty easy to parse and use. All events are logged - not just events from devices you have configured for dataMine.

This is not correct - maybe your USB stick has an issue (??) or something else is wrong, however I have thousands of files in my DM directory and have no problems (I log 85 variables right now, so that means 85 files per week). I know of others that are doing similar (and I know of one user who is logging hundreds of variables). USB usage on Vera does have some issues, and you may be hitting against them…

Also, as Garrett says, the latest dataMine update changes the format so there are fewer files in each directory (only 1 file per week).

Cheers
Chris

6

Chris & members
please don’t take my post as a slight to DM… DM is fantastic, I just did not think it was ideal for an all-in-one, real-time, Event & Activity Log.

I have had only 1 problem and it seems like that might go away with the latest version, I’ll find out in about a month :wink:

The intent of the thread was to get a single, human readable (yet still CSV) file with all the ‘user mode’ actions (Events/Activities the user did manually or created automation for), without any ‘system’ debug/background actions.

Example, in DM’s Notifications files:

Don’t want this…

{“description”:“CheckLogs: 5004”,“type”:7,“source”:0,“code”:“nm_logs”,“id”:1020654803,“timestamp”:1369347604,“value”:“”,“device”:0},

Want this…

{“description”:“Front Door Lock”,“type”:4,“source”:4,“code”:“DL_USERCOD”,“id”:1021655823,“timestamp”:1369364114,“value”:“UserID="3" UserName=”,“device”:4},

And want it to be more human readable like…

2013-05-24; Front Door Lock:Unlock; UserID:3

Chris, is this something you can add to DM - and make it exposed to the web front-end, directly accessible via a static URL?

7

[quote=“Aaron, post:6, topic:175433”]Don’t want this…

{“description”:“CheckLogs: 5004”,“type”:7,“source”:0,“code”:“nm_logs”,“id”:1020654803,“timestamp”:1369347604,“value”:“”,“device”:0},

Want this…

{“description”:“Front Door Lock”,“type”:4,“source”:4,“code”:“DL_USERCOD”,“id”:1021655823,“timestamp”:1369364114,“value”:“UserID="3" UserName=”,“device”:4},[/quote]
I’m slightly confused - aren’t they the same? Or, do you mean you want to filter out the “useless” notifications like the checklogs you show above? This would probably be useful since I get these checklogs notifications every 45 minutes or so, so it really fills up the log. I could probably be tempted to remove them from the log, but the next thing is how to make this more generic so that other entries can also be removed…

[quote=“Aaron, post:6, topic:175433”]And want it to be more human readable like…

2013-05-24; Front Door Lock:Unlock; UserID:3

Chris, is this something you can add to DM - and make it exposed to the web front-end, directly accessible via a static[/quote]
The problem with this as a log format is it’s human readable, and therefore not flexible as a computer readable format. I chose the json format for this data so that it was extensible - CSV (or some other fixed format delimited file) isn’t as flexible if it was changed in future.

I’ve been thinking for a while of adding a notifications page in the dataMine GUI. This would allow you to forget about the file format, and just interface with the data - is this effectively what you’re looking for? Of course there is an interface like this in UI%, but like many parts of UI5, it’s very clunky!

Exposing it outside of the local Lan is a problem. You can always open up your firewall, but this really isn’t recommended. The best way to do this, and I think it would work (??) is to see if MCV would host a version of the DM GUI on their website in the same way as UI% is hosted. So, you’d go to cp.mios.com, and instead of hitting Dashboard, you’d hit dataMine. This would potentially work, and would be secure. However, it requires MCV to do something, and we can all guess how long that could take. This has come up before though and at the time I meant to email MCV but forgot - I’ll send them a mail now…

There is another option, and that is to locally host the GUI on a stick. This would also be mildly secure in that access to your Vera would still be through the MCV forward servers - the problem is you’d need to type your password into it, and this could be open to malware. However, it’s worth thinking about…

Cheers
Chris

8

the actual format is really not that important but cleaning it up a bit would be nice… JSON is readable, but takes some work, unlike the example I provided in the first post which is dead simple. And, both can easily be parsed and input into Excel ect.

as for access… I was not so concerned with outside access, but if desired the users can pull the file (SSH, SCP, FTP, etc) or open ports, which most of us do for other things anyway. I was thinking the web UI view would be nice for real time monitoring. Just like the Vera Log file app does…

http:///port_3480/data_request?id=lr_al_log&app=localApp

9

Hi Aaron,
Agreed - a format like CSV, or whatever, can easily be passed, but as I said, it’s not flexible. For something like the notifications where there’s quite a lot of potential for different data, I decided that was important as I might want to change what’s stored over time.

Sorry - when you said “web front-end” I misread and thought you meant “the web” as in external. So, yes it’s relatively easy to do this to the standard DM GUI, and I will probably look at doing this.

Cheers
Chris

10

Chris,
Are you still planning to put in a “clean” text log option in DM?

This would REALLY help troubleshooting when making complex & interconnected Automation scenes - e.g. when several triggers and/or schedules interact with multiple scenes - sequencing, if it is working, etc.

11

To the OP: Would this: http://forum.micasaverde.com/index.php/topic,15245.msg115823.html#msg115823 be of any use? It’s what I do to keep track of events/activity…

12

Hi Quinten,
Funny you mention it, I saw and read your thread a few nights ago while doing my 2nd try at, “I wonder if anyone did something like what I’m looking for”

While your method can provide a ton of functionality, it is much more work to setup (MySQL & changing each device/event) than I need or want.
I want something dead simple… basic, human readable, text logging of Device & Event ‘activity’.

like this…

Date & Time ; Type ; Item:Action Taken; Item State(if applicable) ; Source of Action

2013-05-24;12:01:14; device; Front Door Lock:opened; ; UserID:03
2013-05-24;12:01:27; device; Front Door Sensor:triggered;armed;
2013-05-24;12:01:34; device; Front Door Motion Sensor:triggered;armed;

2013-05-24;22:10:34; event; Night TV Lighting; schedule;
2013-05-24;22:10:35; device; Living Room Fan Light:dim60; event:Night TV Lighting;
2013-05-24;22:10:36; device; Kitchen Lights:off; event:Night TV Lighting;

13

@Aaron, are you happy just to see events which have notifications set (as a simple way of selecting what you’re interested in), or do you want every device?

A simple modification of the plugin here [url=http://forum.micasaverde.com/index.php/topic,9525.msg128860.html#msg128860]http://forum.micasaverde.com/index.php/topic,9525.msg128860.html#msg128860[/url] might do the trick.

14

[quote=“akbooer, post:13, topic:175433”]@Aaron, are you happy just to see events which have notifications set (as a simple way of selecting what you’re interested in), or do you want every device?

A simple modification of the plugin here [url=http://forum.micasaverde.com/index.php/topic,9525.msg128860.html#msg128860]http://forum.micasaverde.com/index.php/topic,9525.msg128860.html#msg128860[/url] might do the trick.[/quote]

Every Device & Automation Event… as I provided in the examples above.

This is the ONLY way to see a true ‘activity log’ and assist with troubleshooting Automation events.

Vera already logs everything, yes? What is really needed is a human readable version of it with all the ‘garbage’ removed.

15

Yep. It baffles me that this doesn’t already exist.

16

Vera Alerts has what you want … if you have an Android Phone!
Sort by Date … or Sort by Alert Message …

Easy to see the most recent messages … or just the ones where you entered your front door.

You can keep all messages or just the last N of each message type (Automatically cleans up old messages, N is specified by user).

17

[quote=“RichardTSchaefer, post:16, topic:175433”]Vera Alerts has what you want … if you have an Android Phone!
Sort by Date … or Sort by Alert Message …

Easy to see the most recent messages … or just the ones where you entered your front door.

You can keep all messages or just the last N of each message type (Automatically cleans up old messages, N is specified by user).[/quote]

How does Vera Alerts capture all event and device activity without setting up notifications?

18

The alternate event mechanism only works for Notifications and Scene Triggers (Same as Vera Alerts).

Do achieve all events you have to do a variable_watch on EVERY device.

You would likely want to filter out some devices types …
Do you really want to see every time the temperature or light level changes in the house ?
Do you really want to see every time there is a timer tick on PLTS and/or the countdown timer ?
You will likely have to have a filter to only log specific variable changes for specific device types.

For me if I am interested in something … I add it to Vera Alerts.
I add a lot of these during testing … then remove them once an automation component is stabilized.
All of it with a few clicks on the Vera Control panel … minimum typing.

19

Richard,
The goal is to be able to have a ‘clean’ activity long.

Something that looks like this…

Date & Time ; Type ; Item:Action Taken; Item State(if applicable) ; Source of Action

2013-05-24;12:01:14; device; Front Door Lock:opened; ; UserID:03
2013-05-24;12:01:27; device; Front Door Sensor:triggered;armed;
2013-05-24;12:01:34; device; Front Door Motion Sensor:triggered;armed;

2013-05-24;22:10:34; event; Night TV Lighting; schedule;
2013-05-24;22:10:35; device; Living Room Fan Light:dim60; event:Night TV Lighting;
2013-05-24;22:10:36; device; Kitchen Lights:off; event:Night TV Lighting;

… this would provide several benefits. Including: Ease of troubleshooting Automation Events, tracking house ‘activity’ in general which could lead to ideas for creating new Automation and/or ideas for devices/energy savings/etc. And simply a nice, clean way to figure out what is happening and how things are working - or are not :slight_smile:

20

I read that … And I have that in Vera Alerts … for the devices I want to see in the log.

Various sort orders … I have mine setup to display a timer order (most recent on top) set of events.

It has options on how long to keep messages … you can keep all if you want …