Web logs to watch sceneEvents, valueHistory, process flow, debugging,...

I missed a feature to log the triggers, the event-flow of scenes or the temporal development of device variables at a glance. I want these logfiles outside of vera because of the limited space and the access restrictions.
So I created a way to write logs into specific files on a webserver: Now I get many specific logfiles per scene, per process etc…

          • Examples: - - - - -

logfile for heater thermostates (danfoss,…) and the room temperature

2018-02-19 14:00:01 Scene Office SetpointNight
2018-02-19 14:00:01 Scene logCheck lastWake.13:50:35 TempRadiator:22.55 SPT_old:22.50 TempRoom:21.67 SPT_new:12.0 Trigger:ONight Action:setSPT
2018-02-19 14:11:10 Scene logCheck lastWake.14:10:08 TempRadiator:22.09 SPT_old:12.00 TempRoom:21.67 VariableOk noDo Trigger:check


logfile for security events

2018-02-20 08:41:01 Office Trigger: DoorOpen
2018-02-20 08:41:01 OfficeHall Trigger:MotionTrip
2018-02-20 08:41:05 Office Trigger: DoorClose
2018-02-20 08:42:12 OfficeHall Trigger:MotionUnTrip


logfile for envirement values

2017-12-06 11:37:15 Humidity RoomA:50.51 RoomB:41 RoomC:60.54
2017-12-06 17:37:15 Humidity RoomA:49.78 RoomB:42 RoomC:58.41
2017-12-06 23:37:15 Humidity RoomA:46.84 RoomB:42 RoomC:57.92


For me these logfiles are a neccessary help to watch the events of the last days and to debug unexpected behavior of the devices or the scenes.
The solution cosists of a lua function in startupLua, a php-script on the webserver and lua “log-calls” at appropriate places in scenes.
Maybe someone has a similar requirement and thus parts of this solution maybe helpfull for him.

  1. The lua function “strLog(arg)” and the help function “mySplit” are pasted into startup LUA.
    function mySplit(inStr,sep)
    if sep == nil then sep=“%s” end
    local t={}; i=1
    for str in string.gmatch(inStr,“([^”…sep…“]+)”) do
    t[i] = str
    i = i + 1
    end
    return t
    end
    function strLog(arg)
    local sArg=mySplit(arg,“!”)
    local logFile=sArg[1]
    local aStr=sArg[2]
    local aaStr=string.gsub(aStr, “%s+”, “+”)
    luup.inet.wget (“myDomain/path/strVera.php?tgt=” … logFile … “&aStr=” … aaStr)
    end

  2. The function “strLog(arg)” can be called at appropriate places in all scenes with:
    → strLog(“targetFile!Text example”) or better: luup.call_delay(‘strLog’,0,“targetFile!text example”)
    The result in this case is an updated file “targetFile.log” with the content “timestamp example”


A way to include values or to identify one out of multiple scene actions:
local aStr=“Text1 Text2”
local ValueA=123
local ValueB=luup.variable_get(“urn:upnp-org:serviceId:SwitchPower1”,“Status”,devNr)
aStr=aStr … " ValueA:" … ValueA … " ValueB:" … ValueB
strLog(aStr )


A way to identify and include scene triggers:
Add a piece of lua into the scene’s trigger definition luaBox. For scene 123: scn123=“openDoor”
This defines a global variable you can evaluate in the scene’s lua box with:
local scnTrig=scn123 or “”
scn123=“”
aStr=aStr … " scnTrigger:" … scnTrig
strLog(aStr )

  1. The function “strLog(arg)” transfers the logData per luup.inet.wget to a small script “strVera.php” on my webserver (see attachment). The behavior of strVera.php to generate the logfile “LogDevCheck.log” can be tested per browser url: //csvVera.php?LogDevCheck!testString .
    The results are specific logfiles on the webserver.

A variant of this method to monitor the zwave net variables for all devices in one table (Neighbors, WakeupRating…) is described in:
http://forum.micasaverde.com/index.php/topic,60546.0.html

pit,

Great resource and excellent documentation. I was also looking to log specific events that occurred but used a slightly different approach. The Event Watcher plug-in contains all the necessary substructure to do very detailed custom logging. You just need to add a couple lines of code to get things working.

If you are just looking to log events or variables in scenes or the underlying Lua code, simply add these lines to your Lua startup:

function EW0LogIt (var_txt)
  var_url = require "socket.url"
  var_txt = "Scene: " .. var_txt
  var_txt = var_url.escape(var_txt)
  luup.inet.wget ("http://127.0.0.1:3480/data_request?id=lr_EventWatcher&event=" .. var_txt)
end

Then call the function in your luup code like this:

EW0LogIt "Alarm - Arm All Ext Door Zones"  -- use any name you like to identify the scene

To view your log file go to the Control panel of Event Watcher, select Control then Variable and Event Log. You’ll see something like this:
[tt]
3,295 2018-02-20, 17:56:56.582 S 67 LR Motion Sensor SecuritySensor1.Tripped 1
3,294 2018-02-20, 17:56:32.478 K 144 T-Stat 1 UserOperatingMode1.ModeStatus AutoChangeOver
3,293 2018-02-20, 17:56:32.023 K 144 T-Stat 1 TemperatureSetpoint1.CurrentSetpoint 76
3,292 2018-02-20, 17:56:31.983 E 0 User HVAC PLEG: Mode set to AutoChangeOver
3,291 2018-02-20, 17:56:31.972 H 12 Humidity HumiditySensor1.CurrentLevel 81
3,290 2018-02-20, 17:56:31.954 T 11 High Temperature TemperatureSensor1.CurrentTemperature 86
3,289 2018-02-20, 17:56:31.885 T 9 Temperature TemperatureSensor1.CurrentTemperature 74.0
3,288 2018-02-20, 17:56:24.353 L 79 LR Light Sensor LightSensor1.CurrentLevel 29.61
3,287 2018-02-20, 17:56:24.185 H 80 LR Humidity Sensor HumiditySensor1.CurrentLevel 63.91
3,286 2018-02-20, 17:56:23.928 T 78 LR Temp Sensor TemperatureSensor1.CurrentTemperature 77.80
3,285 2018-02-20, 17:55:38.910 S 67 LR Motion Sensor SecuritySensor1.Tripped 0
3,284 2018-02-20, 17:55:35.449 E 0 User Alarm PLEG: Alarm Mode set to Home
3,283 2018-02-20, 17:55:27.169 K 145 T-Stat 2 OperatingState1.ModeState Cooling
3,282 2018-02-20, 17:55:26.830 K 145 T-Stat 2 TemperatureSensor1.CurrentTemperature 82.00
3,281 2018-02-20, 17:55:23.328 L 118 Kit Light Sensor LightSensor1.CurrentLevel 50.87
3,280 2018-02-20, 17:54:59.098 X 53 Bird Room 2 Light SwitchPower1.Status 0
3,279 2018-02-20, 17:54:18.448 E 0 User Lighting PLEG: Kitchen Motion
3,278 2018-02-20, 17:54:18.432 S 81 Kit Motion Sensor SecuritySensor1.Tripped 1
3,277 2018-02-20, 17:53:37.985 S 81 Kit Motion Sensor SecuritySensor1.Tripped 0
[/tt]

Of course, what gets logged is dependent on what parameters you watch. My settings are:
[tt]
LogDirectory /www/
WatchCategories DOGYFHTLSKX
[/tt]
These settings are found on the Variables tab of Event Watcher.

If you want to expand on this and also log PLEG events, add the following code to the StartUp Lua of EACH PLEG instance:

function EW3LogIt (var_txt)
  var_url = require "socket.url"
  var_txt = "Lighting PLEG: " .. var_txt
  var_txt = var_url.escape(var_txt)
  luup.inet.wget ("http://127.0.0.1:3480/data_request?id=lr_EventWatcher&event=" .. var_txt)
end

And call the function like this:

EW3LogIt "All Lights Off"  -- use any text you like to identify the process

Notice the difference in the Function Name. Each PLEG must have this code in its startup section and I use a different prefix to differentiate log entries for each PLEG. (EW1LogIt, EW2LogIt, EW3LogIt and so forth…)
Also note that the LUA Startup code above (for scenes) will NOT work globally with PLEG. EACH PLEG instance will require an individually named function. Since the function is only six lines and it is only one line to call, the entire process is very lightweight!

To capture variables, the procedure is very similar.

  1. define the variable and set its value: varStartTime = yourvariablevalue
  2. Call: luup.inet.wget (“http://yourVeraIP:3480/data_request?id=lr_EventWatcher&variable=varCurrentTime:” … varStartTime)
  3. If your variables contain strings with spaces, you may want to add the var_url.escape(var_txt) lines here as well.

Do keep in mind that this log is written to Vera (or to a USB drive with a little more work) and will take up space on the controller, so a little routine maintenance is necessary if you log a lot of data.

Many thanks to AKBooer for the excellent plug-in and his help to dial in this code. If you’re already using Event Watcher and want to expand its abilities, I highly recommend trying this.

kartcon,

your approach seems for me easygoing. Thank you!

Your clear instruction is surely helpfull for the users of Event Watcher. But my problem ist: Event Watcher doesn’t work for me. In feb 2017 I tried to install this app, because the features looked promising for me.
But I can’t get going it on my veraEdge / UI7. (“startup lua failed”, “no handler”). Today I tried again, but without success.

On the other hand I learned to appreciate the easy access to “my” web logs in my daily experience - local or remote - and for long periods without space problems.

pit,
I suggest you reach out to AKBooer, the author of Event Watcher. I have found him to be quite helpful and very knowledgeable. He helped me fine tune the code above and has been (very patiently) trying to help me with openLuup on a Raspberry Pi. That whole platform was very foreign to me and I made a ton of novice mistakes and blatant oversights. Despite my bumblings he has continued to be both responsive and helpful.
Best of luck getting Event Watcher running. I believe it will be worth your effort, as the plug in provides a wealth of information that I use on a daily basis.
Kartcon

@kartcon, I’m here! Thanks for the comments!!

@pit, it looks like EventWatcher is not correctly installed. How did you do that in the first place?

akbooer,
I installed Event Watcher in 2.2017 from “vera - install apps” on veraEdge UI7. But I got the error “startup lua failed” and if I tried to start an Event Watcher link I got “no handler”.
I tried to update via “vera - my apps”. But no success. So I uninstalled the app via “vera - my apps”.
Today I repeated this procedure without success. So I uninstalled again.

Ah yes, I see that I tried to help a year ago and we didn’t get anywhere.

My recommendation now would be to install files manually using the latest GitHub development version here: 2017.07.09

If you need help with the details of that, just ask again. EventWatcher has been around for quite a while and really quite a lot of people have used it, so it does work. There’s just something we’re missing in your configuration.

sorry akbooer,

seems the Event Watcher don’t like me. I downloaded the files from GitHub and reloaded them into vera per “vera develop apps” (the files were existing). I also reloaded the existing icon per WinSCP. Then I manually created the device following the description in your manual. But now I got “System error : Device: 280. Fail to load implementation file D_EventWatcher.xml”. This file exists, I can view it in “vera develop apps”.

Reload and cold start do not help. In vera UI is the new device not visible, per …/data_request?id=invoke I can see it. Log:

1 02/21/18 15:51:09.986 JobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 280/D_EventWatcher.xml so device 280 is offline <0x7781e000>
1 02/21/18 15:51:11.159 Device_LuaUPnP::CreateServices Aborting device 280 because the topmost device has no interface <0x7781e000>
2 02/21/18 15:51:11.643 Device_Basic::m_eConfigured_set device 280 was 0 now 0 startup <0x7781e000>
9 02/21/18 15:51:22.485 JobHandler_LuaUPnP::Run device 280 ptr 0xf8d018 EventWatcher room 0 type urn:akbooer-com:device:EventWatcher:1 cat 0:-1 id parent 0/0x103e348 upnp: 0 plugin:0 pnp:0 mac: ip: <0x7781e000>

Apparently not!

I downloaded the files from GitHub and reloaded them into vera per "vera develop apps" (the files were existing).

This bothers me tremendously. It’s quite possible to have multiple versions of files (one compressed, one uncompressed) and likely that you will end up using the wrong ones. The solution is to delete everything with “EventWatcher” in it and try again.

I also reloaded the existing icon per WinSCP.

That’s fine. You shouldn’t have to do it again. Not currently the issue.

Then I manually created the device following the description in your manual. But now I got "System error : Device: 280. Fail to load implementation file D_EventWatcher.xml". This file exists, I can view it in "vera develop apps".

As per the above, this doesn’t entirely surprise me.

Reload and cold start do not help. In vera UI is the new device not visible, per .../data_request?id=invoke I can see it. Log: --- 1 02/21/18 15:51:09.986 JobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 280/D_EventWatcher.xml so device 280 is offline <0x7781e000> 1 02/21/18 15:51:11.159 Device_LuaUPnP::CreateServices Aborting device 280 because the topmost device has no interface <0x7781e000> 2 02/21/18 15:51:11.643 Device_Basic::m_eConfigured_set device 280 was 0 now 0 startup <0x7781e000> 9 02/21/18 15:51:22.485 JobHandler_LuaUPnP::Run device 280 ptr 0xf8d018 EventWatcher room 0 type urn:akbooer-com:device:EventWatcher:1 cat 0:-1 id parent 0/0x103e348 upnp: 0 plugin:0 pnp:0 mac: ip: <0x7781e000>

Well, that’s worrying too. Start with a clean slate. Delete the device (using an HTTP request, I assume device #280 is what invoke shows) and check it’s gone after the reload.

Upload the files once more and create the device. Good luck this time.

No success, I fear it’s time to surrender!

I deleted the device per http and reloaded. http-invoke didn’t find it anymore. I searched files with EventW per winSCP in all folders and deleted all “EventWatcher. …”. I reloaded vera and checked in “develop apps”: No EventWatcher files were existing.

Then I loaded the 5 xml, lua and json files from GitHub, reloaded vera and created the device again.
Again the same error: “System error : Device: 281. Fail to load implementation file D_EventWatcher.xml”.

Again I can view this file (and the other uploaded files) in “develop apps” and in winSCP under “/etc/cmh-ludl” and “/overlay/etc/cmh-ludl”. Cold reboot doesn’t help.

Log looks like before:


1 02/21/18 17:07:51.033 JobHandler_LuaUPnP::CreateDevice_LuaUPnP failed to load 281/D_EventWatcher.xml so device 281 is offline <0x77001000>
1 02/21/18 17:07:51.507 Device_LuaUPnP::CreateServices Aborting device 281 because the topmost device has no interface <0x77001000>
2 02/21/18 17:07:51.736 Device_Basic::m_eConfigured_set device 281 was 0 now 0 startup <0x77001000>
9 02/21/18 17:07:54.603 JobHandler_LuaUPnP::Run device 281 ptr 0x107f0c0 EventWatcher room 0 type urn:akbooer-com:device:EventWatcher:1 cat 0:-1 id parent 0/0x1130520 upnp: 0 plugin:0 pnp:0 mac: ip: <0x77001000>

Well, fair enough, but I’ve just deleted all [tt]/etc/cmh-ludl/EventWatcher[/tt] files on my Vera Edge, uploaded new ones from the GitHub repository, and successfully recreated an EventWatcher device.

I DID do one step that you may not have done… on the Plugins pages, DELETE the EventWatcher plugin before you create a new one. The new plugin that you create will NOT appear on the Plugins page (because it wasn’t downloaded from the MiOS repository.)

The Apps > Develop apps > Create Device form looked like the attached…

Give it one more try?

WOW - The Event Watcher is finally running!

The Mios-Plugin was already deleted. But the fields for manual device creation you adviced to fill were others than in your manual.
In the night I made another try adding “I_EventWatcher.xml”. It doesn’t help.
In the morning with fresh power I repeated a clean installation and now with the manual device fields “D_EventWatcher.xml”, “I_EventWatcher.xml” without devicetyp (previously filled according to manual). Apparently this was the breakthrough!

Thanks a lot for your help and your patience!
I’m glad to use now your sophisticated and helpfull tool.

That’s great. I’ll check the documentation, sorry.