Resolved issue that caused frequent LUUP engine restarts

Developer's Corner Scene Scripting
1

TL;DR – if you’re seeing frequent LUUP engine restarts, confirm that you’re not having network traffic issues!

Just wanted to share my experience in hopes that it might help someone in a similar situation.

My (UI5) Vera had been stable for several months. I’d hit a point where nothing was broken, so I was leaving well alone, and was happy with the performance of my Vera and code.

Then I started seeing issues where Vera was acting erratically. I use a lot of PLEGs and I was having situations where timed events (10 min after a door closing, etc) weren’t happening and in general my PLEGs didn’t seem to remember when a condition had become true or false.

After keeping an eye on my Vera using the System Monitor plugin, I realized that the LUUP engine was restarting every 20-30 minutes. In response, I asked a couple questions on the forum and I opened a ticket with MicasaVerde. MV kept an eye on my logs and over the course of several follow-ups suggested that different PLEGs were at fault.

I made a backup and spent a day removing one plugin at a time while keeping an eye on uptime of LUUP. No luck - removed every plugin, deleting their devices, and I still had frequent restarts.

While pruning unneeded code from my startup Lua one day, the power at my house went out for several seconds. After the power was restored and my Vera restarted, it began to run stably. I was disappointed because I couldn’t point at a single fix, but happy that I was on the road to stability.

Then I discovered that another machine on my network had not resumed (by design) when power was restored. After turning this machine on, my LUUP restarts kicked back in!

After some trial and error (disabling Vera’s device discovery, playing with DLNA, disabling my media server), I found that a perl script that I use to retrieve callerID status from my OBI device was working overtime. I had it running once a second, checking the status of the phone line. Once I made this script wait 5 seconds between status checks, my Vera became stable again. For the last several days, the only LUUP restart I’m seeing is the typical nightly restart. The script in question was not communicating with my Vera, but the traffic was going through the same router.

2

@ih8gates, many thanxs for sharing your insights. I too had been experiencing frequent LUUP restarts and discovered that my PLEX DLNA was causing high CPU usage, once I disabled it my CPU usage dropped to 20%-50% which stopped my frequent rebooting. Out of curiosity you indicate

the only LUUP restart I’m seeing is the typical nightly restart
is at the same time or timeframe, I’m assuming that it a result of the nightly heal?

Any tools/hints as to monitoring network traffic issues? Mike

3

Yes - the nightly restart I’m seeing coincides with the CMH restart. For me that’s around 2-3am.

My router has some tools to get a general idea of traffic, but I’m not sure if that would’ve helped here. Wireshark may be able to help, but I haven’t experimented with it.

4

@ih8gates, again many thanxs. The reason I was asking is it seems that there are number of different tools, LUA snip-it, and techniques that various folks are using to monitor memory, software usage, cpu utilization, and logs. When I look at the logs other than something blatantly obvious there is a lot of traffic that are a result of normal system operations and only if your intimately familiar with how VERA works is some one able to make some deterministic conclusion to identify the root causes. It would be nice if some one had a filter that could hide the common/normal system operations msgs and only leave the anomalies to be reviewed which should leave much smaller log to review. Mike

5

It should be noted that I got at least a half-dozen mis-diagnoses from the experts at MCV.

My aim in posting this is to add one more item to the Vera debug toolbox.

6

It is actually interesting that how busy the ethernet network is can cause this type of crash. How frequently do you see a Luup restart now?
I seem to be encountering a Luup restart every couple of days and I am not sure what it is due to.

7

I’m only seeing the one restart at ~2am that coincides with Vera’s heal.

Occasional restarts aren’t a big deal. Vera saves its data to disk every 6 min (I think - might be 4) or so. So my frequent restarts would often lose uncached data.

8

I know the occasional Luup restart aren’t much of an annoyance. I have disabled the nightly heal for a long time since I did not understand the idea behind it and saw no benefit from it. I was just curious as to whether these occasional restarts could be avoided.

9

If on a network switch should this really be an issue?

My Vera reboots frequently. Too frequently.

I have it email me log snippets when it reboots. But I don’t have a clue what to look for.

I have a Plex server but turned off DLNA some time ago.

But who knows – there could be other things…

I wish there was good documentation on how to troubleshoot all of this.