There has been a lot of exchange about the perils of allowing LuaUPnP to grow significantly over 100% memory. And much of this is born out through practical experience, as it grows stability suffers and the more if is greater than 100% the more likely we are to see restarts.
So my question is why does this occur? Why does it reach this point and why does it not simply grow to meet the available memory? I’m not a UNIX guru so I’m along with the intent of learning. Why isn’t it coded so this they’ddoesn’t happen if its such a problem?
I’m guessing that the heap gets fragmented easily and this not as much out of memory as it is out of address spaces (selectors). Too bad one couldn’t periodically force a garbage collection when it gets fragmented and the system is idle. Maybe it can be done but maybe it won’t matter if references to objects are held longer than they are needed.
If the Lua engine does perform these such things the other thing that would cause memory to drop is references to the heap i.e. global variables that are referenced in the operating state. This would make them ineligible for collection thus leaving them hanging around. If there are a lot of small ones then they will fragment the snot out of the heap. In this case there is plenty of memory just not a space large enough to fit/reserve memory to be allocated. In other words if there is not a contiguous space for say a 1k json file then the system will throw an out-of-memory exception even when there is plenty of heap + virtual memory space.
I also suspect that Lua has a lot of immutable objects since it is a managed heap. This means that a new copy of the object is created each time it is modified. For instance, when a string is concatenated each concatenated item creates a new instance of the string and since it is interpreted the possibility of an optimizer to reduce this is non existent.
This scenario also plays out when parameters are called by value rather than by reference. For instance, you call a function and pass it a string. The call stack creates a copy of the string in the scope of the call and when the function executes it pops it off the stack as a new string in memory.
Now after bloviating about memory management I have not written a plugin yet nor do I have experience with Lua. But am kicking around the possibility of one soon. Are there finalizers in the Lua code or does one simply set values to nil when they want objects to be collected?
I’m wondering if some best practices could be used to reduce the memory pressure for common tasks.
It’s very complex … and the 100% is not a hard number … It’s the size of Virtual Memory … not real memory.
Virtual memory can be:
- Backed by files (i.e. programs) and are paged as need into real memory (transient).
- Not backed by anything (Program Stacks that have not been used yet this is where much of the memory listed for LuaUPnP is) But when Vera get’s busy it actually starts to use this.
- Backed by real memory.
The number I use is a RULE of THUMB and not a hard number. As the number get’s larger … the amount of real memory permanently allocated increases … and the amount of free memory available goes down. There are programs that run outside of LuaUPnP … these need memory as well … many are periodic jobs when these run … instantaneous free memory can go down … and if LuaUPnP needs memory at the same time … you get a crash … You might never see the reduction in free memory … because the periodic processess (not part of LuaUPnP) come and go quickly.
It sure would be nice to have even ONE memory slot in the Vera. A $50 SIM card would go a long way to making memory issues go away.
I appreciate the discussion, a complex subject indeed. When I look at memory in my Veras I never see less than 32M available. Plus I have allocated at 64? Swap space. Since LuaUPnP is the heart of things, the worker bee it seems to me that it would be a safe bet to start by allocating it 50 pct our more of working memory so that it would never encounter operating problems. in a Real Time system any ancillary program that has to run can’t be as time critical as LuaUPnp so if they get delayed a bit so what, let them wait for swap space.