Where to store persistent data from LUA

[dunked]

I’m developing some LUA code that does boosts and scheduling of TRVs (using my own wakeup code all from within LUA not using UI5 at all) and need somewhere to store some data that will survive reloads. So far I’ve tried several approaches and they all work - but I’m wondering if there is a best practise or recommendation that I should be following. I should state that the writes are not massively frequent at most once a minute or a lot less and seeing as the luup log files get written every few seconds I’m assuming my writes should not add too many flash cycles (but prepared to be told otherwise). Here’s what I’ve tried so far

1). Reading and writing to files in /etc/cmh-ludl/ the advantage is I can list them from the UI5 luup files page (as an aside you can also serve your own web pages from there which is kind of fun)
2). Reading and writing to files in /tmp/log/cmh/ - which is where the logs get stored, and if you use USB logging that directory is remapped to the USB drive so there’s no writes to Veras flash memory.
3). Saving and getting data from Variable Containers - that seems to work ok and I assume must use a supported method, but I don’t know if there’s a limit on how much you can store in a single variable in a VC (I treat each VC as having 10 variables I can use, I treat the name as just another variablel). There is always the risk of course that using the UI to edit one could completely mess it up.

The variables I store tend to be serialised LUA tables.

I’d be interested in any thoughts or recommendations

[RichardTSchaefer]

Logs are buffered … and written to regular memory or an external USB log file. Flash memory is not in play (Logs for OTHER purposes do use flash memory).
The persistent state of device state variables … (i.e. the the /etc/cmh/user_data… file) is only written when Vera Crashes or periodically … also not a flash memory problem.

You could write your Variables to ANY device (i.e. attach the variables to the TRV directly. (Just create your own ServiceID and Variable name strategy).

[dunked]

Thanks for that info. I might explore attaching my own variables to a device - something I’ve not tried yet…
If the /etc/cmh-ludl directory doesn’t use flash then I’m guessing I’m safe to just write files to that as I tend to read a file in at startup and populate some global tables - in the belief that that will be faster to access data stored in globals than doing variable_get all over, I only re-write the table to a file when something changes in it.

[RichardTSchaefer]

No the /etc/chm area is FLASH … it just only writes the file ever 5 minutes or so, and also when it crashes (if it can)!
The only place to write files that are NOT flash is /tmp/log/cmh
The files use a tmpfs filesystem … which means the files are in normal RAM (or if a USB is attached … they are on the USB drive)

[dunked]

Ok - got that I think. So if I write to /etc/cmh I should do it sparingly as it’s using flash cycles.
If I write to /tmp/log/cmh then it’ll write to USB if I have USB logging on, if not then if it’s using tmpfs I’ll lose the data on a complete reboot (but would I lose it on just a LUA reload I wonder?)… either way that wouldn’t be persistent.

I’ve been checking out adding my own variables to devices - it seems fairly straightforward with 2 ways to create a variable on a device

luup.call_action(“urn:micasaverde-com:serviceId:HomeAutomationGateway1”, “SetVariable”,DeviceNum=36,Service=“myservice”,Variable=“myvar”,Value=myval},nil)
or
luup.variable_set(“myservice”,“myvar”,myval,36)

I prefer the shorter version.
It seems I can put anything I like as ServiceID field - the IT Architect side of me says I should be using a proper urn:upnp-org:serviceId:something, but the practical side says that serviceId will only ever be used by code that I write so I can call it what the heck I like e.g “myveryownservice” as long as it doesn’t clash with someone else’s service

[RichardTSchaefer]

It seems I can put anything I like as ServiceID field - the IT Architect side of me says I should be using a proper urn:upnp-org:serviceId:something, but the practical side says that serviceId will only ever be used by code that I write so I can call it what the heck I like e.g "myveryownservice" as long as it doesn't clash with someone else's service

An early goal of Vera was to be UPnP compatible … they never really achieved the goal … but it has heavily influenced the existing design.

[dunked]

The more I play with adding variables to devices the more useful it becomes!
I’ve just tested and variable_watch also works with variables that you add yourself. So it seems to tick a load of boxes - my two remaining concerns are

does / will UI7 still support this technique?
if a device gets updated e.g via a plugin update or similar would that reset all the device variables to the default and wipe mine out?

maybe just to be safe I can run a job every hour to collect all my variables and write them to a file in /etc/cmh

[akbooer]

does / will UI7 still support this technique?

It certainly should.

if a device gets updated e.g via a plugin update or similar would that reset all the device variables to the default and wipe mine out?

No, not in UI5, and I presume the same for UI7, precisely because updating would otherwise reset the device.

[RichardTSchaefer]

The only way to get rid of the variable is to DELETE the containing device.

[dunked]

Having to delete the device to get rid of a variable is a bit of a pain, I’ll just need to make sure I get them right.

I did a quick fairly crude test to see what the speed differences are between variable_get and variable_set vs reading and updating LUA global variables - doing 10000 updates and reads of each the results are below
test writing using variable_set ended took 6.58 seconds
test writing direct to global ended took 0.019999999999982 seconds
test read using variable_get ended took 0.54000000000002 seconds
test read direct from variable ended took 0 seconds

So what that tells me is that for time critical code I should avoid variable_set - though to be fair I’m normally updating one or two values not 10000 so it wouldn’t really hurt and for reading variables variable_get is fast enough though to be really fussy direct reads of globals is faster

Thanks for your advice Richard - as ever always valuable!