I just placed an order for an Universal Devices ISY Insteon Controller to extend the capabilities of my insteon gear and to overcome the limitations of the altsteon plugin. I have been reading up on the api documentation and it seems pretty straight forward to controlling devices via REST / HTTP requests. To receive real time event changes, the api requires you to send a soap subscription that stays open to receive new events that might happen (e.g. device status changes, etc). I want to develop a plugin that will allow Vera to interface with the ISY. I have a few questions and need input.
I would like to develop a daemon (similar to what Futzle has done with her upnp proxy plugin) that will open a http connection and send a soap subscription request to the ISY and have it listening to event changes. What would be the best way to develop the daemon? Is this possible to do in lua? I have done some searching and I am having a hard time finding information on lua code to maintain a persistent http connection. The daemon would than send those requests to the Vera plugin for processing and updating the appropriate devices.
Does anyone have any suggestions or a better way of accomplishing this?
I’m sure it could be written in Lua using LuaSocket http.request. You would just write your own sink function that read the incoming stream of an HTTP request that doesn’t close. If you make the request with the proper headers, the server will know to keep the connection open and keep writing. I hope there is a clean solution to this because it will solve an entire class of plugin status polling.
The effect of a long standing READ … You will basically have a thread blocked on a read … How will that effect Vera … I believe Vera allocates two threads per plugin … But it’s usage model is not well described.
Will it consider this a hung thread ? and restart ?
I have seen some bugs reports that leads me to believe that the IO layer in Vera (Serial and Socket) can interfere with each other. Having an outstanding read increases the probability of bugs like this to show up.
[quote=“RichardTSchaefer, post:3, topic:175215”]I would have two concerns with the Lua approach.
The effect of a long standing READ … You will basically have a thread blocked on a read … How will that effect Vera … I believe Vera allocates two threads per plugin … But it’s usage model is not well described.
Will it consider this a hung thread ? and restart ?
I have seen some bugs reports that leads me to believe that the IO layer in Vera (Serial and Socket) can interfere with each other. Having an outstanding read increases the probability of bugs like this to show up.[/quote]
My suggestion that Lua would work as the language was with the assumption that the Lua code would be completely outside of the Vera context, running in its own process – I agree with Richard that a blocking read in a plugin is not OK (unfortunately).
Looking at the documentation, when you send the soap message to subscribe to the events, in that message there is data to tell it to reuse socket and duration is infinite.
When you are done listening, you tell it to unsubscribe. I believe it also states that you need to send an ack when a heartbeat is sent. Still trying to learn all of this and read up on soap, etc. I placed an order for and ISY 994i Pro and a serial Insteon PLM (I wish it could use the usb PLM’s so I did not have to order another PLM).
What I am not sure of is if the LuaSocket http.request will close or maintain connection. I have seen reports that it closes upon response, but I am not entirely sure of this.
[quote=“watou, post:4, topic:175215”][quote=“RichardTSchaefer, post:3, topic:175215”]I would have two concerns with the Lua approach.
The effect of a long standing READ … You will basically have a thread blocked on a read … How will that effect Vera … I believe Vera allocates two threads per plugin … But it’s usage model is not well described.
Will it consider this a hung thread ? and restart ?
I have seen some bugs reports that leads me to believe that the IO layer in Vera (Serial and Socket) can interfere with each other. Having an outstanding read increases the probability of bugs like this to show up.[/quote]
My suggestion that Lua would work as the language was with the assumption that the Lua code would be completely outside of the Vera context, running in its own process – I agree with Richard that a blocking read in a plugin is not OK (unfortunately).[/quote]
This would run outside of Vera due to that very limitation of not causing the thread to be blocked. It would run similar to what Futzle has done with her upnp proxy plugin where it runs as a daemon. The daemon would then notify the plugin of any changes.
In that context LUA should be fine.
I know there is an http.TIMEOUT … I am not sure if it can be set to infinite.
I would be surprised if it closed on a response … There are streaming protocols that do NOT have a valid size in the headers … and you keep reading data until a known EOM is detected or the other end closes.
Interesting project. I think that it will probably work. Probably, yes, it’s best to do this outside of the LuaUPnP process so that you are not at risk of breaking it with an apparent deadlock. (But still, why not try it inside a plugin first? If you’re writing the code in Lua anyway, you won’t have wasted that code if and when you have to move it to a daemon.)
The thing that’s going to be most tricky is doing it all in a single thread. Running as a daemon doesn’t absolve you of this requirement: Lua is single-threaded by design and doesn’t give you access to fork() either. The UPnP Event Proxy faced this too, faking the two network connections it has to do by cooperatively switching back and forth between listening on UDP and sending notifications to LuaUPnP on localhost3480. For this reason, although you can happily set the LuaSocket timeout to infinite (I think this is the default), I wouldn’t recommend it, because it will prevent other processes (say, a plugin running in LuaUPnP) from talking to the daemon.
If you really, really need to fork a separate process, then doing os.execute(“some-shell-script”) where some-shell-script backgrounds with & does work (it’s what the UPnP Event Proxy wrapper implementation does). But be sure to close your file descriptors manually, because os.execute() doesn’t set the close-on-exec flag, so your shell will inherit all of the parent process’s open sockets.
You may have to sidestep the LuaSocket http.* functions, because they do tend to close the socket after sending the request. http.request() is written in pure Lua on top of socket.tcp(), so you’ve got the code to refer to and can alter your own version as you need to keep the connection alive. Writing cooperative socket-conversation code that successfully avoids deadlock is “fun”. In a single thread, “lots of fun”.
I’ve licensed the UPnP Event Proxy code the same as Lua itself (MIT) so you’re invited to snaffle whatever code from there you like. You may be interested in the XML parsing code in particular. As for the rest, ask me. I’m not a world expert on SOAP, but I think I have a pretty good handle on LuaSocket, and I have HTTP down pat (I wrote a RESTful web service for a commercial product in my day job).
I would just put the process of waiting for asynchronous events in the child LUA process …
When it sees a request … send an event you tour plugin … and go back to listening …
There are multiple techniques to forward an event to a Plugin.
Handle all other IO in the Plugin. If you try to do other things in the child you will be forced into using asynchronous IO requests … and the complexity that comes with it.
I have a lot of thinking and planning to do on this. I appreciate everyones feedback. I wish the ISY controller event api was similar to the upnp subscription where you would subscribe and it would create the connection to notify of the event instead of having a persistant connection. I’ll have to do more reading on the ISY api and most likely email the company with some questions.
What I’ll try and do is see if I can use lua socket to create the connection and issue a subscription and hopefully it can maintain the connection without closing. Once I can get that to work, I’ll work on parsing the soap request and pass that onto the plugin via some means (taking into account of the single thread limitation).
Reading on the Universal Devices forum about the soap subscription connections, I found a post where you can put the address of a tcp socket connection to allow the ISY send events to. This is similar to what futzle is doing in her upnp proxy code. If this in deed works, I can create a tcp socket listening on a port and wait for the ISY controller to send data to and not have to have an open socket all of the time. I’ll have to test this out when I receive the device.
So in the soap request, you would do the following:
Where the url in http://192.168.X.X:9810 would be the lua daemon listening on port 9810 for connections and process the incoming data. Hopefully this will work.
Would the ISY open a connection per event ? Or just to setup a long lasting connection ?
It believe the option (to specify a port) in Soap is so that a managing agent can direct communications to a managed component. You still have a long standing IO … either a read or a socket accept.
To send the Data to Vera … you could use HTTP (Action or Handler) … Vera already is listening for request.
No threading concerns … You can do a pipe … but then you have to have a read on the pipe in the LuaPnp process.
[hr]
If you do not mind a little latency … you could poll the IO Read from the ISY with a read with a timeout … Then you could keep the entire application in LuaPnp.