Expose http client sockets to "luup-plugins" / "requests.lua" namespace

Plugins & Plugin Development openLuup
1

Hello Ak, I’m trying to get socket object from:
“development/openLuup/io.lua”, line 491
to “development/openLuup/requests.lua”, line 787

Please, would you show workarounds to get that socket-object and make scheduler.socket_unwatch() for it in “requests” code? or, if not, maybe, would it be possible to discuss the possibility to provide patch to openLuup that will expose altUI<–>luup sockets" to “openLuup/requests.lua” code (so it will be possible to take control on socket). I can write a patch.
I just need to know for now, if such a patch be acceptable ideologically to the project.

This will allow to provide continuous/delayed response sending for luup requests but to avoid complex “async_send” mechanisms or openLuup-core major modification.

1 Like
2

@akbooer Here is a old implementation approach but We believe with your opinion it is possible to tune it a get it just right.
from GitHub - ronluna/openLuup: a pure-Lua open-source emulation of the Vera Luup environment
openLuup/requests.lua, line 1243:
client_sock = shared.get_current_client_sock()

http server puts clients sockets into global table into “shared.lua”, and requests.lua takes them from there

3

Hi there.

You are trying to get access to the socket related to an incoming data_request?id=request_image call, is that correct?

If so, then be aware that not all requests arrive over TCP, so there may not even be a socket. This would be the case for a luup.inet.wget() request issued by the same openLuup instance to itself (plugins do do this!) So whatever special code you add would have to deal with that case.

scheduler.socket_unwatch() is already available to you in the requests module, so all you need is the socket itself. I’d be interested to know what you’re planning to do with it after unwatching it!

It seems to me, after a quick look, that the simplest way to get at the socket would be to have it passed as an optional fourth parameter to the standard request handler, so the parameter list would look like:

local function request_image (_, p, _, sock)

and you could use this directly to unwatch and do the rest of the I/O asynchronously yourself. You might get spurious “EXPIRED” messages in the log, depending on how you handle the socket and whether the specified timeout for the HTTP server is exceeded.

I think that this would be a straight-forward change which could be in the baseline, and the fourth handler parameter would be available to ALL luup request handlers (including user-defined ones.)

Does that help?

AK

4

@akbooer

You are trying to get access to the socket related to an incoming data_request?id=request_image call, is that correct?

Yes.

I’d be interested to know what you’re planning to do with it after unwatching it!

There are two practical cases I could need that socket for now:

  1. request_image retrieves image from camera for too long sometimes and blocks engine for several seconds during process. In case of many cameras in configuration it become a problem. In this case I need image retrieving “in background” (let engine continue working until image on camera will be ready, transfer data by small portions, not blocking engine for too long, then close it after use for simplicity of implementation. but, maybe it would be good to return socket-to-user to scheduler again)
  2. In several past implementations, request_streaming requested streams from video camera. It captured socket and forwarded data from camera continuously after posix.fork(). In many cases, socket was closed on user side, so there were no need to return socket to luup-engine-scheduler after use. Video data forwarding over Lua-code was inefficient, so we search more efficient (but easy to implement and support) way to forward videostreams for home automation these days. It is no final solution yet. I guess, request_streaming that forwards video stream over Lua-code can be used several more times in future, until good solution will be implemented or as backup to this “good” solution.

It seems to me, after a quick look, that the simplest way to get at the socket would be to have it passed as an optional fourth parameter to the standard request handler…

so simple. let me do this then.

Does that help?

Yes, Thanks. This was almost all info I wanted to get.

If so, then be aware that not all requests arrive over TCP, so there may not even be a socket… [INTERNAL REQUESTS]

Can sock=nil as fourth parameter of request callbacks mean that it is non-TCP request?
or it would be better to place flag
server.current_request_is_internal -- is true or false
or
server.is_current_request_internal()
somewhere in global namespace? What do you think?
In case it will be a flag or is_*() function, in what module will be better to place it?
Or, maybe, it better to place it into 2nd parameter “p”:
p.request_is_internal = true?

5

No need, it’s already in the latest development version v19.11.29

Yes, that’s exactly what it means.

I’ve updated the id=test request handler to show the value of this additional parameter.

This request:

http://openLuupIP:3480/data_request?id=test&foo=garp&output_format=xxx

yields:

data_request:  id=test
output_format: xxx
client_socket: tcp{client}: 0x2197018
user_parameters:
  foo=garp

but runnning this Lua Test code:

local _,x = luup.inet.wget "http://0.0.0.0:3480/data_request?id=test&foo=garp&output_format=xxx"
print (x)

gives:

data_request:  id=test
output_format: xxx
client_socket: nil
user_parameters:
  foo=garp

Enjoy.

AK

1 Like
6

Many Many Thanks!

7

is it the way to disable responding of luup_http_server in “mid-of-request”?

situation is:

  1. there is incoming test request “SERVER:/data_request?id=test&a=1&b=2”. I handle it by modified openLuup/requests.lua/test() function on server side.
  2. I do scheduler.socket_unwatch(client_socket) in middle of body of test() function.
  3. I want to take exclusive control on client_socket since now and send my own response to client’s browser after delay. I tried to return nil. I tried to return “” (empty string) from test() function. but server sends response (even empty response) to client anyway.

My propositions to change openluup (and send patch to github-openluup-dev branch), if there are no mechanisms to interrupt response sending yet:

=========
variant A. I’ll write code somewhere inside openLuup/servlet.lua or openLuup/server.lua that allows to interrupt request by returning
return nil, "dontrespond"
from test() (or from any other “request”) function.

=========
variant B. I’ll add code to openLuup/server.lua/HTTPservlet()/respond() that makes server don’t respond to client_sockets that are already “unwatched”.

8

Give me a chance to look at this.

I would strongly recommend that you avoid any changes in the standard openLuup distribution files, since this will require some degree of effort to keep current during further updates.

As an alternative, I suggest that any functionality you need could/should be added as a CGI file. Some parts of the system are implemented in exactly this way (console, backup, graphite, sysinfo) and it offers full access to all system functionality. You’re then isolated from any (most) system changes. The WSAPI API is an open standard and is guaranteed not to change.

I’m happy to incorporate good proposals, as I have done previously, if they are of general use. The only application I can envisage for this sort of thing is for Ajax-style calls, which are of potential benefit. Let me consider this further.

AK

9

OK. The latest development release (v20.1.29) implements your “variant A”

I’ve not tested it fully, but it’s essentially a one-liner in the servlet module.

So anything (luup request handler, CGI, built-in request) that returns a mime type of “dontrespond” will simply not try and do anything with the client socket. For most clients, this will probably result in a timeout unless the socket is subsequently handled correctly by whatever handler you’ve written.

1 Like
10

Thank you! It works exactly as I wanted on my configuration.

These two upgrades (initial, from topic, and this) give great opportunities to delay tasks and stream data through openluup without losing engine responsivity. Many thanks!

11

Great, glad it works for you.

Any chance you might share a few more details on how you use this?

Certainly, my camera handling needs improvement. Also, as I mentioned, some Ajax calls to give the console pages asynchronous updates.

Always open to good ideas! (as, no doubt, are others here on the forum.)

AK

12

I wrote earlier:

request_image retrieves image from camera for too long sometimes and blocks engine for several seconds during process. In case of many cameras in configuration it become a problem.

new request_image is not ready yet. I tried just tests. but I’m sure it will work fine. Tests cover most important things that will be in new function. I’m going to write it this way:

  1. function will receive request
  2. …then it will catch exclusive control for client_socket using scheduler.socket_unwatch(client_socket)
  3. it will arm luup.call_delay(“bg_handler”, …) for waking in background
  4. and will return nil, "dontrespond" from request_image
  5. then it will be waked up periodically by bg_handler(), and will do next things in background by small parts:
    5.1. it will connect to camera device (waiting in background if need)
    5.2. it will receive http headers from camera (maybe, by parts)
    5.3. it will edit and send modified headers to client (it will strip security sensitive info from camera’s HTTP headers)
    5.4. then it will forward chunks of image data from camera_socket to client_socket, chunk by chunk, falling asleep periodically if blocking engine for too long
    5.5. it will close both sockets in the end

so there will be new [1] socket_exposing_mechanism, [2] socket_unwatch() and [3] luup.call_delay() used to reach the goal. not fast but stable “async/non-blocking” for anything in openLuup can (but not should) be implemented this way, IMHO. Additionally - maybe, universal “pump()” function can be implemented for forwarding chunk from one socket to another and falling asleep from time to time using luup API (as for data in 5.4). This “pump” functionality can be duplicated for different requests, so it could be better to write and call a function.

English is not my native language, sorry if ambiguous words are in answer. I will try to explain, if hard to read.

2 Likes
13

A post was merged into an existing topic: Scene State