Lua XPath and XML Processing libs?

Developer's Corner General Dev Discussions
1

MCV,
Have you given thought to including an XML Processing (Parsing, DOM, and XPath) processing library into the default install?

A number of the Services/Devices that people have asked for (Weather, Calendar, random RSS Feed data) would be trivial to write if these libs we’re included in Vera by default. It would avoid writing a ton of [overlapping] Lua code in their Device defns

There seem to be a few Lua-native, or Lua-wrapped-C implementations floating around.

Once that’s present, it would be simple to build a few lines to extract anything from a General XML feed.

eg. For Weather, you’d run the URL:

http://weather.yahooapis.com/forecastrss?p=89502

and the per-device code could reduce this using a simple (well, ok, not TOO simple) XPath Expression like:

 [tt]/rss/channel/item/yweather:condition/@temp[/tt]  - Extract the current Temperature
 [tt]/rss/channel/item/yweather:forecast[1]/@high[/tt]  - Extract today's High Temperature
 [tt]/rss/channel/item/yweather:forecast[2]/@high[/tt]  - Extract tomorrow's High Temperature

(my Xpath syntax might be a little off)

With the XML Lib, you could then package a simple “XML Processor” Device that, when provided with:

[ul][li]a URL[/li]
[li]a HTTP “method” - GET (default), POST, PROPFIND, or one of the many CalDAV methods[/li]
[li]an optional POST Body (for CalDAV and a few others)[/li]
[li]an XPath Expression indicating what to extract[/li][/ul]

Just a thought… been working with XPath today for work, and this seems like a natural “less code” solution for many of the problems with Web-based data (RSS, CalDAV, etc).

2

Guessed. that’s a good idea. Do you know which Lua XML parser is best?

3
Do you know which Lua XML parser is best?

Well, given I have ~2wks experience in LUA Programming, I may not be the person to comment on the “Best” one…

For XPath, this package has a simple enough interface, assuming folks can get the data using lu_wget (or something like Curl if you need CalDAV request methods)

http://luaxpath.luaforge.net/

and a lot of these different libraries all work on this XML (SAX) Parser, so you’d end up dragging that in…

Account Suspended

and they mention it’ll fit on an Linksys SLUG (NSLU2), so it should be small enough for the ASUS boxes Vera runs on :wink:

The downside is that you’ll pull in a C dependancy that you’ll need to compile up for the platforms.

I’ll leave it to you guys to decide the best one…

4

@MCV: Any further thought to this?

The same underlying library set is needed to achieve what @anker is looking for in this thread:

http://forum.micasaverde.com/index.php?topic=2001.0

5

MCV, it definitely worth the effort as it would pave a way for developing principally new features.

6

I strongly agree! An XML-parser would be great!

7

Add a mantis: http://bugs.micasaverde.com/view.php?id=591

8

luacurl
luaexpat
luafilesystem
luaprofiler
luasec
luasocket

Are already installed. Send me a test program for the lualib that’s not working to see why.

9

CJ,
After looking at Vera’s filesystem (1.0.918), I see only the following

root@HomeControl:/usr/lib# find . -name \*.lua\* -print ./lua/lxp/lom.lua ./lua/mime.lua ./lua/socket.lua ./lua/ssl.lua

the following “includes” work

require("lxp/lom") -- http://www.keplerproject.org/luaexpat/ require("socket") -- http://luaforge.net/projects/luasocket/ require("ssl") -- http://www.inf.puc-rio.br/~brunoos/luasec/0.2/index.html

the following “includes” fail

require("mime") -- part of socket perhaps? require("ltn12") -- http://www.tecgraf.puc-rio.br/~diego/professional/luasocket/ltn12.html

the failure is of the general form:

20 09/18/09 5:37:07.731 LuaInterface::StartEngine 0x7f8bd8 device 0 <0x6017> 01 09/18/09 5:37:07.734 LuaInterface::StartEngine failed run: [string "require("lxp/lom")..."]:4: module 'ltn12' not found: no field package.preload['ltn12'] no file './ltn12.lua' no file '/usr/share/lua/ltn12.lua' no file '/usr/share/lua/ltn12/init.lua' no file '/usr/lib/lua/ltn12.lua' no file '/usr/lib/lua/ltn12/init.lua' no file './ltn12.so' no file '/usr/lib/lua/ltn12.so' no file '/usr/lib/lua/loadall.so' (null) <0x6017> 01 09/18/09 5:37:07.735 JobHandler_LuaUPnP::RunLua failed: require("lxp/lom") require("socket") require("ssl") require("ltn12") <0x6017>

Not sure where we get luacurl, luafilesystem and luasec (and to a lesser extent luaprofiler). Can you elaborate Vera vesion and include syntax?

Also, do you mind creating a Wiki page for Luup Lua Supported Libraries (or similar). This will let folks know what we can rely upon being present in “the platform”.

10

CJ, if you include the LuaXPath lib referenced above:

http://luaxpath.luaforge.net/

then things like the Weather Plugin would reduce down to a few lines of code… with way more efficiency than the cruddy XML parser I hacked into it :wink:

It depends upon LuaExpat which you already include.

11

@CJ:

After trying out some of the socket stuff, there are a few bits missing from the standard [tt]LuaSocket[/tt] implementation.

Specifically, I was trying the [tt]socket.smtp[/tt] module so I could send email and bypass findvera.com’s mechanisms (still getting wild delays here).

Unfortunately it’s missing, along with [tt]socket.tp[/tt] which [tt]smtp.lua[/tt] relies upon.

Both of these files ([tt]tp.lua[/tt] and [tt]smtp.lua[/tt]) can be manually downloaded from the complete LuaSocket source:

http://luaforge.net/frs/?group_id=23

and then manually copied to [tt]/usr/lib/lua/socket/ [/tt] but then it fails loading [tt]ltn12.lua[/tt]. Seems like I’m digging a deeper trench…

It also looks like the option to upload Modules (“Extra Files”) has also been removed from Vera, so it’ll be hard for folks to do this stuff without reverting to the command line.

Was there a reason for excluding the sub-component parts of the LuaSocket module?

Can you include the full LuaSocket module so we can take advantage of some of it’s functions?

12

@CJ, thoughts?

I’d still like to be able to write a Mail (SMTP) Device/Plugin that can be used to replace the one in FindVera.com for Mobile/SMS and Regular Email. I can make it more reliable that way, and ensure delivery. It’ll also avoid the need for any third-party interaction.

13

@MCV, could you tell anything on XPath?

I need to parse XML response, with nesting, and it’s way too complicated to do it manually…

14

We are working on adding support. CJ is working adding this.

15

Any updates?

16

check out pugilua on github

17

This is an old thread, the Lua Expat library has been available on Vera since UI4 or earlier.

18

… But the lua xpath lib seems to be missing.

19

I don’t think the XPath library was ever added.