Luup Core APIs - a discussion to generate some ideas

[guessed]

I’ve been building a wishlist since starting work with Vera & the LUA/luup extensions, a few weeks ago.

Whilst I’m new to lua, I figured I’d float these out there to start the discussion before we get too far down the road. Some of the ideas might be a little crazy, some might not even be implementable, but I wanted to float them to get the discussion going before things get locked down too much.

For the most part, they deal with solidifying/clarifying existing functionality. I specifically avoided anything adding new functionality, since those requests can be handled in other discussions.

So here’s my list, in order of what I think is needed first:

1. Use a module, instead of a prefix, to contain all of the fn defns in Micasaverde’s space.

You could use something like “[tt]luup[/tt]”, since that goes with your existing naming model and code then becomes

[tt] luup.[/tt].

This would eliminate the need for the current prefix:

[tt] lu_[/tt].

This variable could be established/setup with the appropriate:

[tt] luup = requires(‘libLuup.lua’)
[/tt]
… in the code block that’s generated for the XML defns we upload - similar to how event handlers are generated off of the <[tt]incoming>[/tt] (etc) sections of the XML.

2. [tt]lug_device[/tt] member attributes
This global has a number of attributes with varying naming conventions.

eg. [tt]DeviceType, Category_Num, ID[/tt]

I think these could be standardized to a single convention, likely in lowercase, to something more like:

[tt]room[/tt] - the Room number, or a ptr/ref to a Room structure (description, etc)
[tt]devicetype[/tt] - 
[tt]category[/tt]
[tt]parentdevice[/tt] - currently an int, but see below for improvements
[tt]ipaddress[/tt]
[tt]macaddress[/tt]
[tt]id[/tt]
[tt]udn[/tt]

I don’t mind which formatting convention is used, as long as it’s done consistently throughout. I’m sure there’s a “prevailing wind” in the LUA coding conventions that can be applied here. I chose lowercase, as that seemed common (along with short, often one-word, names)

Personally, I’m not a fan of globals, but I understand this is common in LUA. If this must remain a global, then please change the name (see below for alternate)

3. Make [tt]CallFunctionTimer[/tt], [tt]CallFunctionDelay[/tt] support non-string Parameter types
The current definitions of CallFunctionTimer/Delay limit the consumer to a single “string” parameter.

Remove this restriction, and permit an arbitrary list of parameters to be specified, where any of the parameters may use the full complement of Parameter types (“string”, “number”, “table”, “function” etc).

This will help avoid having code that converts parameter lists into string form and back again for anything complex needing to be scheduled.

4. Model the sub-packages (io, jobs, child-handling)
This seems to be possible in LUA, but if would clarify things if the logical “groupings” of sub-interface were declared/used that way.

This request is a counterpart of (1), covering the sub-module groupings that exist within the APIs.

Given the current interfaces, one could imagine:

[tt]luup.io[/tt] - General IO Routines for interfacing with devices
[tt]luup.debug[/tt] - Logging and any supporting/future Debug hooks
[tt]luup.device[/tt] - Routines for Discovery, and Manipulation, of Vera's Device table.
[tt]luup.job[/tt] - Routines to handle fn and method queuing
[tt]luup.net[/tt] - routines to acquire data over the net (these should be a core/standard "module", not really part of Vera, but pre-loaded for us)
[tt]luup.ui[/tt] - stuff like registerHandler, and any routines to generate "web" output.

There could be others also, depending upon how much you want to separate Device manipulation, such as adding/removing Child devices, from data-transmission, such as setting or getting Variable values from UPnP devices.

Additionally, some of the “[tt]luup.net[/tt]” items ([tt]lu_wget[/tt], etc) could potentially be replaced by more standard OpenSource modules.

5. Model the luup IO apis on the internal lua io api naming
These names can probably be condensed down to:

[tt] luup.io.write()
luup.io.intercept()
luup.io.read()
[/tt]
I’d guess these names will be easier to read than the existing API’s of the form:

[tt] lu_iop_send()
lu_iop_intercept_incoming()
lu_iop_recv_block()
[/tt]
… at least for folks starting off. It also uses (4) to group the sub-modules, to avoid the “[tt]iop[/tt]” prefix.

6. Provide the ability to upload LUA modules, but include some more common ones
I suspect this is the intent of the “Add extra LUA file” option, but haven’t had time to use it.

I’d like to be able to upload “standard” modules like the XMLParsers and Xpath Libs, so that I can then use [tt]require(xxx)[/tt] to load/use them.

If this is the intent of the existing, write the example code to use this to let people know it can be done.

7. Add methods to the Device structure to “lookup” Child Devices
In the event handlers, our code is provided with [tt]lul_device[/tt] as an implied parameter. This is an int representing the “[tt]Device_Num[/tt]” of the device firing the event.

In the Alarm device I’m writing, I have lots of child-devices. Given the [tt]lul_device[/tt] handle, there’s no obvious way to get the [tt]Device[/tt] table of each child device.

Eventually I worked out how to traverse [tt]lug_device[/tt], skipping over all the devices that dont belong to me ([tt]parent ~= lul_device[/tt]).

This operation is something virtually every device writer is going to have to do to get “useful” information (children, or IP address) so it would be useful if you passed us the Device table (entry from the Global table) instead of [tt]lul_device[/tt].

If you did this, you could eliminate the need for a “public” l[tt]ug_device[/tt] variable (or you could add a [tt]luup.device.getroot()[/tt] function)

Additionally, extend “Device” to contain the functions:

[tt]findchild(id)[/tt] - find a specifically named-child, by it's "ID" (spec'd when creating)
[tt]childpairs()[/tt] - return an enumeration of the children, as a table of Device

If you go this route, then Device becomes a true tree-structure, with Parent (device ptr) and ChildList (table of device ptr) type members (and accessor functions)

The current flat-global-list is going to be hard to work with, particularly the more devices it gets in it. Modelling this with API’s should also cut down the lead-time to authoring new devices.

My single [Alarm] device will add ~50-300 “sub-devices”, representing all the sensors of a fully populated alarm system. Anyone with 1-2 children is going to be penalized in lookup by my “fat” device being present in the system

Anyhow, I hope this generates some healthy discussion. I’ve gotten a long way with the APIs as they stand, so most of my items here are intended to make it quicker/faster to learn, so that many more components can be built, by many more developers/scripters.

[325xi]

Great suggestions! As a developer myself, I definitely support the entire list.

It will, however, come to the question whether the required refactoring worth the effort. It definitely does if the idea to OEM the engine takes off… Otherwise I would rather spend resources on marketing - how many developers are currently playing with Luup - three, five?

Besides, once API is published MCV can’t just burn and re-write it - they will have to support both models for backwards compatibility.

[micasaverde]

Thanks for all the valuable suggestions. A couple thoughts…

1. Agreed. Yeah, this is much better. I’ll work on figuring out how to make our addons appear as a module.

2. Hmmm. The way the madness came to be is that in my code I always used mixed case since I think it’s more readble (MacAddress instead of macaddress). So the tags that come from user_data are mixed case. But the UPNP xml makes the first word be in all lower case, with capital letters for each successive word. And some of the tags from javascript come in all lower case. So, in the main database, the tags are inconsistent case-wise, in order to be identical to their source. For example, since the UPNP official tag is deviceType, I thought it might be confusing to a person who knows the UPNP convention if the name within lua was: devicetype because it would be 1 way in the xml file and another way in Lua. So I’m not sure what’s more important: consistency in the case/naming convention (like you proposed), or consistency between with the different architectures (ie that tags from a UPNP XML file use the UPNP naming convention)…

3. Good point. I’ll do that.

4. Agreed.

5. Agreed

6. That is the intention already

7. That’s another tough one… When you get a callback with lul_device (btw, lul_ means Luup local variable vs lug_ means global variable), then there’s no overhead to get your device table, for example:

lug_device[lul_device].IP

But the issue you’re describing is when you want to iterate all your children. At present the structure isn’t nested / object oriented, it’s all flat. For iterating your children I agree a nested approach is much better. However, there are a lot of times when you want to iterate all the devices without nesting. For example, I want to turn on all lights in the same room as my device. Z-Wave devices are children of the Z-Wave interface, Insteon of the Insteon interface, etc. So I’d have to make a recursive function to extract all the lights, as opposed to just iterate the list and do an if type==light and room==myroom. So there are probably cases when a developer needs either depending on whether you’re looking for your children, or all devices that match a given criteria. And, it would be nice to have a hierarchy by device type and room too. If I want to turn on all lights in the house, and the alarm panel has 500 sensors, I don’t really want to have to recurse through trees with 500 devices to pick out the lights.

[micasaverde]

Well now is a good time to make the changes. 3-5 devs is enough to get good feedback, but not too many if we have to change the API. We definitely want to get the API right the first time so we don’t need to have the nightmare of backward compatibility and implementing things multiple ways.

[325xi]

@micasaverde

But the issue you’re describing is when you want to iterate all your children. At present the structure isn’t nested / object oriented, it’s all flat. For iterating your children I agree a nested approach is much better. However, there are a lot of times when you want to iterate all the devices without nesting. For example, I want to turn on all lights in the same room as my device. Z-Wave devices are children of the Z-Wave interface, Insteon of the Insteon interface, etc. So I’d have to make a recursive function to extract all the lights, as opposed to just iterate the list and do an if type==light and room==myroom. So there are probably cases when a developer needs either depending on whether you’re looking for your children, or all devices that match a given criteria. And, it would be nice to have a hierarchy by device type and room too. If I want to turn on all lights in the house, and the alarm panel has 500 sensors, I don’t really want to have to recurse through trees with 500 devices to pick out the lights.

This is just asking for a nice OO design

Seriously, if this becomes the bottleneck, first thing comes to mind is to build supplemental indexes, representing the convenient data structures for fast search and/or conditional traversing, holding just an index of the actual element in the original array. As usual it’s memory vs. performance…

[guessed]

Thanks MCV.

2. Cases consistency.
Generally I like to make it consistent for the consumer, and “hide” some of the implementation mechanic. For this, I’d like to see one of the conventions:

[tt]camelCaseWithXML[/tt] - first-letter upper, with acronyms being all-upper (they're a series of first-letters)
[tt]allruntogether[/tt] - all lowercase
[tt]separated_words[/tt] - lowercase with _ as word separators.

Right now we have a mix, and we have to work it out on a case-by-case basis. This had me flipping between doc constantly - doc that would be easier if “grouped” (and ldoc generated perhaps?)

for the globals, you can follow the same convention. Either the Global is somehow in the package, so it becomes something like the following (since it’s more-or-less a constant):

 [tt]luup.DEVICES[/tt] (plural)  OR;
 [tt]LUUP_DEVICES[/tt] (plural, since it's a list-of)

and the parameter would simply be:

 [tt]device[/tt]  (instead of [tt]lul_device[/tt])

I’m somewhat dyslexic, and was having grief with interchanging [tt]lul_device[/tt] and [tt]lug_device[/tt]. It was funny the first few times I tripped on it…

7. To tree, or not to tree
Both are possible. If you use the LUA “Object” simulation stuff, then you could also lazy-instantiate the children into the sub-tree (by playing with the metatable to do populate-table on demand)

Then you can have both. What I’m mostly trying to avoid here is the effort that each developer will have to go through to “get started” if they have child-objects… and they likely will.

If we can lower this type of barrier, re-factor in some minor ways, then I suspect we’ll get a few more people jumping in to write stuff.

325xi, as you likely know, supporting two API’s in production, with 100,000’s of devices, where they have quite different models is extremely hard to do in practice. If there are only a few users of the existing one, and it’s “quick” to make some cleanup, it would be better to get it over with.

My goal in writing this post was to lower the barrier to entry in performing the common Device-authoring tasks. (Including my other post on XML/XPath)

As a side-goal, once modularized a little, it wouldn’t be that much of a stretch to write a crude simulator with “alternate” implementations for the method calls, something that folks could use directly in envs like Eclipse… (aka a test harness) … but I digress.

[micasaverde]

So I changed the functions/variables as follows. I’m trying to figure out how to do the tree/Object simulation stuff. It’s trivial to just duplicate the object pointers, but I want to try to figure out how to do it without wasting memory. So you have luup.devices[3].children[5].description which will return the same value as luup.devices[5].description and room[3].children[5].description. That way, given a room, parent device, etc., you can iterate all the matching children. The problem is that so far in the C Lua API I haven’t figured how to store the ‘description’ only once. I can easily do it by just pushing the same values over and again and making multiple copies of the same table. But, for example, when you want to push a number/string pair to a table:

int i=1234;
const char *sz=“abc”
lua_pushnumber(L, i);
lua_pushstring(L, sz);
lua_rawset(L, -3); // Set the table entry

I haven’t figured out how to re-use the same actual lua string over and over without duplicating the memory… I also could make the entries be a C API function which does the work and builds the response dynamically, although I assume it’s less overhead to do what I’m doing now, which is to just push all the variables into a Lua table and use pure-Lua code to iterate the table without having to invoke the C api for every iteration. Still trying to figure that one out…

Let me know if these changes work for everyone and I’ll do a new build tomorrow after I do a search & replace on all my existing Lua modules.

lu_log → luup.log
lu_CallFunctionDelay → luup.call_delay
lu_CallFunctionTimer → luup.call_timer
lu_CallAction → luup.call_action
lu_SetVariable → luup.variable_set
lu_GetVariable → luup.variable_get
lu_RegisterHandler → luup.register_handler
lu_WatchVariable → luup.variable_watch
lu_GetDevicesByService → luup.devices_by_service
lu_DeviceSupportsService → luup.device_supports_service
lu_SetCommFailure → luup.set_failure

lu_ir_ProntoToGC100 → luup.ir.pronto_to_gc100

lu_chdev_start → luup.chdev.start
lu_chdev_append → luup.chdev.append
lu_chdev_sync → luup.chdev.sync

lu_iop_open → luup.io.open
lu_iop_intercept_incoming → luup.io.intercept
lu_iop_send → luup.io.write
lu_iop_recv_block → luup.io.read

lu_GetJob → luup.job.status
lu_job_set → luup.job.set
lu_job_get → luup.job.setting

lu_GetXmlNodeText → luup.xj.xml_node_text

lu_wget → luup.inet.wget

And I changed the variables as follow:

lug_device → luup.devices
lug_device[x].Room_Num → luup.devices[3].room_num
lug_device[x].DeviceType → luup.devices[3].device_type
lug_device[x].Category_Num → luup.devices[3].category_num
lug_device[x].Device_Num_Parent → luup.devices[3].device_num_parent
lug_device[x].IP → luup.devices[3].ip
lug_device[x].MAC → luup.devices[3].mac
lug_device[x].ID → luup.devices[3].id
lug_device[x].Description → luup.devices[3].description
lug_device[x].UDN → luup.devices[3].udn

lug_room[x] → luup.rooms[x]

lug_scene[x].Room_Num → luup.scenes[x].room_num
lug_scene[x].Description → luup.scenes[x].description

[guessed]

MCV,
Here are a few minor comments based upon the changes proposed. In general I think they’re a great step to making it easier for scripting folks to read/use.

8. Most of the function names are of the form [tt]set|call|get|read|write_something[/tt], except these:

[tt] variable_set
variable_get
variable_watch
devices_by_service
device_supports_service
[/tt]
For consistency, I think these should be listed as:

[tt] luup.device.set_variable
luup.device.get_variable
luup.device.watch_variable
luup.service.get_devices(urn)[/tt] - I’m guessing the parameters as I dont have the doc handy, added/modelled “[tt]service[/tt]”
[tt]luup.service.has_support(urn, device_num) [/tt]- Again, guessing the parameters…

** Longer term, the “service” notion could also be extended to cover Packages installed, but not yet “instantiated” by the user. They haven’t yet hit “new device” but potentially code could be called upon within the device class to handle things like Automated Device Discovery. In this way, every uploaded “Device Implementation” would get an opportunity (a set of events) for it to participate in DNS Allocation, Serial Bus Device arrival, etc. These hooks could then be used to save the user an install/configure step.

ie. Plug in USB Device, the “Alarm” codebase gets called to see if it should “own” the device, it makes and assessment and then “Creates” a new Top-Level device to handle it. This would be like the auto-install features of most OS’s. If you’re concerned about security, you could put up a “The following Devices are trying to claim message” to let the user act.

Might be a little over the top, but it would save the user a step to get going.

9. I would rename [tt]luup.chdev[/tt] to something more natural.
Perhaps luup.device, which would result in something like:

[tt] luup.device.start
luup.device.append
luup.device.sync
[/tt]
I’m not sure I get what “[tt]chdev[/tt]” means. I consider these like Class-static methods for manipulation of the Global device list. Where the “[tt]device[/tt]” area would contain both methods for manipulation of the Device, as well as ones for manipulation of the DeviceList.

10. What does “[tt]xj[/tt]” stand for in “[tt]luup.xj.xml_node_text[/tt]”
    Also, what does this function do? Guessing, but would a name like [tt]luup.???.get_node_xml[/tt] make sense?

11. “id” attribute naming convention
    In your naming, you’ve used attribute names like [tt]something_num[/tt] when it represents a numerical Id. These are examples:

[tt] room_num
category_num
[/tt]
except for this one:

[tt] device_num_parent
[/tt]
Following the pattern, I think this last one should be:

[tt]parent_num [/tt]("[tt]device[/tt]" is inferred as it lives in the device table)

If you want to go a step further, then the “_num” suffix could be removed, since you always ref by Id. Either way, the lul_device parameter to the [tt][/tt] section, should follow the same naming convention, which would change it to:

[tt] device_num
[/tt]

• luup.rooms
• luup.scenes

Cool, I had no idea these existed!

[micasaverde]

8. For accessors (set/get), I think you usually put that as a suffix, not as a prefix. Also in C it’s the same. When you have several functions that operate on the same type of data, it’s a suffix. For example: you have write/read/printf, and they can operate on a file (f) or a string (s), so it’s fprintf and sprintf. I also personally prefer it as a suffix because it’s closer to what OO is about. If you have a variable, “Color”, you have Color=“red” or Color.set(“red”) or Color_set(“red”). As opposed to: set_Color(“red”)

9. chdev standards for ‘ChildDevices’. I hesitate to use ‘device’ because it’s really children. If ChDev isn’t intuitive, maybe ‘child’ or ‘children’?

10. xj stands for ‘xml+json’. I’ve got to add a bunch of other functions in there for editing the main json database, and retrieving data from the xml files. I’ll probably separate it into ‘xml’ and ‘json’ which functions to walk the tree, set the nodes, etc.

11. While there’s a lot of inconsistencies which you helped correct, in this case there actually is a logic, so let me explain it and see if you concur… Variables are named:

[scope] _ [variablename] _ [modifier]

In C++, Microsoft always puts m_ in front of variables that are member variables to a class so when you look at the variable you know the scope. I think a lot of programming styles use the same convention. So, lul_ is just a prefix meaning this is a Luup Local variable, as opposed to lug_ which is Luup Global. Personally I really prefer having a scope prefix because then, by just looking at the name, you know whether you can set the variable to a value and it will stick when you use it on subsequent calls or not. This is typically an issue when reading C++ code. i think it’s a lot more readable when you have m_ to know that it’s a member variable and not on the stack.

Now as far as the ‘variable_modifier’ syntax, that’s my own convention which I used when creating sqlCVS and database naming. In your device example, you may have a ‘device’ (think a table, or row in a database), which has pointers to several other devices (the ‘parent’ device, the ‘first child’, the ‘sister’, the ‘source’, etc.), and you may have references to another table or set of data. For example, there may be a ‘Person’ table which lists people’s name, and so in the device you reference the ‘author’ (who wrote the device), the ‘tester’, the ‘auditor’, etc.

If the field names (same for variable name) are just: parent, first_child, sister, source, author, tester, auditor then when you look at them, you can’t tell what it’s referring to. Is ‘tester’ referring to a ‘person’, or maybe another ‘device’? Source could be the same. Similarly, if ‘parent’ stands by itself, you don’t know by looking at it what it refers to. If you use the convention variable_modifier, then it’s longer, but clear:

You have a device, and ‘device’ is the ID. So, device_first_child, device_sister, device_source, person_author, person_tester, person_auditor

Now it’s clear that if device_source is 5, that 5 corresponds to a device ID. So, given that ‘device_num’ is the main device id, ‘device_num_parent’ means it points to another device id which is the parent. If the field was just ‘parent’ who do you know by looking at it that it’s another ‘device_num’? Admittedly, this is just a preference and not one that most people use. Most of the time you just see names like ‘parent’, ‘sister’, ‘author’, etc. But I personally find it hard to remember what those refer to and do find it easier if those ‘modifiers’ are appended to the actual piece of information they refer to.

[guessed]

8. Accessors.
   In the languages I’ve seen/used, the set/get goes as a prefix - mostly as a convention, but Java further entrenches this design with it’s introspection methods where it must be of the form:

   setAttribute OR;
   getAttribute

I’ve seen similar in Javascript, PLSQL and C (not C++) code.

This makes things consistent across the board, with everything being basically action-attributeName or action-object in naming model.

btw, I dont find printf particularly natural in a scripting language. I like that Lua uses string.format() which has a more natural name.

For this example:

set_Color("red")

it depends upon how much “convenience” you’re shooting for. If attributes like “color” are already part of the table structure (assuming object-like interfaces in Lua) then there’s no reason to provide a function/method.

Ultimately it comes down to what’s going to be easiest for everyone to understand. This tends to make me stray away from “C-isms”, since they’re cryptic, and shoot for more natural/consistent naming.

check out what AnscaMobile is using for it’s LUA-based Scripted iPhone programming interface. There are some surprising similarities.

9. Child Devices.
   ok, I’ll leave this for now. It might be cleaner to add these methods to the “devices” table itself so we have manipulator methods like:

   devices.open(…)
   devices.add(…)
   devices.sync(…)

then you can eliminate the luup.chdev module altogether by having people work directly with the devices table.

10. xj module
    ok.

11. Naming…

So, lul_ is just a prefix meaning this is a Luup Local variable

It’s more of a parameter, than a Local variable, no? You add this when you generate out the XML into the associated LUA block.

Can I hand-code the impl block, without the XML Wrapper? If I can do this, then what conventions are used will be irrelevant to me since I’ll inline the fully method defn (it’ll also avoid the XML-escaping-while-coding problems with “&”, “<”, “>” etc in my code)

i think it's a lot more readable when you have m_ to know that it's a member variable and not on the stack.

agreed, but this is more of a scripting language. I presume that the scripting audience prefers/wants a simpler naming model. They’ll apply whatever local var naming standards (etc) they want.

You have a device, and 'device' is the ID

I’d argue that most people think of the device as the actual “device” with all the attributes attached to it, but perhaps I’ve been OO for too long.

The deviceId is the C-based “index” to the device within devices table which, in general, I should never need to know … as a scripter, since that’s your datastructure.

again, take a look at the Ansca stuff, since it has a lot of similarities. They appear to have taken an OO-approach to using Lua as their base language.

ok, I’ll stop now, since it’s in a lot better shape. Many thanks for making these changes so far. It was about 1hrs work to cutover my Alarm Module code to it, and it’s certianly easier to read now (it’s about 700 lines of LUA).

I suspect the next “jump” will be to move it to a more OO model.

Are you thinking of this for a future release?

[micasaverde]

Thanks for all the feedback. It’s definitely helped us improve the API.

Are you thinking of this for a future release?

Yeah, but the truth is, right now we want to get the current one released as soon as possible. We’ve naturally stopped all development on the old 6xx branch, and several users are anxious to get going with the new Luup version. So at this point we’re mostly focused squarely on bug fixes and stability. Our goal is a very solid beta that’s usable for the general public by the end of next week.

[325xi]

Could you guys publish a finalized summary on the new syntax, or at least some example, long enough to cover most system functions and variables?

[guessed]

325, if you take the comments from MCV “reply #6” that’s exactly the changes that were made. You can use that to string-replace fn usage (I did) and it’ll all work.

Some of the Wiki doc has been updated - updates seemed to go in over the last 2 days or so according to the Wiki change logs.

Looking at the I_GC100.xml file from the 844 build, it’s been converted over to the new format also. I didn’t check the others, but assume the same there.

If you’re not seeing all these files, then likely something is wrong with the upgrade on your box. I’d “Reset to Factory Defaults” and restart. I’m upgraded now and it’s functioning ok. I’m still having issues with my 2-Serial Port config, per the other thread.

PS: I’m not really using the ZWave side of the house much just yet, so haven’t tried it’s functionality much…

[micasaverde]

There may be a web/javascript error why you don’t see all the files. There are 2 directories with lua/xml files: /etc/cmh-lu and (which is what comes with the stock firmware), and /etc/cmh-ludl (which is what’s downloaded in response to installing new plugins). The web UI may only be displaying the contents of cmh-lu, since before 844 the download plugins on the fly stuff wasn’t working, so all the plugins were put in the static directory. I’ll ask the web developer to look into it on Monday.