Parsing JSON from external source

Developer's Corner Scene Scripting
1

I’m rather new to the Vera3 and trying to parse a valid JSON from a external source.
I searched the forum and found allot of posts about including json libs etc. but i did not found a (for me) working solution.

This is what i tried;

[code]local http = require(“socket.http”)
result, status = http.request(“http://192.168.1.110/websolarlog/api.php/Live”)

luup.log(result)
local json = require(“json”)
local jsonResult = json:decode(result)
luup.log(jsonResult)

local jsonResult2 = json.decode(result)
luup.log(jsonResult2)[/code]

I uploaded a file “json.lua” from http://regex.info/blog/lua/json through Apps->Develop Apps->Luup Files.

I believe the log shows that the json.lua is found and used, but there it could not parse the json.
But i checked the json in JSONlint and its OK.

the logs of the vera3 show:

08 04/02/14 22:31:08.195 Scene::RunScene running 12 powerUsage <0x2e308680> 50 04/02/14 22:31:10.042 luup_log:0: {"0":{"type":"metering","id":"2","name":"Slimme Meter","data":{"id":"1","time":"1396470592","invtnum":"2","deviceId":"2","status":null,"name":null,"type":null,"highUsage":"1006978","lowUsage":"1521106","highReturn":"1739961","lowReturn":"712561","liveUsage":"0000390","liveReturn":"0000000","liveEnergy":390,"gasUsage":"1069464","liveGas":"0","pvoutput":null}},"1":{"type":"weather","id":"3","name":"Weather","data":{"id":"86276","deviceId":"3","time":"1396470000","temp":"15.87","temp_min":"13.33","temp_max":"17.78","pressure":"1004","humidity":"72","conditionId":"803","rain1h":null,"rain3h":null,"clouds":"68","wind_speed":"2.06","wind_direction":"49"}},"2":{"type":"production","id":"4","name":"PowerOne4.2","data":{"id":"763","INV":"4","deviceId":"4","I1V":"125.114975","I1A":"0.013695","I1P":"1.713475","I1Ratio":"54.764","I2V":"117.128502","I2A":"0.012084","I2P":"1.415381","I2Ratio":"45.236","I3V":null,"I3A":null,"I3P":null,"I3Ratio":"0","GV":"223.463928","GA":"0.588602","GP":"17.77169","GV2":null,"GA2":null,"GP2":null,"GV3":null,"GA3":null,"GP3":null,"SDTE":null,"time":"1396463995","FRQ":"50.009003","EFF":"567.993","INVT":"29.482861","BOOT":"27.044428","KWHT":"3422.368","IP":null,"ACP":null,"status":null,"name":null,"trendImage":null,"trend":null,"avgPower":null,"type":null}},"totals":{"production":{"devices":1,"GP":17.77169,"GP2":0,"GP3":0},"metering":{"devices":1,"liveEnergy":390}}} __LEAK__ this:12288 start:1048576 to 0xe7d000 <0x2e308680> 01 04/02/14 22:31:10.042 LuaInterface::CallFunction_Scene Scene 12 failed /usr/lib/lua/json.lua:373: unexpected termination of JSON while looking for object ({ or [ or ' or " or number or boolean or null expected) __LEAK__ this:4096 start:1052672 to 0xe7e000 <0x2e308680>

What is it that i’m doing wrong or could somebody point me to a working solution?

2

I note two things:

One, you are loading the library differently to how the docs say you should.
Your code: local json = require("json")
Jeffrey’s code: JSON = (loadfile "JSON.lua")()
I’m not at all certain that these will leave the same object in the json/JSON variable.

Two, one of these is almost certainly going to fail:

json:decode(result) json.decode(result)
By putting them both in your code at the same time you’ve guaranteed that it will generate an error.

3

Thanks for your reply.

The filename of the uploaded file is “json.lua”, so i believe it should be;

JSON = (loadfile "json.lua")()

i tried both because the doc says to use “;”, but my experience in coding tells me to use “.”.

json:decode(result)
json.decode(result)

is the no out of the box option to get this work or is there somebody how already figured this out?

i now keep getting the following error;

01 04/03/14 8:18:50.663 LuaInterface::CallFunction_Scene Scene 12 failed [string "function scene_5()..."]:142: attempt to call a nil value __LEAK__ this:73728 start:1282048 to 0xf2c000 <0x2be1b680>
But i can't figure out where its going wrong... debugging in LUUP is (as far is i known) very hard....
4

[quote=“freemann”]i tried both because the doc says to use “;”, but my experience in coding tells me to use “.”.

json:decode(result)
json.decode(result)

Your experience is coding is leading you astray. In Lua the . operator and the : operator are similar but not interchangeable.

Your current error is not because of that though. It is because you are assuming that JSON and json are the same variable. They are not. Just as in C, case matters. You’ve never assigned to json, so it is nil.

is the no out of the box option to get this work or is there somebody how already figured this out?

No one who has figured it out is posting on this forum topic.

5

OK i’m getting the “.” and “:”.

Also i indeed made a mistake with the “json” and “JSON” vars and already corrected it;

[code]local http = require(“socket.http”)
result, status = http.request(“http://192.168.1.110/websolarlog/api.php/Live”)

local json = (loadfile “json.lua”)()
local jsonResult = json:decode(result)
luup.log(jsonResult)[/code]

And this code returns the earlier mentioned error.

In /usr/lib/lua i found a file “json.lua” and thats the same file as Jeffrey’s.
Is this a standard file or could this be the file i uploaded (app>developer>luup files)?

Also i check Jeffrey’s json.lua and found that the following error is from his scripts because it in the file;

unexpected termination of JSON while looking for object ({ or [ or ' or " or number or boolean or null expected)

For me this means that the loadfile is working OK and json:decode() is working.
the var “result” is filled with the json (check with a .log()), but decode is not parsing it…

6

Do you have an example of the JSON that you are trying to parse?

7

I missed this earlier. Transferring files through the Lua interfare compresses them, and that’s going to interfere with loadfile(). But this works:

I transferred JSON.lua to my Vera using scp, copying it to /usr/lib/lua/JSON.lua. Then this code:

local json=loadfile("/usr/lib/lua/JSON.lua")() local jsonResult = json:decode("123")
correctly produces the output 123.

Apparently loadfile() on Vera needs a full path. Who knew?

8

[quote=“freemann, post:5, topic:180525”]In /usr/lib/lua i found a file “json.lua” and thats the same file as Jeffrey’s.
Is this a standard file or could this be the file i uploaded (app>developer>luup files)?[/quote]

It’s not on my Vera (1.5.621). Either some plugin you installed put it there, or you are on a different firmware.

9

Json.lua was originally part of the datamine and later replaced json_dm.lua

10

Here is an example of the JSON i want to parse;

{"0":{"type":"metering","id":"2","name":"Slimme Meter","data":{"id":"1","time":"1396510544","invtnum":"2","deviceId":"2","status":null,"name":null,"type":null,"highUsage":"1007585","lowUsage":"1523429","highReturn":"1740247","lowReturn":"712561","liveUsage":"0000000","liveReturn":"0000900","liveEnergy":-900,"gasUsage":"1069716","liveGas":"0","pvoutput":null}},"1":{"type":"weather","id":"3","name":"Weather","data":{"id":"86321","deviceId":"3","time":"1396510200","temp":"10.27","temp_min":"8.89","temp_max":"11.67","pressure":"1001","humidity":"92","conditionId":"701","rain1h":null,"rain3h":null,"clouds":"36","wind_speed":"4.11","wind_direction":"348"}},"2":{"type":"production","id":"4","name":"PowerOne4.2","data":{"id":"764","INV":"4","deviceId":"4","I1V":"320.028259","I1A":"0.776598","I1P":"248.533432","I1Ratio":"20.882","I2V":"305.428894","I2A":"3.083031","I2P":"941.64679","I2Ratio":"79.118","I3V":null,"I3A":null,"I3P":null,"I3Ratio":"0","GV":"225.529861","GA":"5.033944","GP":"1145.149902","GV2":null,"GA2":null,"GP2":null,"GV3":null,"GA3":null,"GP3":null,"SDTE":null,"time":"1396510545","FRQ":"49.998001","EFF":"96.217","INVT":"31.310949","BOOT":"27.342432","KWHT":"3423.165","IP":null,"ACP":null,"status":null,"name":null,"trendImage":null,"trend":null,"avgPower":null,"type":null}},"totals":{"production":{"devices":1,"GP":1145.1499,"GP2":0,"GP3":0},"metering":{"devices":1,"liveEnergy":-900}}}

JSONlints says;

Results Valid JSON

my vera is on firmware version 1.5.622 and i also think that a plugin installed the json.lua.

But i still get errors…

Just looked again to the debug lines i posted earlier;

50      04/02/14 22:31:10.042   luup_log:0: {"0":{"type":"metering","id":"2","name":"Slimme Meter","data":{"id":"1","time":"1396470592","invtnum":"2","deviceId":"2","status":null,"name":null,"type":null,"highUsage":"1006978","lowUsage":"1521106","highReturn":"1739961","lowReturn":"712561","liveUsage":"0000390","liveReturn":"0000000","liveEnergy":390,"gasUsage":"1069464","liveGas":"0","pvoutput":null}},"1":{"type":"weather","id":"3","name":"Weather","data":{"id":"86276","deviceId":"3","time":"1396470000","temp":"15.87","temp_min":"13.33","temp_max":"17.78","pressure":"1004","humidity":"72","conditionId":"803","rain1h":null,"rain3h":null,"clouds":"68","wind_speed":"2.06","wind_direction":"49"}},"2":{"type":"production","id":"4","name":"PowerOne4.2","data":{"id":"763","INV":"4","deviceId":"4","I1V":"125.114975","I1A":"0.013695","I1P":"1.713475","I1Ratio":"54.764","I2V":"117.128502","I2A":"0.012084","I2P":"1.415381","I2Ratio":"45.236","I3V":null,"I3A":null,"I3P":null,"I3Ratio":"0","GV":"223.463928","GA":"0.588602","GP":"17.77169","GV2":null,"GA2":null,"GP2":null,"GV3":null,"GA3":null,"GP3":null,"SDTE":null,"time":"1396463995","FRQ":"50.009003","EFF":"567.993","INVT":"29.482861","BOOT":"27.044428","KWHT":"3422.368","IP":null,"ACP":null,"status":null,"name":null,"trendImage":null,"trend":null,"avgPower":null,"type":null}},"totals":{"production":{"devices":1,"GP":17.77169,"GP2":0,"GP3":0},"metering":{"devices":1,"liveEnergy":390}}} __LEAK__ this:12288 start:1048576 to 0xe7d000 <0x2e308680>

i noticed;

luup_log:[b]0:[/b] {"0":{

and specially the the bold text. Could it be that http.request returns an array and that the var “result” is an array?

I tested this by trying to parse “result[0]”, but then i also get error;

01 04/03/14 11:27:20.963 LuaInterface::CallFunction_Scene Scene 12 failed /usr/lib/lua/json.lua:362: attempt to get length of upvalue 'js_string' (a nil value) <0x2b625680>

my conclusions for so far;

  • the JSON string is valid
  • http.request returns something
  • json.lua is loaded correctly, because its returns errors that are coded in the script
  • it still doesn’t work…
11

Can we consolidate what you’ve done into one post? It’s getting spread thin across the whole thread and while it’s obvious to you what’s still valid and what’s not, it isn’t to me.

  • What does your code look like now?
  • Where are you testing this? (I’m guessing the “Test Luup Code” box, which BTW is not a splendid idea because it has a habit of reporting failure if your code happens to return false.)
  • What are you expecting [tt]luup.log(jsonResult)[/tt] to print?
  • What lines are now being printed in the Luup log when you run your test? There will be more lines than you are quoting, and I think they are significant.

The :0: you noticed is not part of the data. It’s telling you which device (0, the Vera Scene Controller) is doing the logging.

Tip: If you SSH into the Vera you can run lua on the command line. It’s easier to do interactive commands on the command line because you can just run individual print() statements and see the output right away.

Edit: where I’m headed here is that I think that your result variable is being correctly parsed. But when you go luup.log(jsonResult) it’s logging something like “table: 0x7fc270700600” and you’ve edited that out of the log that you posted because it didn’t look significant. Lua won’t pretty-print a data structure for you; you need to either select a specific element to print (jsonResult[“0”][“type”]) or use one of the many pretty-printing functions that will recursively spit out the data structure.

12

ok, here is an summary;

- What does your code look like now?
[code] -- create socket local http = require("socket.http") -- request data result, status = http.request("http://192.168.1.110/websolarlog/api.php/Live")

– load json
json = require(“json”)
– parse json to ?
LogVariables = json.decode(result)
– print ?
luup.log(LogVariables)

– email me the results so i could see whats and when its happening.
luup.call_action(“urn:upnp-org:serviceId:SmtpNotification1”, “SendEmail”, { Recipient_Name=“Marco”, Recipient_eMail=“x@x.nl”, Subject=“json” , Message=LogVariables…" **** "…result }, 30)[/code]

- Where are you testing this? (I'm guessing the "Test Luup Code" box, which BTW is not a splendid idea because it has a habit of reporting failure if your code happens to return false.)
I test it in a scene that i created and run the seen manually.
- What are you expecting luup.log(jsonResult) to print?
anything except an error or nil/null
- What lines are now being printed in the Luup log when you run your test? There will be more lines than you are quoting, and I think they are significant.
see the above code.

about the :0: ;
thanks for explaining!

could i just ran some random lua on the prompt? That would be great, because “save lua”, “confirm”, “save changes”, “reload” and wait for X minutes takes allot of time and make debugging one hell of a job…

13

ok, here is an summary;

- What does your code look like now?
[code] -- create socket local http = require("socket.http") -- request data result, status = http.request("http://192.168.1.110/websolarlog/api.php/Live") -- print result so i could see that the request is handle well luup.log(result)

– load json
json = require(“json”)
– parse json to ?
LogVariables = json.decode(result)
– print ?
luup.log(LogVariables)

– email me the results so i could see whats and when its happening.
luup.call_action(“urn:upnp-org:serviceId:SmtpNotification1”, “SendEmail”, { Recipient_Name=“Marco”, Recipient_eMail=“x@x.nl”, Subject=“json” , Message=LogVariables…" **** "…result }, 30)[/code]

- Where are you testing this? (I'm guessing the "Test Luup Code" box, which BTW is not a splendid idea because it has a habit of reporting failure if your code happens to return false.)
I test it in a scene that i created and run the seen manually.
- What are you expecting luup.log(jsonResult) to print?
anything except an error or nil/null
- What lines are now being printed in the Luup log when you run your test? There will be more lines than you are quoting, and I think they are significant.
see the above code.

about the :0: ;
thanks for explaining!

could i just ran some random lua on the prompt? That would be great, because “save lua”, “confirm”, “save changes”, “reload” and wait for X minutes takes allot of time and make debugging one hell of a job…

14

Yes, that’s exactly what the “TestLuup code (Lua)” box is for.

I’ve just tried your example with my JSON parser (at http://forum.micasaverde.com/index.php/topic,17499.msg137677.html#msg137677) and it successfully returns decoded nested arrays and objects.

15

i found that box, but didn’t understand what is was doing and/or what i could expect from it.
I’m now running some code on the console in LUA and that looks great, simple, fast except that luup functions are not available.

Great that its working with you parser! How could i implement it in my code?

16

How about you take the networking, and the big JSON structure out of the picture, and try to reduce the problem to something trivial.

This code works for me on the Lua interactive shell:

[code]> json = require(“JSON”)

print(json:decode(“123”))
123
[/code]

But if I do this (with . instead of : ) I get an error:

[code]> json = require(“JSON”)

print(json.decode(“123”))
./JSON.lua:266: JSON:decode must be called in method format
stack traceback:
[C]: in function ‘assert’
./JSON.lua:266: in function ‘onDecodeError’
./JSON.lua:519: in function ‘decode’
stdin:1: in main chunk
[C]: ?
[/code]

The fact that you’re not getting that error when you use json.decode() is interesting.

There are tons of JSON parsers out there. Maybe try another one?

Edit: spaces to prevent the forum interpreting my text as a smiley.

17

ok its working!

This is what i did:

Download the json parser of akbooer;
http://forum.micasaverde.com/index.php?action=dlattach;topic=17499.0;attach=13117

Placed the file in the following dir “/usr/lib/lua”

And this is the final script that sends me an email with the value from the JSON that i was looking for.

[code]
---- this codes needs the following plugins:
– eMail Notification (2498)

local endecoder = require(“akb-json”);
local http = require(“socket.http”);
result, status = http.request(“http://192.168.1.110/websolarlog/api.php/Live”);

local jsondecoded = endecoder.decode(result);

luup.call_action(“urn:upnp-org:serviceId:SmtpNotification1”, “SendEmail”, { Recipient_Name=“marco”, Recipient_eMail=“xxxx@xxxx.nl”, Subject=“liveEnergy in W” , Message=jsondecoded[‘totals’][‘metering’][‘liveEnergy’] }, 30);[/code]

Now i could pick any variable from the JSON string that i want.
For example;

luup.log(jsondecoded['totals']['metering']['liveEnergy'])

I hope this helps others and akbooer thanks for writing the parser!