No worries, I finally found time to finish implementing all the debug messages throughout all the functions. We should be able to isolate it now (more messages in the HTTP calls).
Update: I ran my latest file last night with just the debug messages and very minor corrections (typos, URL updates etc.) to Joey’s code. I’m having issues (code keeps stalling) and it’s probably not related so much to the code but my Vera. I’m really going to have to do this at home on a weekend if I’m to gain any ground.
My initial impression after thoroughly going through his code is lack of persistent storage. He doesn’t store the cookie (string) to a variable within the plugin for repeated use (it’s kept global) and he’s not writing it to a file on our Vera like Google Calendar does this with it’s sem/pem. I can do this either way but will try to store the key along side the credentials.
It seems jlind can get much further than I and it has way more debug messages in it … Very odd, perhaps my Vera3 is can’t handle all the messages… Regardless, we’ll keep dredging forward.
What I found is that the https.request returns nil if an error occurs. I imagine not having a valid cookie is an error. Trying to find something more in depth about this is a little difficult, but I did find this page:
Nil is what’s being returned from the refreshStatus function after a given period of time.
My understanding is that although he’s setting cookies to “keep-alive”, it doesn’t guarantee that the server isn’t going to kill the session. The cookie is on the client side and the session is on the server side. The fact (I tested overnight) that you can reset the username/password on an hourly basis will keep the process running fine is somewhat proof of this. In the SetUsernamePassword function it runs the updateCredentials which sets the variables and then calls retrieveLoginCookie and stores/restores the cookie into the global variable. This logic is fine, but we need to break out the refresh cookie variable and call it additionally when nil is returned from the refreshStatus function. I’m going to try making these mods to my lua file and see if it fixes it.
Any suggestion on how to get this to log to a text file rather then running putty and greping the log?
He has a log function that I could modify and direct all messages through that and in turn just log them to a text file. Will Vera like that, not sure … Also not sure when I’ll get around to doing that if that’s how we decided to proceed. I’m curious though to see if this will run for you, it has a lot of messages that it outputs during the process of making HTTP calls. Mine kept stalling, if yours grabs all the data then I might look into purchasing a new unit so I can test (I can always use another z-wave controller).
Give this a go and let me know if it runs…
[quote=“jlind, post:323, topic:185402”]What I found is that the https.request returns nil if an error occurs. I imagine not having a valid cookie is an error. Trying to find something more in depth about this is a little difficult, but I did find this page:
Nil is what’s being returned from the refreshStatus function after a given period of time.
My understanding is that although he’s setting cookies to “keep-alive”, it doesn’t guarantee that the server isn’t going to kill the session. The cookie is on the client side and the session is on the server side. The fact (I tested overnight) that you can reset the username/password on an hourly basis will keep the process running fine is somewhat proof of this. In the SetUsernamePassword function it runs the updateCredentials which sets the variables and then calls retrieveLoginCookie and stores/restores the cookie into the global variable. This logic is fine, but we need to break out the refresh cookie variable and call it additionally when nil is returned from the refreshStatus function. I’m going to try making these mods to my lua file and see if it fixes it.
Any suggestion on how to get this to log to a text file rather then running putty and greping the log?[/quote]
Do a search in the forums for Info Viewer (not sure if the app store has this), this plugin will allow you (in real-time) to watch the log data. You would basically give it a search term of “>> HYWL:” and it will only display my log messages.
I made a bunch of changes to my current file I’m working from. Give me a bit to work with this one before I switch over.
yeah ,I think the cookie parser is not working properly. I tried the code in a unit test framework i use and it failed to login. I replaced the cookie handling code with what I use (originally given to Joey) and it logged in fine. Hope this helps.
function updateCookieTableFromHeaderString(value,cookies)
– Remove the superflous comma in the date field
value = string.gsub(value, “Sun,”, “Sun”)
value = string.gsub(value, “Mon,”, “Mon”)
value = string.gsub(value, “Tue,”, “Tue”)
value = string.gsub(value, “Wed,”, “Wed”)
value = string.gsub(value, “Thu,”, “Thu”)
value = string.gsub(value, “Fri,”, “Fri”)
value = string.gsub(value, “Sat,”, “Sat”)
value = value … “,”
local next = 0
while true do
_,next,cookie = value:find(“%s*([^,],)“, next+1)
if next == nil then break end
,,cookie,cookieval = cookie:find(”([^=])=([^;]*)”)
cookies[cookie] = cookieval
end
end
also, it is only looking at the first set-cookie header instead of all that may be present. Not sure if this is an issue …
Thanks Mikee, I’ll patch the code I’ve been working on and see if it corrects the issue. I can’t speak to what Honeywell expects regarding the cookie, I’d have to see what others are doing on Github.
yeah, add “return cookies” to the function. ooops. what did you change in the url ? Joey and I started developing a plugin at the same time. I helped as much as he needed. I use my own (which you are more than welcome to). It did not do the auto configure. I have a bit of a Noah’s ark of devices in my system, 2 zwave honeywells, 2 wifis I have mini splits but am afraid of the redlink units used to control them and they are expensive but I’ll probably grab 2 of those as well one of these days.
For this test I left everything exactly as-is (URL’s that is and all his original code), with the exception of your cookie changes (which I adapted to his variables). Here’s what I’m using at the moment.
function updateCookieTableFromHeaderString(cookieString,cookieTable)
-- pass the cookie string that is returned from the lua socket
-- the lua socket returns all set-cookie keys returned as a single comma delimited string.
-- Unfortunately, the contents of some of the values /attribues might also include a comma which
-- makes cookie handling somewhat messy.
-- Thanks to mikee for a crash course on how cookies work!
-- returns a table of name / value pairs
local rawCookie = {}
local curCookie = ""
-- remove all the commas related to dates / expiration dates.
-- ORIGINAL CODE
--cookieString = string.gsub(cookieString,"Sun,", "Sun ")
--cookieString = string.gsub(cookieString,"Mon,", "Mon ")
--cookieString = string.gsub(cookieString,"Tue,", "Tue ")
--cookieString = string.gsub(cookieString,"Wed,", "Wed ")
--cookieString = string.gsub(cookieString,"Thu,", "Thu ")
--cookieString = string.gsub(cookieString,"Fri,", "Fri ")
--cookieString = string.gsub(cookieString,"Sat,", "Sat ")
-- MIKEE REVISION CODE
-- Remove the superflous comma in the date field
cookieString = string.gsub(cookieString, "Sun,", "Sun")
cookieString = string.gsub(cookieString, "Mon,", "Mon")
cookieString = string.gsub(cookieString, "Tue,", "Tue")
cookieString = string.gsub(cookieString, "Wed,", "Wed")
cookieString = string.gsub(cookieString, "Thu,", "Thu")
cookieString = string.gsub(cookieString, "Fri,", "Fri")
cookieString = string.gsub(cookieString, "Sat,", "Sat")
cookieString = cookieString .. ","
-- now we can parse the cookie string by commas
--rawCookie = string.split(cookieString,",")
--local nvPair = ""
--local theName = ""
--local theValue = ""
--for i,v in ipairs(rawCookie) do
-- disregard everything except the first name=value pair of the cookie.
-- v is a single cookie including name, value and attributes.
--nvPair = trim(string.match(v,".-;"))
print (nvPair)
--theName = trim(string.match(nvPair,".*="))
--theName = trim(string.gsub(theName,"=",""))
--theValue = string.match(nvPair,"=.*")
--if theValue ~= nil then
--theValue=string.gsub(theValue,";","")
--theValue=string.gsub(theValue,"=","")
--else
--cookieTable[theName] = nil
--end
--cookieTable[theName] = theValue
--end
-- MIKEE REVISION CODE
local next = 0
while true do
_,next,cookie = cookieString:find("%s*([^,]*,)", next+1)
if next == nil then break end
_,_,cookie,cookieval = cookie:find("([^=]*)=([^;]*)")
cookieTable[cookie] = cookieval
end
return cookieTable
endEdit: I REALLY appreciate the help on this (as I’m sure others do), I have 3 mini-splits in the house which are controlled using Honeywell stats. All linked to a Redlink gateway. I haven’t had time to really focus on this and although I understand Lua, I know nothing of how these plugins are constructed.
no problem at all. we’ll get to the bottom of it. did you get around the 500 error in getDevicesInLocation ? You mentioned you had with a change to the url. I haven’t set up my sniffer in a while to capture the traffic but I can try. Will also check the re-login logic.
Honestly, with as little time I’ve had to really focus on this I’m afraid I’ve been hacking at his code. I’ve also been bouncing back and forth on openLuup and helping some of the users getting their Linux environments setup. With your revision Joey’s original code, we can now work forward ruling any issues along the way. If it’s the URL I’ll just grab the cookie and use Fiddler to submit the request so I can analyze the payload. I got some time to really focus on the core functions he uses (and learned a little along the way) today and it’s impressive for as fast as he turned this out.
For documentation though, the URL change was this [getDevicesInLocation(locNumber)]:
url = "https://mytotalconnectcomfort.com/portal/Locations", -- correction to URL; was /Location/GetLocationListData; now /LocationsThis seems (at the time) to correct the problem but then again it may have just suppressed it…
yeah, it might not be doing what you want. I’ll try a different URL to get the devices and replace the getDevicesInLocation function…
Sounds good and thanks again for jumping in here and adding clarity…
Rebooted my system and noticed the error cleared. Before I bounced, I added a couple of select debugs, things look good so far… Tracking the core functions and assembly of the cookie string.
Will report back once this has been running long enough…
Update: No errors and “Refreshed:11:04:20 AMStatus:Successful” … It’s been almost 12 hours so keeping fingers crossed…
Do you have an updated lua file you would want to share for additional testing?
Nice. I have a fix for the automatic setup. It logs in and gets the devices IDs ok but when it updates the dialog there are two entries for each device. Not yet sure why but am looking at it … only one of each device gets created so it seems to work but I don’t like the appearance of the dialog. It is also overwriting some entries on the screen so is hard to read … perhaps my thermo name is too long ?
The auth cookies seem to have an expiration of a couple days so that is the point at which reauth needs to take place. I think we can handle this with an internal timer that reauthenticates before this time. The way it is setup now it will not try to reauth until a failure during status polling. I fear this can mean failed event triggers until the polling is done so probably best to be proactive.
Anyway, progress …
I like that plan (proactive auth refresh prior to expiry). I am very curious to see what happens when the auth ultimately expires, mine shows the following:
expires=Sun 20-Dec-2015 20:07:37 GMTSo 2:37 PM Sunday, Central Time (CT). As for the automatic setup, I know that Cohibafan was having issues installing the plugin and it ultimately detecting his Wifi stat (id = 7 chars rather than 6). I haven’t had a chance to identify why (yet), wanted to get this stable before then. Might need to shorten the name, I remember Joey struggling with the screens.
Progress is good !
[quote=“mikee, post:339, topic:185402”]Nice. I have a fix for the automatic setup. It logs in and gets the devices IDs ok but when it updates the dialog there are two entries for each device. Not yet sure why but am looking at it … only one of each device gets created so it seems to work but I don’t like the appearance of the dialog. It is also overwriting some entries on the screen so is hard to read … perhaps my thermo name is too long ?
The auth cookies seem to have an expiration of a couple days so that is the point at which reauth needs to take place. I think we can handle this with an internal timer that reauthenticates before this time. The way it is setup now it will not try to reauth until a failure during status polling. I fear this can mean failed event triggers until the polling is done so probably best to be proactive.
Anyway, progress …[/quote]