Retrieving and processing HTML pages

Developer's Corner General Dev Discussions
1

Not exatly sure what I’m doing and don’t know quite enough to ask the most pointed question…but here goes anyway.

I’d like to in essence “go to an https address, grab the contents of the returned HTML web page, and process it”.

This is the relevant code:

local https = require("ssl.https")
local webPage = ""
local auth_response={}
local response, status, header = https.request
     {
        url = "https://mytotalconnectcomfort.com/portal",
        method = "GET",
        headers = {
	    ["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
            ["Accept-Encoding"] = "gzip, deflate, sdch",
            ["Accept-Language"] = "en-US,en;q=0.8",
            ["Connection"] = "keep-alive",
            ["Host"] = "mytotalconnectcomfort.com",
	    ["User-Agent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36",
		},
        sink = ltn12.sink.table(auth_response)
      }
webPage = table.concat(auth_response)

At which point I would search through the HTML “webPage”. The HTML status is coming back as “200” which I assume indicates “OK”, but I get nothing in the auth_response table.

Is there something obviously wrong…or is it likely some issue with the request headers?

2

Well, instead of using a table as a sink I changed it to a file for easier inspection.

I am actually getting data back, but it looks to either be encrypted, or otherwise an encoding issue (the file is not readable text).

Any suggestions / pointers?

3

I think I have it sorted…changed Accept to “/” and Accept-Encoding to “plain” and I’m getting expected results so far.