So guys,
I have a project where I log into my OWA (Company email), scrape the page, and have my email read to me in the morning while I have breakfast. Something to that nature.
I have actually figured out a lot so far, when it comes to making https post calls, parsing the correct cookies, and dealing with the redirects. Thanks to Postman, and luatester.
I am stuck on something now I can login and make a successful call however the response body is always encrypted. For the point of this post I have pruned everything down to a simple Google call.
[code]local https = require “ssl.https”
local ltn12 = require “ltn12”
local util = require “util”
local response_body = { }
local res, code, response_headers = https.request
{
url = “https://www.google.com”;
method = “GET”;
protocol = “tlsv1”;
headers =
{
[“Accept”] = “text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8”;
[“User-Agent”] = “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36”;
[“Accept-Encoding”] = “gzip, deflate”;
[“Accept-Language”] = “en-US,en;q=0.8”;
[“Content-Length”] = 0;
};
sink = ltn12.sink.table(response_body);
}
print(“Status:”, res and “OK” or “FAILED”)
print(“HTTP code:”, code)
print(“Response headers:”)
print(response_headers)
if type(response_headers) == “table” then
for k, v in pairs(response_headers) do
print(k, “:”, v)
end
else
– Would be nil, if there is an error
print(“Not a table:”, type(response_headers))
end
print(“Response body:”)
if type(response_body) == “table” then
print(table.concat(response_body))
else
– Would be nil, if there is an error
print(“Not a table:”, type(response_body))
end
print(“Done dumping response”)[/code]
Most of this has been from googling around and I admit I still don’t have a great grasp. The output is very hopeful. A successful 200 call with response headers that are legible. It is the response_body which is jibberish.
Response:
Print output
Status: OK
HTTP code: 200
Response headers:
table: 0x15eb3d8
set-cookie : PREF=ID=1111111111111111:FF=0:TM=1441251094:LM=1441251095:V=1:S=b1ID5JBw3l8ZTjUz; expires=Thu, 31-Dec-2015 16:02:17 GMT; path=/; domain=.google.com, NID=71=2JsP70BMZBg2TyiNBrAmbYd0CNMEtwl6IEzVywtPQecbQIz_8vtad0kvz9pXkRZwql42SMfp4ghU6AbYpTURSCMi8heuJTHUuN4_pr3JeRiIZ9XDCEznTR3OWSLE-ly9; expires=Fri, 04-Mar-2016 03:31:35 GMT; path=/; domain=.google.com; HttpOnly
p3p : CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
date : Thu, 03 Sep 2015 03:31:34 GMT
x-xss-protection : 1; mode=block
content-encoding : gzip
transfer-encoding : chunked
expires : -1
cache-control : private, max-age=0
connection : close
alt-svc : quic=":443"; p="1"; ma=604800
alternate-protocol : 443:quic,p=1
x-frame-options : SAMEORIGIN
content-type : text/html; charset=UTF-8
server : gws
Response body:
????i[?(?}????N6&???? @??q?????8??????oi?-??????s????J?R?T?*???%?V?????{???7cz??U??.g?????kk???a?`;??+]9F==???'g?a0?H???@?PF???>???????zfj;????C?v?3???5??a/3?`t!m@?S?m7??Zf`;z!?m?=?Enf??/w??&?2?????????uk?y??Är?????????S?;z??'\!PQ?f?h ?tWu??)eo???????? ?5J???|?GH??I3???? 9?j[?? ??0?(?G?=?? `???S???F
j????f?v?????8?%g??x???@R?A?c??!T?????M7t??(?????I??c?>e2??K???P??????g{Z ??oO?n????0?T6_??k?7=?0?;7??w??s}?EQ??B?$??zI(??rQ??k?V???$Ib_??y.VD?\??:??J??2????"?V?2)W?I~ ?\??*?5Xz?ZdW??.J??^??s?L?e?2._??*?J%?H?BGY???z?<?J?\?b?^???H??D? LOH? ]$?T?$z%t?k?^e??RE,?k???T?s??AEUz-?k?L?+WJ?\I}Q??k??WX??j?^?ev???J??BW??T??%JO??K?Z.???F?WkRM?B????Z??X?A??_???:tv??Uj???Z?? A?(?K?4??cWw????|?MIP&?bY ??G?q???PeR?????@???\}.<??e<???-*????}??>???s?/??R??{?2e???^?+aP? ?Kn????B??%?A????????r?M?euS????VA?90\mMo9:??aFy???t?\??????c??z??g????9?P2S??,3?`?*???Z? ?/?0???3???3A+%`rC}??D????L.???A4Az?7$???e?L#??.??5? ?bu???S?B?`?fK?3?e???!?K???e#?? t/O??[??gp?y+?-??#???n?$?qz*?4?:??8?4V?9?hn*Kh?Y?i??{6???-/?J2??%?w.???2H)?y???y???~+V?"?@?r??W(?????dEo???P??????s$k@?Zv%?kO>?r6?????Z??e????]V?K?+??b???,???i???Z6f??]?????w?k??;;?E??$ o??&z?`??y/??@AA??o??,??Ny?CS??? ??aX??>?fD{0(:??Z?<???mEY?9?yO ??r???<9??T???????k????-??r???(/?{?+;? Gn??F1???????sA ?eYJU????6?P?s? Q-??: ?8?C?;????*?6?#?U?oza4v??7ezh?T?iS??[????4?3??o?` ?N??;???e??<?? @.??
?6?e>~???a?D????!?#???85?S?R"???%?q9???5?>Ef??????emF??\?os?????TJ????k}??-eyY)?i8(???J?y???50?`*??y?D??8?`?Z???*?Ae?
x? Vu??a???)??r?o??????P
?>?y????O???)w????t??S2:T?QL/}h??,b7????9?(???d???e?????k#n???Z?[?R??H?|8??Oeg?OUP0)KbPnJ4O??#?=a??Ay\??????#??^r?????????5O?L???mj??.?Q???<~Zx?g????Od???/??a`??"??.?~??D"J?"?? ? A)L??4???>?h&?8-?X?1:?????{-?j?N?M|?^_??]???I?!ã@m???p?i?(??IE??1??u?q??????2g?n6O&A??9/???#??=?}w??c????????k?N?-???+????a??U2??????????v??}g??7]?????u??{?^?-a??=<)??)'??V?a??jY???o]?G??,?6?e??4?J??_Fz?7?>=Vj?=?M[`v:?]??M0? ????u?tT???Y?R???SY??V^2???v%m?,????C ;????x L3? "?????
?? b`????c???????DR????$l???P? s?k?f?x?XC41{???????C?v?k?
?
??m?0???gh??@A?p@UW??gz?????Mw?/??z?>o?vp?Yg?????/?????x????0??Z?c?0^??!l?%??(l?t6???????qt?~??`????????????=z=;g?a??(?/??Sv?=?p??0??=?????.??&@??f???????;{???y????Ds??6e?k!)I??e???/"e? ??n??????>e3f???;
?W???R????c???O?(???t?????C??viP@8$?}?v??B????w?X3'XC8wD(?????LK???<??C???5?q=????T=??????#?zn?_?+kx?a???$??dx?g?? :?\WS#X#s??YM??eZ$&???Zx??? ;4?xU:??,?0?? ????????yT=?"?E?O?????#{\^.?{P??=$?H?4<(??2????lKa?Po??*<?/e0??mksy?$??c?R??TZ? ?"\??|\y?"<?A,8??d???
????z??>*x????r?z?\O?B?0? qh????,?O???o?$g?u?????d???ggWG??????>?????????!<UEk??kiP0? ????L)??o??y@?R/?e??<:8?8|?,?.V?????e??????>?????4?}??D?????\ha?D?e1a????
d=?0u ?/? M?A?*]Nx ?.n?J???N37??c???3?ne'a?9'????????bN/ ?s,????(??~?gaFRS ??,??G?
@???i??C?^?g?t??4?????!??|s ??Nf!???o}?UxV ???a??"?C?rR=2???7??6!?0?s?xT?}T?<?c? e?n|?#?=?!?r?EC_Y??C0T_?A
?_~??Y?)?^(????B?r?W?? m?) 8??18?W?%?-?H>?y?s
?Xw '
?|??t??D3?+d:???eg>???9?x????;W?{?{???~uKx8 ?2M??HC?/??4{E????q?X &_?^?o??G??u?????? ?? -#?=??`D: _??a????',?Y??}}???wN?g?=F??<?Py??\^b??/??W?}?w?2TN?^???x???|CY?26??&x??/???{???????C?KK?0X_B-?cA?1G??ehly?? ?|?1?\z??18??????wa3az?$?37? ?qQ?O???a???'?b?3??=?Pa ?hA?V??4a?????A?l???>??=?FU?U? qTPAayT?,/G??G]??8?4?Sr`??Oe?2?? 7-?y?6?a\=????St? ?)????T?RMD?e?a? g??m+`?28?T?2???4???U!?M]d? ??P?????U?b??O?v?? ?J XHa'W{????.Sl?5?u ??{)?z????e#?5??u#l?`??9???(?$??
M??? ?(?(pPa?U?]`?^??}y???-)??d??+??
?l t>^@??@??M?? J?/?V
v"Z??un?g??152??;??an???|u???s???lR??N??>?Uj???=h?????BS?f??Y?r??w?9??%6E?U?5?w?6?d?Z$N??????%^%"2?e??p? ?Q?!?|\7^?????l??G??6???}?:g???jõy?~?M????|D.??p??p(?? ????p?`?? ???T.k?uBl ?{?Q??????&?Qa??¢???t?Z?FdD???X?D ?'s??Bg>?#:?w???2_??H
???g/X[~???????"??~??k?"%?B"?`/??(AhtyY??UC?tH??|l?F??.6.??@Hw??7x(.??'???
? ??%No8;?s???!}????????M r9Dn???|?????<?bh;??Ml]`?i???5??gY???????N*?3]=#ea??>?tv????????8(?Û????w????O ??d|i?PMre}}?$? :J-?fYp???Y???>U?x+P?U???3?.?????\?j9?X????y??Dq????? ??w????-??~/?{??????
?zU15??:?'?[5]e???H&K?+??#S???Ud?10v?i ??2p?|h? k?P4?3A?W?e??M?lAe?0A?s&????????za??2??y???b{?=`?M2???b?C?PA????8j2K?oZACo^?)??`SS?e?c"Kp??]uu?4Z?*?P?F0?$???a???????.h??z?$"m? _a??aZ=?A|?i?p?h?????@5?Z?q?U?|???'??Q?? ??"?U??a?U Kaf???J??????002?%?-FI?f ?{???o??U|y?2B$?A?9v?!??F3<}?11U??dV?v#??B???|?I??f?q???(S??"?A???D ????O#????????:?0?[?p(S|?>V>kx????????G0??bM??9=?D????? 58?1?9?C/?d?Qd]???*??*K?>?$?z??u0s?0[????9?????'?=?}+k ????g?}?^#?HIK?Zk????X!??,??R ?^k ]???ogE?RCz=?4?7Y???"+?R V????????"&??? ?Y?????=R?ie?? ??r?TMe?EV?B?????J?JZb???j?T+?RYA?~??/?"(?`?"?jE' F(2J??d?Wf7?h??|?I?->1???'VjN??????(?"?$?:?tMB?~???????b[????QmH??b?~VT??E? Y??
??WY??J?B?B??0a?
??\0$???
??\ ????(??*?? ?J?2??\[???s??_e)??l????-?R?L??/???b?/f?7?G?[??>6?????sC?????#?-t??? `?:??????2pu?a$D??(?i?CM?5???g?^-?????T????m?????????&~lO7?7pY?nO;p??t????i?8.^??????u??L???????m??/f?=w?b?<?n?zw]????Apr?Nv???Yg?I?????U???b?|,?[?Ú?.n?7?????k?`?w6w?sm_??lN?n?j?V????????????5???kc?x;Z?j?h????zR?qOn???w?]?>W?][?<???rvV<?*a?Q?3;:tN?u?? ?b[???zrX;~~i[?Fw??9???.???g???4??????????D????aGO???Mo{??p??=??g[???t?9?^??y???vq?e????%?j4?O7ow?7?????????{c_??w:???3?)oLz[???r??n?Ok{?+=??h????y8?^?z;j?X?.??F?~A???uwe`????i{e??j???I????YG#?l??O??^???&????9?-?>?O?[?tû~?lW???W}?]{?u?;???????v??v??Wa?????xc??F?n/????@9}-=?f????8??nqz???Y?t??????v?G??lL??Fy??N??=?vq?q????Z??Na???l?;jw?z{?=????n?7e??.z?l?]nnVm???w??+??/??';??]????1?co?O?j?????N_?+?????JW????T???nw?2??va?e????]?Y??5!c?????7?a??Y?v????3??+???????:*?h??^??\?????K???z??????g?QT??s??????FmX?V?o????Fcp?g?{?y>T????????J]????k?????????n{???m>]o??w+k?Y??.?k???l|2????i3{wt??;V??? ??= ? jIy6??2????8??????x?H?ht1e+? %???U??????3???^??? 7S???( ?-B???~???p? ???BX_??~?6Z??hT%?Y???_???????c?J???Y?*a?D?I>?9^'?qga??w ?2??2@o???@?L????2??????D?e ?/??8?????b??<??m.J!$????A?B?>?p?a?g?????
[s(?]JAP?????/?????HxZdaG?gLC???B??U,?5??
?V1?QL#?-P'?Q 2?PL?*????2-???-???????e???jgh2??x?k???H??e=?????g-????$?B_???kh??-?\?PG????z9??%??Ha?_m?*Fz???QX,gI?"k7u?&v??P??pn??????\?"k??Q??G??c?)=???=?4??,?"S )?H????j??d%?
??)?e?e%?0$T???(??R?8???0 ??????t?3~?Fa3??%I ???*K???F|?g?M?W}?l?l?,???7<??j??[Oy? ?????U???b8h ??@o????8?pO?|??? ??bt%????? ???bqKSG????JK??5???M#~?
?D?ZeS??,De??D
?>O???A??g??[???J
?F?s?!???j?]?S???E?+?i7?V??k??Zn?}~?,R? ??b3Q?;??3_1???YkA*7???z?(?I/???5C?k?^?#???<1xô??n??gj??kr??9?2??Sul?6?B??l3??:???G?5.? ?Y?;_?N?lU?m??TIYDM???????z??:?.??#?.F??$??*??O
]# ?J>0??????????eip???j?Z?si??????F??"OH+?d&??a?s??-D?)u?q?W7l;?Hg=?%?s}???Q3??gi??G??d??e??-U?????4S5?"Y???|?N??i???r???L??|?OQ?|??M?w?H\??$??0"??????t?6????N$~?X?3n?}??|b8????H?<??[??0h;?????ae??d???&??`?a?0??|p?1?*tw/e????8?v?nsY????N e????pK|K? >-???Q8%???PL??0>m??a41???ZU?
*???Z?:9_/e#3|!??o?J??D??a`?K?"??[4??Ii?e???? ?e6>
???b?4E?U?A?5????A,???i?~?+q??l?_2?b??Rf??P??6?o?M#?R"t???_????C??(?t{?g??es???l???UB?92?;?@?? )2???'|??\.s?[V?HF|????????_2(?_????*?*? ?BFC??"w??f????s?????a??BY??~ 6-*]??z?,M??+?+z)?^?1??
???\?#@??Y?u?a??@IwK^?^%???d?R??g1????_?V IDY?? ?Y?WH?C%? ?*Y1' GJJ [U??Rm??_OL?u??e?????G???eZ)Rl 8u??m\`?,?????8Gx??0??+Q?;E???1e??2{:?=??S??????4??zb?(Q+;S???r??\???Z?;??{?p|??o0?j|?O?U????v???? r>???Z*7?D?JBr??F?z?D=?1&?$?R??`??v}?Q? ?e?/-`?X?K_q?I8?M??nt?"G ?RT#oxV*?[;?YWL?F?O?V*?WL??~?a??ef?V2?'???????????#5?tV?"?C???W??????k?(?y??_Z??{???g???H?MK+B??7?E??(??51~<?"?F??????4???K?}??q|C?bM*'??j?????l*:?r?Nn4?N???P`H???A<?k0vu-p?(?a???i?!??qF?T*2B~v?R?b?*?%??@?q????[v??M%??*E??i??s??}??:??VU?]g??Kee?Df>??_? :? ?g%??$????Gg??[??SD???*9O ?!?R?.$<???L???d??V$ ??tI/U $?a?_??}6H??8?& ?_?m??Q3??&??[??5(??@?f???X-???????D?@N??C??????8F?IH????^q?0|?(?????D????W???H?jo???b??S(????wu'X~?W? e?6$?'?b/?_??????>?????S???QQ5?^W?|F??|F? _(&?J?Z??z?+??????5??f?d???l $?7?I?e???q?!?E??S#??Ib??1?R ????rR????F?Sl?a?Q?Ve???H??S??L??p?G?? e?e9?????????<?R???T}+??O???i?????q??g??Qd?tI?/???_????????d?Ut??w0???W??G???_7???_??????X`"??m?f!??ZT??Of|??$??8???af??7??
??vM1<??X-???? ?*??B???3?)D???pv9e~E?d??!z??????C?????g½@??.?&??K??X>??#??R??H?&???e_?a??M"?q???J%N?J?$+?U??^?r?[?zX?~v\1?E??]?Gs*?b?gD?)')??{?H?l?Y?1?????/????}???????p~?0ii?????N??c??c???/?/???? ?b?-??AL?5b???8Ew?????a?Ob?09W?<? ?F^?W?{?z?Y5C?Na?'?c< t{?<;?d??1d8?/1Hd????p?Atg|3[0????]Tf???We<?N?H?bEe?fy!?K-???????#s?[??] >?}?wy????^^?W?H?k?[?I5??%????_ P????\?2um!?e??/oG??s??7w?}y?Y??f???4??w?e???U??l??
Any idea on how to decrypt the response body? I have tried changing the protocol to sslv3 with no difference.
Just fyi there really aren’t that many question marks but when I copied and pasted I guess a lot of special characters were lost.
