Voice Automation (Jailbroken iOS or any Android)

Community Playground General Discussions
#1

Hey all,

I’ve had my vera for a long time and it’s been great. I’ve always wanted to take my house to the “next level” and add a level of voice control. I’ve been playing with a few nice microphones and arduino based controllers. I wondered if there was a way I could utilize my phone, which I usually have on me.

I have an iPhone (for good or bad) - you cannot do this with a non jail-broken iOS device (thanks Apple). If you are jail-broken, you can install two Cydia apps. One called “NowNow”; which lets you replace Siri calls to the Google Now API. And a second one called “Googiri”; which allows you to intercept calls to the Google Now API and direct them to a URL of your choosing.

Basically, you ask your phone “ turn on the kitchen lights”. The Googiri app listens for “” (Jarvis in my case, of course) and then calls my custom URL (“https:/somethere.com?q=XXXXX”) where “XXXXX” is the actual string that the Google Now API returned (minus the , so in this case “turn on the kitchen lights”.

If you have an Android device, you can do this with Tasker (see this life hacker article: How to Create a Custom Google Now Command for Anything on Android)

I wrote a really quick/dirty PHP script that takes these commands and parses them lexically to figure out what you want to do by room and device type. It works amazingly well. If anyone wants to try it, let me know and I can either give you the source code to put it up on your own server or even set it to forward requests to your MiOS account for a test run. Trying to re-factor my test jig into a more portable set of modules.

So far, I can:

I. turn lights on/off
II. Set dimmer values (either at a set percentage or adding/subtracting for current value)
III. Run any scene
IV. Open/Close my garage doors
V. Lock/Unlock locks

I even wrote a module to control my now-defunct Logitech Squeezebox systems I have all over my house so I can say “Jarvis living room music pandora station the killers radio” and it just works.

My script just uses the web calls for MiOS to make commands (part of my URL is the user/pw for MiOS). It sends a request to get a list of your devices/rooms and then uses that to figure out what you are trying to say. For example, if I just said “Jarvis kitchen lights on” it would turn on every “light” in the room “kitchen”. If I said “Jarvis kitchen island lights on”, it knows I have a device in the “Kitchen” called “Island Lights” - and it turns that on.

Let me know what you think? I’ve talked to the Googiri developer a couple of times and he’s a really nice guy. He’s working on a version that actually waits for, and parses, the response from the URL to take appropriate action. That would be nice because you could then as a question and have your phone reply via voice “Jarvis, is my garage door open”.

It’s getting there! Hope this gives you guys some new ideas.

Cheers.

#2

I’m not an apple lover for the reasons you stated, but there is alot of apps that have voice commands for VERA on andriod. With a 20-100 andriod tablet mounted on the wall you can make voice comands, view cameras or turn on and of lights, a/c, alarm and more. Better tablets have wireless or QI charging for a removable tablet. Really there are a ton of options. You just have to stop using an overpriced, featureless, closed box phone.

HAL and AutoHomationHD come to mind first for voice command apps.

#3

I have a Samsung Galaxy tab mounted in my lounge, and another one in my kitchen. I have them displaying my security camera, and use autovoice set to continuous listening which is running in the background listening for voice commands. This works fairly well, I even have some of my IR stuff (via SQ blaster) controlled via voice now. The only issue I have, you have to be close to the tablet for it to hear the command. Also autovoice sometimes goes deaf for a few seconds, and does not hear any commands, probably a bug in autovoice. I am using a external microphone in the lounge which has increased the usable range for voice commands being issued. I am still looking for a decent microphone for this purpose. Ideally I would like a Bluetooth or wifi microphone, as where I ideally want it there is no power and I cannot run a cable to it. I do not think that exists, and I do not want to wear a headset in the house… So all in all not perfect yet, but when it works its great.

#4

Unless others can redirect me, I think this is the first post on iphone voice automation. I have gone ahead and loaded NowNow and Googiri and I’m trying to figure out what it offers. In the meantime, I would be interested in yr preliminary php script to give me a head start.

#5

It never really occurred to me to just mount a tablet on the wall :slight_smile: I’ll have to give that a go.

I’m out and about, so this is the most easily available version - it’s really rough. I haven’t even factored it out into objects or function calls - it works so well I didn’t get around to it immediately. I have a newer version that is much cleaner and also grabs all your devices from MiOS (In this version, you just define an array at the top with the things you care about). That is kind of handy because you can define things in anyway you want without messing with your VERA settings. The devices are organized by “Room” and then “Name” (or names). If the “Room” is “*” it will be global and it will run it regardless of what room you specify.

Device Types:

  • Dimmers (urn:upnp-org:serviceId:Dimming1): You can say “on/off” or “XXX Percent”
  • On/Off switches (urn:upnp-org:serviceId:SwitchPower1): You can say “on/off”
  • Door Locks (urn:micasaverde-com:serviceId:DoorLock1): You can say “lock/unlock”
  • HVAC (I wrote a special type for my NEST thermostat and it works great if you have the NEST plugin installed in MIOS)
    • You can say “Temp XXX” or “Temperature XXX” (it will figure out if you are in Heat/Cool mode currently and set the appropriate set point (Heat/Cool)
    • You can say “Increase Temp/Temperature by XXX”, “Up XXX”, “Decrease Temp/Temperature by XXX”, or “Temp/Temperature Down XXX” to increase or decrease the temp relative to the current temp
    • You can say “Set Temperature to XXX” to just set the temperature
    • You can say “Fan On/Off” to turn the fan on and off
    • You can say “Mode Heat/Cool/Auto” to set the operating mode

I didn’t include my Squeezebox control library (a simple telnet wrapper); I will if anyone is interested.

At the top you’ll see the device array - just make it fit for you. In the Googiri settings, under “WEBSERVER ADDRESS”; I have:

https://myaddress.com/jarvis_test.html?u=&p=&sn=<VERA S/N>&q=

Googiri will fill out what’s after the “q=” with the results of the Google Now API call. As I stated earlier - currently it just makes the call to the URL and forget about it; the author is working on dealing with various responses now, which would be great.

#6

Not that this is too helpful now, but pre-iOS 7 I use to use SiriProxy to do home automation via voice control via Siri. It was awesome while it lasted but once IOS 7 came out the siri protocol was too much and the people responsible for SiriProxy couldn’t figure out a way to get SiriProxy to work with the changes.

In case it helps anyone here’s my source code for the plugin i wrote for SiriProxy. It’s written in ruby.

#7

I loved SIRI Proxy - and I did the same thing back then :slight_smile: I didn’t like that all traffic went through it though.

This is a bit better, in that respext. It only sends the request to your URL if it matches the prefix list. I think in a bit we’ll even be able to set multiple URL’s for different prefixes so that you could have two houses, for example.

#8

Try vox commando it’s great

#9

Tried but failed sofar, e.g. When I say jarvis goodnight, google now responds with "here are some pictures, though it is supposed to run a scene. Just wish to exclude typo issues. In your second post you quote: “https://myaddress.com/jarvis_test.html? etc.” Should this not read jarvis_test.php instead? The WEBSERVER Addr in Googiri, should that between quotation marks or without?

#10

I finally got the file jarvis_test.php working. For windows IIS it is necessary to change the beginning of the php schript with <?PHP instead of just <?. Morover, the “p=” Googiri webserver address: https://myaddress.com/jarvis_test.html?u=&p=&sn=<VERA S/N>&q should be replaced with ‘pw’
With no success yet in forwarding command from Googiri, I went ahead and tested the script from the browser URL line. This revealed that the https:// did not work, only http. Curl by default check if the SSL certificate is valid. Quick fix, add this in your options: curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false) However, turning off CURLOPT_SSL_VERIFYPEER allows man in the middle (MITM) attacks, which you don’t want!
If your PHP installation doesn’t have an up-to-date CA root certificate bundle, download the one at the curl website and save it on your server:
curl - Extract CA Certs from Mozilla
Then set a path to it in your php.ini file, e.g. on Windows: curl.cainfo=c:\php\cacert.pem.
Now commands like: https://yrwebserveraddress/vera/jarvis_test.php?u=‘username’&pw=‘password’&sn=‘serialnumber’&q=Goodnight works, amongst other commands, perfectly fine. However, passing the same command via Google Now has failed sofar. Moreover, the voice recognition leaves a lot to be desired unless I’m doing something wrong.

#11

I’m still working on a much cleaned up version that is more modular (Vera being just one plug in for support) - that way you could use this as a gateway to any service you wanted to fill out an interface for. Maybe 50% done.

As far as the voice recognition - it’s been pretty good for me. There have been a few common mistakes that I just decided to account for by using device aliases. What do you mean that passing the command via google now fails??

#12

Thanks for yr response. To clarify “command via google voice fails”, the following applies.
Using the command https://myaddress.com/jarvis_test.php?u=MIOS username&pw=MIOS Password&sn=VERA S/N&q=Kitchen%20Hallway%20On in web browser (IE, Chrome and Safari) perfectly switches on the lights.

Putting https://myaddress.com/jarvis_test.php?u=MIOS username&pw=MIOS Password&sn=VERA S/N&q= under webserver into googiri and ask for jarvis Kitchen Hallway On confirms the message on the screen/by voice but fails to turn the lights on. Hence I suspect the problem is within googiri not properly forwarding the url. I’m looking for ways how to test the performance of googiri to solve this issue.
Hope this clarifies my previous comment.

#13

I understand; throw some logging in the file to output to stderr or stdout and see what the query being passed actually is? I wonder if it’s not being URL Decoded properly; or if it is not even executing the script? Shoot me a PM or something and I can hook you into my test jig to see what’s going on.

#14

Matt (the googiri developer) has recently made some pretty cool updates to this. Between that, and the widespread release of the jailbreak for pretty much every apple device out there right now, I think this is even more useful.

The interface now allows you to return a JSON object that can alter the behavior of the phone when it completes the request. You can now respond with text to populate a dialog box. You can also specify how long the dialog should be visible, what “style” it should use (Message or Error). It also allows you to respond by asking for another question which is great. This means you can say “Jarvis close the garage door,” and the system can respond with a message basically saying I don’t know which garage door you mean, which you are immediately prompted to say.

My first use of this as to make it so my home automation code (Jarvis) can query the state of devices “Jarvis is garage door 2 open?” - and it will tell me. Cool stuff!