Project Patrick – build your own personal assistant

Look for my short talk on Project Patrick daily in the Microsoft Marquee.

Setting Up Patrick

PPProject Patrick is a UWP application that can be run on a PC/Tablet/Phone or supported IoT device like a Raspberry Pi 3.

Clone the repository that can be found here on GitHub:






Prerequisites are:

  • Windows OS needs to be 10586 or higher
  • The Speech Recogniser you use across devices will depend on what is installed on the device. We use Adaptive Coding to identify IoT Core from other Windows editions and change the speech recogniser accordingly on line 85. Edit En-GB to read En-US (Line 85) if you don’t have EN-GB on your PC.
  • Use AllJoyn Explorer (Windows Store App) to test connection to the Alljoyn network connection. If you have an Alljoyn network, you need to have a supporting service running somewhere on your network. If you have a Raspberry Pi you can enabled the ZWaveAdapter headless service to do this for you.

The Billy solution uses a key word to identify actions it should interpret and execute. It runs a continuous dictation loop using the Windows 10 Speech Recogniser. On discovering the use of the CallSign the recognised text is sent to the LUIS language model to discover the intent. The query and any entities recognised by the LUIS model are returned with the discovered intent in the service’s JSON response.

You can (and should) define your own CallSign value on line 62. The call sign should be a name that is clear to pronounce, doesn’t rhyme easily, and you find gets recognised easily. Your mileage will vary if you have a strong accent or are using the generic en-US recogniser rather than a cultural specific one e.g. en-GB

Example phrases you can say:

What is the time?

Tell me a joke?

Look up <something, where something is a thing on Wikipedia. Note current speech recogniser build on the raspberry pi insider preview does not capitalise nouns correctly which upsets Wikipedia… try Look up machine learning

What is the weather doing today (will provide weather for Cambridge, UK)

If you have an Aeon Smart Switch on your Alljoyn network you can ask to turn it on.

Turn on the lights

Turn off the lights

Get the lights

Phases can be modified or added to in the LUIS language model utterances.


Raspberry Pi Device Setup

We use a Makibes 5” HDMI screen with our raspberry pi. This provides a nice self-contained device. You need to make the required display Config.txt changes on the OS SD card.

Config file for 5” HDMI screen:

# Video
framebuffer_ignore_alpha=1 # Ignore the alpha channel for Windows.
framebuffer_swap=1 # Set the frame buffer to be Windows BGR compatible.
disable_overscan=1 # Disable overscan
hdmi_cvt 800 480 60 6 0 0 0


Wi-Fi setup – via device default IoT Core start up app. You’ll need a keyboard to enter the network key. Note after connecting to a network on the network list page, you don’t see any indication you have connected. Go back to the home page and see if you have an IP address.

Device Portal – Use the device portal to set up your Raspberry Pi device from your PC. You can set up speaker and microphone volume on the Audio page. You can start up the ZWaveAdapter from the Apps page, and after the initial deployment you can set up the Billy solution to run at start up or when you want to. http:// (IP Address):8080


Note that Windows 10 IOT core requires a proper shutdown before removing power to stop corruption of the SD card.


Creating a New Intent in Billy

It is really easy to add a new service to the Billy solution in two steps:

1)     Define the new Intent in the LUIS.AI model the Billy solution is linked too. See Editing LUIS.

2)     Add a new Intent class with the same name as the Intent added in 1. The common name is the link by which the Billy app interprets what LUIS determined and the class to execute back on the Billy device.

Example – Weather, Time

weather exampleEach intent is derived from intentBase class and IIntent interface. Only the execute method requires explicit implementation. It returns the message string that is spoken at the completion of the task.

 time example

 Rotate Intent

The rotate intent drives a servo that is connected to signal pin 26 on the Raspberry Pi. The Billy Solution includes the PWMService which provides a PWM buss provider on the Raspberry Pi to drive servos. Servos should be wired up so that a common GND exists between the Servo battery and the Raspberry Pi. The servo signal cable is connected to Pin 26. The Servo positive cable is connected to the Servo battery.

Multiple servos can be controlled to create a JIBO like character by using additional GPIO pins on the raspberry pi.

Editing LUIS

The default Billy solution uses a very simple LUIS application that is available for you to import into your own LUIS account service.

Alternatively, the LUIS service provides a Cortana Intent model which can be used. To use this model just reference the Cortana service but be aware that you need to implement the Intent class functions for the built-in Cortana intent model intents – and there are a lot of them. Alternatively, you can build your own simple model, as our example, and then grow the number of intent functions over time.

  • In your LUIS Model, add an intent
  • Enter sample utterances
  • Train your model
  • Publish your model and copy the URL into your Billy Solution


The Billy solution will run and can be delivered on a Windows 10 PC. But is altogether more fun when you build a Patrick device to dedicate to it J






Raspberry Pi ‘Patrick’ materials list:

1)     Raspberry Pi 3

2)     8GB micro sd card

3)     Makibes 5 inch HDMI LCD (A) 800×480 High Resolution with Bicolor Case for Raspberry Pi 2 Model B/Raspberry Pi Model B/B /A

4)     Sienoc Mini Microphone USB connector for PC Laptop Studio voice recording Skype MSN Video

For Alljoyn device control:

5)    Aeon Labs Stick S2

6)     Aeotec Smart Switch Gen5 (UK) ZW075-C07 – Z-Wave Plus


EMF Camp Sumo Competition – win a drone.

ZumoAs part of the Microsoft activities at EMF Camp we are offering the opportunity to win an Parrot AR Drone 2.0 to the winner of the micro-sumo robot competition.

Friday thru Saturday, you’ll have the opportunity to create/modify Arduino Sumo code to create you customer Sumo robot control program. Participants will then be invited to compete in a fun sumo competition Saturday late afternoon.

Physical robots will be allocated randomly for testing and competition. Participants will have a short time to deploy and test to their allocated robot prior to their combat round.

We have only a small number of robots and will limit the number of competitors. Robots will need to be shared during development and testing which will take place at the Microsoft marquee behind Stage C. Look for the Microsoft Village on the EMF Camp map.

Details of the code available to get you started:


Micro sumo robots!

The robot is a Pololu Zumo Robot ( ).

You need to have the Arduino IDE installed on your machine, download it from

Software files are available from the Pololu GitHub:

The basic getting started sumo program is within the examples directory called SumoCollisionDetect

The objective of a sumo match is to push your opponent out of the Dojo ring before the match time runs out. The robots have only minimal sensors making opponent detection possible only by touch.

The basic sumo program provides the key requirements of a sumo player:

1)     Button timed start, and stop

2)     Dojo ring edge detection – keeps your robot within the ring.

3)     Collision detection

4)     Basic battle strategy

1 to 3 are required to participate in a sumo match.

4 is where you can weave your magic! Build a battle strategy to win. This may be a random act strategy (run, fight) or a smart strategy guided by sensor collision data (where did I hit, which direction should I run/fight?).

Our Zumo robots have one additional feature – a Bluetooth adapter. This enables you to pass a telemetry stream to another machine for off device processing and the return of action command.

Unlike formal Sumo robot competitions, we are allowing and encourage off device analysis of the telemetry stream as this represents a realistic IoT scenario in industry J

We will hold a Sumo Challenge competition Saturday look at the online EMF Camp schedule for exact timings.

The winning team will get an AR Parrot Drone 2.0.




Microsoft is a Gold Sponsor of EMF Camp 2016.

We have our ‘Make. Invent. Do’ village at the start of Camping C. You’ll easily spot our marquee.

We have a variety of talks, activities and workshops on offer including:

  • robofloatMicrobit programming
  • Project Patrick: build your own household assistant
  • IoT device and cloud connectivity
  • Micro-sumo robots: building a battle strategy

We’ll have some exhibits as well including:

  • Microbit Connect 4 Robot
  • Microbit Robo-Float
  • Sea Perch Underwater Remotely Operated Vehicle
  • Magic Mirror
  • Microbit Hexapod robot
  • Project Patrick Raspberry Pi digital assistant

The activities are offered as ‘walk in’ tasks, pop in a do as much as you like. Don’t be shy, the Microsoft team will be happy to help you get started or if you get stuck. We are planning a guided microbit programming workshop to help beginners of all ages explore this new device.

Each activity will have a short introductory talk scheduled across the camp days so you can get a broaWP_20160602_10_39_57_Richder understanding of the activity before having a go.

Here are some more details on each activity:

Microbit programming:

The BBC microbit provides a highly accessible Internet of Things device which can be easily programmed using a number of different programming tools, all from your web browser! Explore the programming tutorials using our microbits and then look at the various microbit robots we have to start planning your microbit invention!

Project Patrick:


Project Patrick is a cloud connected speech controlled digital assistant – like Amazon’s Echo. You can easily build your own ‘Patrick’ to run on your PC or your Raspberry Pi 3. All base code and services are available from GitHub. Patrick uses Windows 10 speech recogniser and speech to text voices to listen and respond. It uses the LUIS cognitive service to easily determine the command to execute.  A simple services model means you can easily build your own services and extend your Patrick.

IoT Device and cloud connectivity

Learn how to build, program and connect a Feather M0 to Microsoft Azure’s IOT Hub, a commercial IoT platform designed to support millions of devices globally. Use our Azure IoT Starter Kits to build a simple sensor solution and connect it to the cloud.

Micro-sumo robots: building a battle strategy

UZumosing the Arduino based Pololu Zumo V1.2 robot, come and build a battle strategy using the Arduino coding tools and test it in combat at our mini Sumo Dojo. Our Zumos are Bluetooth enabled so now you can use remote computing power to! Micro-sumo is a fun way to get into real time IoT device solution design. Come play, and test your metal in our sumo competition!