Cortana for Games

Because Microsoft is still working on porting Azure Mobile Services for Windows 10 Universal apps (you may cast your vote at the Visual Studio User Voice site), I’m going to pause the blog series I started last time and present you with a different topic today: Voice commands in games.

In Windows 10, Cortana can recognize your voice to tell you the time, check the weather for you, look up your next scheduled meeting or simply chat with you. As a developer, you may add as many additional voice commands to Cortana as you like – a great opportunity to increase the retention and lifetime value of your game!

Cortana

In this blog post, we’re going to build a simple app that resembles popular mobile strategy games and extend that app with voice commands that allow the player to ask Cortana for their current gold, last attack or building construction progress.

To get idea of where we’re headed, take a look at this YouTube playlist illustrating our desired results.

Creating The App

1. Launch Visual Studio and start by creating a new Universal Windows App project.

New UAP App

2. Build and run the empty app once to have Visual Studio download any referenced libraries.

Setting Up The Model

Cortana will run as a registered service later, so it can help the player even when the app is not running.

Because we need the game model in both the game itself, and in the registered service, we’re going to create a new class library for the model classes. That way, we can share the same code between both projects later.

1. Add a new Universal Windows Class Library.

New UAP Library

2. In this library, delete the empty Class1.cs code file and add the model classes we’re going to use later.

3. Next, we’re going to write a small serializer class that helps us writing the model to disk and reading it again.

There are two Load methods: The first one takes a file name parameter, opens that file from the local user folder of our app, and converts its XML contents to our model in memory. The second one just builds a default file name from the type of the model class to load. The same goes for the two Save methods.

If you’re unfamiliar with how storage in Windows 10 works, check the Files, folders and libraries section of the official documentation.

Showing The Current Game State

Now that we’ve got our game model in place, we can focus on showing the current game state in our app.

1. Define four view model classes for presenting the model to the player.

If you’re new to data binding, check out the official data binding overview.

2. Write a fifth view model class that combines the four view models for showing them at the main app page.

3. Add texts to the main page, showing the current values of your game model. The namespace specified by the xmlns:viewModels attribute should contain the above view model classes.

 

4. Finally, let’s build a dummy model to show. Add a reference to the FantasyKingdomCortana.Model class library to the FantasyKingdomCortana main project, and fill the MainPage C# class file.

The MainPage class holds a reference to our view model. As soon as the app is started, it creates a new view model and passes it on to the view. After that, it initializes the view model by trying to load any existing data. If none is found, it simply uses dummy values for a start.

Making It A “Game”

Right now, our app is not very game-ish. For sake of simplicity, we’ll keep that part short, but let’s at least add some basic functionality so we can see that Cortana is able to properly access the current game state.

1. In MainView.xaml, add a handler for the Collect button.

2. In MainView.xaml.cs, handle the button click event by increasing the current gold value, and saving the game model to disk afterwards.

You’ll now see that you can click the Collect button to increase your current gold value. Close the app and start it up again, and everything will be exactly the way you left it.

Enabling Cortana

We’re ready to add voice command support to our game. All available commands of our app are defined in a voice command definition (VCD) file. This file uses XML to tell Cortana which phrases to listen to, and which service to pass these commands to.

1. Add a new XML file called FantasyKingdomCortanaCommands.xml to the app project.

New XML File

2. Because the user doesn’t need to wait for the voice commands to be registered, we can do so in an asynchronous method.

3. The voice commands defined in the VCD file should be registered as soon as the app starts. This enables Cortana for our app as soon as it has been started at least once. The bottom of the OnLaunched method in the App class (in App.xaml.cs) is a good place for this.

4. Finally, we need to write the service that handles to voice commands. Add a new Universal Windows Runtime Component project to the solution.

New Windows Runtime Component

5. Add a reference to our model library to FantasyKingdomCortanaService.

6. Delete the empty Class1.cs file and create the FantasyKingdomCortanaService class we’ve registered in the VCD file.

The new class implements the IBackgroundTask interface. Its Run method will be called whenever Cortana recognizes one of our voice commands. In the Run method, we’re fetching a connection to the Cortana voice command service, and get a reference to the voice command data. If it matches any of our defined commands, we should show the user that we’ve heard him or her, as done at the beginning of the ShowTreasury method. After that, we access the data we want to provide to the user, and return the answer.

Note that the background task class itself, and all other classes in the background task project, need to be sealed public classes.

There’s one last thing we need to do now: Add a reference to the service to our app.

6. Add a reference to our service library to the main app.

7. Right-click the Package.appxmanifest file of our app and click View Code. Find our Application inside the Applications element, and a new Extensions element for installing the service.

That’s it! Cortana will now recognize our command for checking the treasury.

CortanaThe commands will automatically be unregistered when the app is uninstalled.

For an example of handling other commands, head over to the public Git repository. There you’ll also find the full implementation to check out and run.

If you’ve made a game with Cortana, and/or are running into any issues, I’d love to hear! Just let me know in the comments below.

Author: npruehs

Nick Pruehs is a Co-Founder of Slash Games, Hamburg. In 2009, he graduated as “Best Bachelor” in computer science at Kiel University. Two years later Nick finished his master’s degree in “Sound, Vision, Games” at Hamburg University of Applied Sciences, becoming the Lead Programmer of Daedalic Entertainment shortly after.

Questions? Comments? Suggestions? Your turn! :)