Lua in C++ Windows Store Apps

Many games support script languages to be able to change their logic with having to re-compile: Blizzard allows customizing your World of Warcraft user interface with Lua, for instance, and CCP built EVE Online around Stackless Python.

I’ve been messing around with C++ Windows Store Apps recently, more specifically with DirectX games for Windows Store. It required quite a bit of an effort to get Lua to run (and validate) with Windows Store apps, and I’d like to share my learnings.

  1. Download and install Microsoft Visual Studio Express 2013 for Windows, and create a new Visual C++ Windows Store DirectX App.Create a new Visual C++ Windows Store DirectX App.
  2. Download and extract the Lua 5.2.3 source code.
  3. Copy and include the contents of the Lua src folder in your project.
  4. VC++ does not support C++/CX extensions in files compiled as C. Select all Lua C source code files, and open the properties dialog. In Configuration Properties > C/C++ > General, change Consume Windows Runtime Extension to NoChange Consume Windows Runtime Extension to No.
  5. The precompiled header file (pch.h) for the Windows Store App is C++ and can’t be used from C. With all Lua C source code files still selected, open the properties dialog. In Configuration Properties > C/C++ > Precompiled Headers, change Precompiled Header to Not Using Precompiled Headers.Change Precompiled Header to Not Using Precompiled Headers.
  6. Some of the functions in Lua call system and IO functions that are not supported by Windows Store Apps – but we’re not going use them anyway:
    1. First, we need to disable deprecation warnings for these functions treated as errors. In the properties of your C++ project, in Configuration Properties > C/C++ > Preprocessor, add _CRT_SECURE_NO_WARNINGS to your Proprocessor Definitions.Add _CRT_SECURE_NO_WARNINGS to your Proprocessor Definitions.
    2. Next, in luaconf.h, remove the LUA_DL_DLL define in the #if defined(LUA_WIN) section.
    3. In lualib.c, remove the io_popen and io_pclose functions, and remove the popen entry from the iolib array in the same code file.
    4. In lua.c, loadlib.c and loslib.c, replace all calls to the getenv function by NULL. These functions are designed to handle NULL as a failed call to getenv.
    5. Finally, in loslib.c, remove the os_execute function and its entry in the syslib array in the same file.
  7. In the source file luac.c, remove the main function.
  8. Build and deploy the solution, and observe that validating it using the Windows App Certification Kit succeeds, except for the Branding validation test that requires you to replace the app icon images.

Congratulations, you’re good to go! Go ahead, and take your first steps with the overview of the Lua C API!

References

1. Justin M. Chase. Embedding Lua into a Windows Store app. January 13, 2014.

Unity3D Windows Store – Part 3: Reflections

In the first two parts of this series, we’ve talked about how to build and run a Unity3D game targeting the Windows Store platform, and we took a look at the limitations of .NET for Windows Store apps and how to pass the Windows App Cert Kit tests with our game.

One of these major limitations is the .NET for Windows Store apps Reflections API. Many of the methods you’re used to call for accessing types and type information are not available here, so we were forced to re-write them for our current Windows Store game. In this article, I’d like to summarize our learnings, hoping that I can spare you some headaches in this regard.

Accessing basic type information

In .NET for Windows Store apps, some functionality of the System.Type class has moved to the System.Reflection.TypeInfo class. The official explanation for this from the .NET documentation is as follows:

Type object represents a reference to a type definition, whereas a TypeInfo object represents the type definition itself. This enables you to manipulate Type objects without necessarily requiring the runtime to load the assembly they reference. Getting the associated TypeInfo object forces the assembly to load.

So basically, many things you were used to do with Type objects need to be done with the associated TypeInfo objects now:

 Retrieving all loaded assemblies

One might think that this might be a fairly easy task. Usually, you can refer to the current app domain using the static property System.AppDomain.CurrentDomain and call the System.AppDomain.GetAssemblies method to access a list of all assemblies that are currently available. As we learned in the previous part of this series, there’s a workaround out there on StackOverflow. This approach uses File I/O to find all libraries in the install location of your app, loads these libraries, and refers to the loaded types of those. Obviously, doing so every time results in an immense performance hit, so you’re better off caching these assemblies, which are very unlikely to change during app run-time anyway. Furthermore, trying to load an assembly for which the C++ compiler stripped the relocation addresses, such as Unity dlls, fails with a BadImageFormatException you might want to catch. Our final solution, getting rid of the nasty async keyword, looks like this:

 Finding a type by name

Sounds even easier than the previous one, doesn’t it? We first ran into issues doing so in the general .NET framework when we started using multiple assemblies in a single project. The default Type.GetType method requires you to pass the assembly-qualified name for finding a type. However, in our project, we were

  • storing some type names in XML data files for instantiating these types at runtime, and
  • automatically increasing the assembly version in each build.

Thus, every build would have invalidated all of our data files, because the assembly-qualified name would have changed due to the new version number. Thus, we were using Assembly.GetType instead, just passing the namespace-qualified name of the type we wanted to look up. With .NET for Windows Store apps, finding all assemblies was a problem, but we solved that in the last section. Finally, we wanted to get rid of the version, culture and public key token of all types that were used as type parameters for generic types (e.g. System.Collections.Generic.List`1[[System.Int32,mscorlib, Version=2.0.0.0,Culture=neutral,PublicKeyToken=b77a5c561934e089]]). Doing so with regular expressions was crazy slow on our Windows RT devices, so we’re doing that the good ol’ substring way now:

 Accessing the members of a type

Finally, if you wanted to get the field, property or method of a type, you’ve been using the Type.GetField, Type.GetProperty and Type.GetMethod methods before, respectively. In .NET for Windows Store apps, none of these is available. The TypeInfo class mentioned before offers a method called System.Reflection.TypeInfo.GetDeclaredField, and similar methods for properties and methods, but these don’t look for the specified members in the base types of the type you’re accessing. Thus, you’re forced to recursively look up the member in the base types yourself:

As always, feel free to share your thoughts or ask any questions in the comments below!

Static Code Analysis and you

There are a lot of ways of improving your code quality: Pair programming, code reviews, unit testing, just to name a few. Static code analysis is one of them. While good code analysis tools can cost thousands of euros and will highlight issues in your code you’d have most probably missed otherwise, it is important to understand why these issues can lead to subtle bugs.

I’d like to shed light on some the issues Resharper points out from time to time, and why taking theses issues seriously can help you to write better C# code.

Possible compare of value type with ‘null’

The following code snipped is taken from our A* pathfinding implementation and has led to the R# issue Possible compare of value type with ‘null’:

Strictly speaking, the type T could be a struct that implements the IAStarNode interface. For structs (and other value types), this comparison will always fail, which might be as intended. The clean solution would be to compare to the default value of the specified type:

Assignment to a property of a readonly field can be useless

This can be a tricky (and acutally dangerous) one. Consider the following code snippet:

Resharper will complain about the assignment of the value 42 with Assignment to a property of a readonly field can be useless. Field type is not known to be reference type.

The problem is that readonly fields can only be initialized in the constructor. As GenericClass doesn’t do so, its field will always have its default value. For reference types, this is null, and sooner or later you’ll run into a NullReferenceException – but you’ll find the issue at least. For value types, the result is a little more subtle: In this example, the field will be initialized to the default value of our struct, having its Value property set to 0. While you are absolutely sure of having assigned it the value of 42, this did never happen.

Non-readonly field referenced in GetHashCode

Let’s say you’re working on your own vector struct. At some point, you start writing unit tests, which always fail – you forgot to override Object.Equals. Doing so, the compiler will complain about having overridden Equals, but not GetHashCode.

So you provide your own implementation of GetHashCode:

Now, Resharper will complain about a “Non-readonly field referenced in GetHashCode”. Eric Lippert has written a long and detailed explanation of why this might become a problem for you.

Basically, the GetHashCode method is used whenever you’re putting an object in a hash table.  Hash tables store their items in buckets, trying to achieve an equal number of items in each bucket for faster lookups. They are using the hash codes of these items for indexing into the correct bucket.

Now, let’s say you put your vector into a hash table, for any reason. If you change the values of your vector now, its hash code will mutate, and the containing hash table will fail to ever look it up again: If the hash code of your vector was 42 when it was added to the table, then it was put into the bucket with the index 42. Modifying the vector could make the hash code change to whatever value, say 73 for instance. If you’d want to check whether the table contains your vector now, it would try to look it up in bucket 73, and fail to find it.

Unity3D Windows Store – Part 2: Windows App Cert Kit

In the first part of this series, we’ve talked about how to build and run a Unity3D game targeting the Windows Store platform. We succeeded in building the Unity player, building the resulting Visual Studio solution for the Windows Store game, and running the app on our local machine along with all of our fancy Unity plugins.

Today, we’ll take a look at the limitations of .NET for Windows Store apps and how to pass the Windows App Cert Kit tests with our game.

Supported APIs Test

The Windows App Cert Kit (WACK) has been shipped by Microsoft to enable you to run all of the tests on your app that you are required to pass for being listed in the Windows Store. The kit can be launched for your app right after you’ve ever run it from Visual Studio on your local machine.

The most usual, annoying and complex issues we’ll be facing are the supported API tests. Microsoft prohibits calling certain APIs from your game in order to increase general security and stability of all apps in the store. Most of the violating calls are related to File I/O and reflection.

File I/O

File I/O issues can be solved by defining a new compilation symbol and adapting your project and solution configurations (see part 1). We’ve defined a new symbol windows_store and wrapped all I/O-related code, which was part of our framework but used for Unity editor extensions only, anyway.

Reflection

The more complicated issues are related to reflection. In the Slash Games Framework, we’re following a heavily data-driven approach for configuring and running our games: Almost all entity data, such as hitpoints, damage or speed, is stored in XML files. Whenever a game entity is created, we’re instantiating and attaching the required components (i.e. HealthComponent, AttackComponent) via reflection. Thus, just replacing or wrapping all reflection code with ifdefs clearly wasn’t an option.

Luckily, our reflection code entirely resides in a dedicated library called Slash.Reflection.dll. The few classes of this library were the only ones that we needed to modify. Our most difficult problem was how to access all loaded types in our Windows Store app. As there’s no AppDomain for this target platform, AppDomain.CurrentDomain.GetAssemblies isn’t available, and thus there’s no framework method that allows reflecting all loaded types. However, we’re not the first ones facing that problem, and there’s an acceptable workaround that has been posted on Stack Overflow.

We created a convenience method GetLoadedTypes and provided two  implementations for the different target platforms. As the approach for Windows Store apps requires you to reference other libraries (i.e. .NETCore, Windows), we were forced to create a new project Slash.Reflection.WindowsStore and put our new reflection code in that project. Giving both libraries compiled from Slash.Reflection and Slash.Reflection.WindowsStore the same name, Unity automatically picks the right one for your target platform. This will come in handy later, when we’re implementing Windows Store-specific features such as the snap view or suspending the app.

Performance Test

Then, the performance test was the last one to fail:

Performance launch: The Native Image Generator failed

The Windows App Certification Kit generates native images for all the managed assemblies of your app package. There’s a manual out around at the Microsoft support pages, helping you to identify the file that’s failing, getting information about the failure and how to pass the test.

In our case, the combination of package name, executable file name, and user name was too long: The generator stores the native images in a user directory, which has a long path name because of the fixed parts that it includes. As all of our package and executable names were already very short (i.e. Slash.Reflection), we opted out of automatic native image generation by  including a file named nongen.txt in our app package at the cost of a minor performance hit in our game.

Exporting the App Package

Now that our game was passing all tests, we exported the app package just as explained on the official Unity website. More detailed information about the process of deploying Windows Store apps can be found at the MSDN, but for now, the steps provided by the Unity documentation were enough.

If you aren’t running a Windows 8 build server, you can easily put the exported package in your SkyDrive, access it on your ARM device and install the game by running the Powershell script Add-AppDevPackage.ps1.

Microsoft recommends running WACK  on your device as well, in order to check whether you pass required maximum load times, for example. The Windows App Certification Kit for Windows RT is available at the official MSDN website. Running the kit for our game on the mobile device immediately passed all tests.

In the last part of this series, we’ll talk about how to include Windows Store-specific features, such as live tiles, contracts or toast notifications in our Unity game.

As usual, feel free to share your thoughts or ask any questions in the comments below!

Next: Unity3D Windows Store – Part 3: Reflections

Unity3D Windows Store – Part 1: Building The App

At slash games, we’ve started working on our first very own game project now, and Windows Store is one of our target platforms. While being quite experienced with Unity3D, there were some challenges building our first Windows Store app with that engine, and I’d like to share some of our learnings with you.

If you’re thinking about porting your Unity game to the Windows Store platform as well, take a look at the official Unity porting page.

Initial Situation

We’ve been starting with a more or less empty 2D Unity project. More or less means: While there wasn’t any game logic or other game-related code yet, we planned to use a handful of plugins we’ve been using in our previous projects:

  • NGUI. Probably the best-known UI toolkit for Unity. We didn’t want to wait another five years for the new Unity GUI system, and NGUI served us well in previous projects.
  • FingerGestures. Unity plugin that abstracts from the underlying hardware and detects common input gestures, allowing you to handle touch events and mouse clicks the same way.
  • Spine. 2D skeletal animation for games with Unity support. Allows your artists to create 2D animations just like you would for 3D games.
  • log4net. .NET library that allows logging to different targets and to enable, disable or change the log level without having to recompile the application.
  • Slash Games Framework. We’ve been using and improving our own game framework at slash games for over a year now. The slash games framework is a collection of .NET libraries that provides a lot of features like AI, entity systems, custom collections or math classes for all of our games.

After having added all of these plugins to the empty Unity project, the first thing I tried to do was building the player for the Windows Store target platform, before creating any game objects or scenes.

Building the Unity Player

The first try failed immediately. Verbose log output told me that log4net was relying on a whole bunch of namespaces that aren’t available for Windows Store apps, most of which are related to File I/O:

Error building Player: Exception: Failed to run Reference Rewriter with cmdline –target=”Temp/StagingArea\log4net.dll” –framework=”C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETCore\v4.5″ –platform=”C:\Program Files (x86)\Windows Kits\8.0\References\CommonConfiguration\Neutral\Windows.winmd” –support=”Temp/StagingArea\WinRTLegacy.dll” –system=Unity –dbg=pdb –alt=System.Xml.Serialization.[Temp/StagingArea\log4net.dll]

Error: type System.DBNull doesn’t exist in target framework. It is referenced from log4net.dll at System.Void log4net.Appender.AdoNetAppenderParameter::FormatValue(System.Data.IDbCommand,log4net.Core.LoggingEvent).

Error: type System.DBNull doesn’t exist in target framework. It is referenced from log4net.dll at System.Void log4net.Appender.AdoNetAppenderParameter::FormatValue(System.Data.IDbCommand,log4net.Core.LoggingEvent).

[…]

Error: method System.Boolean System.IO.File::Exists(System.String) doesn’t exist in target framework. It is referenced from log4net.dll at System.Void log4net.Appender.RollingFileAppender::ExistingInit().

As we had experienced similar issues with Unity web players before, there was already a logging abstraction layer in our framework, and a compilation symbol log4net that was used for wrapping log4net-specific code. Thus, for eliminating the above errors, I just had to switch to a configuration without that compilation symbol defined in Visual Studio.

Rebuilding all libraries and giving it another try created a Visual Studio solution for a Windows Store app that you need to build in order to generate the app package itself, similar to developing iOS apps with Unity.

Building the Windows Store Solution

At first, trying to build the solution in Visual Studio failed as well:

Error: DEP0700 : Registration of the app failed. Windows cannot install package MobileProject because the package requires architecture ARM, but this computer has architecture x64. (0x80073cf3)

The solution exported by Unity is initially set to the active solution platform ARM. Opening the configuration manager and switching to x86 (not x64) allowed me to successfully compile the app.

Running on Local Machine

While you could start the app on the local machine now, it crashed almost immediately with the meaningless error message

Template.exe has triggered a breakpoint.

You could step two or three times through mscorlib before the app was killed. After some research I found out that another configuration setting was wrong: For Unity games, you are supposed to build the solution with the Master configuration. Opening the configuration manager again and switching the active solution configuration fixed the problem.

In fact, not doing so would lead to a failed certification test when running WACK for the app:

The debug configuration test detected the following errors:

  • The binary UnityPlayer.dll is built in debug mode.

I’ll go into details about WACK and how to pass all tests in the next post of this series. I’d like to add that I highly recommend checking in the Visual Studio solution into version control at this point. Re-building the Unity player will replace the Data folder of your solution, only. Thus, the other files, particularly your manifest and any native code you write can be stored in your Git repository (or whichever VCS you prefer).

Feel free to share your thoughts or ask any questions in the comments below!

Next: Unity3D Windows Store – Part 2: Windows App Cert Kit