Engineering Game Development

Unity builds. I don’t like them.

Of all the tools at your disposal to make a build faster, this is the worst. And it’s not just the “hey let’s #include .cpp files” weirdness, but the way that it can make a well structure, modular code base become worse than spaghetti junction at rush hour, and the worse thing is that it’s not the fault of the programmer, especially when something as exceptional as Visual Assist can start helping you create non-modular code because of the way Unity Builds collect the code you write.

What Is a Unity Build?
Unity Builds have nothing to do with the excellent Unity Tool Set. I’ll just clear that up right off the bat.

These Unity Builds are a way of reducing build over-head (specifically opening and closing files and reducing link times by reducing the number of object files generated) and as such are used to drastically speed up build times. I say drastically because they are usually used after a code base has starting generating build times that totally cut down on a programmer’s productivity. This might be 10 minutes, 30 minutes or even hours.

Because Unity Builds give you a quick gain, it’s seen as a pretty magical solution, and when a full build is reduced from 30 to 3 minutes you can see why. But the caveat in that statement? Full builds.

I won’t go through how to generate Unity Builds here as the blog post The Magic of Unity Builds does a great job so have a look there. It’s interesting that I’m linking to a blog post that is in the total opposite direction that I’m coming from, but unless you know both sides of a tale, you don’t know the tale.

So without any more delay, why don’t I like them?

Say Goodbye to Local Properties
Well written code is modular, and modular code relies on a few assumptions. One of those being that a single translation unit (a cpp file and any associated include files) is isolated from other translation units, so (unless you extern the contents) anything defined within a cpp file is not available outside, and as a consequence you can have local objects being called the same things without conflict.

Take for example the following simple code (easily expanded to a real world example I’m sure)

// In VectorTest.cpp
namespace
{
   const uint StandardTestCount = 10;
}

// In ListTest.cpp
namespace
{
   const uint StandardTestCount = 3;
}

In a normal compilation environment these variables are local to the file they are working in, and this is a good thing. Why should the vector tests file care what is defined within another test file?

But if you have a Unity Build with the following then you’re going to have issues…

#include "VectorTest.cpp"
#include "ListTest.cpp"
#include "ArrayTest.cpp"
#include "QueueTest.cpp"

Specifically errors relating to the variable ::StandardTestCount already being defined. So we now have to be aware of what is defined throughout the entire project and resolve any issues, and inevitable this will end up with all variables being pre-appended with (in this example) VectorTest_ and ListTest_ etc.

I don’t want to refer to the ‘Magic’ post to much, as it’s a good article, but there is a statement in there referring directly to this. Specifically the following

“Technically you shouldn’t have this problem if you’re writing “proper” code”

This is wrong. “Proper” code is well structured and well maintained, meaning it’s modular and only refers to what it needs to refer to. In a lot of cases you will have variables with the same name, functions with the same name and other elements that you naturally write as you go along. That’s why the anonymous namespace (or the C style ’static’) is so useful, and so useless if you switch to Unity Builds.

Using Is (even more) Lethal
Never use the ‘using’  keyword in a header file. It’s a standard statement and one that stands up easily. The reasons behind this are pretty clear (declaring ‘using’ in a header file forces all code including the header file to use that statement – in effect wiping out a namespace).

But using it in a .cpp file isn’t a crime. In fact it’s remarkably useful, even within the whole files scope. As a developer, I should know what I’m using in a local file, what elements I’m bringing into the file and what would/could conflict. Some people might not agree that ‘using’ at the top of a cpp file is a good idea at all, and I see their point, but on a whole file scope, it can be massively useful and as it’s localised it’s not going to pollute anything other than the file it’s in.

But in Unity Builds, using (that is not scoped within a function or type etc.) is lethal. Declaring ‘using ‘ in a cpp file makes it available throughout the code base (or more confusingly only in those files that are included after the one declaring it). Suddenly, using is everywhere. And your namespaces are now useless.

Every Change is a Rebuild
This is one of my biggest problems with Unity Builds. In well structured projects (more on that below) changing the content of a cpp file (or even a local header file) shouldn’t require everything to be rebuilt. Every. Single. Time. Yes, the unity build rebuilds are faster than doing a full build without a unity file, but doing even a fast rebuild every time will build up. Quickly.

When-ever I change a .cpp file, all I need to build is that .cpp file. Granted, link times are a tad long because it still has to re-link the other object files, but it takes a second to compile that one file. On average (I took count today) when I changed a header file it compiled on average 5 .cpp files. And it (again on average) took about 5 seconds to build.

Very rarely should I be required to re-build the entire project, and most of the time I’m don’t. And that saves me a lot of time. Every single day.

Multiple Configurations
This is mainly a bugbear of mine rather than a direct issue with Unity Builds, but I see it in nearly every Unity Build project I use. The main project is the Unity Build project, but there is another project that is built and maintained that doesn’t use Unity Builds. Now there is a point here – by having an additional ‘normal’ project you are forcing the modularity that can collapse with Unity Builds to be checked (usually a Continuous Build server will be building this as well as the Unity Build every time).

But we have problems with this.

Firstly, the non-unity build is only ever being built on the CB server. So any problems are going to break the build, and it will break if people are not using it day-to-day. Secondly you now have multiple projects to maintain. Not too much of a problem if you have an automated project generation step (see below) but it is still another project to maintain.

People may occasionally have to use the non-unity configurations, especially if they are getting an unknown error on the CB server. So now they are left working on a configuration that is uncared for and probably builds so slowly and erratically that they are probably losing all the time they saved from those quick unity builds they have been doing all day.

But What About My Build Times Then?
Well structured software builds quickly (or as quickly as they can anyway). But what is a well structured project when it comes to build times?

• Sensible file inclusion – Only include the files you need and try as best you can to limit them to .cpp files. This means the compiler only needs to bring in what’s necessary and when you change one of these header files only those things that directly rely on it will change. If you find yourself having to include files that constantly change into a large number of cpp files, then I’d wager that a refactoring of the large header file would be in order. You should be building only a small number of cpp files when you change the content of a header file.
• Forward Declarations – Where possible use forward declarations rather than including the header file in your class or function declaration. Annoyingly you cannot forward declare enums (on standard compliant compliers anyway) which sometimes throws this out of the window. But by not including the files until use, you’re cutting down on the amount of code being included and the number of files being opened.
• Pre-compiled Headers – Use Pre-compiled Headers (PCH). Using PCH’s is the one (built into the IDE) feature that will cause your build times to plummet, especially if you are being sensible with them (such as including files that don’t change – SDK’s and common, fixed headers for example). Using pre-compiled headers across multiple platforms can be a pain, but it is possible to unify them and get a massive boost. I’ll cover these a little bit more below.
• Library Files – Modular code usually leads towards easily extracting common code into a collection of libs. Reducing the amount of code in a project (and as a result how much you need to build) can speed up your productivity quickly.
• Parallel Building – If you’re IDE supports it (and it might do and you just don’t know) and you have a multi-core machine, turn on parallel building of files. Building individual files at the same time is obviously going to cut down on your compile times no matter how quick they are at the moment.
• Get a Better PC – It goes without saying (but I will anyway) doing this will speed everything up.

Pre-Compiled Headers
Pre-compiled headers are one of the best ways to get a massive boost to your compile times. So how fast are my build times compared to that of a Unity Build version when using PCH’s and taking into account the other build improvements suggestions?

As stated above the ‘average’ build time of a minimal build throughout the day was around 5 seconds. On a Unity Build it was a full build every time and was around 2 minutes on the slowest platform.

Changes to ‘core’ files, which are included by a higher than average number of files, resulted in builds of around 30 seconds on around 20 files. Again on a Unity Build this would have been around 2 minutes regardless.

Full rebuild (which I did twice today) was around 3 minutes. Granted a whole minute slower than a Unity Build but I did it twice rather than every single time.

Pre-compiled headers are not a silver bullet solution. Nothing is. And because of this here are issues that you do need to be aware of

• Compiler Support – Some compilers out there simply do not support PCH’s. On a daily basis I use between 4 or 5 different compilers and while I’ve never encountered one that doesn’t, they are out there. This can shoot my argument in the foot, but it is a rare problem and one most people won’t encounter.
• PCH Quirks – While I use a number of compilers that do support PCH’s, every single one of them has a slightly different way of building them and slightly different requirements before they can be used. This doesn’t affect the code that is written but does affect the content of your project, especially if you want to make them seem as consistent as possible.
• Over-Inclusion – Because your PCH usually includes common files and files that rarely change, it does mean that some files are being brought into the project in a compilation unit that wouldn’t otherwise be required

Unity builds are a solution for the wrong problem. Long compile times are not caused by not using Unity Builds, they are the result of having badly structured and badly referenced code. Fix that (and have better code to boot) and you’ll be able to use ‘proper’ build tools without resorting to a quick fix.

Making Unity Builds Better
I don’t want to just leave it at that, because no matter how much I argue, Unity Builds will always exist and people will always use them (after all it’s quicker to rescue a broken down build by making it a Unity Build than doing it the hard way). So what can people do to actually make Unity Builds a bit more desirable and less of a problematic fix?

• Automate Adding Files – A lot of teams have auto project generation code already (XML files, template projects etc.) so it’s important that you automate adding files to the project, otherwise people will forget to remove the file from the build step and they will forget to add it to the unity build file.
• Multiple Unity Files – Running a Unity Build doesn’t require you to have one unity file with every cpp file inside it. You can have multiple build files (usually per module or per component) which means at least some of the translation unit leaking is limited to each module rather than the whole program.
• Additional Project – No, this isn’t a contradiction from the above “Multiple Configurations” comment above. In this situation you will have a project that contains the cpp and header files so you can reference them but this project isn’t built. Instead, you have the ‘proper’ project simply contain the unity file(s). This isn’t something I’ve personally tried, but it does get around the issues of adding files if you don’t have an automated step.

I’m not one for bringing up promotions from companies but it was brought to my attention the other day that SmartBear are offering 5 licences for their Code Reviewer tool (a more light weight version of Code Collaborator) for the tiny sum of $5. And in the way that marketing departments like things to flow together, they are only doing this for 5 days – from July 13th to 17th. 5 licences for $5, only for 5 days.

I’m a fan of SmartBear’s tools, I’d go as far as to say that Code Collaborator is one of my favourite code review tool out there, but to be honest what makes me want to mention this is that they will donate all the money raised from this promotion to a start-up to help them keep their heads above water (as of writing this it’s not been announced which one).

They are also providing a mid-week Webinar that will step you through how to use Code Reviewer, so check that out if you need a bit more information about it.

Anyway, if this might persuade you to give code reviewing a go or could help you formalise you process a little bit more, then check it out and see if this is right for you.

I was going to write a couple of reviews covering the various code reviews tools I’ve tried over the past year or so, but the notes I made on each one seem to have mysteriously vanished. And since it wouldn’t be fair on the various tools we used to try and remember why we moved on, I’m just going to cover the one tool we stuck with… ReviewBoard.

Web Based
This was one of the features that I really wanted from a review tool. No installs, no applications that need to be running in the background, simply a web address to access and an e-mail address for notifications.

Easy To Use
When you first look at the UI it’s pretty obvious what you need to do and how you can review the work in front of you. As with any UI, it has it’s quirks but I’ve not had anyone say “I can’t use this!” like I did with some other tools we tried.

Conversations
It’s easy to have a conversation in ReviewBoard and this is what I always want to see when code is up for review. Clicking on other people’s comments and seeing a clear ‘Reply’ button and being able to see the entire thread on the main page allows people to dive right in.

Openness
ReviewBoard doesn’t block people who are not on the review from seeing the code and making their own comments. As long as they are signed up to the site they can see anything being reviewed and what has been reviewed in the past. If I’m writing an internal blog post about a feature or implementation I am working on, I always add a link to the code review allowing everyone to have their say. After all, there is a good chance that they might use the code at some point in the future, so why shouldn’t they?

Quick to Review
Once you get notified of a review you have been assigned, getting to that review is as quick as hitting the provided link and selecting the diff. to review. Everything is done in-page and with a click on a button your entire set of comments and reviews will be published for everyone to see. It can be hard to persuade people that reviewing code will make them more productive, and having a fast process can make the argument easier.

Actively Developed
Nothing is perfect and ReviewBoard does have its bugs, but when you can raise an issue on their bug tracker and know that someone will actually look at it you start to feel a little more secure about any problems you find.

Widely Used
Having only one group of people use a tool can sometime be a benefit as it can be tailored specifically for you, but that can be a dead-end. ReviewBoard is widely used by some pretty big companies and you know it stands up to a certain level of quality and that the development being done will be for a wide range of users.

Stable
I’ve used tools in the past that didn’t work. They would fall over, become unresponsive or simple not do what they should. I’ve not encountered any of that with ReviewBoard. A few people using it have found some issues that get in their way, but as I’ve already mentioned these are quickly noted and will hopefully be resolved in the future.

So those are some of the main reasons we’ve stuck with ReviewBoard. It’s very easy to get hold of and install, so I really would recommend that if you want to try something new, then give it a go.

What Else Did I Try?
While I don’t want to go into the whys and why-nots of each tool I tried, I thought I would list some of the tools we did try so you can give them a go. What might not have been suitable for us could pretty much fit the hole you are looking to fill, so go and check these out too.

SmartBear’s CodeCollaborator – Try this out, it’s fantastic
SmartBear’s Code Reviewer
CodeStriker

Thanks and remember to enjoy your code reviews!

In the last post I went over some of the reasons why code reviews are important in building a stable and secure project. I wanted to follow that by covering the different review methods we’ve used in the past and the ways in which each one worked or in some cases didn’t.

Paper Based Reviews
One of the simplest, and first, methods we used – simply take the code you want to review, print it out in whatever format you want (we found small A5 booklets worked best) pass them around and then start reviewing. Usually people would be given a couple of days to review the code before everyone involved would sit around a table and discuss what they had found. The advantage to this is that it gets people talking. You are sat around a table together and it’s a great way to get people participating. Unfortunately it also forces people into thinking they must contribute rather than simply being happy with what is in front of them and it’s a drain on resources and time. It’s surprising how long it takes to get code from an IDE into a suitable (and readable) format, especially if all you have is a word processor. And wasting all that paper is something that I simply don’t want to continue doing.

Forums
We also tried posting code onto an internal forum when the review didn’t justify a full sit-down session. This worked very well in making the code easily visible – even to people outside the review group though that might not be suitable in every case. It also reduced the time it takes to set up and cuts down on waste. But discussing specific parts of the code simply didn’t work. Referencing line 316 of PlayerCamera.cpp in a post makes it very hard for people to link to the work, and while forums are good for limited threaded discussions, trying to quote and re-quote just becomes hard to track, read and find out what needs to actually change.

Automated Code Reviews
By automated I mean using a tool that has been created specifically for code reviews, one which can display code correctly, generate or show diffs and allow people to reference files and lines of code easily. I won’t go over the actual tools we have used (I’ll do that next time) but suffice to say this is what we use and the method we have stuck with.

Of course, when using automated tools you have more scope as to when to do you reviews (do you review individual diffs, whole files or something in-between?) and I’ll spend the rest of this article going over the different methods we’ve used so far.

Blocking Code Reviews: Blocking code reviews are used when you want every piece of code reviewed before it is even committed to the repository. When a developer has made any changes to the code base they fire off a review and the developer cannot check in their work until everyone in the review group has flagged it as shippable. Any changes or requests need to be done before the code review can be submitted and checked into the repository.

This didn’t work. Developers need to get into a work-flow and having to constantly stop and wait for permission to continue or to review someone else’s code stops them in their tracks and is a serious drain on their productivity. And let’s face it. Game development might be a multi-million dollar industry but lives are not on the line if someone checks in something that might not be perfect.

Non-Blocking Code Reviews: The next obvious step. Every check-in needs to be reviewed so the review is generated, sent off to a selected group, but the developer is then free to commit the code with any changes or improvements being done in another commit. This is beneficial on two counts. The original developer is able to continue working, either on the same area of code or another without it conflicting on any waiting commits, and the reviewer is able to continue their work, leaving any reviews the need to complete for when it’s suitable, maybe performing a group of them in one go.

While this worked well in regards to making sure that everything is reviewed, we found that you needed to monitor the size of the review groups and to check that people are not being over-loaded. Filtering your reviews only to come to them at the end of the day with 10+ of them waiting for you can be quite disheartening.

Feature Based Reviews: This is generally where we sit when it comes to diff based code reviews and it puts the responsibility back into the hands of the developer. If the developer is working on a new feature then they are free to check in their changes with a view to putting the whole feature up for review when it is finished. This greatly reduces the number of reviews and for new code allows the developer to put the whole code into context. It’s very similar to paper based reviews but in a more suitable environment. But there is one caveat. It only works if there has been a decent design stage to the feature being written. Having people bring up basic design flaws when the thing it practically finished can be a killer blow but this is showing up the work flow process more than anything else.
This can obviously be extended to large scale changes that someone might be working on. The can check in their small changes, which a review done on the final product, maybe generating the diff based on a collection of commits rather than just one.

Bug fixes are another matter. Once the main implementation is done and people are into a bug fixing stage then these should be reviewed as and when they are being committed (usually in the same manner as the non-blocking reviews). This is especially important when it’s coming up to a milestone and is the way continually developed projects (for example with Middleware or Technology) will eventually spend most of their time.

Other Methods

Obviously there are other methods we could try.  Review buddies, review masters (one person assigned responsibility for checking all commits – sometimes this will usually be the lead or programming manager), e-mail based discussions, ”important changes only’ reviews, and plenty of other variations on the same theme.  We haven’t tried these because the automated reviews have been working so well for us.

Review buddies would be the one I’d like to try (alongside, rather than replacing, the automated review process) but I would usually have this done between two developers working on similar parts of the code – though I would hope these developers would be talking to each other already!

Review Groups
One thing I’ve not covered is who does these check-in reviews, which can be just as important as what is being reviewed.

Initially we used review groups, which contained 2/3 developers and were mixed quite randomly. Every review would be assigned to a random group and fired off. While this worked by spreading the work and the knowledge of the code base, it wasn’t producing the best results. Having one person involved in 2 out of 3 reviews for a particular feature, only for someone else to replace them on the last one would produce duplicated, or sometimes conflicting, results. And the new person isn’t as up-to-date with the feature as the original set of reviewers, which can limit the scope of the review and lead to it taking longer.

While this does spread the knowledge throughout the team, it isn’t necessary for everyone to know everything about the code base.

So now reviews are generally more focused, being sent directly to specific people who will have the best input on a particular feature. This does need monitoring to make sure people are not being over-loaded (or over-looked) but that’s a pretty simple management issues. People can also be asked whether they want to be involved in a certain review. Do they have a particular interest in that area, or does it use a new piece of technology that they would like experience with?

So that’s a brief over-view of the code review systems we’ve used over the past couple of years. We’ve settled on our automated code reviews as they produce the best results in the shortest time and give you much more scope in regards to how and what is reviewed. Choosing the correct piece of software to run this system is an important decision and I’ll go through the tools we used and most importantly the ones we stuck with in the next post.

Code reviews have a bad reputation.  Claims of wasting time, pedantic reviewers and hot-shot programmers all crop up as reasons why teams shouldn’t and don’t bother reviewing any of the code that is being used in what could be a multi-million dollar project.  But why would you want to have a code review?  What does it bring to the project that would otherwise be missing – because that’s the point really, if it is a waste of time then should we be doing it at all?

The Reason For A Review
Every review needs a reason.  Throwing some code in front of a group of developers without purpose will generally end up with a bunch of incoherent comments, suggestions and ideas that might be good, but depending on the project or the time scale, totally irrelevant or unfeasible.  The following are just some of the reasons you might want people to look over a set of code.  There are, of course, many more reasons but I think the majority will fall under one or more of these banners, regardless of who is doing it or what you hope the outcome will be.

Finding Bugs
It’s funny, and it might just be me, but every time I’ve asked someone to put up something for the first time, at least one bug is always found.  It might be a misplaced define, or an = instead of an == or any number of things but there’s always one.  And I know that the time it took to post and complete that review will be much less than the time it takes for that bug to be found, a bug report written, the author to read the bug, find the code in question, do some tests and then finally fix it.

Obviously not every bug will be found by a review.  It might be 1 in 10, it might be worse or it might be better.  But the point is that bugs will be found that otherwise would sneak through into QA or milestone builds, and each one will take time to find, report and fix.  Time that could be better spent doing something much more productive.

Stability
While I can pretty much guarantee that a bug will be found in the first review someone does, I would also place bets on the number of bugs being found as the reviewing continues goes down.  The number of suggestions and improvements people make will become less demanding and more of a conversation.  This isn’t because the reviewers are losing heart, or that less code is being reviewed, it’s due to one simple fact – the build is become more and more stable as more reviews are done.

This could be down to a number of reasons, but one of the biggest reasons in my opinion is peer pressure.  Programmers are suddenly aware that they are not the only people who will see this code, that their peers will be looking at and critiquing the work being done.  And most people want to look good next to their peers.  This is not a bad thing despite the connotations ‘peer pressure’ can bring to the mind.  Mutual respect with your co-workers and the need to drive yourself forward are often driven by peer pressure, and when one or two people start to move forward, a good team will always follow.

Sharing Domain Knowledge
The more people know about a particular system the better.  Illness, team moves and resignations will always be part of the work environment and it means that the key team member you are relying on could be gone before you know it.  And if you need someone else to dive in there and fix a couple of issues that have been found by QA, how long do you think that will take?

Code reviews can help.  As code is written, modified or fixed it is being looked at by more than one person.  The author has to explain their decisions, how things work and how things plug together.  Because of this you might have 2 or 3 people on your team that are happy to dive in and help.  Even if you need two developers to pair because the knowledge is spread between them, it’s still better than having no-one know what the heck that particular piece of code is doing.

Sharing Experience
People learn best when they are surrounded by people they can learn from.  Books and blogs are excellent resources (as an avid reader of books I would say that), but there is no substitute for someone sharing their years of experience with you in a 5 minute conversation.  Having people suggest different ways to structure, optimise and write code might seem a little late when it’s already written, but it’s something people can take with them into their next task, rather than going back and re-writing large chunks of code.

And it works the other way too.  There is nothing better than graduates coming into a company with their crazy ideas about how code should be written or used and teaching the old guard a new trick or two (C++? Inheritance? Madness I tell you!).

Coding Standards
This is one of the reasons people hate code reviews.  They don’t want them to turn into conversations about where the bracket should go, or if that variable should have an ‘i_’ prefix or not.  And on the whole they are right, and if this is what your code reviews turn into then you are simply ‘doing it wrong’.

But this has its place.  Developers new to the team can be given documentation about coding standards, but it is one of those things that tend to easily slip, and there is no reason why this shouldn’t be picked up in a review.  If it continues and every review becomes about how the standards are not being met, then you have an issue that should be resolved outside the review before it makes the whole process a pointless and much less useful exercise.

Problems With Code Reviews
Of course code reviews have their problems but then what doesn’t?  Most of the time these have nothing to do with the actual review process and more to do with the management of the team, or what has come before.

• Arrogant reviewers & victim syndrome – Code reviews are there to improve the code base for the benefit of the team.  They are not there to ‘go on the attack’, to rip someone’s code apart or to make someone feel like they are simply not good enough.  If you start to find that one or two programmers are simply making blanket statements or not taking anything on board then this is certainly a management issue and should be resolved outside the review process.

• Pointless reviews – This was pretty much covered in the Coding Standards section, but there is always a chance that code reviews start to only skirt the surface of the code without looking at anything to deeply.  If this starts to happen, you need to reassess what each review is for and what each person’s role is.  And remember, there is nothing wrong with simply saying “This looks great”.

• Lack of outcomes – People will argue, and people will disagree, especially when it comes to code.  ”You should do it like this”, “I don’t want to change that because…”.  Discussions are great and should always be encouraged, but at some point there needs to be a decision.  You have to know who has the final say – whether this is a manager who is on every review, or a senior who has the last say.  A simple comment along the lines of “That’s a great suggestion for the future” can put an end to most arguments with most people being happy with the outcome.  If they are not then again this needs to be resolved outside the review process before it goes any further.

• Lack of time – It’s an important milestone week, code needs to be fixed quickly and you don’t have time to review it.  This can happen and sometimes you need to take it on the chin.  But isn’t this the time when it’s even more important that everything is checked and double-checked?  If you have a milestone week and your changes are coming thick and fast, then maybe you don’t have time to review everything, but why is so much changing at such a critical time?

This is the true story… of seven(ish) programmers… picked to make a game… to work with the tools they created… to find out what happens… when tools programmers… start using the tools they make… in the real world.

Ok, so it’s terrible but I couldn’t resist – it puts across exactly what I’m about to describe.  Last week we did an interesting experiment on BlitzTech by taking a group of our dedicated tool programmers and asked them to use the tools they develop to make a game in exactly the same way a BlitzTech end user would.

But what can you get out of this?  A day certainly isn’t long enough to stress test a large scale SDK (and we have some great games doing that for us already) and to get a true appreciation of a toolset you certainly need a good length of time to see everything.  But what it does is take people out of their comfort zone, away from the areas that they are most comfortable with and gives them a real opportunity to experience much more of the environment that houses their work.  It’s also a very good chance to get solid feedback on the accessibility and initial impressions of a set of tools.

So was it a success?  I would certainly say so, and it’s something I wish other tool developers would do.  One group of people will have a very different outlook on an application than another group – which is why we also have a group of artists on the BlitzTech team full time.  By making sure that the people creating the tools actually use them (hummm, that sounds very similar to one of the main unit testing advantages) will always lead to a better user experience as people appreciate exactly what is needed and how the tools will be used.

So will we do it again?  I certainly hope so, with a different group of programmers, but even extending it so the last group can take what they did even further will simply allow us to expand on what was worked on and give the group a chance to see even more.

And that can only be a good thing.

Our current Continuous Integration process works quite nicely and has greatly benefited our team, but these systems can always be extended and improved upon.  The following post will simply cover some of the ideas I have for the future and how it might improve the whole process.

Visual Unit Testing

Visual Unit Testing is the process of creating small, scripted tests that run the game and allow you to test that certain states have been reached.  For example, having a test which runs the game, scripts the input so one character shoots another and then tests if the enemy is dead or the score has been increased would be a very basic test.  It could also be used to perform long soak tests with more intelligent input rather than the kind of tests that are used at the moment.

Allowing the process to take screen shots and post them to a web server with the test results would allow a huge number of systems to be tested in the environment they will actually be used in.  Extending this to allow QA Technicians to create rafts of tests would also remove the need to continually test the more mundane aspects of games, such as does the menu flow behave as it should, do the leaderboards work as intended etc.

Obviously running this process would require a large amount of time, especially if run on multiple platforms.  But it could easily be hooked into the nightly build process meaning hours of testing would be carried out when no-one was in the office.

Some Arcade titles originally supported simple scripting using Lua to perform automatic controller input, but this wasn’t taken far enough, and the BlitzTech engine now has its own state machine system which would most likely be used instead.

Automated Code Reviews

Teams at Blitz carry out code reviews at different times and to different levels, but one thing that is usually done is checking that the code conforms to a given Blitz coding standards (with people moving around teams as they do, a consistent standard is pretty much essential).  But again, checking something that is based against a set of rules is something that an automated system is much better at that our human co-workers.

Adding a code metrics step to the Cruise Control build step, meaning the code is run through after every check-in could catch if the code is not standards-compliant below a certain threshold.  If this was the case, the build would simply fail (and this step could be removed if we had a designer/artist build machine as discussed in a previous post) and the programmers would have to re-examine the code they posted.

While this could seem quite draconian, it would free up code reviews to concentrate on the bigger picture, rather than getting stuck in the rut of commenting on the use of ‘m_’ or class name pre-fixes.

Extending The Nightly Build

As I mentioned in Extending Cruise Control we collect information on how often the build is broken and the ratio of who breaks the build the most.  We also generate a massive amount of information by using Subversion, such as check-in rates and lines changed per check-in etc.

The nightly build could easily be extended to incorporate all this information and, along with links to the Visual Unit Testing web service, could produce reports on the activity and stability of the game for that day.  This would be a very useful tool for the programming managers, especially as some people already use this information, but it is usually done on an ad-hoc basis when the tools are manually run and the information manually collated.

Automatic CI Support

A large portion (at the moment pretty much all) of my current work is involved with extending the Arcade technology I developed and integrating it into the BlitzTech engine.  It has the ability to automatically generates the required configuration files to set up a Cruise Control server, but the current plug-in system (like adding automated e-mail generation, stats collection etc.) is pretty much set up to do only what it needs to do, without much flexibility.

I fully intend to make all these plug-in’s come as standard, allowing developers to cherry pick the elements they want and hook them into their CI process with very little work.  While people may start to ask why I do not use NANT since they are all hooked into CruiseControl.net, I can simply do more, and do it faster, with Ruby.

So I have a few ideas on where I want to go with our Continuous Integration process in the next few months (years?) but there is nothing like lofty goals to keep you going.  Visual Unit Testing is obviously the biggest, and will require the largest amount of time to developer, but will generate the biggest payback once it’s being used to its fullest potential.

So what kind of things would you like to see in a CI process?  Or is there something your process is doing that you think is pretty special?

So far we have a process which guarantees that the game builds and that the assets can be generated.  We have processes for doing this automatically and for making sure people are as up to date as possible.  But there is still one question which none of this answers…

When I run the game, will it actually work?

Games are a mass of systems, all interacting together to produce something that is fun and interesting to play.  It’s QA’s job (on the whole but not exclusively) to make sure that when it plays it plays well and there are no glaring bugs, but if we are automating so much, we can also automate some of the testing too?

Unit Testing

A Self Testing Build in this case is really just a fancy term for Unit Testing, something a lot of people have heard of but just as many havn’t.

The basic premise behind this is that at the end of every single compile small pieces of code are run that test various systems in the game.  These are run as post build steps so any changes to the code are instantly tested against the tests that already exist.  If some code has changed that makes one of the tests fail then the whole build fails.  This means Cruise Control fails, the nightly build fails, but most importantly everyone knows it’s broken and they should avoid updating until it is fixed.

The following are a couple of very simple tests, that are run every time the code changes to make sure the vector’s behave the same and as expected.  This might look simple (one of the main reasons people don’t unit test is because it will ‘obviously work’).  But with vector code being platform centric, a change on one platform can effects all the others.  The tests are written using UnitTest++ which is the library we use to write all our tests.

Feed back to the developers tells them something has broken, the test results are displayed in Cruise Control and the person responsible instantly sets about fixing the build.

Game Based Unit Testing

It’s often difficult to think about games from the point of Unit Tests but it often results in code being written in a more encapsulated and less coupled way, which in itself results in code being easier to write and maintain.  But as an example, the following could easily be tested in a small suite of Unit Tests…

• Does and AI character ’see’ the player when they are in front of them
• Is the player ignored if they are outside the enemies cone of vision?
• What state do they enter when they hear a noise?

By testing the under-lying logic of the system we can be confident that when something breaks we will have a better idea of where the problem lies that having to look at every system at every level.

Running Tests On Different Platforms

Unit Tests are only useful if they are run often and on the platforms the code is destined to be used on.  This can cause problems when trying to run Unit Tests on development kits as the code usually needs to be loaded onto the machine before it can be run and the results fed back to the host.  While this might only take 10 seconds, doing it after every build would be a massive drain on a programmers time.  While this isn’t a problem on the PC platform, we generally expect the console tests to be run as part of the nightly build when it has all the time it needs to run and feed back the results.

But, even if it isn’t being run on every platform all of the time, we still have a system running on the PC and some testing is still better than no testing at all.

So this is how the Blitz Arcade system adds a level of testing to the Continuous Integration process.  It’s one of the hardest parts of the process because Unit Testing does take a while to get into, and to understand how code can be tested when at first it looks like it is a series of black box objects.

In the final part, I’ll take a look at what I would like the process to do in the future, and the kind of things that would be cool to have in a fully integrated Continuous Integration process.

Games are not just code.  Art, design, music and QA all get in the way, making the whole CI process harder, but actually making the game fun!  It’s just as important that the CI process benefits these guys otherwise there are still going to be a hurdles to getting a solid game released and making the process as smooth as possible.

The Nightly Build

Due to using Cruise Control, we know the build can compile and because we never leave on a failed build we know we can run a full ‘rebuild’ of the game at night.  This guarantees that we do a full rebuild of the game at least once a day, and can remove any niggles that crop up from Cruise Control only running ‘build’ steps.  And since we are doing it when no one is around, we have the time to build all the game assets, including animations, textures, models, music and anything else that we need.  Since this can take hours, it doesn’t matter if we start the build at 11pm, as by 9am the next day everything is up and ready to go.

So we finally have a fully integrated build that is being generated for the start of every working day.  It means that QA have a full build to test at the start of the day, but also the designers, artists and musicians have a fully up-to-date project in which to add their assets and tweak the game play.

But this can be difficult to first set up.  Since you are automating the entire process, everything needs to be command line driven.  This isn’t always easy with off the shelf tools, or even bespoke software, but the time it can save is definitely worth the time.  Luckily the BlitzTech SDK is fully configurable using a custom scripting language so we can not only build the assets, but run additional processing steps such as compression or pre-processing of assets.

Generating More Regular Builds

But we still have a slight problem.  What happens if 10 minutes into the day a programmer implements a new feature that the designers want to play with it right there and then?  Do they have to wait until tomorrow morning just to get hold of it?  What if the assets being generated by the artists change in a way that are no longer compatible with the build they have?

Since we have a Cruise Control machine constantly building a new version of the game, we can easily add additional steps to the end of the CC process.  We have a step at the end of a successful build that copies the executables to the network and (similar to the failed build mails) sends a mail to the designers and artists telling them a new build is available.

By providing them with a simple tool that allows them to get ‘latest build’ from whereever they want, the designers have access to the latest game features pretty quickly (this is also another reason to trim down the amount of time it takes to finish a Cruise Control build).

So the programmers are no longer disrupted with requests for new builds, and the artists on the team are no longer using half-built, half complete builds during the day. It means the full game is being used at all times and is never in an unknown state.

Some teams at Blitz add additional information to their Cruise Control builds and this is something I hope to incorporate into the Arcade process at some point.  Each CC build will display the latest Subversion revision on screen at all times.  By requesting that the programmers specify the SVN revision in bug reports or feature check-in’s, designers will instantly know if the build they have does what they need it to do.

Generating The Game Assets

It would also be cool if the Cruise Control build also came with up-to-date assets so QA could get the latest build and go from there.  Obviously time is a serious concern here, but since our entire build process is command line driven, we can add steps to build the assets at the end of a successful build and before copying it to the network.  In our process this is an optional step, as at the start of a project, when it takes minutes to generate the assets this is pretty useful, but near the end of a project this can add hours onto the process and has to be removed.

One way to avoid this, and make the process more suitable for the artists, is to have a separate build machine, which only generates the release build (which is usually the one used by the artists and designers) and builds only the common asset packages.  This is sometimes done on the bigger projects at Blitz as it can take slightly too long for artists to wait for the programmers Cruise Control machine to finish.

Generating Submission Builds

One more thing that comes from this process is our submission builds.  Since we can generate a nightly build automatically (usually by adding a single script as an automated task) we can generate all our builds by running the same script.  This means that our submission builds hook into the same process as everything else which avoids the most important builds being built in an un-tested environment and make the process even more secure.

So we now have a process that allows us to automatically generate the whole game at any point of the day, and at least once a night.  The designers, artists and very importantly QA can hook into the CI system giving them an up-to-date build at any point and no one is every playing a build of the game that is half-built or doesn’t have everything that is currently available.

In the next part I want to cover the process of self-testing builds and how this can make a build not only compile, but actually make sure it is doing what it is supposed to be doing.

Ryan Wiancko approached me a week or so ago with a view to putting some of my Articles on IndustryBroadcast.com, a site which provides audio versions of popular blog posts written by members of the Games Industry community.

I think the idea of Industry Broadcast is fantastic, especially as I look around an office that is filled with people wearing headphones while they work or who have long drives to work.  You can listen or download the audio casts from the website or you can subscribe in iTunes.

There is some great content on there and I’m honoured to be included in a list that includes people like Clinton Keith, Jamie Fristrom and Noel Llopis.

You can find my articles here, so head over to IndustryBroadcast.com, download the files and give them a listen.