In these cases its handy to be able to deploy a debugger onto the user’s machine so you can dive in and see where your code is going wrong. To be successful at this you need a couple of components: the debug symbols from the compile, the source code, and a debugger.

On Windows the debugging symbols are separate files from the executables. PDB files contains the information debuggers need to map addresses of code and data in a running tool to the source code counterparts. In Visual Studio, PDBs are only generated in the Debug configuration by default, so assuming you distribute something like a Release build to your users you will need to turn on PDB generation in that configuration. Its under Linker… Debugging… Generate Debugging Info. Set it to Yes (/DEBUG). When you prepare and publish your tool set, make sure to include these PDBs with the executables (EXE and DLL files).

PDBs can get quite large, so it may be a good idea to not always pull down PDB files when users get the latest tools. Insomniac’s tools retrieval script has some command line flags to pull down PDBs and code only when we know we want to debug something on a user’s machine. Using -pdb will get the executables and PDBs, and -code will get the executables, PDBs, and source (all from a label populated when the tools executables were checked in).

Once you have the PDB and code on the machine you just need a Debugger to dig in with. On Windows you have a choice: Visual C++ Express Editions or WinDBG (from Debugging Tools for Windows). Both are free to install so you aren’t bending any license agreements here. Visual C++ should work pretty much like you expect on your development box, but can take a while to install and patch to the latest service pack. WinDBG on the other hand is very quick to install, but takes a little getting used to. Typically you must show the UI you want to use (Callstack, Memory, etc…), as well as potentially manually setting the PDB search path (from File… Symbol File Path). It’s a very different experience but its so quick to deploy it may be worth checking out.

First, you can have the tools and the game depend on a shared library of resources. The tools have their own UI, tick or don’t tick at their own rate, and may or may not use the same renderer as the actual game. By creating tools this way, you have tools automatically update their own behaviors as new features are added to the game, or even automatically generate their own UIs if you’re using a dlls and a reasonably robust reflection system. The problem here is that if you’re not using the game’s update and render loops, you still have to boot the game to see everything in action. This can create long turnaround time for assets, especially if your game takes a long time to boot and / or load. That said, this allows you to keep your tools slightly smaller, it reduces dependencies on what are often unnecessary game libraries (like threaded sound and in game UI), and can make the tools less prone to break due to game changes.

Your second option is to either make the tools dependant on the entire game, or make the tools embedded in the game. Now, I think every studio has a small amount of “on the fly editing” capabilities in their engine, but there are very few that are willing to take the full plunge and make their game the editor. In some cases, this is because they can’t afford the extra memory or processor time to fit an editor on a console dev kit. That said, in game editors, or editors that can run the full engine stack, can reduce asset turnaround time significantly. Given a properly designed tool, artists, designers, and scripters can actually edit objects on the fly, and see their changes affect the environment immediately. Once given a tool like this, few would want it taken away. However, these tools can also get cumbersome as well. Because you’re working in the game engine directly, few tools developers in this scenario will take the time to develop good user interfaces, as it tends to clutter up the screen quickly, instead relying on weird key or button combinations to achieve the desired result.

In both cases of dependency, the one thing you need to avoid is the dreaded #IFDEF EDITOR block, and I know you’ve all seen them, and they almost always defeat the purpose of having shared code in the first place. The idea behind shared code is to make sure the editor behaves the same way the game would under the same circumstances. #IFDEF EDITOR blocks, by definition, create inconsistencies between editor and game. But these blocks almost always become a necessity at some point, which is why some studios prefer to forgo the dependency question entirely, and instead opt for a tool that outputs a platform independent format (XML or JSON for example) then has various versions of the game decide what to do with them. The game can ignore blocks it doesn’t understand, and fill in missing blocks with default values if needed. This keeps game and editor independent, so that mismatches can be safely dealt with. However, it still incurs the wrath of the slow turnaround time, and the necessity to maintain two separate code bases.

In my opinion, dependency is a requirement. The idea here is to protect against game changes, include the ability to preview your game, and shorten turnaround time to the game whenever possible. Toward this end, you should limit yourself to dynamically linking in only the libraries you need, which is usually your rendering libraries and your game object libraries. Try using reflection where possible to generate your UI against game objects so that changes in game objects don’t require complete editor recompiles. Save and load objects to a text format (XML is my favorite, but you can choose whatever you like) at least during development so that you can protect against mismatched resource bugs, and, lastly, use a robust shared command system to transmit changes to a running game to shorten turn around time.

Many game companies struggle with delivering tools quickly and cheaply.  Money is always an issue wherever you go.  After all, the bottom line is what keeps a company afloat and it’s employees employed.  No one wants their company to fail, to lose their jobs, or to lay off their workers.

Game companies are in an especially difficult position.  Attempting to balance a workforce spread over multiple disciplines — art, design, programming and production — is hard enough, but when you consider that those disciplines have their own specialties within each one, the task is even more difficult.

The obvious solution is to cut corners wherever possible, and that oftentimes falls squarely on the shoulders of  the tools team.  Why?  Because most game companies don’t make money selling tools.  Tools programmers serve in a support role, and therefore (in the minds of most game execs) are less valuable than those working directly on the games.

Tools teams very rarely get the full support of management, and game teams can’t be stalled waiting for tools to be completed.  The unfortunate sentiment among those in power is that there’s no time for tool design.  Get it done and get it done now.

There is no time for design, so the thinking seems to go, but what does that really mean?  Does that mean that the programmer implementing the tool charge blindly into development without thinking about how the tool needs to function?  Of course not. 

The programmer has a vague idea of what to do, and without ever writing it down or validating his thoughts with the end users in any formal way, begins to implement the design from his own mind.   He still thinks about it a great deal.  Perhaps 75% of his time is spent thinking and only 25% is spent typing.  There are probably still many  unanswered questions, but as the tool begins to take shape, some answers may start to become more obvious one by one.  The tool seems to practically design itself, but in reality, design is going on quite informally.

But wait!  What if one of those unanswered questions causes a serious problem?  What if the best answer to that question requires a rewrite of major portions of current code-base?   What if the other possible answers are so undesirable, that the rewrite actually appears to be the best option.  Because the design had been postponed until the code was in the process of being written, redesign is now very expensive.  Code that has been written will go to waste, and new effort must be exerted to replace it.

If the programmer takes this problem to a (non-technical) manager, concerned with cost and speed of developing the tool, the manager may come to the very justifiable conclusion that a rewrite is not the way to go.  Instead, just make a work-around for this one problem, in other words, a hack. 

As long as that’s the end of the story, then that’s probably okay.  Unfortunately, more issues may arise, with similar outcomes.  Also, once delivered, the end users will likely have feedback.  After all, without any formal design process, many of their needs/wants/concerns went unheard.  And now the real fun begins.

It’s already been established that the quickest solution is more desirable than better architecture and code, and so as feature requests are delivered to the programmer from the users, more and more workarounds are put into place to deliver a new tool quickly.  This leads to code that is difficult to maintain and potentially very buggy.

The end users are now saddled with a tool that does basically what they want but perhaps has stability or performance problems.  The difficulty  to fix those issues increases as time goes on as the code becomes more brittle and spaghetti-like.  Fixing one thing breaks something else, leading to a never-ending maintenance cycle that really makes no net improvement whatsoever.

Team sizes are increasing.  So are the assets, themselves.  The more users stressing the system, the more fragile it becomes.

NxN had the right idea with Alienbrain but never really got anywhere due to serious technical issues with their back-end.   It’s been a few years since I used it last, so they may have fixed a lot of those problems.   Anyway, it also had some very nice features you don’t get in other source control solutions.  You could easily redesign the whole interface (it was mostly html and javascript as I recall), and they included a feature that was very art-centric.  Previews.

You could generate previews of assets and view them right in the Alienbrain interface.  It was a very slick feature and a selling point of the software.  Finally, a user could see a preview of a model or texture (and many other asset types) without doing a get and opening the files in Maya or Photoshop, etc.  That’s a real time-saver if you don’t remember the filename that was used for a specific asset.  You have the chance to browse all the assets of that type and find the one you want pretty easily.

Like I said, though, NxN had its share of troubles.  Still, I believe we can do better than the source control status-quo.  I imagine an asset database solution that integrates with every asset generating tool, as well as the build process, generates a preview for each asset (even if it’s a bitmap that says “Preview Not Available”), and is searchable by its meta-data, including tags, creator, last modified, and so on. 

The classic view of assets as a collection of files inside of folders, with users having to know exactly what files need to be checked in and out of source control when changes are made seems a little antiquated.  Instead of searching through folders ten layers deep, how about using a tag cloud to find assets instead?

I imagine being able to open a web-based interface, searching for an animated character from an old project and clicking a button to copy it to a new project, including all of it’s vertex, texture and animation data and using it as the starting point for a brand new character, or maybe just as a placeholder until a new character is created.  How many walk cycles does one studio need to recreate every time a new project is started, anyway?  Why not take something you have and modify it to fit a new character in a completely different game?

I really beieve that asset databases are the wave of the future for game development.  When the Xbox360 and PS3 came along, team sizes doubled, and assets got bigger and more complex.  What’ll happen next time there’s a hardware revolution?  We need to streamline the way we manage assets, or else, it’s going to bite us in the ass… even more.

Unfortunately Perforce was not designed as a distributed revision control system, so we needed to come up with our own solution. We needed to allow sharing code across Perforce server instances. p4share is a Perl script I wrote to help solve this problem without involving a huge list of complicated manual steps.

To get the job done it does a lot of deleting, syncing, and copying of files on the local client… nothing too exciting. I was however able streamline the process in a interesting way given Perforce’s ability to open a file for edit at the client have revision (as opposed to the head revision). When you open a file for edit at the client have revision then all of the changes that have been made in subsequent revisions must be resolved into your edits before you submit your changes. This resolve step is only necessary when your have revision does not equal the head revision when checking out a file, or the file in question allows for multiple checkout and someone edited and submit changes before you can submit.

Given the ability of Perforce to open a historical revision for edit, I was able to make p4share less likely to loose edits on files that have changed on both servers. p4share uses a label to store the revision at which each file was last shared. When sharing happens again in the future, the client is synced back to the revision that was submitted the last time files were shared. The files are then opened for edit at that historical revision and overwritten with files from the other server. In this state any file that had changed on both sides will require resolution to submit, but the resolving mechanics of Perforce has all the information it needs to do automatic resolution (two versions of a file and a base revision… which in this case is the revision labeled during the last share session).

Using this technique is a win because it removes the possibility of stomping files and losing changes on either side when manually merging changes from both servers.

You can find p4share on Nocturnal Initiative‘s Perforce server: nocturnal.insomniacgames.com:1666 at //Source/Trunk/p4share/p4share.pl, and via p4web here.

Now comes the real decision point though.  Does a rewrite make sense for the current project or should it be put off for a later time? If you’re in beta, rewriting a tool now isn’t going to help you get your game done.  Consider how long a rewrite will take in man-days and calendar days.  If you can get the new and improved tool into the hands of your developers fast enough to save them more time than it took to develop it, then I say, go ahead.

The key point here, is the suggestion that you “do the math’ on the tool: figure out how much time it will take to rewriting versus refactor, and balance that against the time saved by the number of developers that use the tool.

But doing the math should be a key concept when you’re trying to figure out anything tools related, including trying to convince higher ups that you really need a dedicated tools team or process team. What you need to take to them is real data that shows that you save more money with a tools team, or a tools refactor, than without one. So the question is, how do you accomplish this?

To answer this question, you need to know three pieces of data:

1. How many developers use your tool?
2. How much time does each developer waste because of poor design or poor implementation, OR how much time would be saved if a new tool was implemented?
3. How much does each developer cost?

Number 1 and number 3 are easy to know. Just take a quick head count, and then compare their levels to the average salary for their field and the experience level using the published data from the Game Developer Salary Survey. Estimates here are usually fine. Using averages across the board (about $45k per year per developer, which comes up to about $22 an hour) here’s the numbers you’re going to come up with.

You’ll notice that at about 4 developers loosing 2 hours per day, you’ve basically paid for another developer. Even if you have 10 developers loosing 30 minutes per day, you can afford an intern to fix that problem.

With that said, hours lost per day, or hours of productivity gained is always going to be a best guess, and if you’re trying to sell this concept to higher ups, you’re going to have to make sure that you get that number right, or can somehow convince them that you got the number right. Now, the best way to do this is by having your tools gather metrics concerning how often they crash, time between key actions, build times, cook times, and turnaround times, but that only helps if you already have a team and are just looking to expand it. Otherwise, you have to rely on hearsay, but here are some techniques that should help:

• Ask other developers how much time they think they lose on a given day because of bad tool design or performance and average those numbers. Ask for comments about how tools could improve.
• Show time lost from other developers who are only spending half of their time (or less) working on tools. If you have a bug tracker, you can use those numbers to show amount of time spent on tools bugs. Combine these with well known metrics concerning hours lost in task switching to show real cost for these support requests.
• Show an unfilled developer need. If you hear people having trouble with a specific issue, ask them how much time they think they could save (on average) if a tool were made to help them. Show that it would cost less to hire a tools developer than to leave the problem unfixed.

Of course, once you’ve convinced higher ups to create a tools team, don’t stop there. Show them it was worth it. Too many people stop once they have what they want and don’t show the real and tangible benefits. These are not always obvious, especially to people who aren’t in the “pits” (meaning doing actual development), especially when some developers may not be vocal about their increased productivity, only their frustrations with a new tool. Show the amount of productivity gained, and amount of money saved. That will help prove to you and your boss the value that a tools team can bring.

Luckily, there is a lot of great software out there that is either open source or very cheap, and the very best solutions have risen to the top.  Most of us have heard of Blender, Gimp, the Torque Engine and Unity (which includes a $200 “indie” version), and there are a great deal more.  Check out IndieGameTools.com for a good list of the best stuff.

While some of these tools are really great, there will always be a need for custom tools, that aren’t very useful to other developers.  Smaller developers should consider partnering with a small tools development company or individual contractor.  there are a few people out there who are doing this sort of work and banding together can be mutually beneficial.  Getting someone experience to help out in this area can give a leg up on the competition, after all, the better the tools, the easier it is to make a great game.

This sort of problem always comes down to time and money.  The perception among management may be that this is going to waste time.  After all, why rewrite something that seems to work fine, and if there are issues, isn’t it easier (and cheaper) to fix a few bugs than to write something from scratch?

Well, that may be true, but not necessarily.  After all, buggy tools waste the time of everyone using them.  If ten people waste just ten minutes per day, over the course of a single project that lasts a year, then you’ve lost  over 10 weeks worth of work during that project.  The actual amount of time may be much greater of course.  I’ve worked in some studios where tools were so slow and buggy, it wasn’t uncommon for individuals to lose several hours in a single day.

I knew a programmer who would write almost every piece of code twice.  He would completely scrap the first implementation in favor of his second try.  The first was basically a learning experience, and once he figured out how to solve the problem, he could do it much more cleanly on the second go.

Rewriting a better tool may be much faster than the initial implementation.  The team has learned from their mistakes and may have a much clearer vision for how the tool should be designed and implemented.  There may also be some re-usable code, so not everything needs to be wasted; individual modules may be able to be salvaged.

Now comes the real decision point though.  Does a rewrite make sense for the current project or should it be put off for a later time? If you’re in beta, rewriting a tool now isn’t going to help you get your game done.  Consider how long a rewrite will take in man-days and calender days.  If you can get the new and improved tool into the hands of your developers fast enough to save them more time than it took to develop it, then I say, go ahead.

There are a lot of approaches to this problem, but I’m going to narrow down the solutions to three that tend to be more efficient and should be used when developing a mature tools pipeline: Using in game editors as opposed to stand alone tools, implementing dynamic resource loading and unloading (through something like a developer console), and improved communication between game and stand alone tools.

Right now, I’m going to focus on the third possibility. The use of a game-embedded editor versus a standalone tool set is an ongoing argument in the tools community, and each side has its positive and negatives, but regardless of which way you go, some of your tools are not going to be game-embedded, and it is important that any “stand-alone” tools be able to communicate with your game. By creating even a simple a communication library, you’ll be able to issue commands to the game remotely, grab and analyze information without using game resources, and smartly organize, load and save diagnostics information, which might otherwise create large amounts of special case code in your game. By creating a slightly more complicated communication system, you can dynamically run scripts, save and load resources, and even set up a system that communicates changes in seconds to running games. Talk about turnaround time.

The key to creating a good communications library is understanding the limitations of each console, and when the console (or running game) can initiate communications with a PC, and visa versa. For things other than debug output (the topic of another article), you can assume that a running tool can communicate with a running game, but not the other way around. This means that the tool must initiate the communication before the console can send the necessary information back. In addition, most communication libraries perform this communication in a background thread and, if they don’t you should design them to do so. The last thing to keep in mind is that some commands may require a lot of data be sent back and forth from the tool and the game, and it is advantageous to split these commands into multiple sends of packet data, both from the tool and back from the game. A well defined command system will be able to specify just how much data will be sent, and how many packets it intends to split the data across.

So how do we go about doing this? First, consult your console’s documentation on communication. For PC, your best bet is to used named pipes. From there, the diagram at left offers a very high view of things, using command factories to create defined commands and issue responses. Here’s the basic rundown.

1. Have your game open a well known named pipe, either public (if you want to communicate across PCs) or private (if you don’t). The game can then sit in a wait state on the pipe, looking for commands from your tool. Remember, this is in a separate thread, so having it in a wait state shouldn’t impact your game.
2. Have your tool connect to the same named pipe, and issue a command string and parameters.
3. Have the game, on receiving input, look up the command in a command map. This should point to either a command factory class or command factory method (I prefer the later for memory reasons, and a class is usually overkill). The factory should return a class that inherits from a base command.
4. Run the returned command with parameters. The command should always generate some sort of simple response, be as simple as Succeeded / Failed or as complex as Need More Data, Ready To Send Data, or Ready To Initiate Communication.
5. Send this response back to your tool, which should display the result to the user.

From here, the amount and type of communication is up to you, though this can become very complicated very quickly, as you’re essentially creating your own network protocol.  However, there are a few things you should keep in mind. First, as I said before, you’ll want to design your protocol to be able to push multiple packets of information, usually of fixed size. This will dramatically reduce your memory requirements game side and will improve response on your tool side, as you’ll be able to offer more information to your users faster than if you were waiting for one large response. Second, develop a system for communicating with persistent items, such as pieces of debug information or your AI. This way you don’t have to go searching for the AI or object you’re watching or manipulating on every command, it will just always be there.