When I worked in Quality Assurance and people asked what I did, saying “QA” returned only blank stares. Even saying “Quality Assurance” is little better for those who have never played video games. But if you say “I’m a tester at a video game developer” then there’s instant recognition: “Oh, you play video games for a living. Hahaha! Life must be so tough! Lucky you!”
This perception infiltrates through the game-development community and inter-office as well. Throughout the game-development industry, the title “QA” (often mistakenly listed as “Q&A” – my job is not “Questions & Answers”) signifies exactly what the perceptions are: high-school students working part time for a low wage, tasked with the responsibility of playing through a game for a month or two and, hopefully, articulating, at least at a very basic “yes/no” level, whether or not they had “fun”. These employees generally work for the publisher (as opposed to the developer) in the time period around the game’s release, and are hoping to parlay this light introduction to development into an “actual job” – as a writer or designer or programmer or producer, etc.
A few companies, developers and publishers alike, BioWare being one, employ more specialist teams of QA. At BioWare, we were called QA Analysts. It’s not really much of a separation in terms of name, but the roles and responsibilities form the casual tester are quite different – even if the Analyst is still required to “play the game”.
The key differences are that the Analyst is paired with developers and that the Analyst is there throughout the development cycle. And the Analyst isn’t just playing the game. Instead, the Analyst is responsible for ensuring every facet of the game is the best that it can be, while simultaneously understanding limitations in time, scope, and budget (the staff).
So how does an Analyst handle such an immense task? Well, first of all, most projects employ a streamlined, focused team propped up by contract QA. From there, the game is divided into parts to simplify roles.
For example, at BioWare we split the process into two branches – with one side focused on the game and one side focused on the systems.
A System Analyst would be responsible for, amongst many other items, the duty of testing animations; to perform this job, the Analyst might create a specific “dummy level” to allow him to quickly ensure behavior works properly in all situations.
One level I looked at often for Mass Effect was a large room with every monster type standing at ready. Since there were no scripts running (AI) and the level was just a plain room, it wasn’t taxing on the system to have so many creatures in one place. A lever in the room allowed me to make every creature cycle through their animations – from walking (in place) to running to dodging to attacking, and so forth. In this way, it was very easy for me to look over every creature in the game and quickly report any failed animation cycles. Ultimately, many levels are designed to aid in quick testing and a System Analyst will spend most of their time working with programmers.
A Game Analyst looks at created content filtered through the context of the game experience.
If a creature plays a broken animation and the system test shows that all animations are working, then the Game Analyst knows something is going wrong inside the game – perhaps with scripting. Maybe the game is telling the creature to play an animation that it doesn’t have; maybe the game is telling it to move at a rate it cannot process; maybe something else is acting on the creature to prevent it from properly playing its animation. It is the Analyst’s job to figure out what that bug is or, at least, find a sure-fire way to reproduce it so that the designer can figure it out. Regardless, the Game Analyst focuses on everything that happens while playing and works primarily with designers and writers.
An Analyst, either branch, usually has a few “testers” or “contract QA” or “junior QA” assigned to him – who also do more than just “play the game”. These testers take directed routes through the game and look for specific issues.
For example, a game tester might be told to play through one specific level as a female character using a pistol and look for balance issues (judgments on game difficulty revolving around using a “weaker” pistol versus some other more powerful, preferred attack), while a system tester might be tasked with opening a specific level and looking for bad walk mesh (places where the character can “escape” the level and break the game – and, again, specific tools are usually created to aid in this process, such as an exploding “paint” effect to help locate “seams” in the mesh).
Side note: although personal judgment plays a large role in testing, the goal is always to keep things as objective as possible. Research and test plans are great aids here.
The testers report any issues that they find to the Analyst, who is responsible for managing the bug database for his assigned level. And further up the chain of command we have the Tech Leads who are responsible for the entirety of the game or systems and their respective databases. Finally, we have the lead who is responsible for managing the entire process and, in BioWare’s case, communicating directly with the Producer.
But even this isn’t the entire picture, because it still implies a reactive testing (that is, developer creates finished content, delivers it to QA, and then the content goes through the test/fix/approved/regressed process) which is, really, an expensive and problematic model to rely on for all testing.
Thanks in part to the introduction of “agile” development, QA is now more often employed from the very start. It is still “reactive” (and QA is a service department so that’s their intended role) in the sense that a writer might put a story together and deliver it to QA for criticism and feedback, but by providing feedback through the various milestones the cost of changes are much cheaper.
That is, if you decide to change the story when the game is complete, even a small change could mean throwing out tons of scripts, character models, animations, voice acting, and so forth. A small change could effectively force a dozen people or more to spend weeks or months of time to implement the “fix”, and then weeks or months more to test the specific changes as well as everything that was touched by developers in the process. On the other hand, a drastic change when the story only exists on paper means one writer going back to the drawing board for a few hours or days.
In other words, do you really want to put QA on a project a month before the game’s release only to hear them tell you “this isn’t very fun”? No, you’d rather them let you know the game’s weaknesses when everything is in prototype and can be easily changed without impacting the schedule.
In this way, an experienced analyst can drive development, saying things like “we had difficulty testing this system on our last game because of how it was designed. I’d like to see it designed with this in mind to accommodate” or “our research shows that 23% of our customers never found this quest, and your new quest idea is set up in a similar manner. We need to come up with a solution to make it more visible if we’re going to invest so much time into creating the quest.”
As evidenced by these two examples, and the preceding comments, it should be clear that QA is not just “playing the game”. In fact, with so much going on, it is definitely possible for an Analyst to go days and weeks without even getting to play the game – at all, much less for enjoyment. Of course, while this does happen, you want to do everything you can to prevent this, up to and including working overtime. 🙂 Even as a Tech Lead, or my supervisor the lead, it was important to make sure we played through the game at least once a week, if not more, to stay current on the big picture.
While viewing the game drives knowledge and needs to be performed regularly, QA is about a complete involvement in the development process and seeing things from a bigger picture than many developers have access to. Looking for bugs is important, critically important, but it’s understood that QA has a lot of early benefit to offer also. Once everything is done, and solidly put together for beta, then everyone — QA, programmers, artists, etc. — can work together to find and fix bugs.
It takes a lot of study and experience, but it’s enjoyable and, I think, a very exciting career path – more exciting to me than many other available development opportunities. I understand this post does not cover many of the day-to-day details that arise in QA, but I hope it does a fair job at touching on the basics. Questions, comments, and suggestions are welcome.