Learning about production in pre-production

Have you every been on a project with a date scheduled that says "Start production"? This is the date when the team transitions from pre-production to production and a ton of people (internal or outsourced) join the team to create the 10+ hours of gameplay to ship? The idea is that the team knows what's fun about the game and can build a ton of assets in parallel.

Did that deadline really work out? Was the team really ready to "go into production"? Most aren't!

The main goal of pre-production is to build knowledge, or learn about the game. We want to know how fun our game is, how we are going to make it and how much it is going to cost. Building this knowledge requires iterating on the areas where we lack knowledge. We choose what to work on based on a priority of the knowledge we want to build. For example, we’ll iterate on the core mechanics first because the provide the greatest knowledge of the value they provide to the game. We’ll also iterate on building characters and levels to learn how much effort and cost is required to make many more of them in production.

We often focus more on learning about the core mechanics, or the fun of the game, and not enough on how we are going to make it. This results in many projects exceeding their production budgets or schedules. What we need to do is to learn the cost of building assets during pre-production as well as learning about how fun the game is going to be. Both of these go hand-in-hand since we can’t determine the cost of production assets until we know what makes levels fun.

Let’s look at level production. Level production can easily require 50%-75% of the production budget. We need to refine our understanding of the effort to build levels during pre-production so that we do not build mechanics which inflate production costs beyond our budget. For example, a fantastic feature for the consumer would be a mechanic that allowed every object in the environment to be destructible. However implementing this may also double the effort to create levels since building destructible geometry everywhere requires far more effort. By knowing these cost implications, the Product Owner (PO) can better judge the value of adding this feature or not.

Learning about production cost is an iterative process. We begin with a range of estimates based on existing knowledge (perhaps from a previous title) and iteratively refine these estimates during pre-production. As we move forward in pre-production, we are building a vocabulary. At first we are establishing the alphabet. This may be the second-to-second game-play experiences such as battling one enemy character. We then start building words from our alphabet. These are the 10-60 second loops of gameplay, perhaps dispatching a wave of AI characters in a room. From here we build sentences, then paragraphs and chapters of larger loops of game-play.

We follow this same pattern in building levels. We don’t start by building an entire level until we build a vocabulary of rooms. We may build several different types of rooms with different experiences (e.g. lots of AI in a plain room followed by a boss in an ornate room). We don’t wait until our room vocabulary is final before we start building our level vocabulary. Building a shippable level before we have a minimum room vocabulary will waste a lot of time. This waste is seen on many milestone driven projects. Teams feel compelled to show a level of polish to their publisher when the gameplay is still undetermined. These polished milestone levels end up as either waste or debt; They are eventually thrown out or require a great deal of rework when the team learns more about the game-play.

Firing tracer rounds

Sometimes it’s valuable to demonstrate a 20 minute experience with a quick and dirty level filled with stand-in assets to build knowledge of our level goals. This in turn may influence the lower levels of our vocabulary. Another valuable test would be to create a single room with as much graphics polish as your engine can muster to help refine our visual DNA and target. Although these assets may be thrown away, their cost is easily worth the knowledge they create for your project and customer. The key is to not waste more effort than the knowledge is worth.

The PO is responsible for ROI

Since the PO owns the return-on-investment for the project, they need to demand that the team demonstrate improved knowledge of production costs refined through pre-production iterations. They do this through separate stories or conditions of satisfaction on existing stories. An example is “demonstrate the cost (in actual hours) to create the X level”.

The last pre-production release should have a level or two which demonstrates the major range of vocabulary at potentially shippable quality. This is the only way the team can demonstrate that the knowledge of production costs have been learned.

Scrum for artists

“Why use agile for art? When Michelangelo painted the ceiling of the Sistine Chapel he wasn’t using agile. He had a plan to paint the entire ceiling.”

These words were spoken by an artist I worked with when we first started using Scrum. Little did I realize at the time that Michelangelo might have been better off using a more agile approach to painting the ceiling. He had no idea about how to painted images on a curved and segmented ceiling. He signed a fixed price contract calling for 12 figures painted. Four years later he had painted 300 figures. There was a great deal of trial and error and false starts. It’s no wonder he referred to it as one of the worst experiences of his life.

Historical accuracy aside, the questions and impressions about how relevant agile or Scrum is for art creation remains common.

The following are common perceptions about Scrum from artists:

Scrum was created for programmers.

Scrum is largely used in software development projects, but it wasn’t created for programmers.

In fact it has no practices specifically for any discipline. Scrum was defined by studying how many different products (such as cameras and cars) were developed. It’s just as applicable for artists as it is for any other discipline.

Art production runs on a schedule. We can’t be iterative.

With video games, art and technology have to serve the gameplay mechanics together to create a great experience. We have to explore how all these components work together to create the best possible experience. Once we discover this, we can schedule making 10 to 12 hours of this experience. Exploring what we need to create before we mass produce assets is beneficial. Cross-discipline teams support this.

Artists often want to work with other artists.

This is the same with other disciplines. They speak the same language. The problem again is that our product requires an integration of all disciplines. This forces some form of collaboration eventually. Unfortunately a studio structured to separate the disciplines (though a heavily matrix management structure for example) will discourage collaboration or at least make it more difficult. If an artist is evaluated by the number of assets they can create (through promotions or salary reviews), then they will optimize their work along this line. Interrupting an artist to help with a game problem will impact the number of assets they can create, so they discourage those interruptions.

Scrum might be good for preproduction but not for production

It’s true that Scrum is better for preproduction than production. Production requires a more defined work flow that must create a fixed amount of gameplay to ship on a specific date. The Scrum practices should be modified by the team in production. However this doesn’t mean we won’t have challenges in production. The game will still continue to throw challenges at us that we don’t expect. These play havoc with the best planned schedules. Artists creating assets will still need rapid response to problems emerging that they can’t solve. They will need to continually find ways of working more effectively through improved tools and pipeline processed.

While the iterative nature of development will slow during production, the need to incrementally improve what we are making and how we are making it will never slow down.

What specific problems are we trying to solve on the art side with agile? How is this going to make better for the artist and team in general?

We need to know if we are creating the right thing and not wasting time.

Parallel development of assets and technology that the assets depend on is a traditional source of wasted effort. Engine development is often started with optimistic feature sets, performance goals and schedules. We can’t simply shift engine development to iterative development without an iterative asset creation effort. Iteration on technology requires an ongoing conversation and experimentation with what looks and works best. Every artist knows that the quality of a complex asset such as a level depends on a tradeoff between polygon count, texture quality, lighting complexity and the palette of effects available. None of these qualities is independent of the others. Some levels will require more effects than others and there will be a envelope of tradeoffs that occur to balance the overall aesthetic.
We want to build the knowledge of these tradeoffs during engine development before we commit to production. This requires daily collaboration between the engine creators and the first order customers: the artists that use the engine.

We need to have a working build.

Nothing will impact progress more than not having a build that fully works. Graphics bugs that impact the visual quality can prevent an artist from fully evaluating their work. What’s worse, addressing these problems can be put on the back-burner by a separate technology group that is solving problems more relevant to their own local needs. Cross-discipline teams are more likely to have someone on the team who can solve the problem or know someone who can.

We need faster tools and pipelines

Artists are often constrained by the limited or slow control they have over the game. There are often barriers to iteration that slow their progress down. Less iteration means lower quality. The fundamental problem is often that the programmers who have control over improving tools and pipelines are not impacted by their limitations. They don’t see how slow it is to change a texture on a game and see how it looks in the game. They are focused on the tasks that are important to their local team.

As with the build issues, having a programmer on a cross-discipline team experience the drag on the team’s velocity that poor tools and pipelines will help solve the problem. Smoothing the entire pipeline from concept to user experience will allow a better game to be made. Cross discipline teams are the best way for this to happen.

The role of QA on an agile team

"When I was a child we used to suck on pennies...and it was a delight!"

Every year or so I’m invited to speak at one of my son’s schools to speak at one of those parent career day events. It’s always been fun. “Video game developer” is the coolest career imaginable to these kids. I could be standing next to the astronaut-secret agent dad wearing his space suit and they would climb over him to ask me questions. It definitely scores some “dad points” with my boys.

One of the inevitable questions is “how can I be a game tester?”. They believe that the role of tester is eight hours a day of the same experience they have playing games. What could be better than that? Unfortunately I am forced to shoot down this notion. I explain to them that most jobs in the game industry are in very high demand and we get to choose the people with the best grades. Making games requires excellence in everything they are already studying. The math they may not like is what we use to calculate where Mario jumps. The literature class teaches them how to create stories which are essential to games as well. The list goes on. I merely have to point at their work on the walls of their classroom and tie this or that into work that needs to be done making games.

Getting back to the job testing…I describe the long hours with games that crash. I describe how testers often are very frustrated because they have an understanding of what the game is and have very little influence over over how the game is developed. I don’t want to discourage the future generation of testers. I just don’t want to give the impression that there is this wonderful career with a low bar of entry just waiting for them because they are “good at playing games”.

If you saw John Malkovich's monologue on SNL recently, that's how I feel telling these kids "the bad news" (note: if embedded video doesn't show up on your reader, use this link).



The traditional role of QA

The traditional role of QA on a game project is to pile on a project in its end stage. At this stage, the important decisions about the game have been made and the job of QA is to find and report the defects. Many of the defects are too deep in the design or architecture of the game to be addressed so late. For example, if the “shooting mechanic” just isn’t fun, it may be too late to correct it. Let’s just slap a band-aid on it (e.g. hide the poor aiming behavior with an effect).

As you may guess, the QA role attracts people that are passionate about games. The sheer number of people applying for these role gives the industry the ability to choose among the best. We should be leveraging the value these people bring far more.

The new role

Agile teams can extract more value from testing. Much of this value comes from the team’s goal of delivering a playable game every iteration and the principle that results of the iteration can influence the plan going forward. The QA role shifts to take advantage of this:

• Rather than sitting in a pool of testers, testers are spread through teams.
• Stories require QA action for approval before they are considered done.
• QA’s voice is heard throughout development, not just the end.
• QA represents the consumer.

The role of QA on an agile team has to change. The traditional approach of “give me the alpha game and I’ll give you bug reports” are gone. The tester’s role on an agile team has to grow. This starts with the tester’s participation in the Sprint planning meeting. When each story is discussed before the tasks are estimated, the tester needs to understand how that story will be approved. This usually takes the form of how the story and the conditions of satisfaction (CoS) are written.

For example, take the following story:
“As a player, when hit the jump button, my character will jump”.
The team would have a discussion about this story and may ask the following questions:

• Will the character have jump animation? Will it need to be smoothly transition?
• Will the character be able to jump to a higher level?
• Can the character jump from a moving platform?
• Can the character jump while walking or running?

These questions will help the team understand the tasks that will be necessary to implement the story and how to test that the story is actually “done”. For example, the jump story may define the CoS that result from the answer to the questions above:

• The character will have a simple stand-in animation that will not be blended.
• The character will be able to jump up to a ledge, or down to a lower level.
• The character will not be able to realistically jump from a moving platform.
• The character can jump from any starting motion, but momentum of the player will be ignored. It can only jump straight up.

With these CoS, the tester will be able to verify that the story is complete before the end of the Sprint.

The tester should also catch problems that are not part of specific CoS, but appear as part of development. For example if the test level is missing some physical geometry that allows the player to “fall through the world” the tester should raise the issue as an impediment to testing the game at the daily Scrum.

In addition to testing the game, the tester should help to insure that the team is not impacting other teams through their work. This would include:

• Performing regression tests on team builds if major changes are about to be committed that might break some obscure part of the game.
• Testing tools and pipeline changes that a support team might want to release to the content creators.
• Finding ways to improve the testing process. For example, if tests are frequently encountering a bug associated with bad texture imports, the tester might raise a request to have the automated tester or even the exporters improved to catch the problem earlier.

Much of the tester’s time will be spent verifying the gameplay and offering advice on areas of improvement. This requires an ongoing dialog with members of the team.

Since the tester represents the consumer, they need to keep a “consumer eye” on the game. They should point out any issue that the consumer will notice (such as a bad texture or gameplay progress stopper).

The benefit to the value of what the team adds every Sprint will greatly benefit from having a tester on the team. Customers will get what they ask for and stories will be truly “done”.

The challenge

The tester is challenged in this new role. As mentioned above, the traditional approach doesn’t work on agile teams. The tester is required to communicate with every discipline, understand the tools and stand up and be heard. Not every tester I’ve met can do this. Some I've met are more than ready. QA has traditionally been a “gateway role” for other roles in game development, so it’s an appropriate “test of fire” for people with those skills to grow their career in game development.

Scrum as a tool for process and cultural change

Any process can be used to make games, but no process is perfect. Even if we were to find a perfect process, the constant change in our industry would make it quickly obsolete. Scrum is not a process, it’s a framework for creating your own process. It can help any organization create transparency which enables competent leadership to react to change with common sense.

When we first adopted Scrum, every few months we kept observing how things had changed. “When are we going to reach our goal?” we thought. It turned out that you never reach it. Your culture has to come to terms with the fact that there is no process goal. It’s about change and that change is continual.

Take for example a studio culture that was ruled by technology. Games were seen as platforms to demonstrate technical achievement. Tools and pipelines to improve productivity for artists and designers were always lower on the list of priorities because the programmers were always trying to dig their way out of the holes they created trying to accomplish impossible challenges. The build was rarely working due to all the bugs left along the way.

How did Scrum and agile change this? First it made broken things visible. They needed to have a working, and even potentially shippable, version of the game every two to three weeks. This raised a lot of problems. At first the teams spent half their sprints trying to cobble together a working build. This killed their velocity. Since velocity is only measured by the value of the features working in the game.

This simple measurement of velocity is important. Scrum relies on a simple psychological fact; people behave based on how they feel they are being measured. Scrum measures based on the value to the product added by a team. Teams make small changes to improve velocity. Our example team did this. They started to come up with ways to improve the reliability of the build. Because velocity is based on what is seen in the game which includes art and design, they started to form cross-discipline teams to improve their velocity. Because of the overhead of communication when the team sits apart, they moved together into the same area. Because the programmers began to see how the poor tools and pipelines impacted members of their team, they focused more on improving those things.

All of these changes occurred within teams that took ownership of their work. The role of management during all this was to facilitate the change. This is the hard part. This requires management to step back where they traditionally step in. A manager needs to become like the coach on a football team. They can give feedback to the players, but once those players are out on the field, it’s entirely up to them on how the team performs.

Does every team adopting Scrum see this kind of benefit? Absolutely not. Scrum is a tool for the team to build change their process and possibly their culture. It's like owning a hammer; it won't guarantee you will build a great house, but it's a nice tool to use when building it.

We are living in exponential times

If you think the exponential growth of game development budgets is unique, watch this:

Fill your ipod with game development advice

Ryan Wiancko at Industry Broadcast has been recording podcasts of many game developer blogs and articles. Recently he has added a few of mine as well. Ryan does a great job at capturing a wide range of good posts from all over.

Who cares if it was called Scrum or not?

In distant past projects I encountered a strange phenomenon during the beta phase. Our team would enter alpha, the publisher would throw a couple dozen testers on the project and all of us would focus on the bug database. During this time I felt more productive than any other time of the project.

We would address the bugs daily. Weekly we would triage the bug database to prioritize the bug “backlog”. We tracked the total bug count and used a “burndown chart” to measure bug resolution velocity, bug discovery velocity and the projected “zero bug” date that we were all trying to reach. All I had to focus on was solving those bugs and achieving the best possible velocity.

Does this sound familiar? It’s no coincidence that many of the Scrum practices reflect theses practices. When people have a clear idea of what they need to do (in the form of smaller solvable tasks that they take ownership of), what the goal is and a solid empirical measurement that is updated daily, they can achieve a high level of focus and effectiveness.

It was one of those “aha!” moments when I came across Scrum and discovered how similar it’s practices were to what worked elsewhere.