Agile Game Development

Player roles and user stories

Clinton.Keith@gmail.com (Clinton Keith) — Fri, 05 Sep 2008 18:11:00 +0000

Many game development projects don’t put much thought into the various kinds of players who buy the game. They usually add three levels of difficulty towards the end of development as a means of adding replay-ability and accommodating a range of player skills. The levels are differentiated by a simple scaling of challenges in the game such as the number of opponents, the damage from their hits or the damage your hits cause. This reflects the amount of effort we think it’s worth.

Would we benefit from considering a broader range of players and placing more importance in their roles throughout development? Some games do this, especially some online games. An example is the popular Battlefield games which allow players to equip themselves based on specialties. If you are not familiar with the games or the specialties, they are usually divided across these roles:

Assault specialist - Equipped with an assault weapon and grenades for close quarter combat.
Sniper - Carries a high power sniper rifle and a sight that can they can use to call in precision strikes.
Engineer - Has a bazooka, mines and can repair vehicles.
Special forces - Carries a light automatic weapon and C4 explosives for sneaking around behind enemy lines causing problems.
Support - Totes a heavy automatic weapon and a radio to call in mortar strikes.

These specialties require different behavior from each player who assumes each role. They aren’t as limited as difficulty levels because players can try each specialty in any order and with any skill level. These specialties cannot be added at the end of development. They need to be developed somewhat in parallel during preproduction. They have an impact on level design and should be added well before production starts.

User stories provide a mechanism for identifying these roles and clearly communicating features related to each. The template for user stories I like is:

“As a <>, I want <> [so that ].”

A good method for identifying and differentiating goals is to phrase the user story in terms of those roles. So instead of saying:

"As a player, I would like to have a bazooka so I can blow up tanks"

the story becomes:

"As an engineer, I would like to have a bazooka so I can blow up tanks".

What's the difference? It's mainly one of value and priorities. For a generic player, the bazooka is one of a host of weapons, many of which are more important to the game. However for the engineer, the bazooka is probably the most valuable weapon. I wouldn't play the engineer without it. There's nothing more gratifying than taking out a tank with a well placed shot.

Even if your game isn't going to have specialties, like Battlefield, there is a lot of value in brainstorming the various roles of players early in development. Who is buying your game? Are you going after a largely casual market? If you are, it would benefit you to identify the "casual player" role in some of your stories. It will lead to many small decisions such as simplifying the controls or adding more checkpoints so the casual gamer doesn't become frustrated.

How Pixar Fosters Collective Creativity

Clinton.Keith@gmail.com (Clinton Keith) — Tue, 02 Sep 2008 18:16:00 +0000

HBR is posting the article free for a short period of time.

What's equally tough, of course, is getting talented people to work effectively with one another. That takes trust and respect, which we as managers can't mandate; they must be earned over time. What we can do is construct an environment that nurtures trusting and respectful relationships and unleashes everyone's creativity. If we get that right, the result is a vibrant community where talented people are loyal to one another and their collective work, everyone feels that they are part of something extraordinary, and their passion and accomplishments make the community a magnet for talented people coming out of schools or working at other places. I know what I'm describing is the antithesis of the free-agency practices that prevail in the movie industry, but that's the point: I believe that community matters.

Thanks to Clarke Ching for pointing to the article.

Overtime works with waterfall?

Clinton.Keith@gmail.com (Clinton Keith) — Tue, 02 Sep 2008 16:04:00 +0000

Many Scrum teams have found that excessive overtime reduces productivity. The frequent inspection of work done on a daily basis makes measuring productivity far easier with Scrum. One of the reasons for this is that a Scrum team can adapt their practices and see what effect those changes have on their effectiveness.

Jeff Sutherland reports that one of the companies he coaches has measured the productivity of teams using Scrum and waterfall-like practices under different overtime conditions. They produced a graph they call the Maxwell curve:

This is hardly a scientific study (e.g. I'm real curious about how they measure story point velocity in a waterfall environment) but it is a very strong visual argument for what is intuitive about how people work:

- Teams of people who take ownership of their work and make a commitment are more productive, but this high level of productivity cannot be sustained for 60 hours a week.

- When people are treated like cogs in a machine (handed estimated tasks that have to be completed to a predetermined schedule), they can indeed produce more at 60 hours a week that 40. However the productivity of cog teams is not nearly as high as committed teams because their intensity is not nearly at the same level.

Think of a runner sprinting and a jogger. The sprinter will be faster, but cannot maintain that pace as long as the jogger.

The question is "who covers the greater distance?". Does the team that "jogs" go farther in 60 hours than the team that "sprints" for 40? Maybe, but which is sustainable? Which team would you rather be on? Also, is it the same progress? Consider Jeff's comment on overtime with waterfall:

Overtime doesn't work in waterfall. It introduces technical debt. It works short term for the project leader as long as no one discovers he is damaging the code base. Velocity gets slower and slower with overtime but it may be years before management realizes they have to pay for the technical debt. By then the project leader has been promoted.

Jidoka, TDD and asset validation

Clinton.Keith@gmail.com (Clinton Keith) — Mon, 01 Sep 2008 17:51:00 +0000

Jidoka

Jidoka is a Japanese term used at Toyota which means "automation with a human touch."

It's origins lie in the early philosophy of Toyota (at the time it was called Toyoda). Part of this philosophy was to minimize labor costs by reducing labor waste. A example is the creation of "Type-G automated loom" in 1924. Before then, each loom was watched for thread breakage by a single operator. If a broken thread wasn't caught quickly, it would ruin an entire run of cloth. The entire process was very wasteful in operator time (90% waiting) and ruined cloth.

The innovation of the type-G loom is that it would automatically stop whenever the thread broke. This allowed a single operator to support dozens of machines and virtually eliminated bad production runs of cloth due to broken threads. Quality went up.

When Toyota started making cars, the philosophy of Jikoda was carried over to the manufacturing process. On the Toyota factory floor, a problem can potentially stop the entire line until it is fixed. Once the fix is identified, the standard process is improved to prevent a recurrence of it in the future.

The following diagram shows the Jikoda flow

TDD

Those of you using TDD (Test Driven Development) should recognize this flow. Unit tests are introduced for every function in the codebase. These unit tests provide validation of those functions that are run when changes are integrated into code base (by a continuous integration server).

When we discover a bug, we must solve it immediately or end up stopping the line (stopping the commits). When the bug is identified, a fix and a unit test, to catch further recurrences of that bug, are checked in and work continues.

Why solve bugs immediately?

Bugs can cause the entire team to lose work. A build that crashes can waste hours of work across the entire team.
Bugs cost the least to fix immediately after they are created. Bugs fixed months later in "alpha" can cost 10-100 times more.

The practices for TDD are well established. Tools like Cruise Control allow an easy integration of TDD into any development environment.

Asset Validation Jikoda-style

What we need are similar tools and practices for a version of asset TDD.

Unit testing for assets should:

Catch assets which break the build
Catch assets with naming convention errors
Catch assets which violate budgets (texel density, memory footprints, poly counts, bone counts, etc).
Identify and track approved assets versus unapproved assets that need art lead approval

Bad assets are another form of debt, like bugs. They cost more to fix later. A more automated approach to checking assets will help keep this debt, which is waste, low.

I've always been on teams that do some of these steps, but they weren't complete and they were always implemented later than they should have. It would be great to have some more standardized tools. Hey Autodesk!.....

Should the Scrum Master also be a member of the team?

Clinton.Keith@gmail.com (Clinton Keith) — Fri, 29 Aug 2008 22:49:00 +0000

This question is raised frequently. I prefer that a Scrum Master (SM) not be a member of the team, but this is not a strict rule of mine. An SM as team member can create problems:

A focus on their own tasks over the team issues.
Prioritizing their own impediments over those of other team mates.
It can impede team ownership through the “Scrum Master as Team Lead” anti-pattern.

These problems are usually created subconsciously, but they do create barriers to team commitment. Sometimes there is no choice but to have the SM be a member of the team. There may not be enough teams for the SM to fully occupy their time. An SM can handle 3-4 teams before it starts becoming a full time job (your mileage may vary).

If the SM has to be a member of the team, keep an eye out for these problems. Raise them in the retrospective. It’s not easy, but it has to be done. One great way to overcome this is to rotate SM duties. If your current SM is the only one certified or experienced, they should coach the other people who rotate into the role.

Rotating the SM role has many benefits. Team members appreciate and understand the role much better after trying it.

Survey - When and where to hold next CSM class?

Clinton.Keith@gmail.com (Clinton Keith) — Wed, 27 Aug 2008 16:42:00 +0000

I've received many requests for another 2 day "Certified Scrum Master for Video Game Developers" class (the one in Austin last May filled up).

So I'd like to collect a your opinions about when and where to hold the next class. If you are interested in attending one of these, please answer a few questions in the short survey below.

Click Here to take survey

Thanks!

Agile values - Individuals and interactions over processes and tools - Part 2

Clinton.Keith@gmail.com (Clinton Keith) — Tue, 26 Aug 2008 20:21:00 +0000

Question: How does a large software project get to be one year late? Answer: “One day at a time!” - Fred Brooks

Our processes and tools to manage ever growing projects have built up over decades. Large teams have driven the creation of hierarchies of management. Project schedules and design documents attempt to predict every requirement and task necessary to make a fun game now require expensive databases to manage. All of these are considered necessary to tackle the complexity the rise from having upwards of a hundred people working on a project for years.

Game development requires developers of widely different disciplines. Take for example a cutting edge AI character that needs to walk around an environment and challenge the player in the game. The creation of this character requires the participation of animators, designers, character modelers, texture artists, programmers audio composers among others.

It’s important that these disciplines collaborate as much as possible to be the most effective. If the animator is experiencing a problem with the animation technology, then it is important for them to work with a programmer as quickly as possible to address the problem so they can continue working. Often the processes and organization will prevent that. In our example with the animation bug, the programmer may be working on a series of tasks that were assigned by a lead. This may prevent that programmer from helping out the animator without permission from their lead. This leads to a chain of conversation as shown in the following figure:

The animator has to pass the request up through the chain of command which then has to make it back down to a programmer who can solve the problem. In this example, the change involves five people and four requests! This flow is prone to failure and delay.

So what has been described?

Over 100 people of various disciplines on one team.
Thousands of unpredictable problems that can introduce wasted time and effort
Detailed plans and tools to manage them that can’t predict these problems and are potentially inflexible at changing.
Hierarchies of management that can lead to further waste.

Agile can address these issues from the bottom up. This is a major benefit from self-managing teams. They self-manage the smallest level of details, not necessarily the highest levels. They aren’t self leading teams. They unburden leadership of the role of managing minor details. They allow leadership to focus on the big picture.

When teams learn that they can take a small amount of ownership to solve the smallest problems, they start taking on larger problems. They begin to ask for more ownership in other areas:

In creating better teams that can solve more problems by reducing external dependencies.
In demanding more of themselves to achieve commitments they own.
By identifying risks early and addressing them before they become problems at all.
By growing leaders among their own ranks.

Agile values are preferences and not binary decisions. We can still have process and tools to support the team, but we value individuals and interactions more to solve most of the problems that occur on a daily basis.

Scrum Rising article posted

Clinton.Keith@gmail.com (Clinton Keith) — Tue, 26 Aug 2008 19:56:00 +0000

I've posted my February 2007 Game Developer Magazine article "Scrum Rising" here.

It's an overview of applying Scrum to game development.

Scrum makes you smarter

Clinton.Keith@gmail.com (Clinton Keith) — Mon, 25 Aug 2008 15:52:00 +0000

http://jeffsutherland.com/scrum/2008/05/scrum-makes-you-smarter.html

"Recent research shows that people who work in command and control environments actually lose a significant number of points on their IQ. They get stupider."

Agile values - Responding to change over following a plan

Clinton.Keith@gmail.com (Clinton Keith) — Mon, 25 Aug 2008 00:22:00 +0000

“Plans are nothing. Planning is everything” - General Eisenhower.

When the allies stormed the beaches of Normandy in 1944, they had a very detailed plan of attack. This plan led to their assumed victory over Germany before Christmas of that year. That plan was doomed from the start. How could it be otherwise? How could a plan for hundreds of thousands of soldiers, millions of pounds of supplies and thousands of missions all work out according to a plan on paper? Eisenhower knew it wouldn’t. That’s why there were always contingencies. That’s why the allies were always adjusting the plan based on the reality of the emerging battles.

Agile project management follows similar reasoning. A popular misconception about agile is that it doesn’t allow for plans. This isn’t true. Agile focuses on the activity of planning rather than focusing on a fixed plan. A certain amount of knowledge about any project is going to be well known. Just as the allies knew the location of the beaches they were going to land on, a game development project will begin with quite a bit of knowledge about the genre and market targets for the game to be developed. It is necessary to plan for these things and to share that plan. The problem occurs when the planning goes too far. I’ve seen a game design document for a fantasy shooter game that had the bullet count for each of the clip types of every weapon. How can we know how many bullets per clip we should have in a game? Why do we need to plan for that detail before we have the knowledge of what we need? This is an example of the source of problems that detailed plans can create. If the team sticks to the detailed plan, then it won’t be the best game possible. Great game emerge.

The agile approach is to plan for what is known and to execute against what is not known iteratively. In other words, agile is focused on gaining knowledge. If we don’t know how many bullets in a clip or what type of weapons will be the most fun, we don’t try to plan away that uncertainty on paper. We address it head on by building weapons early and learning what is best.

Many elements of a plan are assumptions. We assume the things that we put into a game design document are going to contribute to a hit game. Some of these assumptions are correct, some are not. By implementing and testing these assumptions, we change some of them. We then alter the plan to create a better game.

Looking for a great programmer who does TDD?

Clinton.Keith@gmail.com (Clinton Keith) — Sat, 23 Aug 2008 17:00:00 +0000

A former co-worker. Very talented and only interested in a US game developer who embraces agile & TDD. Highly recommended. Contact me.

Iterations and vertical slices

Clinton.Keith@gmail.com (Clinton Keith) — Thu, 21 Aug 2008 21:53:00 +0000

Any game developed using agile makes progress using iterations. The goal of every team for every iteration should be to make progress by adding value to the game or pipeline for a customer. This might be a the improvement to a core feature for the player who buys the game or a function for the animator using the asset pipeline or a tool.

Iterations (or Sprints) are like mini projects by themselves. They often include design, coding, asset creation, tuning and debugging. However we are not always producing full vertical slices of a game every iteration. We’ll use a an example of what we might deliver for a team which is focused on creating AI behaviors: One of the most difficult aspects of AI behavior is navigation in a complex environment. The AI logic has to identify objects that will prevent the NPC from moving and calculate a path around them. Throw in some other moving characters and objects and the problem can become very complex to solve. Navigation can become the most riskiest problem to solve AI and therefore one the the riskiest problems to solve for the entire game.

Naturally we would want to solve the navigation problem as early as we could. The problem with doing this work early is that other related systems such as character animation and physics might not be developed enough to support the sprint goal of having a polished AI character walk through a complex environment. In this case, the iteration goal for the team might be to demonstrate simple cylinders navigating a complex test environment.

Does this solve all the risk associated with AI characters navigating complex environments? No. It probably addresses a big part of the problem, but there may be other problems that show up when progress is made with the animation and physics. The benefit is that those systems will have some basis to work against. If we find that our cylinders have problems navigating up stairs, then that might influence animation and physics work over the upcoming iterations.

It's a much more intractable problem when you discover your nicely animating AI character can’t climb the stairs that are duplicated in every level.

Iteration goals are prioritized based on value, cost, risk and knowledge. Addressing high risk and creating knowledge sometimes drives the highest priority stories on the backlog. A Product Owner has to understand the risks of the project as well as the value of the features when prioritizing.

Agile values - Individuals and interactions over processes and tools

Clinton.Keith@gmail.com (Clinton Keith) — Wed, 20 Aug 2008 22:30:00 +0000

The agile manifesto defines four values that the criteria for any method which calls itself "agile". I'm going to cover the ideas behind each:

Individuals and interactions over processes and tools

Back in the early days of mass-production, when companies like Ford were discovering the efficiencies of assembly lines, the theories of Fred Taylor were being applied in factories. These theories were based on the idea that complex jobs, such as assembling a Model-T car, could be performed by relatively unskilled and cheap labor if each step in the assembly line was broken down to a simple job that could be standardized. Workers were considered to be interchangeable cogs in a large machine.

Though “Taylorism” has been discredited, many management practices are still based on the idea that as long as a well defined process and set of tools exist, then the work that can be done by a group of people can be predicted and defined up front. Workers are considered “fungible” or replaceable and anonymous on a large project plan that attempts to predict what each part of the whole will take a worker to complete. Managers find comfort in this. As long as people and hours are are interchangeable, then we can play with the amounts of each to achieve predictability. This idea has been disproved for decades. It was the basis of Fred Brooke’s book “The Mythical Man Month” in the sixties. People and months are not interchangeable.

This is why agile values individuals and interactions over processes and tools. Teams of people make games. Artists, programmers, designers and testers all have to communicate to be the most effective. Collaboration and shared vision are key to great teams. Tools and process are still important, but they can’t replace this.

So why do we cling to this idea? Because it’s easier. Process and tools, interchangeable hours and people, spreadsheets that are meant to be crystal balls only require management. Collaboration and shared vision is harder. They requires leadership. Leadership is harder than management.

Production Debt

Clinton.Keith@gmail.com (Clinton Keith) — Tue, 19 Aug 2008 16:35:00 +0000

Have you ever seen a schedule for a game project that picks a day for production to begin? I think I've seen that on every project. Some day is chosen, a year or so in advance, and you have to start creating "production assets (levels usually)" on that day. I've never liked it and it's never worked out very well.

The problems are:

Teams never transition from pre-production to production as if a switch were thrown. Different asset types transition at different rates. E.g. we may have level production down, but our animation system is still lagging.
Where does this date come from? How do we know how much time we need for pre-production, and more importantly how do we know how much time we need for production, especially when we have a fixed ship date? How do we know if the time slot we're giving ourselves is enough so that we don't paint ourselves into a corner? How many times have you entered a nine month production phase with 12 months of work to do?
What are the criteria for determining we CAN be producing assets? The date comes and we have to be in production, so we usually tell ourselves and our publishers "sure...we're in production! All those assets you see there are really production assets that aren't done...yeah". You're still iterating to find the fun and you can't iterate as much in production!

Measuring Production Debt in Preproduction

Production work is a necessary debt that we create in preproduction. One of the goals of preproduction releases (or milestones) is to measure that debt. During the first few releases that debt will be uncertain. You might say "we believe that we have 1000 people months of production work, plus or minus 200". The project would continuously refine that number during subsequent releases. Towards the end of preproduction you want it to be a much more accurate number.

Why Measure Production Debt?

Imagine you are on a first person shooter project that has 12 levels planned and 12 months of production scheduled. During development your team implements some cool technology that allows every piece of non ground geometry to be destroyed or have holes blown in it. This is a great new addition but it creates a problem. It doubles the amount of effort required to build a level.

How do we react to this? The first problem is that we have to know the problem exists. In many cases we don't do a proper job of measuring the production impact of design and technology changes. We paint ourselves into the proverbial corner. This is why demonstrating the cost of production should be a part of every release review.

If we know that we will double the level production debt with a new feature developed, such as the destructible geometry feature, we can make solid decisions early. Some choices would be:

Drop the feature because we can't afford it
Drop half the levels
Start production earlier
Extend production and ship date
Scale up production resources

Some of these choices are harder than others, but they are all better choices than the one that is usually taken: trying to stuff 24 months of production effort into 12 months of schedule.

How to Measure Production Debt

The only way to measure production debt is to make a small set of production quality assets, measure the effort and scale the measurement. For our example, our release would demonstrate a small level with a couple of buildings that are destructible. If we estimated that the demo level was 10% of an entire level we could say that the level asset debt would be 10 times the effort spent on the demo times the 12 levels we are planning.

As mentioned this would be an inaccurate measurement in early releases. We can't always make truly shipping quality demo levels early in a project. Subsequent releases would get closer and the estimate more accurate.

The good thing is that we can expect that we'll get more efficient at creating levels during production and that will free up time to incrementally improve the levels during that time.

What Happens to the Production Date?

Publishers will rarely allow a team to tell them "we'll pick a production date when we know more". It's OK to pick a production date with the caveat that the date will be refined per asset type every release. Back the refinement up with higher quality asset demos every release. Publishers and developers should have ongoing discussions of the questions raised above when it's clear that new features are impacting production debt. Publishers rarely get good information about cost and benefit information about major features.

Production debt (like every other debt it seems) always grows. The problem with production debt is that we can't retire it in an ongoing was as other debts (design, technical, etc.) are. Don't make your team attempt to pay it off with crunch.

Cooper Product Design applied to Video Games - Part 2

Clinton.Keith@gmail.com (Clinton Keith) — Sun, 17 Aug 2008 23:14:00 +0000

In Part One, I described Alan Cooper's model on how agile fits in with his Goal Driven Design process. It made a lot of sense to me and it bears examining if it can be applied to game development.

As discussed, Cooper doesn't shy away from applying phases to product development...something that many agilists do. However his phases are more integrated and overlap their responsibilities throughout the entire project. So rather than calling them phases, he refers to them as "stages".

Comparing this to games, we typically have two distinct "stages" that be identified in game development: preproduction and production. I added the concept stage in part one since games do not start iterating from a blank slate on day one.

To better apply the Cooper design principles, I'd now like to divide up the pre-production stage into two stages: design and engineering. These are not phases divided by time. Both are intertwined, but they are answering two different questions. Design is focused on "what" we are building. Engineering is focused on "how" we are going to build it. This would include the tools as well as the engine. They are really both design stages, but the goals of what they are designing are a bit different, so I'll separate them here.

Here are the four stages we are going to talk about:

We ask questions of the first three stages and make a statement at the production stage:

The stages that ask questions seek correctness. They want the right answers. In the production phase, we seek to maximize efficiency while we are building. We know the answers from the previous stage. Entering production without these answers is one of the most common reasons for cost and schedule overrun.

The "toolsets" for each stage are different. In the concept stage we use our intuition to determine which path to take. In the design stage, we're focusing on player and stakeholder needs. The engineering stage focuses on how these ideas are going to be realized and if it's possible to produce with the resources we have . The production phase focuses on efficiency; how can be build things faster, better and more inexpensively.

There are three different states of mind that for the four stages. The two middle stages are where collaboration occurs among mixed discipline teams. The concept stage is more solitary. It's driven by vision and insight. Consider visionaries such as Miyamoto. Their major contribution comes from their vision and concept of game ideas. You can't take a group of non-Miyamoto (lesser) game designers and expect them to scale their vision to his level.

The production phase is not as collaborative either, especially in cases where we are trying to outsource production assets. It's more about the flow of productive work and improving the flow. This does require some collaboration, but it's not the same. We shouldn't stop to discuss and change the goals with the customer at this point. Changing genres when half your levels are produced is a bad thing.

Finally, let's look at the procedural approaches we use in each stage. In the concept stage, it's pure iteration. New ideas are tried and old ones are discarded. We have total freedom to explore in any direction necessary. Insights are not incremental.

The design and engineering stages are iterative and incremental. We increment towards production through time-boxed iterations that improve value toward customer value (humans) and the means to provide it (technology).

The production phases is incremental. We are not iterating on things we discovered in preproduction. We're mass producing the assets and finding ways to incrementally improve how we create assets.

Where does agile fit in?

The two center stages are the agile stages. This is where fully collaborative cross-discipline teamwork takes place to push the product forward and achieve the best results.

The other stages Cooper refers to as the "fragile stages". The concept stage is "magical" and unpredictable. It's hard to know when it is successful. You can't create a large team do it faster. The production state is fragile because of its size, length of time and potential for waste.

However with the production stage, we can apply Lean principles which are largely derived from product manufacturing (e.g. Toyota). These focus on eliminating waste and focusing on where value is added. It focused on a constant set of incremental improvements that do support collaborative work but also work standards for larger groups of people.

Conclusion

I feel this model has some value. Over the past five years, I've considered & discussed a couple of the issues with adopting agile for video game development:

- Game development is a combination of product development (preproduction) and manufacturing (production). Applying the same Scrum practices to both is not ideal.

- Our one release model for large scale games isn't an ideal fit with the regular release model of other products where agile is more commonly applied. We can't get proper customer feedback for our iterations. Focus/play testing is not enough. We need visionaries.

Stages seem a good fit to describe what we do on an agile game project. Cooper's breakdown adds value in discussing the practices, tools and goals of each stage. Stages also address the problems of phases by not handing off responsibility between them.

Scrum is a good fit for the design/engineering stages. Based on production experiences, Lean is an ideal process for the production stage. Many studios do the concept stage, but as Cooper says: "it's magical". How did Apple create the concept for the iPod? What practices do we create for the concept stage that allows many ideas to be inspired and filtered? How do we know when we are done?

Cooper's model is a step forward is defining that.

Article - How to fail with agile

Clinton.Keith@gmail.com (Clinton Keith) — Sun, 17 Aug 2008 20:15:00 +0000

Better Software Magazine just published an article that Mike Cohn and I wrote called "How to Fail with Agile". It's a tongue-in-cheek set of advice for those seeking to fail at agile.

You can find it here.

Certified!

Clinton.Keith@gmail.com (Clinton Keith) — Sun, 17 Aug 2008 17:26:00 +0000

After months of review, I've finally been approved as a Certified Scrum Trainer by the Scrum Alliance!

What does this mean? It means I can "officially" train Scrum Masters anywhere. It means that every Scrum Master I train is certified through the Scrum Alliance and has full access to the resources there. It also gives me valuable feedback and instruction on what and how I teach these courses.

Having access to members of the SA such as Ken Schwaber, Mike Cohn, Jeff Sutherland, etc has been a huge benefit to me as it will be to every CSM I teach who automatically becomes a member.

Plus I get to use this nifty logo:

Cooper Product Design applied to Video Games - Part 1

Clinton.Keith@gmail.com (Clinton Keith) — Wed, 13 Aug 2008 19:25:00 +0000

One of the keynotes at Agile 2008 was by Alan Cooper on his views on agile. Alan is an agile proponent who has differences with some of the other popular "agilists". Alan a more design and interface driven approach to product development. I feel that Alan's approach to design and construction as necessary stages of product development, surrounding an iterative and incremental core, mesh well with what I have experienced in game development.

When we talk about the waterfall methodology these days, we're usually not talking about the original definition of waterfall defined by Winston Royce in 1970. We're talking about projects that are mostly planned up front (using BDUFs) and designed/optimized/tuned at the end. The issue we have with this is that the documents & schedules created up front are usually wrong and we end up with a poor product or blown schedule and budget.

Conceptually, waterfall looks like this:

The arrows are hand-offs from one stage (or phase) to another. Alan refers to these as not only hand-offs of work, but hand-offs of responsibility. The difference with agile is that responsibility is not handed-off, but continues throughout the project.

The popular opinion is that agile does away with these stages or at least mixes them up every iteration. While this may work for mostly software driven projects that have regular and frequent releases to customers, I don't believe it's ideal for games. A typical console game has one release in 2+ years and has a large content component that must be mass-produced at the end.

In 2002, I read about the Cerny Method for making games. This was a major inspiration for how games should be developed. Later, when reading about agile, I felt that an agile model for development could support the Cerny Method very effectively.

The Cerny Method identifies that the goals of preproduction and production should be entirely different and managed differently. Preproduction should be exploratory and minimally planned up front. The goal of preproduction is to find the fun of the game and produce a few shippable levels which would answer all the major questions of cost and quality for the production phase. The production phase would be entirely planned against the knowns from preproduction and executed to that plan.

What agile does is provide a framework for these stages. Preproduction is iterative and incremental and can be based more on the Scrum practices. Production is mostly incremental and can leverage the lean production practices. Production is less iterative due to the cost of iterating on a large number of produced assets. For example, we don't want to iterate on the jump height of a platform game after we have created a dozen level with hundreds of platforms based on the previous jump height!

I've left concept phase in to represent the idea that we don't start creating incremental value on day one. Concept is purely iterative phases where we develop our "big ideas" about the game before pre-production. A opposed to production, concept is where we are meant to throw out most ideas. Concept could be very short, but the responsibility of the designers in concept does not end when the project moves to preproduction.

If we look at the stages with Cooper and Cerny in mind, the responsibility for the concept of the game continues throughout the entire project. This doesn't mean that the concept will iterate throughout the project, but that there will be collaboration and overlap between the the groups. It also means that the boundaries are a bit fuzzy. The same goes for preproduction. We're not going to finish pre-production on Tuesday and start production Wednesday on all assets. We're also going to encourage refinement and incremental improvements during production.

In the next part, we'll look at how the tools and techniques differ for each of the three stages.

Warnings from Chris Deering

Clinton.Keith@gmail.com (Clinton Keith) — Mon, 11 Aug 2008 16:28:00 +0000

From Gamasutra this morning, from former Sony Europe boss and current Codemasters chairman Chris Deering:

----

Despite all this growth, however, Deering warns that current development costs, currently in excess of $10 million for major titles, are unsustainable, given that less than 3 out of 10 games actually recover their costs.

Moreover, he says software sales may decline even as hardware proliferates. "Traditional revenue sources will not be sufficient to fund games development," he says. "Especially as global retail sales will be 20 percent lower in 2011 than in 2008."

----

We're still a hit-driven industry (where a profitable game will pay for a larger number of unprofitable ones), but the margins of profit for the hits are declining. This is due to higher development costs and what he refers to as a decline in software sales.

I hadn't considered the decline in software sales part, but if fewer games are made that leads to fewer choices for the consumer which leads to the consumer buying fewer titles overall then that makes sense.

IGDA Leadership Forum

Clinton.Keith@gmail.com (Clinton Keith) — Sat, 09 Aug 2008 19:55:00 +0000

The IGDA has recently opened up registration for the upcoming Leadership Forum. This is the second year of this forum. Last year's event was a bug success and very worthwhile. Any producer or leader should sign up quickly before they fill up. I suspect it's going to sell out quickly again.

I'll be presenting a talk called The Myths of Scrum. We're going to have some fun with this topic!

Agile 2008 Conference

Clinton.Keith@gmail.com (Clinton Keith) — Sat, 09 Aug 2008 16:01:00 +0000

Just returned from the Agile 2008 Conference in Toronto. This is a great conference that applies the agile principles in its organization and execution. Lot's of impromptu "knowledge jam sessions" and even an area for the musically talented to get together and play. It's can be a bit chaotic, but not dull.

The conference had ~ 1600 attendees. It's been growing pretty steadily since it was formed from merger of two smaller conferences about five years ago (including an XP conference). Many popular agile, XP, Scrum and classic CS authors were there. Most were very approachable and friendly.

The attendees were divided up equally between coaches, developers using agile and people new to agile. There were about ten tracks and about 40-50 talks going on at any one time. The sessions dived deeply into agile problems, developments and successes. I gave a 90 minute presentation on agile game development and participated in a CTO panel run by Jim Shore and Dianna Larsen.

The conference was very lightly attended by game developers. I believe there were only three of us, but we smelled each other out soon enough. The other two were a couple of developers from a large company that makes "computerized gaming machines" in Nevada. We had a great conversation over some tasty Indian food about their challenges and stories of making video gaming machines for casinos.

My focus at the conference was attending sessions on building and facilitating great teams. There were plenty of sessions on this topic. I received an overwhelming amount of information on theory, team practices and coaching in the area of hyper-productive teams. You just can't fully appreciate the value of a daily scrum until you've heard about the decades of research on group theory and complex adaptive behavior systems that reinforce the practice. :)

Toronto is a great place for conferences like this, as long as it's in the summer. There are plenty of after hours things to do and great places to eat. All the city dwellers take full advantage of the perfect summer climate. Due to it's terrible winter weather, there are many malls below street level that you can get to quickly for lunch or a quick Starbucks fix.

Next year the conference will take place in Chicago. Hopefully more game developers will attend (think great food and Rush street).

Scrum experience at Nokia

Clinton.Keith@gmail.com (Clinton Keith) — Sun, 03 Aug 2008 15:06:00 +0000

Interview with Scott Foe, Producer at Nokia: http://www.gamasutra.com/php-bin/news_index.php?story=19210

Quote:

You said you're using Scrum.

SF: That's the project development methodology that we've employed. You know, Scrum is - especially with the recent [Game Developer magazine] article from, I believe, last year, “Scrum Rising” - a hot topic in the games industry of late.

People have strong opinions.

SF: People have strong opinions. I definitely have my own strong opinions. It's definitely not a silver bullet, nor is any development methodology. I mean, for example, yes, Scrum leads to greater team communication and cooperation, but if one of your team members is an axe-wielding barbarian, then that might necessarily be a good thing.

But there is certainly, in publishing minds, this kind of idea that Scrum is this evil dragon that - “Oh, if we don't have a waterfall project planned with everything mapped out in detail, then this project is just going to go way overboard, and way over budget.”

I have to say that if you're doing a quality-driven title - so I'm basically separating the universe of game projects into release-date driven, and quality-driven; release-date being, “We gotta get it out by this date,” and quality driven being, “It's gotta be out when it's ready” - for a quality-driven product, using an iterative development model like Scrum is excellent for bringing that home, and for revisiting issues, and making sure that everything is just perfect.

For a release-date driven model, Scrum is excellent because, at the end of every sprint - Scrum project cycles are broken down into a number of sprints - you have something which could conceivably go to quality assurance. So if your due-date has to be - cannot miss! - this certain date, then at the end of every sprint you have something that you can conceivably ship with. Whether that's good or not, that's in debate. But you could theoretically launch it.

From the publishing side, Scrum has very few artifacts, so when you go into a Scrum product development methodology, instead of having, like, oodles of Microsoft Project files and Excel files, and this and that, what you have is the product backlog. People take product elements off the product backlog, the team puts it in the sprint backlog, they complete it, then they give you a sprint report and a playable build. The sprint report and the playable build are probably the best visibility into how the health and well-being of the development project of any of the corporate artifacts that come from the different development methodologies out there.

So, for example, if you're at a publisher, and you really want complete visibility into how is the team doing, what are they accomplishing, you have that sprint backlog and can see, at the end of each sprint, what the team did, what they failed at, and why.

You can't get more clear-cut than that. Having the actual sprint review build, that build of the game, to be able to take around internally and say, “Here, look. This is what the game is about. This is how we're doing. This is how we're going.” There's no better feel for how a project is going, instead of waiting however long for a given milestone build.

From the contractual side? I mean, it requires flex. Say you're doing a release-date driven title, and you're making a contract, and you know the team is operating on a Scrum development methodology. Well, it's easy to say, “This is the due date. These are the payments. And go. These are the number of people you have, and we'll be out on this date.”

Now when you go for a quality driven title, contractually it becomes a bit more sticky, in that you either have to, one, plan on going back to amendments to the contract to say, “Okay, we need two more sprints. We need three more sprints. We need this many more sprints.” Or actually building into the contract the possibility of approving and paying out more sprints on an as-needed basis. And of course, most business development people, and people who write the contracts in the publishing arms of organizations everywhere are probably not familiar with this, and very married to the standard way of doing things, which, of course, causes friction within the publishing organization.

Again, not a silver bullet, but I definitely couldn't see myself using any other development methodology that's in current practice today. I definitely am a huge Scrum fan.

I imagine that kind of heavily iterative method would be particularly suited to a game like this, where you've got fairly disparate gameplay styles that you're bringing together. I suspect that took a lot of trial and error.

SF: Most definitely. I mean, a piece of paper is never fun. Right? First person shooter is easy. Well, easier. You've got a point of reference, you know you're going to need some weapons, and some enemies, and maybe you're going to do an amazing narrative structure, and blow everything else out of the water. But for the most part, you know what you're aiming at.

For something where you are planting the flag, and you are running out ahead of everybody, and there are rocks, it does take time and iteration, and polish to get the jetpack to be as fun as the jetpack could be, or to make sure that the princess-rescuing is satisfying.

One other point, going back to the transparency, I don't think it's really part of the standard Scrum development methodology, although I may be incorrect about this, but one of the things we found really useful on this project is doing our burn-downs by discipline, so you have a burn-down chart which is basically, here's all the work that needs to be done on the project, and you watch it burn-down as elements are taken out of the product backlog.

When you burn-down by discipline, and you see the velocity of how the project is going, you can say, “Oh, we need more artists, or we need more designers, or we need more programmers.” And it becomes more obvious, sooner, although, again, that might already be written down somewhere. I haven't read all the Scrum books. That might be out there.

Hyper-productive Teams

Clinton.Keith@gmail.com (Clinton Keith) — Wed, 30 Jul 2008 21:23:00 +0000

A hyper-productive team is a phrase often used in Scrum literature to identify teams that have achieved a state of ownership, commitment and collaboration that allows them to be far more productive in creating “product value” on a regular basis.

I view hyper-productive teams as independent of Scrum. There are many examples of teams that have been hyper-productive independently of the process they’ve used. I’ve been on several in my career from the defense industry to the games industry. They are the highlights of my career. I never worked harder and never had more fun than when I was on those teams.

The first was the YF-23 avionics integration team that I worked on for 10 months in 1990 at the McDonnell Douglas plant in Saint Louis. We were led by a retired F-14 Tomcat squadron commander. We actually had 10-15 minute stand-up meetings every morning to go over the issues and daily goals. He called it the huddle. We were a team under siege in a hostile environment as contractors working the midnight shift.

Another hyper-productive team was the Midtown Madness One team. Most of the team (including me, the product director) had never worked on a game before. However, we were given almost complete autonomy by Angel Studios (and largely by even Microsoft) to build the game we wanted. I recall all of us stopping work to play the game at 6 PM every night and then stopping at 9 PM to drink beers in the conference room and discuss ways of improving the game. I would arrive some mornings to find that members of the team had reworked the rendering engine or vehicle dashboard models overnight to greatly improve a weak part of the game. We didn’t use Scrum either, but we had some TDD like practices and even posted burndown charts which visibly tracked key statistics such as polygon budgets for the level, etc.

Hyper-productive teams are hard to define. To paraphrase a common quote: “it’s like porn…you can’t define it, but you know it when you see it”. In addition to talent, hyper-productive teams are the greatest contributors to the creation of a hit game.

I try to define conditions that support, or are common to, hyper-productive teams. In addition to educating teams on how to use Scrum, a main goal of mine is to facilitate teams on becoming hyper-productive.

So what are the conditions that allow a hyper-productive team? I’ll identify a few I feel are important:

Independence and a sense of ownership - The team needs to feel that they can contribute creatively and have some control over the game. This isn’t necessarily total ownership, but some.
Leadership - There needs to be one leader on the team who can communicate a vision between the team and the customers and keep the team focused.
A core competency - Not everyone on the team needs to be an expert, but on the hyper-productive teams I have seen there have been at least one core expert. This isn’t a lead position defined by a role, but by actions. This person supports the vision with brilliance that the team can be confident in and rally around.
Team collaboration - Teams grow into great teams organically. There are countless articles and books written about how this happens (here is one of my favorites). This is where facilitation can help a team transform itself.

As mentioned, these teams form independent of process. However, Scrum can support and nourish hyper-productive teams based on its practices and principles. Team self-organization, Sprint goal ownership and commitment and a daily dose of visibility can support and grow hyper-productive teams.

Product Owner as Pig?

Clinton.Keith@gmail.com (Clinton Keith) — Mon, 28 Jul 2008 22:21:00 +0000

Scrum defines two types of individuals associated with a project using Scrum; one type is fully committed in a project, the other is merely involved. A commonly referenced joke about the commitment of the pig and the involvement of the chicken in a ham-and-eggs dinner illustrates this and is used to name the roles. The product owner is often referred to as a “chicken” because they aren’t committed to the work to be accomplished by the team every Sprint. They are only involved with the outcome while members of the team are called “pigs” because they truly commit to the work of the Sprint.

The product owner (PO) for a video game project can be compared to the role played by key visionaries such as Shigeru Miyamoto, Will Wright, Tim Schafer and Sid Meier on their projects. A PO represents the ultimate customer during development: the player. The PO has to foresee what the market will want up to three years in advance. They have to know the mind and emotional responses of the player. Pure iteration is not enough for video game development. We typically have one true release to get things right. Most of our games can’t slowly grow their feature sets and market simultaneously like other products. This requires great vision. This is unavoidable.

You might ask “can’t we iterate and focus test our iterations?”. I wish it were so, but over the years I’ve discovered it isn’t enough. Focus testing will tell you how your game compares with the focus tester's current favorite games. Additionally there is a huge discrepancy between the people you focus test and the people you have to impress to get market awareness (e.g. reviewers and hard-core-gamer-mavens). That’s a bigger topic.

Originally the Product Owner role was considered a “chicken” role. That’s to say it was a role that required no commitment to the Sprint goals. This has changed over the past as the product owner role has been identified as a key to success or failure of a project. This is no more true that for game development. As a result, many projects consider their POs as “pigs” (we used the terms “ninjas” and “pirates” for these concepts of commitment because, for some reason, people dislike the labels "pig" and "chicken").

“PO as pig” is needed because we can’t always know what’s fun two weeks from now. Questions about how “bouncy” a physics response should be or how “snappy” an animation transition needs to be, etc, etc are subjective and may need daily feedback from the visionary.

The question this raises is “how can the team commit to Sprint goals if the PO is giving direction every day?”. The answer is that they can’t commit to same level of detail. They have to commit to higher level goals, especially when they are in pre-production. This can be difficult for veteran Scrum teams to accept, but when the PO is a pig, the PO is making a commitment as well. Their “skin is in the game” with the rest of the team. They should commit to goals with the rest of the team. Highly details Sprint goals are meant to improve communication between pigs and chickens, but if your #1 customer is on the team, then the communication problem is reduced.

A PO can’t be a pig on 10 teams however. Some games have a “hierarchy of POs” with the main product owner and a number of POs for major mechanics or functions reporting to them. This is a common solution used in many industries adopting Scrum.

One word of warning. If you are working on a game (agile or not) without a single strong visionary who is clearly calling the shots about the features regularly and transparently, then you may have a problem.

Feedback: "Top 10 Pitfalls Using Scrum Methodology for Video Game Development"

Clinton.Keith@gmail.com (Clinton Keith) — Tue, 15 Jul 2008 21:09:00 +0000

Gamasutra just posted an good article by Paul Miller called "Top 10 Pitfalls Using Scrum Methodology for Video Game Development". I found it interesting. I agree with some points and disagree with others. The article highlights some misconceptions and myths about Scrum. Since they come from a person which has gone through a product cycle with Scrum, it's very useful to discuss them.

I'll start from the top:

10. A Game Design Document (GDD) isn't needed anymore because the backlog spreadsheet replaces it.

This is one of the biggest misconceptions of Scrum. Agile does not eliminate documentation or planning. It does focus more value on a working game over a design document, but that doesn't mean that we should eliminate all documentation. It just means that we don't rely entirely on the GDD alone to define what the game will be. GDDs can often become too detailed. I've seen GDDs for shooters that try to define the number of bullets in a clip for a fictional shooter. Why can't some of those things be decided while developing the game?

On the other hand we should document what we know about the game. You can't plan away uncertainty, but we should communicate what we are certain about.

I assume that by "backlog" he means "product backlog". If your product backlog is becoming too big to manage, then perhaps it is too granular for future releases.

9. Interrupt what people are doing to have them come to the daily 15 minute stand up meeting.

This sounds like the "daily scrum is about tasks" smell. The daily Scrum is for the benefit of the team and the product. If there are "developers had already been through a couple development cycles on various projects and had an established track record of working on the correct tasks and shipping products on time" then they should help mentor the others who don't have this experience.

If the Scrum is starting late, then the Scrum Master should fix that. Lateness to the Daily Scrum is an impediment because it wastes time. It's the SMs role to fix impediments.

It always amazes me to hear the argument that 15 minutes is too costly an amount of time to spend per day to insure that all 8-10 people on the team are on track. Do companies track bathroom, coffee, lunch and web surfing breaks to insure that we don't waste 15 minutes or more per day? It's not about the time. It's about the resistance to collaborate. As a former CTO I can clearly identify this issue as one which affects senior programmers quite often! ;)

8. Moving all team members into a dedicated team room or team area does not cross-pollinate with cubicle assignments in the per-discipline-organized cubicle farm.

I agree. One point is that when teams reach the proper level of collaboration, they drive this themselves.

7. A team member says "I'm not doing something right now because I was told there needs to be a task entered into the backlog and the task needs to be assigned to me by the project manager."

Teams should have total control over how they achieve the Sprint goals they've committed to. The Sprint goals are increments of value for the customer. I'd suggest to this team that:
- They don't have project managers assign tasks (that's not Scrum).
- They put the Sprint backlog on a task board (3x5 cards) and ditch the spreadsheet.
- They take control of the execution order and creation of tasks within the Sprint.
- Focus on release goals. He mentions that the team is only focused on 2 weeks. This is why there are 2-6 month releases where the team focuses on Big Hairy Audacious Goals (BHAGs).

6. Start managing stories and sprints before all the people are on the team. Decide that your Scrum stories and sprints are 100% completed months before the project even has a QA tester.

Very good point!

5. Scrum Pitfall: The 15-minute daily stand-up meeting degenerates into a daily go-around-the-table-and-each-person-give-a-status-report.

Many teams new to Scrum hit this issues and the result can often lead to managers saying "let's manage the team out of it". There was a recent post on this topic. It is very difficult for teams to achieve the collaboration to become hyper-productive. Sometime the team is not formed correctly, sometimes managers won't let it happen.

4. Make Scrum mandatory.

He makes a very good point about management feedback. One of the biggest problems I find is to get management to understand that each Sprint should show improved value and to have honest discussions about what is working and what is not. Too many times, the "work in is in progress and we'll show more value someday" pattern remains after a team adopts agile.

3. Implement only two weeks' worth of work and write only the code based on what you currently know about the game design.

I agree. Studies have shown that we solve problems best by a combined top-down and bottom-up approach. I've never been very comfortable with XP's "do just what's needed for the next two weeks" philosophy. Experienced developers will have more knowledge and can create farther reaching solutions. However, the problem is that they usually go too far. The opposite side of the problem is with programmers who only code for the current iteration and don't refactor constantly.

2. "Sprint Zero" is pre-production time.

I don't get this one. Prototyping is for knowledge. Knowledge has great value in building and refining the product backlog. I won't limit prototyping to a set number of Sprints (much less one).

1. Start managing the project by prioritizing tasks and asking for time estimates even before any GDD is written.

So what does the team do while the GDD is being written? If I know I'm going to be working on an FPS, I might want to get a camera and character controller working while the customers are trying to define the characters themselves. Perhaps there are questions that come up in the GDD writing process that would benefit from some early prototyping? This relates to #10, but it also reflects the impression that the GDD will answer all questions. If they can honestly say that the games they ship completely match version 1.0 of the GDD, then power to them. I've never seen it.

I'm also getting the impression that their product backlog is a huge spreadsheet of tasks that is prioritized and assigned up front. Tasks should only exist in the Sprint Backlog. Let the team worry about the tasks and their estimates. Estimate the Product Backlog in Story Points or Ideal Days. Don't create too granular of a Product Backlog. It becomes too difficult to manage.

I'm really glad to see such articles written. It reinforces what I've seen as a coach these past six months. The issues that teams adopting Scrum are seeing have patterns. Patterns can have solutions which can be shared.