2008-09-06 17:24:37 UTC
Xbox 360 defects: an inside history of Microsoft’s video game console
Dean Takahashi | September 5th, 2008
When his fourth Xbox 360 video game console died in April, Chris
Szarek wasn’t surprised.
The Chicopee, Mass. gamer was accustomed to the hardware failures that
became known throughout the Internet as RROD, or the “red rings of
death” which flash when the console becomes inoperable.
A 40-year-old photographer, Szarek was a hardcore Microsoft fan who
spent more than $1,000 on his games. But each time one of his Xbox 360
consoles failed, he had to spend time convincing Microsoft’s tech
support that they should send him a new console. Each time he got a
refurbished console as a replacement (a machine that had been returned
to a repair center in Texas, fixed as much as possible, and then
shipped back out). When he complained on the Internet and to the media
about the shoddy product and poor customer service, people branded him
a cry baby and wrote him off as a statistical anomaly. But by the
spring of 2008, Szarek was vindicated. There were at least a million
or two other people like him.
Szarek’s fourth machine lasted almost two years, experiencing the same
short life that many other Xbox 360s suffered. Microsoft replaced
these machines for free under the warranty that it announced on July
5, 2007, for defective Xbox 360s exhibiting what it more politely
called the “three flashing red lights.” That warranty program cost
Microsoft up to $1.15 billion, but the loss of face and loyalty among
gamers in the fierce console war with Nintendo and Sony has been
immeasurable. Szarek, who became a spokesman for dispossessed
defective Xbox 360 owners, played a part in making Microsoft
acknowledge its console quality problem.
This is the unauthorized tale of how Microsoft lost its chance to
become the leader in the biggest market it has attacked beyond its
twin monopolies in Office and Windows software. Rival game console
maker Nintendo out-thought the larger players Microsoft and Sony by
designing the Wii game console with a clever, intuitive game
controller. Even so, Microsoft could have captured more gamers during
this product generation, yet the RROD problem held it back. The Xbox
360’s defect problem will go down as one of the worst snafus in
consumer electronics history.
Its own worst enemy
Microsoft knew it had flawed machines, but it did not delay its launch
because it believed the quality problems would subside over time. With
each new machine, the company figured it would ride the “learning
curve,” or continuously improve its production. Even though
Microsoft’s leaders knew their quality wasn’t top notch, they did not
ensure that resources were in place to handle returns and quickly
debug bad consoles. There were plenty of warning signs, but the
company chose to ignore them. The different parts of the business
It reminds me of the German war machine just before World War I, as
chronicled by Barbara Tuchman in the classic history book, “The Guns
of August.” The German generals were intent on keeping their trains on
time; but the leaders overlooked their chances for stopping the war
altogether. The Schlieffen plan called for them to strike first. Once
the Russians and French mobilized, the Germans had to move into
action. They marched off blindly into tragedy.
Likewise, Microsoft’s strategy depended on beating its rivals to
market. It couldn’t afford to stop and delay the launch in order to
solve its quality problems, or so upper management believed. What
Microsoft’s leaders didn’t realize was that getting to market first
with a flawed machine would only win them a battle; and it risked the
loss of the war.
“They got enamored with the idea of the Microsoft army rolling
everything out at the same time,” said one knowledgeable source who
asked not to be identified.
The quality problem negated much of the advantage of going first, and
it has delayed the company’s plan to aggressively market the console
and slash its prices. (Microsoft disputes this point; it cut the price
of all three versions of its Xbox consoles by $50 to $79 on Wednesday.
And the company believes it will sell more boxes than Sony will. But
prices ought to be lower still during this stage of the console life
cycle). That has stopped the company from reaching the broader market
of consumers that Nintendo has won over. It has lowered its ambitions,
hoping instead just to get a clear edge on third-placed Sony. The
future profits that the company once hoped for are now likely to wind
up in Nintendo’s pockets.
Microsoft’s top game executive, Robbie Bach, president of the
Entertainment & Devices group, said at a dinner in July that
Microsoft’s own research shows that gamers have largely forgiven the
company for defective Xbox 360s. Microsoft has still sold more Xbox
360 consoles than Sony to date. But there is no doubt that the company
has lost considerable good will among gamers. Before Microsoft offered
free replacements, connsumers grumbled that they had to turn to
forums, such as those on Ars Technica, to vent and to find solutions
to problems that the company didn’t openly discuss. And for a couple
of months now, Sony’s PlayStation 3 has been outselling the Xbox 360
in the U.S. for the first time.
“Fundamentally, their thinking shows that they are a software company
at heart,” said one veteran manufacturing executive. “They put
something out and figure they can fix it with the next patch or come
up with a bug fix.”
The terrifying part of the story is that this kind of problem — where
technology fails and no one knows what to do about it — can happen to
About this story
I asked Microsoft to confirm or deny 35 different facts contained in
this story. Instead, I received a formal statement from a Microsoft
spokesperson, saying the company had already acknowledged an
“unacceptable number of repairs” to Xbox 360 consoles and responded to
the hardware failures with a free replacement program. The statement
also said, “This topic has already been covered extensively in the
media. This new story repeats old information, and contains rumors and
innuendo from anonymous sources, attempting to create a new
sensational angle, and is highly irresponsible.”
I don’t think this story is sensational. I have tried to verify the
facts over several years. I view this story as the last chapter for my
book on the making of the Xbox 360, “The Xbox 360 Uncloaked: The Real
Story Behind Microsoft’s Next-Generation Video Game Console.”
The facts revealed themselves slowly, emerging from the day-to-day
stories that I wrote about the game business. Some people might
consider this post mortem to be ancient history. But the
reverberations are still playing out today. They help explain why
Microsoft isn’t being aggressive with its price cuts and why gamers
aren’t getting bargains on hardware as they did the last generation.
While I talked to many people for this story, few were willing to let
me use their names. As you will see, not every source is anonymous,
and we have included the viewpoint of Microsoft executives from past
The details are interesting because they offer a deeper look into how
the console business runs than is otherwise available. Microsoft, for
instance, still hasn’t perfected its Xbox 360 manufacturing process.
In the absence of a precise chronology from Microsoft, some anonymous
sources have tried to describe what happened. But the history of the
decision making and inside story of what happened on the RROD has
never been told, until now.
Microsoft’s mea culpa
In many ways, the Xbox 360 was a big achievement. Microsoft had beaten
its rivals Sony and Nintendo to market by a year. Sony had outsold
Microsoft five-to-one in the previous generation of consoles, but
Microsoft has outsold Sony’s PlayStation 3 with the Xbox 360.
Microsoft’s to game executive Bach (left) had repeatedly promised that
Microsoft would be profitable in his Entertainment & Devices group in
the fiscal year ending June 30, 2008. The company met that target. Big
losses have turned into profits.
But when the company launched its Xbox 360 video game console in
November, 2005, it didn’t have a handle on product quality and it was
not prepared to systematically analyze its product returns and debug
Microsoft has admitted those mistakes. On July 5, 2007, the company
said it would take a billion-dollar write-off to pay for free
replacement of Xbox 360s by up to three years from the date of
purchase. Peter Moore, who at the time was head of the games business
at Microsoft, said in an interview the decision to take the write-off
was due to an “unacceptable number” of returns. The company made
improvements to the console’s quality, but Moore said it was unclear
how many units would fail in the field. Microsoft then followed up its
new policy with numerous public apologies, including a statement from
Bach, (pictured above).
Microsoft never disclosed its actual return rates. But according to
data obtained by VentureBeat, the total number climbed above 1.2
million consoles in early 2007. That is a huge amount, considering
Microsoft had only shipped 11.6 million into stores by the time of the
announcement in mid-2007.
The company was dragged kicking and screaming to its admission of
widespread defects — with gamers and the press doing the dragging. In
interviews in April and May of 2007, Microsoft executives denied that
quality problems were haunting them. Todd Holmdahl, (pictured left)
general manager in charge of the Xbox platform for Microsoft in
Redmond, Wash., said in an interview with me on May 9, 2007, while I
was at the San Jose Mercury News, that the “vast majority” of
Microsoft’s Xbox 360 customers were having a great experience with
their consoles. He declined, however, to say what percentage of
consoles were being returned for defect reasons. In doing so, he was
as coy as any of the console makers are about such sensitive data. But
everyone wanted an answer to that question.
It’s not clear why it took Microsoft so long to announce its warranty
extension last year, almost 19 months from the time when the first
complaints arose at the outset of the launch. As it was investigating
the cause of the problems, the company stayed quiet, acknowledging
problems only little by little. The company delayed its free-
replacement announcement until it understood the scope of the problem.
In July, game executive Bach said that the number of returns was not
measurably big until a year after the launch. It was a problem that
became evident only over time, he said.
“It wasn’t related to things we were seeing in testing or some
judgment call we had to make about whether the product was ready,”
Bach said at the dinner. “We were confident the product was ready. We
did a lot of testing. The problem that shows up with the three red
lights on the console is a complex interaction with some very complex
But the evidence for the quality debacle was there to see even before
Microsoft shipped any machines. In August, 2005, as Microsoft was
gearing up production, an engineer raised a hand and said, “Stop. You
have to shut down the line.” This wasn’t just a brief moment. The
engineer spoke up repeatedly.
That engineer, who asked not to be identified, had deep experience in
manufacturing. When production results were really off kilter,
stopping a line and tracing a problem back to its roots was the
answer. But the higher-ranking engineers, managers and executives
chose to risk going forward. There wasn’t a universal backlash from
the engineering ranks, according to one engineering source.
Nobody listened to that engineer — who spoke on condition of anonymity
— apparently because console launches are always hurried affairs.
Yields — the percentage of working products in a given batch of total
products produced — generally start low. As the manufacturers conduct
statistical analysis and tight controls on every step in assembly,
they learn how to drive the yields up.
Still, the picture wasn’t pretty. The defect rate for the machines was
an abysmal 68 percent at that point, according to several sources.
That meant for every 100 machines that Microsoft’s contract
manufacturers, Flextronics and Wistron, made at their factories in
China, 68 didn’t work. At the recent dinner, Bach denied that there
was a big concern about defects at that point in time.
At that point, it is likely true that the engineers weren’t raising
enough red flags for the executives to pay attention. Early yields on
electronic goods are almost always lousy. Those veteran engineers
figured that they would be able to debug the problems and bring the
yields up quickly. But the expected rapid improvement in quality just
didn’t happen. The communication between upper management and the
engineers wasn’t clear. Nor was the strategy aligned between marketing
and customer support.
There were plenty of warning signs. Early reports on the problems were
myriad. In an Aug. 30, 2005 memo, the team reported overheating
graphics chip, cracking heat sinks, cosmetic issues with the hard disk
drive and the front of the box, under-performing graphics memory chips
from Infineon (now Qimonda), a problem with the DVD drive, and other
things. At that point, the contract manufacturers were behind schedule
and had only built hundreds of units. They were supposed to have been
in high gear, making thousands every week.
The illusion of a well-oiled machine
Yet around that same time, in mid-August 2005, Microsoft gave me a
tour of the hardware test center in Mountain View, Calif. Hundreds of
boxes were being tested in heat chambers and test labs. Leslie Leland,
an engineering manager, said all was on schedule. Microsoft wanted to
convey its own confidence to its partners, game developers, and
gamers. To my untrained eye, it looked like a smooth operation.
But on the inside, it was a frenetic time. The initial yield on the
most critical chip, the three-core microprocessor designed and
manufactured by IBM, was only 16 percent. For every 100 produced, only
16 worked. This low yield was surprising because a plaque on the wall
in Microsoft’s campus in Mountain View proclaimed that IBM had “taped
out,” or completed the design, of its first microprocessor prototype
on Dec. 8, 2004. The timing of the tape out was right on schedule,
which was good because it is normal for six months to a year of
debugging to follow a tape out. IBM had an easier time than the maker
of the graphics chip.
ATI Technologies (a graphics chip maker that has since been acquired
by Advanced Micro Devices) was late to deliver working graphics chips,
or GPUs. Like IBM, it had promised to deliver one of the most complex
chips it had ever made in about half the time it usually took to
create such chips. But ATI only had half the number of the 400
engineers IBM had on the project.
The fact that both companies had designs done at all was the result of
a Herculean effort. Microsoft’s engineers started working on the Xbox
360 at least a year after Sony’s engineers began work on the
PlayStation 3, yet Microsoft wound up shipping a year ahead of Sony.
Everything Microsoft did was under time pressure, including the
creation of the IBM microprocessor and the ATI graphics chip. The only
way to pull it all off was through a virtual organization, where
Microsoft outsourced many of the tasks to companies that specialized
in particular tasks. Microsoft’s hardware engineers in Redmond, Wash.
and Mountain View, Calif. were the master integrators of all of the
suppliers. The Microsoft chip designers in Mountain View also designed
an important video processing chip themselves.
Considering all of the work, Microsoft had too few hardware people.
Some of the designers of the Xbox 360, including engineering chief
Greg Gibson, were stretched thin. Gibson and J Allard (pictured left),
who led the console design effort, had begun work on Zune, Microsoft’s
portable media player. Top brass had approved the project to dethrone
Apple’s iPod, but Microsoft kept Zune secret from the outside world
until much later. Some engineers were pulled of the Xbox 360 at a
critical moment to join Allard’s effort to create a music player.
Those who were left to work on the test team worked around the clock,
traveling to China to work in the factory.
Seasoned hardware team executed well on the original Xbox
Outsiders frequently slam Microsoft’s hardware engineers as having a
software (fix it later) mentality. But Microsoft’s engineers had been
trained to put quality first. Most of them had years of training.
Holmdahl, for instance, had been with Microsoft in the early days of
its hardware division. By the time he joined the Xbox project, he was
a veteran, having worked with contractors such as Flextronics to make
20 million mice per year. Making consoles was an altogether different
task, but the team had expert help through its Web TV acquisition,
which brought veteran console hardware engineers aboard in 1998. One
of the best at debugging was Nick Baker, one of the architects of the
Xbox 360’s chips.
In 2001, the team went through the paces making sure that the original
Xbox had good quality. On that launch, Microsoft was 20 months behind
Sony’s launch of the PlayStation2. Microsoft’s original Xbox team
moved as fast as they could, but they took a long time defining the
box and picking vendors.
By the time they hired Nvidia in March 2000 to make the graphics chip,
they were hopelessly behind. Nvidia combined two different chips into
one in half the time it took to design a chip, but it fell behind
schedule in part because of a bug. Someone else had specified a sub-
par power supply for the Xbox, and that made the graphics chip hiccup
now and then. Once the team fixed the problem, they raced ahead.
But Flextronics, the contract manufacturer, wasn’t ready. It had to
build a bigger manufacturing database to accommodate the design.
Microsoft had to postpone its launch by a week. It also decided to
launch only in the U.S. market and would wait until the next spring to
launch in Europe and Japan. No console maker had ever pulled off a
worldwide launch before, and now Microsoft understood why. The delays
in other territories put it even further behind Sony. Sony eventually
outsold Microsoft by more than five to one — scarring into memory a
tough lesson about market timing.
But the Microsoft hardware team had come out shining. Aside from a
problem with a supplier’s DVD drive, the number of defective Xboxes
was low. After the DVD drive issue was fixed, yields rose above 90
percent and stayed there for the duration of the manufacturing.
Microsoft thus had a seasoned team to work on the Xbox 360.
Lessons of round one drove grand strategy for the Xbox 360 business
Despite the good quality manufacturing, Microsoft’s game business was
deeply flawed in the Xbox generation. In the second round, Microsoft
vowed to learn from its mistakes.
In the first generation, the company lost $3.7 billion over four
years. Most of that was because the costs of the box — particularly
its hard disk drive — were too high. Microsoft’s planners thought the
hard drive would give them an edge over Sony’s PS 2. It didn’t.
Bill Gates didn’t really care about the losses in the first
generation. That was simply the ante for getting into an exciting new
business. Microsoft has always viewed its ambitions in the game
business as strategic. It isn’t in the business just to dominate video
games, but to own the living room. The game console is a kind of
gateway to the Internet and all of the future entertainment services
that can be piped into the living room.
But Steve Ballmer, who took over from Gates as CEO during the first
generation, really wanted the Xbox business to be profitable second
time around. He wanted to keep the foothold in the living room, but
not at the expense of draining Microsoft’s cash, which was needed to
fight Google and other rivals.
This time, Gates said he felt that Microsoft would have a fundamental
cost advantage over Sony. One of the key ways Bach hoped to outwit his
rivals was with speed: getting a machine to market in four years
instead of five. Secretly, Microsoft had planned on selling 50 million
Xbox 360s at that point.
The Xbox team met for a series of retreats. One such retreat took
place in early 2004 at the elegant Salish Lodge beside the Snoqualmie
Falls made famous in the Twin Peaks TV show. After that meeting, Bach,
who had headed the Xbox business as its senior executive from the time
of its inception, decided that company would aim for profitability
with the Xbox 360. As much as possible, the machines would not be sold
for a loss.
Bach believed the hardware itself should be profitable over the life
cycle of the system. His planners counted on good revenues from
accessories and games to ensure profits the second time around. And,
in the three-page paper that outlined the grand strategy, he said that
Microsoft wanted to hit the market at the same time as its rivals.
Microsoft’s best guess was that Sony would launch the PS 3 in 2005.
That turned out to be a year off, since Sony ran into its own
development problems related to its decision to put Blu-ray drives
instead of DVD drives into the PS 3.
Microsoft had more preparation for Xbox 360
There was more time to design the box than the first-generation team
had, but it was still a race. Different teams sped into planning. The
different groups included chip designers, hardware planners, game
designers, software operating system engineers, marketers and
With the Xbox 360, Microsoft was guilty of packing a little too much
into its box. The company sent out its scouts in late 2002 to ask
gamers what they wanted. The first Xbox came off as “The Incredible
Hulk.” The second Xbox, Microsoft had mandated, had to be smaller, if
only to fit in the smaller homes in Japan, a key target market. The
designers thought it had to be more like Bruce Lee. Microsoft’s own
user-experience team ran the show, while outside industrial design
firms competed in a bake-off to define the soul of the machine.
Hers Experimental Laboratory of Japan and Astro Studios in San
Francisco won the bake-off. They wanted to create something that
looked elegant and iconic, qualities that were sometimes at odds with
the computer that it was. Those firms gave the box its signature look
— curved sides that looked as if the box were inhaling air and was
about to explode with energy (that was the marketing image, anyway).
There was a constant tug of war between the industrial designers, who
wanted something small, and the engineers, who wanted to pack a lot of
performance into the box. The more powerful the system was, the bigger
the box had to be to provide air flow for heat-sensitive components.
Microsoft decided late to add a hard disk drive to most of the
machines. It also came up late with a plan to add wireless
controllers; all of the previous consoles shipped with wired
controllers. The hard drive blocked a lot of the air flow on one side
of the machine. And the wireless modules had to have enough of their
own space to ensure that there was no electrical interference. In the
end, the machine was a series of compromises. The gigantic power
supply needed to run the machine sat outside in the power cord, while
the console shell was poked full of holes on the sides to ensure air
“It turned out in the end that this was all going too far, too fast,”
said one source. “They were adding too many features after things were
locked down. That incremental feature adding just made it fragile.”
The marketers also decided the box had to be backward compatible with
the older games, including “Halo” and “Halo 2.” And they also felt the
box had to have enough horsepower to make games look beautiful on high-
Those parameters, in turn, gave direction to the chip and hardware
designers. This time, hardware chief Todd Holmdahl wanted Microsoft to
have much more control over its fate. And he believed he had built up
a team that could act as the “system integrator.”
Microsoft had a team of 100 chip engineers and 100 more hardware
engineers between Redmond and Mountain View. They were the overseers
of the integration of the parts into a working console. It was a
relatively small team for such a large project compared to a hardware
company such as Sony, but Microsoft had plenty of help from its
In contracts with IBM and ATI, Microsoft set early deadlines for
delivery of the first working chips. It also decided that it wanted to
own the chip designs. That way, if it wanted to, it could take the
designs and have them made by a factory of its choosing and wasn’t
locked into using IBM and ATI (or ATI’s chosen partner) chip
factories. The reason for this change was that Microsoft wanted to
avoid dependence on its chip partners. It had learned the hard way
after getting into a bad dispute with Nvidia over the prices charged
for the graphics chip in the original Xbox. The case had to go to
arbitration and Microsoft ultimately lost that battle. However, this
was a mistake: it was now responsible for failures related to the
In Microsoft’s favor this time was the fact that it had credibility
among game developers thanks to hits such as “Halo,” (pictured right)
which sold over 10 million units. Microsoft believed it could charge
more for each machine and more for its games; and not every machine
would have an expensive hard drive. In the second generation,
Microsoft planned on reducing the cost of the hardware over time.
Contract manufacturing had matured since the last generation.
Flextronics was willing to make each console for just $10 each, while
another manufacturer, Wistron, would do the same for $7, according to
sources. They hoped to make profits on large volumes of machines sold.
They were also practiced in the art of reducing the costs of the
components they put into their machines. Overall, Microsoft had a much
more formidable hardware operation, when you consider all of its
virtual resources, on the second time around. But it still wasn’t
enough to do a perfect job.