Jinn of Quality and Risk

With iPod, listening to music will never be the same again. —Steve Jobs

Much of the underlying iPod design was performed by outside companies. ... Realizing that the MP3 market was still in its infancy, Apple developed a layered design chain tuned for an early-stage market to create the iPod. [via the strangely named Electronics Design Chain Magazine... I suppose we'll soon have "strategy" people touting design-chain management in addition to supply-chain management, or are they related?] 

Code reviews are more effective than debugging. An abundance of studies have shown that code reviews find more bugs per unit of time than any other technique. Code reviews are beneficial for all team members and projects. Junior engineers learn good habits; experienced ones teach good habits; bugs not visible to some become visible to others; the general quality of code improves. It encourages developers to feel that they are part of a team. People who can't deal with constructive criticism of their code make for bad team-mates. Note that code reviews are unlikely to identify architecture or design mistakes. However, they will enforce what the best developers have automatized: standards, e.g. style, naming conventions, and documentation. This makes the code more maintainable. One last thing: engineers who are under stress because of the amount of code they're expected to write every day will tend to avoid or fake code reviews. This in my experience is always an omen of very low code quality; never a good thing. Team leads should track the amount of time spent in code reviews and the number of error/defect reports generated, because these are good indicators of coding quality. (Some of my notes were prompted by discussions on /.) 

At the 4th USENIX Windows Systems Symposium (August 2000), Mark Lucovsky, Windows NT Lead Architect, gave a keynote address entitled From NT OS/2 to Windows 2000 and Beyond—A Software-Engineering Odyssey.

The Windows NT project started in 1988 with 12 engineers. Windows 2000 involved teams totaling more than 5,000 people—the largest group of engineers Microsoft has ever assembled for a single project. What did it take to scale the development effort by orders of magnitude in just ten years? How well has the initial design stood up over time? Lucovsky was one of the founding members of the Windows NT group, joining Microsoft in 1988 from Digital Equipment Corporation.

Here are the salient lessons learnt, grouped by topic (note: an optimal debrief process should generate a list of challenge/solutions rather than a list of injunctions; see notes on my debrief framework below).

• Team management: to scale a team, establish a culture, in particular around problems (for discovery, reaction, ownership, and solution); goal-setting is a good foundation for a software-engineering culture; maintaining a culture as a team grows is hard (note: it requires explicit awareness and actions within the culture); structure the project into modular sub-projects, in order to foster independent teams (rather than a bunch of individuals who report to one person).
• Objectives: set goals and prioritize; define the vision (high-level goals — in the case of NT it was: portability, reliability, extensibility, compatibility, and last performance); ingrain the vision within the team culture (note: use it as the explicit and objective standard for all decisions).
• Quality: spread review duties broadly; make reviews part of the culture; accept that mistakes will happen and deal with it in a non-abusive manner (i.e. don't call people "stupid"—even the best developers check-in compile-time mistakes); the test team size is about the same as the development team size; the average number of defects per developer and time required to fix defects increases in relation to team size (the effect on productivity is close to linear, as a 50% increase in team size tends to double total defect fix time per day).
• Technical: design for future evolution, based on a solid architectural foundation; pay early attention to the development environment; communal (shared) ownership of the code fails when there are more than 150 people involved; excessive process management (in e.g. version control and build constraints) serializes and slows down development work; the build lab should run automatically, email reports of build failures per branch, and consolidate only successful builds into the main branch/build.

I would love to help such a team work through its experience and identify lessons learnt, using my framework for focused project debriefs (based on after-action reviews). Maybe I could sell/run that activity as an external consultant? There are so many IT projects/teams in the world who could profit from it. Someone also needs to follow up on the lessons learnt and necessary change management...