Software Engineering: How Teams Change Code for Years

You have never used the same app twice. The icon is the same, but last Tuesday it had a bug, on Wednesday somebody fixed it, and on Thursday a change meant for something else broke it again. Behind the single button you tap sits a team that has been changing the same code for years, often with nobody left who wrote the first version.

Writing a program that works on your laptop is one skill. Keeping a program working while dozens of people change it for a decade is another, and it is mostly not about code. It is about arguments settled in advance: what the thing should do, who may change what, how you know a change is safe, and what you do at 3 a.m. when it isn't.

That second skill is called software engineering. It is the one subject on the Computer Science map whose main material is people, time and mistakes.

Software is built by teams, for years

A student program is written once, by one person, and thrown away. A real program is read far more than it is written. Its first version may be a small part of its life; most of the work is changing it: new features, new phones, new laws, bugs nobody knew about. The people who change it did not write it. So the central problem is not "can I make the computer do this?" It is "can a stranger understand and safely change this next year?"

This turns the usual priorities upside down. Clever code is a liability if only its author understands it. Plain names, small functions and clear structure are not decoration; they are how knowledge survives when people leave. Frederick Brooks, who managed the development of IBM's OS/360 in the 1960s, wrote the classic account of building such a system in The Mythical Man-Month (1975), and most of its observations are about people and communication rather than programming.

Requirements, design and the art of hiding things

Before code comes a question: what should it do? Requirements are the answer, and they are always incomplete, because users often don't know what they need until they see something working. Design is how you split the work so that people can build it in parallel.

The key idea in design is abstraction: using something through a simple description without needing to know how it works inside. You drive a car without knowing the engine. A module is a chunk of code with a small, stable public face and a hidden inside. The classic paper is David Parnas's 1972 "On the Criteria to Be Used in Decomposing Systems into Modules". Its argument: don't cut a system into modules by the order of the steps it performs. Cut it so that each module hides one decision that is likely to change, such as how data is stored. Then, when that decision changes, only one module has to.

Good design is deciding in advance what you will be able to change without telling anyone.

Parnas's idea sits underneath nearly everything later: libraries, classes, services, even the interface idea in Systems Engineering (34/22). Brooks, who argued against the idea in 1975, later wrote in the 1995 edition of his own book that this was one of the few places he'd been wrong.

Version control: many people, one history

If five people edit the same files, how do you combine their work, and how do you undo a mistake from three weeks ago? Version control is the answer: a system that records every change, who made it and why, and lets people work in parallel on separate branches that are later merged.

The dominant tool is Git, created by Linus Torvalds in April 2005 for the Linux kernel, after the proprietary tool the kernel team had used was no longer available to them. Every developer holds a complete copy of the history. Each commit is a snapshot of the whole project, identified by a hash of its contents and linked to the commit before it, so history cannot be quietly rewritten without everyone noticing.

When two people changed the same file, Git merges line by line. If the changes are in different places, it just combines them. If they touch the same lines, or lines right next to each other, it stops and asks a human, by marking a conflict. Try it: pick lines for Alice and Bob to change.

With the starting choice (Alice edits line 1, Bob line 3) the merge is silent and clean. Move Bob to line 2 and Git, correctly, stops: it cannot know whether the title and the flour belong together. This is also why teams make small changes and merge often. Long-lived branches are where conflicts, and bad surprises, pile up.

The testing pyramid, and what a passing test proves

You cannot check by hand that a million-line program still works after every change, so teams write programs that test the program. There are three layers. Unit tests check one small piece in isolation and run in milliseconds. Integration tests check that pieces work together. End-to-end tests drive the whole product as a user would; they are the most realistic and the slowest and most fragile. The testing pyramid, popularised by Mike Cohn in his 2009 book Succeeding with Agile, says to have many tests at the bottom and few at the top, because the fast ones give quick, precise answers.

The hard truth about tests is that a test only checks the inputs someone thought of. Here is a leap-year function with a bug: it says a year is a leap year if it divides by 4. Real rule: divisible by 4, except years divisible by 100, unless also divisible by 400.

function isLeap(year) {
  if (year % 4 !== 0) return false;
  return true;
}

The suite {2023, 2024} runs every line and every branch (100% coverage), and all tests pass. The bug is still there. Add 1900 and the test fails. Coverage tells you which code ran, not whether you asked the right questions. That is why engineers pair tests with code review, where a second person reads each change before it merges. Review catches design problems, spreads knowledge, and is the cheapest second pair of eyes a team has.

Technical debt: borrowing against the future

Ward Cunningham, who built the WyCash portfolio system, used a money metaphor in a 1992 conference report: shipping first-time code is like going into debt. A little speeds you up, provided you pay it back by cleaning up soon. If you don't, the interest piles up, because every later change has to work around the mess. He later said that he never meant the metaphor as a licence to write bad code on purpose.

A toy model shows the shape. Each release has capacity for 10 units of work. Taking shortcuts delivers up to 40% more features now, but every release adds debt, and each unit of debt steals 3% of the capacity of every future release. You can spend a share of each release on paying the debt back.

With full shortcuts and no repayment you are ahead for the first eight releases, and behind from the ninth; by release 20 the careful team has delivered 200 units to your 132. Spend a fifth of each release on repayment and you finish at 188, close behind. The numbers are invented, but the shape is real: speed borrowed now is repaid with interest, and the interest is paid by people who weren't there when the choice was made. Sometimes borrowing is right (a demo next week). The mistake is not knowing you borrowed.

Estimation, and Brooks's law

Every software schedule is wrong, and nearly always in the same direction. Part of the reason is psychological: Kahneman and Tversky named the planning fallacy in 1979, the habit of planning from the best-case story of this project instead of the record of similar ones. Part is mathematical, and you can see it.

Suppose every task estimate is honest: the median, a coin-flip guess of how long it takes. Task durations are skewed, because a task can take twice as long but not half as short as nothing. The toy below simulates ten tasks, each estimated at 3 days.

With ten tasks and moderate uncertainty, the chance of finishing by the sum of the estimates (30 days) is only about one in four. A date with an 80% chance is nearer 39 days. Each estimate was fair; the total isn't, because the delays do not cancel out the early finishes. This is also why one number ("done in 30 days") is worth less than a range.

Now the famous law. When a project is late, the tempting fix is to add people. Brooks (1975): "Adding manpower to a late software project makes it later." His reasons: newcomers need time to learn and take veterans' time to teach them; and communication grows faster than headcount, with n people having n(n-1)/2 pairs (45 pairs for ten people, 1,225 for fifty). Some work also cannot be split: "nine women can't make a baby in one month."

With 30 person-weeks left, adding people slows the project; with 120 left, it helps. Brooks himself later called his law an oversimplification, and later writers such as Steve McConnell argued that late projects are usually ones that started with an over-optimistic estimate. The robust lesson is narrower: adding people near the end costs more than it seems; adding good people early, or on work that splits cleanly, can pay.

Agile and waterfall, fairly

The waterfall approach does the phases in order: requirements, design, build, test, release. It is easy to plan, to contract for and to audit, which is why it survives where requirements are stable and the cost of error is high. Defenders point out that mistakes caught early are much cheaper than late ones, and that documentation helps new people. A fact often missed: Winston Royce's 1970 paper, usually credited as the source of the picture, called a purely sequential process risky and inviting failure, and went on to propose feedback loops. The word "waterfall" seems to have come later.

Agile grew out of a meeting of seventeen practitioners at the Snowbird resort in Utah in February 2001. Their Manifesto prefers individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. Teams work in short cycles of a few weeks, show something working each time, and adjust. Its critics say the evidence for its benefits is limited, that it can neglect documentation and architecture, and that in large safety-critical systems you can't skip planning. Its defenders reply that waterfall's "plan" is a guess that gets more wrong the longer it runs.

Most real teams mix the two. Hardware-heavy projects, as in Systems Engineering (34/22), lean toward the V; app teams lean agile. The honest position is that the choice depends on how well you know what to build, and how much a mistake costs.

Reliability, incidents and blameless postmortems

Every system that runs long enough fails. What separates teams is what they do afterwards. In August 2012 the trading firm Knight Capital installed new software on eight servers, but one was missed. The new release reused a flag that, on the server still running the old code, switched on a function retired years before. In about 45 minutes it sent millions of unintended orders. The SEC described losses of more than $460 million (the firm reported a pre-tax loss of about $440 million). A missed copy, a reused flag and old code nobody had deleted: none of these was one person's mistake. They were gaps in a process.

That is the idea behind the blameless postmortem, as the Google SRE book puts it: find the contributing causes of an incident without blaming any individual or team, on the assumption that everyone acted reasonably on what they knew. The reason is practical. When people fear punishment, they hide problems and stop reporting near-misses, and the next incident is worse. A good postmortem asks what in the system made the mistake easy to make, and what will make it harder next time: a check, an alarm, a staged rollout, a deleted dead function.

Blame asks who. A postmortem asks how the system let it happen.

What a graduate just sees

• Code is read and changed far more than it is written, so write it for the next stranger.
• A module should hide one decision likely to change; if a change ripples everywhere, the cuts were in the wrong places.
• Small, frequent merges beat big, rare ones.
• A green test suite means the questions someone asked passed, not that the code is right.
• Technical debt is a loan; the danger is borrowing without knowing it.
• An estimate is a distribution, so ask for a range and the chance of hitting it.
• Adding people to a late project costs more than it looks.
• Methods like agile and waterfall are bets about how well you know what to build.
• After a failure, ask what in the system allowed it, not who to blame.

Where it shows up next

• Systems Engineering & the Capstone (34/22): the same ideas (interfaces, verification, margins) applied where software meets hardware. The Ariane 5 failure there is a software engineering story too.
• Computer Security (34/33): most breaches are engineering failures, such as unpatched software and missed review, and security is built on the same code review, tests and postmortems.
• Operating Systems (34/26) and Databases (34/28): the largest codebases you will read, built with these tools.
• In the real world: every app, every website, a car's firmware, a hospital's records. The job titles are software engineer, site reliability engineer and engineering manager.

If you want to go deeper

• 📘 Software Engineering, Ian Sommerville (Pearson, 10th ed., 2015): the standard university textbook, broad and methodical.
• 📘 Software Engineering at Google, edited by Titus Winters, Tom Manshreck and Hyrum Wright (O'Reilly, 2020): how a very large company does it, free to read online.
• 📘 The Mythical Man-Month, Frederick P. Brooks Jr. (1975; anniversary edition 1995): the gentle classic, short essays you can read in a weekend.
• 🌐 MIT OpenCourseWare 6.005, Elements of Software Construction (Fall 2008): free lectures and readings on abstraction, specifications and testing.