Shipping Software: Testing, DevOps & the Cloud: Making Release Day Boring

At 04:09 UTC on 19 July 2024, a cybersecurity company sent a small configuration update to the Windows computers running its protection software. Seventy-eight minutes later, at 05:27, it pulled the update. By then about 8.5 million Windows devices had crashed, which Microsoft estimated was less than one percent of all Windows machines. The update had gone to everyone at once.

Nothing about that is exotic. The code was not malicious, and the company was not careless in the way the word usually means. A mismatch that no check caught met a release process with no brake. The same pattern, a change that is fine nearly everywhere and catastrophic somewhere, is what a whole branch of software engineering exists to manage.

Most of the apps on your phone changed this week without you noticing. A big service may release many times a day. That is only safe because of machinery you never see: automatic tests that run on every change, packages that run the same on a laptop and in a data centre, rented computers that appear on demand, and dials that show whether the new version is healthier or sicker than the old. This piece is about that machinery and the few ideas that make it work.

Integrate often: the pipeline that says no early

Teams that work in long-lived branches discover, late and all at once, that their changes do not fit together. Teams that merge small changes to a shared main line every day discover it within minutes, when the problem is still small. That second style is called continuous integration, and the practice it depends on is trunk-based development: everyone works from a single main branch, and any other branch lives briefly. The research behind the book Accelerate (Forsgren, Humble and Kim, 2018) found short-lived branches and frequent merging among the practices that went with high-performing teams. That is a correlation, and practitioners still argue about how much of it is cause, but the logic is simple: a conflict you meet today is cheaper than one you meet in three weeks.

Every merge triggers a pipeline: a fixed series of automatic checks. A change must pass the whole series before anyone considers releasing it. Continuous delivery (the term spread through Jez Humble and David Farley's 2010 book of that name) means the software is always in a state that could be released; continuous deployment goes one step further and pushes each passing change to users automatically.

The design question that a graduate asks is the order of the stages. Checks differ hugely in cost and in what they can see. A style check takes seconds and sees nothing but formatting. An end-to-end test drives the whole product like a user and sees almost everything, but takes the longest and breaks most easily. Try both orders below. The times are invented but typical in proportion.

Without JavaScript: with cheap checks first, a wrong calculation is reported after 4 minutes (style check 1 + unit tests 3). With the slow end-to-end tests first, the same bug is reported after 20 minutes.

The rule is fail fast: put the cheap, sharp checks first and the slow, broad ones last, so most bad changes are rejected in minutes. Notice the price of the rule. A bug that only the end-to-end tests can see waits behind everything else. The pipeline is a bet that most mistakes are cheap kinds.

How do you know whether your delivery process is any good? The DORA research programme (DevOps Research and Assessment, now part of Google Cloud) tracks five measures, which its site groups as throughput and instability:

Group	Measure	What it counts
Throughput	Change lead time	From commit to running in production
Throughput	Deployment frequency	How often you deploy
Throughput	Failed deployment recovery time	How long to recover from a bad deployment
Instability	Change fail rate	Share of deployments needing a rollback or hotfix
Instability	Deployment rework rate	Share of unplanned deployments caused by a production incident

The finding the programme made famous is that speed and stability are not a trade-off: teams that deploy often tend also to recover faster and fail less. Its 2024 report found that adopting AI tools raised individual productivity but was associated with lower delivery stability and throughput, and its 2025 report described AI as an "amplifier" of whatever strengths and weaknesses an organisation already has. Treat the measures as instruments for a team to learn from, not as targets to hit: a number that becomes a target is easy to game.

Flaky tests: the arithmetic of trust

A flaky test passes and fails on the same code. A network call times out, two tests share a file, a clock ticks over midnight. One flaky test is an irritation. A thousand tests that each flake a tiny fraction of the time are a pipeline that cannot be trusted, and the arithmetic is unkind.

With 1,000 tests that each flake one run in a thousand, the chance of a green build on perfectly good code is 0.999¹⁰⁰⁰, about 37%. Nearly two pushes in three come back red for no reason. Engineers then re-run until green and stop believing red, which is the real damage: when a genuine failure arrives, it looks like the usual noise.

Without JavaScript: 1,000 tests at 0.1% each give a green build on healthy code 36.8% of the time; with one automatic retry, 99.9%.

Retrying hides the noise, which is why teams do it, but it also hides the flake: the test is still unreliable and still slow. Google's testing team reported in 2016 that about 1.5% of all its test runs returned a flaky result, and that almost 16% of its tests showed some flakiness. The sensible policy, which many teams adopt, is to track flakes, quarantine the worst offenders out of the blocking path and fix or delete them. A test nobody trusts costs more than it protects.

Containers: ship the kitchen, not the recipe

"It works on my machine" is the oldest complaint in the trade. The operating system, the language version and a hundred small libraries differ between a laptop and a server, and the difference breaks things. A container packages an application together with everything it needs apart from the operating-system kernel, and runs it in isolation.

A container is not a small virtual machine. A virtual machine carries its own whole operating system and pretends to be a computer. A container shares the host's kernel. On Linux, namespaces limit what a process can see (its own process list, network, file system) and control groups limit how much CPU and memory it can use. That is why containers start in about the time it takes to start a process and why you can pack many onto one machine. The price is weaker isolation than a virtual machine gives, which matters in security (see Computer Security, 34/33). Docker, introduced in 2013, made this easy to use; the format is now an open standard that several tools run.

A container is built from an image, a stack of read-only layers. Each instruction in the build recipe makes a layer. Here is the rule that every engineer learns the slow way, stated on Docker's own documentation: when one layer changes, every layer after it must be rebuilt too. So the order of instructions is a performance decision. Put what rarely changes first, and what changes every commit last.

Without JavaScript: after a one-line change to the source, the careless recipe rebuilds for 122 seconds and the careful one for 32. If the dependencies change, both take about 2 minutes.

The careful recipe copies the short dependency list first, installs, and only then copies the source, so the 90-second install is skipped on most edits. The deeper idea is immutability: you never patch a running container, you build a new image and replace the old one. Servers stop being pets with histories and become cattle that are rebuilt on demand. The same image that passed the tests is the one that runs in production.

Cloud models: who runs which layer

The cloud is rented computing. Amazon launched its storage service S3 in March 2006 and rentable servers (EC2) that August; by now a long list of companies sells the same idea worldwide. The U.S. standards body NIST wrote the lasting vocabulary in 2011 (Special Publication 800-145): three service models called infrastructure, platform and software as a service. The practical question behind the jargon is one a graduate asks of any option: which layers do I run, and which does someone else run?

Without JavaScript: on rented virtual machines (infrastructure) you manage the operating system and everything above it. On a platform you manage your code and data. In software as a service you manage only your data and who can access it.

Notice the last row. In every model, your data and who may reach it stay yours. Providers publish this as a shared responsibility model: AWS, for instance, describes itself as responsible for the security of the cloud, meaning hardware, facilities and the virtualisation layer, and the customer for security in it, such as access permissions and, on rented servers, patching the guest operating system.

Higher layers cost less effort and give less control, and the money works differently too. A server is paid for whether it is busy or idle. A function that runs on demand is paid for per request. Which is cheaper depends on traffic, as this toy shows with invented prices.

Without JavaScript: at 20 million requests, the server costs $45 and on-demand $24. At 60 million, a second server is needed ($90) against $72 on demand. At 100 million, $90 against $120.

The lesson is the shape, not the numbers. Pay-per-use wins when traffic is low or spiky, and renting capacity wins when traffic is high and steady. Engineers also weigh what the sums leave out: your team's time, start-up delays, and how hard it is to leave a provider. The 2024 DORA report added a caution that migrating to the cloud without using its flexibility can leave a team worse off than staying in a traditional data centre.

Releasing without fear: canaries and flags

Tests only check what someone thought to ask. Production has real users, real data and real hardware. So mature teams assume some bad changes will get through and design the release to limit the damage. In a canary release the new version goes first to a small share of traffic. Monitors compare it with the old version, and only if it looks healthy does the share grow: 1%, then 5%, then 25%, then everyone. The name comes from the canary that miners carried to warn of bad air.

A feature flag is the sibling idea: code ships switched off, and a setting turns it on for some users. If it misbehaves, the fix is to flip the setting, which takes seconds, not a rebuild and redeploy. Together, canaries and flags separate two things that used to be one: deploying code and releasing it to people.

Without JavaScript: if the bug crashes 100% of the users who get it and the threshold is 2%, the canary stops at the first stage, having hit 10,000 of 1,000,000 users. Releasing to everyone at once hits all 1,000,000. If the bug hits only 0.5% of users, the monitor never trips, and both strategies reach 5,000.

Look at the second case. A canary protects you from loud failures and does little against quiet ones, which is why teams also watch business measures and error reports, not just crashes. Back to July 2024. CrowdStrike's published root-cause analysis described the failure as a mismatch: the sensor's code supplied 20 input values to a content interpreter, while a new kind of content expected 21. The check that was meant to validate such content assumed 21, so it passed, and the interpreter then read past the end of an array. The analysis listed missing runtime bounds checks, and, among its findings, that this kind of content should have staged deployment. Customers also had had no way to delay these updates; the company later gave them more control. A crash on the first one percent of machines is exactly what a canary stage is for.

Observability and the error budget

Once software runs on machines you cannot walk up to, you need to see inside it. Observability is the ability to ask new questions about a running system from the signals it emits. The usual three signals are logs (a record of events), metrics (numbers over time, like requests per second) and traces (the path of one request through many services). Where engineers disagree is how much more than "three kinds of data" it takes: many argue that observability means being able to ask questions you did not think of in advance, not just draw dashboards for the failures you predicted.

Perfect reliability is not a sensible goal; each extra "nine" costs far more than the last. Google's Site Reliability Engineering book frames it as a budget. A team sets a service-level objective (SLO), say 99.9% of requests succeed. The remaining 0.1% is the error budget: unreliability the team is allowed to spend. While budget remains, releases go ahead. When it is spent, the team slows down and works on reliability. It turns an argument between "ship faster" and "be safer" into a number both sides can read.

Without JavaScript: a 99.9% objective allows 43.2 minutes of failure in 30 days; 99.99% allows 4.32 minutes; 99.999% allows about 26 seconds.

Large outages show the same lessons at scale. On 19 and 20 October 2025, Amazon's N. Virginia region (us-east-1) had a long disruption. Amazon's own account traced it to a latent race condition in the system that manages DNS records for its DynamoDB database service: two automated components interacted badly and left the service's regional endpoint with an empty DNS record, so nothing could find it. The core errors lasted from 11:48 PM PDT on the 19th to about 2:40 AM, with other impacts continuing until 2:20 PM on the 20th. Customer applications and other AWS services that depended on DynamoDB failed together. Three lessons repeat: small automation bugs can have regional reach, dependencies are rarely as independent as diagrams show, and the useful response afterwards is a blameless postmortem that asks what let it happen.

What a graduate just sees

• The cheapest bug to fix is the one found nearest the keyboard. A pipeline orders its checks to find most mistakes in minutes.
• Small, frequent changes are safer than big, rare ones: each is easier to test, to understand and to undo.
• Many small chances of failure multiply: a thousand slightly flaky tests make a pipeline nobody believes.
• Build things so they can be replaced, not repaired. An image is rebuilt; it is not patched.
• Every cloud choice answers "which layers do I run?", and your data and who can reach it are always yours.
• Tests cannot prove production is safe. Staged rollouts and flags limit how many users a mistake can reach.
• A canary catches loud failures, not quiet ones. Watch what users experience, not only what crashes.
• Reliability is a budget, not a virtue: choose how much failure you can afford, then spend it deliberately.
• After an outage, ask what in the system made the mistake easy, not who made it.

Where it shows up next

• Computer Security (34/33): a release pipeline is also an attack surface. Whoever can change the pipeline or a dependency can change what runs everywhere, and unpatched images are a common way in.
• Computer Networks (34/27): load balancers, DNS and regions are the plumbing that canaries and outages travel through.
• Building AI Applications (36/78): an AI feature needs the same discipline, with evaluation sets playing the role of tests and model versions needing staged rollouts like any other release.
• In the world: site reliability engineers, platform and DevOps engineers, the on-call rota, and the status page that tells you a provider is having a bad day.

If you want to go deeper

• 📘 Textbooks. Jez Humble and David Farley, Continuous Delivery (Addison-Wesley, 2010), the book that named the pipeline idea; and Nicole Forsgren, Jez Humble and Gene Kim, Accelerate (IT Revolution, 2018), the research on what separates high-performing teams.
• 📘 Free to read. Betsy Beyer and colleagues (eds.), Site Reliability Engineering (O'Reilly, 2016), free at sre.google/books; and Titus Winters, Tom Manshreck and Hyrum Wright (eds.), Software Engineering at Google (O'Reilly, 2020), free at abseil.io.
• 📘 Gentler. Gene Kim, Kevin Behr and George Spafford, The Phoenix Project (IT Revolution, 2013), a novel about a failing IT department, for the feel of the problems before the tools.
• 🌐 Free course. MIT's The Missing Semester of Your CS Education, short lectures on the shell, version control, debugging and shipping code; and the DORA research site, dora.dev, for the delivery measures and annual reports.