Computer Architecture: Why Your Processor Spends Its Life Waiting

A modern processor core can start several instructions on every tick of a clock that ticks around four billion times a second. Then it needs one number that isn't in its cache, asks main memory for it, and waits about 60 nanoseconds. On one measured chip that is roughly 250 ticks, long enough to have started on the order of a thousand instructions. Nearly everything a computer architect does is a trick for not noticing that wait.

There is a second puzzle in the same machine. The fastest Pentium 4 chips ran at 3.8 GHz, and in 2004 Intel cancelled its plans for a 4 GHz one and turned to dual-core designs. Two decades later, typical clock speeds are still only a few GHz, yet the chip in your pocket is many times faster than that desktop. Speed came from somewhere other than a faster clock.

This piece is about both puzzles: why a processor spends its life waiting, and why it got faster anyway. Along the way you will build an adder out of gates, encode a real instruction by hand, and break a cache on purpose.

Everything is a pattern of bits

A bit is a wire that is either low or high. Eight of them in a row make a byte, and the meaning of the pattern lives in how you read it, not in the wire. The same eight bits can be a small number, a negative number, a letter or the colour of a pixel.

Read as a plain number, each position counts double the one to its right: 1, 2, 4, 8 and so on. Negative numbers use two's complement: the leftmost bit counts as minus its weight instead of plus. In eight bits that makes the top bit worth −128, so 1111 1111 is −128 + 127 = −1.

Tap to make 0111 1111 (127), then press Add 1. The answer is −128. The hardware did nothing wrong: the counter ran out of room and wrapped, the way a car odometer rolls over. Programs that forget this have real bugs, and a 64-bit register simply moves the cliff a long way off: it holds 2⁶⁴, about 1.8 × 10¹⁹, different patterns.

Gates add numbers

Adding is the first thing a processor learns to do with bits, and it needs no new idea. Add two bits and you get a sum bit and a carry bit. The sum is 1 when exactly one input is 1 (an XOR gate); the carry is 1 when both are (an AND gate). Chain one such unit per column, passing each carry into the next column, and you have a ripple-carry adder.

Each column is a full adder: five gates (two XOR, two AND, one OR). Sixty-four columns need 64 × 5 = 320 gates. Try 0111 + 0001: the carry ripples through three columns before the answer settles at 1000. The carry's path through each column is two gates long, so a 64-bit ripple adder has a path of 128 gate delays, which is slow. Real processors use carry-lookahead and cleverer variants that compute many carries at once, trading more gates for less waiting. That trade (spend hardware, save time) comes back in every idea below.

Subtraction costs almost nothing extra. To compute A − B, flip every bit of B and add 1 (that is two's complement negation), and use the same adder. One circuit does both. Multiplication, comparison and the rest of the arithmetic unit are built in the same way, from the same few gates.

A program is numbers in memory: the instruction set

The processor's loop is short enough to say in four steps. Fetch the instruction whose address is in the program counter. Decode it: which operation, which registers. Execute it in the arithmetic unit, memory or branch logic. Write back the result and move the counter on. Then repeat, billions of times a second. If you want to watch that loop run, the toy CPU in Processors (29/11) steps through it. Here is the part that is easy to miss: the instruction itself is just a number.

The list of instructions a chip understands, and how each is written as bits, is its instruction set architecture (ISA). It is the contract between hardware and software: x86-64 (most PCs and servers), ARM (most phones) and RISC-V (open and growing) are different contracts. A register is one of a small number of named slots inside the core, typically 16 to 32 general ones, that instructions read and write directly. Registers are the fastest storage there is, and the compiler's main job is to keep the hot values in them.

With the defaults, add x5, x6, x7 ("put x6 + x7 into x5") becomes the 32-bit pattern whose hex is 0x007302B3. The fields are fixed-width slices: a 7-bit opcode, three 5-bit register numbers, and 10 bits that pick which arithmetic. Because the fields are always in the same place, decoding is mostly wiring, not thinking. That regularity is the RISC idea: keep instructions simple and uniform so the hardware can be fast. x86 instead has instructions from 1 to 15 bytes long and decodes them into simpler internal steps first. Both work; they are different bets.

Memory is a hierarchy, and distance is time

Fast memory is small and expensive; big memory is slow and cheap. So computers stack them. A few registers, then a small L1 cache, bigger and slower L2 and L3 caches, then main memory (RAM), then an SSD or hard disk. Each level holds copies of the most recently used pieces of the level below.

Numbers make the gap real. Here are measured latencies on one desktop chip (an Intel Core i7-6700 at 4.0 GHz), with the disk figures from the widely quoted "latency numbers" list. The last column scales one clock tick to one second so the gaps are human-sized.

Caches work because programs are lazy in a predictable way: after touching a byte they usually touch its neighbours (spatial locality) and touch the same byte again soon (temporal locality). A cache therefore never loads one byte; it loads a whole line of 64 bytes. The rule a graduate remembers is the average-time formula, with hits cheap and misses expensive:

Worked: with a 1 ns hit and a 62 ns miss, a 1% miss rate averages 1 + 0.01 × 62 ≈ 1.6 ns; a 10% miss rate averages about 7.2 ns, more than four times worse. The following simulator makes the point with real code. A 32 × 32 table of 4-byte numbers sits in memory row after row. Sum it in two orders.

Row by row, only 64 of the 1,024 accesses miss, because one load brings in 16 numbers. Column by column with 64 lines it is the same 64, but with 32 or fewer lines it collapses to 1,024 misses out of 1,024 (the two halves of the table fight for the same slots) and the run is roughly thirteen times slower. Same arithmetic, same answer. Choose "In 8×8 tiles" and even a 16-line cache gets back to 64 misses, because each small block of the table is finished while it is still cached. This is why programmers who know architecture tile their loops, and why linked lists scattered around memory lose to arrays.

Pipelining: an assembly line for instructions

Doing one instruction fully before starting the next wastes the hardware, as if a laundromat ran one load through wash, dry and fold before touching the next. A pipeline overlaps them. The textbook design has five stages: fetch, decode, execute, memory, write-back. When the line is full, one instruction finishes per tick even though each takes five.

For 100 instructions that is about 104 ticks instead of 500: nearly a 5× gain from the same parts. The catch is hazards. If an instruction needs a result the one ahead of it hasn't finished, the line must wait, and the empty slot is called a bubble. A branch is worse: until the processor knows which way it goes, it guesses (branch prediction) and starts fetching down that path. On a wrong guess the half-done work is thrown away, which on a modern deep pipeline costs roughly 15 to 20 ticks.

Slide the cost to 2 ticks every 4th instruction: the 100 instructions take 152 ticks, about 1.5 ticks each, and the gain over no pipeline falls from nearly 5× to about 3.3×. Push the cost to 16 ticks, like a mispredicted branch, and a hazard every 12th instruction cuts the gain to about 2.2×. That is why predictors are among the most elaborate parts of a modern core.

Parallelism helps, up to the part you can't split

When one core can't go faster, use more of them. Four cores can run four programs at once, or one program split in four. The limit was written down by Gene Amdahl in 1967: whatever part of the job must run in sequence caps the gain, however many workers you add. In its usual form, with p the fraction of the work that can be parallelised and N the number of cores:

With 95% parallel work, 16 cores give 9.1×, not 16×, and no number of cores can ever beat 20×, because the 5% serial part is still there. Even at 99.9% parallel, 1,024 cores give only about 506×. A GPU is the extreme version of the bet: thousands of simple lanes running the same instruction on different data. It shines on graphics and matrix maths, where p is nearly 1, and is poor at code full of branches and dependencies. See GPUs (29/12) for how that bet turned into the AI hardware industry. Amdahl's pessimism has a famous rebuttal (Gustafson, 1988): bigger machines are usually given bigger problems, which keeps p high. Both are true, in different jobs.

The power wall: why the clock stopped

Every time a transistor switches it charges and discharges a tiny capacitor, which burns energy. Add up billions of them, thousands of millions of times a second, and you get heat that has to leave a chip the size of a postage stamp. The dynamic power of a chip is:

Capacitance, voltage squared, frequency. For decades, shrinking transistors cut C and V so fast that you could raise f for free in power-per-area terms: this is Dennard scaling, from a 1974 paper by Robert Dennard and colleagues. Around 2005 to 2007 it broke, because leakage current and the threshold voltage stopped shrinking with the transistor, so voltage could no longer fall. Intel's Prescott Pentium 4 already drew up to 103 watts, and on 7 May 2004 Intel cancelled its successors, Tejas and Jayhawk, and pivoted to dual-core. Herb Sutter's 2005 article "The Free Lunch Is Over" told software developers what followed: programs would stop getting faster for free and would have to use several cores.

Under that model, doubling the clock from 3 to 6 GHz needs 2³ = 8 times the power. Two cores at 3 GHz give the same ideal throughput for 2 times the power, and four cores at 1.5 GHz for half a unit. The modern chip is the answer: many modest cores, some specialised units, and parts switched off when idle, because a chip can now hold more transistors than it can power at once.

What a graduate just sees

• Bits have no meaning until you choose how to read them; overflow is the odometer rolling over.
• Arithmetic is a few gates, and the slow part of an adder is the carry travelling across it.
• An instruction is a number, and a program is a list of them in memory.
• Distance is time: a trip to RAM costs hundreds of ticks, so access patterns matter more than operation counts.
• Walking memory in order is nearly free; jumping around it is not.
• A pipeline is full only if the code lets it be, and every branch is a bet.
• More cores help only as far as the work can be split (Amdahl).
• Power rises with the square of voltage and the cube of speed, so wide and slow beats narrow and fast.

Where it shows up next

• Operating Systems (34/26) rests on hardware features introduced here: interrupts, the user/kernel privilege split, and the address-translation hardware behind virtual memory.
• Computer Security (34/33): the Spectre and Meltdown flaws, disclosed in January 2018, came from speculative execution (the guessing described under pipelining) leaking data through the cache.
• Algorithms (34/24) and Data Structures (34/23): two algorithms with the same operation count can differ tenfold in speed because of locality.
• Compilers (34/30): a compiler's whole output is instructions in one ISA, and its hardest job is choosing registers.
• Machine Learning (34/32): training is matrix multiplication on GPUs and similar chips, where Amdahl's p is nearly 1.

If you want to go deeper

• 📘 Textbook: David Patterson and John Hennessy, Computer Organization and Design: RISC-V Edition (2nd ed., 2020). The standard first course; builds a processor step by step.
• 📘 Textbook: Hennessy and Patterson, Computer Architecture: A Quantitative Approach (7th ed., late 2025, with Christos Kozyrakis). The graduate-level reference for caches, parallelism and accelerators.
• 📘 Programmer's view: Randal Bryant and David O'Hallaron, Computer Systems: A Programmer's Perspective (3rd ed.). Best for locality and for how code becomes machine instructions.
• 📘 Gentler: Charles Petzold, Code: The Hidden Language of Computer Hardware and Software (2nd ed., 2022). Builds from a flashlight signal to a CPU with no maths beyond arithmetic.
• 🌐 Free, hands-on: Nand2Tetris (Nisan and Schocken, The Elements of Computing Systems): build a working computer from a NAND gate up, free in a non-profit setting.
• 🌐 Free course: MIT OpenCourseWare 6.004 Computation Structures (Spring 2017): lectures and problem sets from gates to a pipelined processor.