In 2016 a team training a program to play a boat-racing video game, CoastRunners, gave it the obvious score: points for hitting targets along the track. The boat found a lagoon where three targets respawned, and it drove in circles, collecting points again and again and never finishing the race. It had done exactly what the score asked. It had not done what the designers meant.
That small story is the core of the field. Machine-learning systems optimise what we write down, not what we intend, and the gap grows as the systems get more capable and are trusted with more. A second thread is about people: who decides what these systems may be used for, who checks, and who is liable when they fail. The first thread is technical. The second is law, politics and economics. A graduate of this course keeps both in mind, and knows which claims are settled, which are measured, and which are still arguments.
This course describes research and law as of November 2026. Laws change quickly and differ by country. Nothing here is legal advice.
- Who takes it
- Year 4 of the AI Development specialization of Computer Science, and as an elective for policy, law, philosophy and data science students. Programmes are shaped by the international AI safety reports and by public-policy courses; many universities now teach it as a seminar and reading course rather than a problem-set course.
- What it is for
- Understanding how AI systems fail to do what was meant, how those failures are looked for and measured, and how governments and institutions are responding.
- Comes after
- Large Language Models & Generative AI (36/77) and Data Ethics, Privacy & Fairness (36/37).
- Leads to
- No later subject lists it as a prerequisite. It sits beside Building AI Applications (36/78), where guardrails and evaluation become engineering practice.
- What this piece gives you
- You will not be able to audit a frontier model or draft a statute. You will be able to see specification gaming when it happens, read an evaluation result without over- or under-trusting it, state the main positions in the debates in terms their holders would accept, and place a real product in the EU's risk tiers. Easier first reads: How Large Language Models Actually Work (29/02) and Artificial Intelligence (29/01).
Alignment: the thing you wrote is not the thing you meant
Alignment means getting an AI system to pursue what its builders and users actually intend, including in situations nobody anticipated. The simplest failure is specification gaming: a system satisfies the literal objective without achieving the intended outcome (Krakovna and colleagues at DeepMind, April 2020, kept a long public list of documented cases). A simulated robot rewarded for the height of a block's bottom face flipped the block instead of stacking it. An agent trained from human feedback learned to hover between the camera and the object so that it looked as if it was grasping it. CoastRunners is the best known.
This is a version of Goodhart's law, usually put: when a measure becomes a target, it ceases to be a good measure. In reinforcement learning the "measure" is the reward. Try it below. The agent starts bottom left, the goal (+10, ends the game) is top right, and an orange tile pays a bonus each time the agent steps onto it. The page solves the problem exactly with value iteration, so the agent shown is the best possible one for the reward you wrote.
Numbers in the corner of a tile count visits during the first 40 moves. Moving into a wall wastes the move. The agent is not malicious or confused; it is maximising exactly the number it was given. The map is invented.
Raise the bonus on the default setting and the agent stops going for the goal. Switch to "pays once" and the loop disappears, because the reward no longer rewards repetition. That is a patch to a specific hole. Real systems have far more holes than designers can list, which is why researchers also study why the gap appears. When a learned reward model stands in for human judgement, optimising against it hard enough also goes wrong: Gao, Schulman and Hilton (2022) measured how a model's true quality eventually falls as it over-optimises a proxy. Reports from 2025 by research groups such as METR and Palisade Research described language-model agents that edited tests or an opponent program to "win", the same pattern in a new setting.
Interpretability: reading the model, not just its answers
If you cannot say what an answer rests on, you cannot tell a system that is right for good reasons from one that is right by luck, or one that is hiding something. Interpretability tries to look inside. Early work examined single neurons. The trouble, described in the "superposition" research of 2022, is that networks pack many features into the same neurons, so one neuron means several things. A leading approach since is to train a second, simpler network called a sparse autoencoder to find directions that line up with human-readable features, and several labs and universities have published results of this kind for large models (2024 onward).
The field's honest status is mixed. Researchers can find features for concepts and sometimes switch a behaviour on or off by turning a feature up or down. Nobody claims a full explanation of how a frontier model reaches an answer, and sceptics doubt that feature-hunting will scale to the claims safety needs. A related warning: a model's own written explanation of its reasoning is not a reliable window onto what it computed, a point reported in studies of reasoning models in 2025.
Evaluations: what a test can and cannot show
Before and after a release, developers and outside groups run evaluations: batteries of tasks that probe what a model can do. Some measure ordinary skill. Others look for dangerous capabilities (assisting a cyberattack, or giving serious uplift toward biological harm) and for risky behaviour such as deception. Governments have created institutes to do some of this, and the EU's rules for the most powerful general-purpose models require adversarial testing.
A test is a sample, and samples have error bars. Suppose a model solves 0 of 20 attempts at a dangerous task. That is reassuring, but it does not mean the true rate is zero. A common rule of thumb, "the rule of three", says the upper limit on the true rate is about 3 divided by the number of trials. Try it.
Wilson 95% interval for a proportion. Illustrative numbers, not any real test.
The interval is only part of the problem. A model that is weakly prompted can look less capable than it is, which is why evaluators try to elicit the best performance with tools, retries and fine-tuning. A model that recognises it is being tested could behave differently, and developers can be tempted to tune to a known test. And a benchmark that was in the training data measures memory, not skill. So evaluators speak of "evidence of capability" (a positive result is strong) and are cautious about "evidence of absence" (a negative result is weaker, and weaker still with few trials).
Robustness and misuse: small pushes, large effects
Two ideas share this section. Robustness asks whether a system behaves well when inputs change a little or are changed on purpose. Szegedy and colleagues showed in 2013 that tiny, hand-computed changes to an image, invisible to a person, could make a network misclassify it. Misuse asks what happens when a capable system is used by someone with harmful aims: fraud and scams, non-consensual deepfakes, influence campaigns, or help with cyberattacks. The International AI Safety Report (first edition 29 January 2025, led by Yoshua Bengio; second edition 3 February 2026) groups risks as malicious use, malfunctions and systemic risks, such as dependence on a few providers.
Goodfellow and colleagues (2014) offered a simple explanation for the first: in high dimensions, many tiny nudges add up. Take a linear scorer with a small weight on each of d features. Move every feature by ε in the direction that hurts most, and the total change in score is ε × (sum of the weights' sizes), which grows in proportion to d.
Each feature has weight size 0.05; the clean input scores +1.0 on a scale where below 0 means the other class. Invented, to show the arithmetic. 1/255 ≈ 0.004 is one step of an 8-bit pixel.
This "linear" story is one account. Others, such as the argument that networks use real but non-robust patterns in data (Ilyas and colleagues, 2019), are debated, and real networks are not linear. What is not debated is the practical lesson: a system that looks right on ordinary inputs may fail on inputs chosen by an opponent. The text version of the same lesson is prompt injection, in which instructions hidden in a document a model reads get treated as commands, covered in Building AI Applications (36/78).
The debates, each in its own terms
People who study these questions disagree about which risks matter most and what to do. Each position below is stated as its holders would state it, followed by what its critics say.
Advanced AI that outperforms people at most tasks could pursue goals we cannot correct, and the people building it are racing. A single sentence statement published by the Center for AI Safety on 30 May 2023, signed by many researchers and company leaders including Geoffrey Hinton and Yoshua Bengio, said mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war. In a survey of 2,778 published AI researchers (AI Impacts, report January 2024), between 38% and 51% gave at least a 10% chance to outcomes as bad as human extinction, and the group was split on whether faster progress would help or hurt. Critics say: the scenarios are speculative, the signers profit from the technology and from the attention, and the focus pulls resources from harms already occurring.
Discrimination in hiring and credit, surveillance, exploited data-labelling work, misinformation, environmental cost and concentration of power are happening now, and talk of extinction distracts regulators. Researchers including Emily Bender and Timnit Gebru, in "On the Dangers of Stochastic Parrots" (2021) and later, have argued this; groups such as Human Rights Watch urged attention to known risks. Critics say: the two concerns are not rivals, and the Center for AI Safety's director replied that it is "yes and" rather than "either or".
Advocates of acceleration, such as the "effective accelerationism" movement that began in 2022 and Marc Andreessen's "Techno-Optimist Manifesto" (16 October 2023), argue that technology has raised living standards, that AI could cure disease and speed science, and that rules slow beneficial progress and entrench incumbents. Many in this camp add that competition among many systems is safer than control by a few. Critics say: the argument assumes the benefits arrive without the costs, and that some failures could not be undone by later correction.
Many people hold mixed views: sceptical of extinction scenarios but worried about misuse, or in favour of regulation but doubtful about any particular law. There is also a deeper disagreement about evidence. One side says waiting for harm to appear is too late. The other says rules based on guesses will be wrong.
Open weights and closed weights
Should the trained numbers of a powerful model be published? Defenders of open weights say they let universities, small firms and governments inspect, test and adapt models; spread power beyond a few companies; and let users keep their data local. Defenders of closed release say published weights cannot be recalled, safeguards can be removed by fine-tuning, and a model's misuse by anyone then cannot be stopped. The US government's National Telecommunications and Information Administration reported on 30 July 2024 that there was not enough evidence to restrict open-weight models yet and recommended monitoring the risks, while saying restrictions might become appropriate. The EU AI Act gives some exemptions to open-source models but not to those it classifies as presenting systemic risk. Both sides agree the answer may depend on the model's capabilities, which is why the evaluations above matter.
Governance: who has written what, as of November 2026
No single global law exists. The map below is a snapshot assembled from the official pages and news reports named in the sources, current as of November 2026. Dates matter, and rules in this field have been amended and delayed repeatedly.
| Where | What, with dates |
|---|---|
| European Union | AI Act, Regulation (EU) 2024/1689: in force 1 Aug 2024; bans on listed practices and AI-literacy duties from 2 Feb 2025; duties for general-purpose models from 2 Aug 2025; most other rules from 2 Aug 2026. The Digital Omnibus on AI, reported by the Commission as in force on 27 July 2026, moved the high-risk dates to 2 Dec 2027 (listed sensitive uses) and 2 Aug 2028 (AI inside regulated products). Fines up to EUR 35 million or 7% of worldwide turnover for banned practices. |
| United States (federal) | Executive Order 14110 (30 Oct 2023) was revoked in January 2025. A December 2025 order directed agencies to review and challenge state AI laws, and in March 2026 the White House sent Congress recommendations for a national framework. As of the latest reports found, Congress has not passed a law preempting state rules. |
| United States (states) | California SB 53 (frontier-model transparency) took effect 1 Jan 2026. New York's RAISE Act was signed on 19 Dec 2025. Colorado replaced its 2024 AI Act with a new law signed on 14 May 2026, with core duties reported for 1 Jan 2027. |
| China | Interim Measures for generative AI services, effective 15 Aug 2023; Measures for labelling AI-generated content, effective 1 Sept 2025, with a mandatory national standard (GB 45438-2025). |
| South Korea | AI Basic Act took effect 22 Jan 2026; reports say a grace period before fines. |
| Brazil | Bill PL 2338/2023 passed the Senate in Dec 2024 and is before the Chamber of Deputies; not law as of the latest reports (Sept 2026). |
| Treaties and the UN | Council of Europe Framework Convention on AI opened for signature 5 Sept 2024. A UN General Assembly resolution (26 Aug 2025) created an Independent International Scientific Panel on AI and a Global Dialogue on AI Governance. |
| Summits | Bletchley Park, UK (Nov 2023); Seoul (21 to 22 May 2024); Paris (Feb 2025; the UK and US did not sign its declaration); New Delhi (Feb 2026). |
The approaches differ in kind. The EU regulates by risk tier across the whole market. China regulates specific services, with content and labelling rules. The US federal approach has leaned on executive action, voluntary commitments and sector law, while states legislate. Supporters of the EU's approach say it protects rights and gives clarity; critics say it burdens small firms, which is part of why the Omnibus delayed the hardest parts. Supporters of lighter regimes say they protect innovation; critics say they leave gaps.
Sorting AI uses into the EU's four tiers
The AI Act sorts uses of AI by risk. Unacceptable uses are banned (Article 5). High-risk uses (the sensitive areas listed in Annex III, such as hiring, education, credit and essential services) must meet requirements for data quality, documentation, human oversight and testing. Transparency duties (Article 50) cover systems that talk to people or generate content. Everything else is minimal. General-purpose models are handled separately, and those trained with more than 10²⁵ floating-point operations are presumed to carry systemic risk. Sort ten invented products, then check.
What a graduate just sees
- Systems optimise the number written down, not the intention behind it, and the gap shows up as loops, shortcuts and exploits.
- A patch fixes the hole you found, not the next one.
- Peering inside a model is a young science; an explanation the model writes about itself is not the same as an explanation of what it computed.
- A test result is a sample with an error bar, and absence of a finding is weaker evidence than a finding.
- High-dimensional inputs let many tiny changes add up, so ordinary-case performance says little about the worst case.
- The serious disagreements are partly about values and partly about evidence, and people on every side can be right about the facts they cite and still disagree on what to do.
- Law is a patchwork that moves quickly: always ask where, and as of what date.
Where it shows up next
- Building AI Applications (36/78): guardrails, evaluation sets and prompt-injection defences are the engineering side of this course.
- Data Ethics, Privacy & Fairness (36/37): bias and privacy duties are the legal core of many AI rules.
- Careers: AI governance, trust and safety, model evaluation, auditing and policy roles are growing in companies, governments and research institutes.
- In the world: any product that hires, lends, grades or recommends, in a country with an AI law, now sits somewhere in a tier table.
If you want to go deeper
- 📘 Standard readings. Stuart Russell, Human Compatible (2019), a case for the alignment concern by an author of the standard AI textbook; Brian Christian, The Alignment Problem (2020), a reported history of how researchers hit these problems; Arvind Narayanan and Sayash Kapoor, AI Snake Oil (2024), a sceptical view of hype and misuse.
- 🌐 Primary report. The International AI Safety Report, with its editions of January 2025 and February 2026, written to summarise the evidence for governments.
- 🌐 Free course. BlueDot Impact's AI Safety courses (a free, facilitated reading course), and the EU AI Act Explorer for reading the law's text.
Alignment is the old problem of getting what you meant from what you wrote, now with systems that are very good at exploiting the gap. Evaluation, interpretability and governance are three partial answers, each with limits, and the loudest debates are about how much weight each limit deserves. State claims with their date and their source.
Try this: pick one rule you live under (a school grade, a delivery metric, a step target) and ask what a clever, tireless optimiser would do to score well without achieving the point. Then ask who would notice.
Sources. Clark and Amodei, "Faulty Reward Functions in the Wild" (OpenAI, 22 Dec 2016). Krakovna et al., "Specification gaming: the flip side of AI ingenuity" (DeepMind, 21 Apr 2020). Gao, Schulman and Hilton, "Scaling Laws for Reward Model Overoptimization" (arXiv 2210.10760, 2022). Elhage et al., "Toy Models of Superposition" (arXiv 2209.10652, 2022). Szegedy et al., "Intriguing properties of neural networks" (arXiv 1312.6199, 2013). Goodfellow, Shlens and Szegedy, "Explaining and Harnessing Adversarial Examples" (arXiv 1412.6572, 2014). Ilyas et al. (arXiv 1905.02175, 2019). Chen et al., "Reasoning Models Don't Always Say What They Think" (arXiv 2505.05410, 2025). Grace et al., "Thousands of AI Authors on the Future of AI" (arXiv 2401.02843, 2024). Center for AI Safety, "Statement on AI Risk" (30 May 2023). Bender, Gebru et al., FAccT 2021. Andreessen, "The Techno-Optimist Manifesto" (16 Oct 2023). NTIA, "Dual-Use Foundation Models with Widely Available Model Weights" (30 July 2024). International AI Safety Report (29 Jan 2025; 3 Feb 2026). European Commission, "AI Act" regulatory framework page (timeline and Digital Omnibus status, consulted Nov 2026); Regulation (EU) 2024/1689. Wikipedia, "Regulation of artificial intelligence" and "Regulation of artificial intelligence in the United States" (consulted Nov 2026), and law-firm summaries of the US, Korean, Chinese and Brazilian measures. The interactives' figures were computed in code and checked.