A passenger asked an airline's website chatbot about bereavement fares. The bot told him he could buy a full-price ticket and claim the discount back within 90 days. The airline's actual policy required applying before travel, and it declined his claim. In February 2024 British Columbia's Civil Resolution Tribunal ordered it to pay him about CA$812, the difference in fare, plus costs and interest. The airline had argued that the chatbot was a separate entity responsible for its own words. The tribunal disagreed: the bot was part of the airline's website, and the airline was responsible for what it said.
The model in that story was not the whole problem. The system around it had no reliable way to ground answers in the real policy, check them, or limit what a customer could be told. Those are the jobs of this course. Large Language Models & Generative AI (36/77) explains what a model is. This piece is about everything you build around one so that something dependable comes out.
- Who takes it
- Year 4 of the AI Development specialization of Computer Science, and developers who want to build with models rather than only call them. Courses on it are new and vary: expect names such as LLM applications, AI engineering or applied generative AI.
- What it is for
- Turning a general-purpose model into a feature that is accurate, affordable, fast and safe enough to ship, and knowing when it is not.
- Comes after
- Large Language Models & Generative AI (36/77) and Software Engineering (34/31).
- Leads to
- No later subject lists it as a prerequisite. It sits beside Data Ethics, Privacy & Fairness (36/37) and AI Safety, Alignment & Governance (36/79) in the same year, and borrows release practice from Shipping Software (36/76).
- What this piece gives you
- You will not be able to run a production AI system after this. You will see six ideas that decide whether one works. It does not re-teach how models work; for that, read 36/77, or start with the easier How Large Language Models Actually Work (29/02). Vendor-neutral: any provider's model slots into the same patterns, and prices below are invented unless stated.
Prompts are specifications, and replies are untrusted input
A prompt is a specification written in prose for a reader that is very capable and has no context beyond what you give it. The habits that work are those of writing a good brief for a stranger: state the task and the audience, give the rules, show two or three worked examples of input and ideal output, say what to do when the information is missing, and keep trusted instructions apart from the untrusted material the model must work on. Treat the prompt like code. Keep it in version control, change it deliberately, and test every change (see the evaluation section below).
Programs cannot use free text, so applications ask for structured output: a reply in a fixed format, usually JSON that matches a schema. Several providers now offer modes that constrain the model while it generates, so the output cannot break the schema's shape. That removes a class of failures, but note exactly what it guarantees: shape, not truth. A model forced to produce an integer will happily produce the wrong integer.
So the rule is the one a web developer already knows from forms: never trust input, and a model's reply is input. Parse it, validate it in ordinary code against the schema, then check what you can about the content. If validation fails, the usual remedy is to retry with the error message added to the prompt. If a single attempt produces valid output 85% of the time, three attempts succeed with probability 1 − 0.153, about 99.7%, provided failures are independent, which they often are not: a prompt that confuses the model once tends to confuse it again. Try six replies to the same task.
The checks run in the page on the text shown: JSON parsing, then each field's rule. The content check compares with the email (12 cartons, pickup).
Without JavaScript: of the six replies, one is correct, four fail validation (stray prose around the JSON, a quantity written as a word, a delivery option that does not exist, a missing item) and one passes validation while being wrong (quantity 2 instead of 12). Validation cannot catch the last one; only a content test can.
The last reply is the instructive one. The schema checks passed, and the order is still wrong. Closing that gap is what the evaluation section is for.
Retrieval: find the right page before the model speaks
A model knows what was in its training text, up to a cutoff date, and nothing about your company's policies, last week's prices or a customer's file. It also fills gaps with fluent guesses. Retrieval-augmented generation (RAG), named in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University, puts a lookup in front of the model: find the passages most relevant to the question, paste them into the prompt, and ask the model to answer from them.
The pipeline has a fixed shape. Chunk your documents into pieces. Embed each piece as a vector, a list of numbers that places it in a space where similar meanings sit close together (Large Language Models & Generative AI, 36/77, shows how). Store the vectors. At question time, embed the question the same way, find the nearest chunks, and send them to the model with the question. Nearness is usually measured by cosine similarity, the angle between two vectors:
Real embeddings have hundreds or thousands of learned dimensions. Below is a toy with a made-up six-page library handbook, so you can see the mechanics. You can rank pages by shared words, or by four invented "meaning" numbers per page and per question. Pick the last question and compare.
Word overlap: cosine between word counts after dropping little words, computed from the real text. Meaning vectors: four invented numbers per page (loans, money, schedule, people and places), standing in for what a learned embedding model would output. Cosine is computed live for both.
Without JavaScript: for "Can I visit on Tuesday afternoon?" word overlap scores every page 0, because the page says "Monday to Friday" and "opens at 9". The invented meaning vectors rank the Hours page first, with a cosine close to 1.
Two lessons sit in that toy. Keyword matching misses paraphrases, which is why embeddings took over (and why many systems combine both). And retrieval is a ranking problem with its own failure modes: the right page can rank fourth. If it does not reach the prompt, the model cannot use it. A well-known test (Liu and colleagues, 2023, "Lost in the Middle") found that models tend to use information near the start or end of a long prompt better than information in the middle, so cramming in twenty chunks is not a free fix.
The other decision is where to cut. Chunks that are too small split an answer across two pieces, neither of which is a good match. Chunks that are too large bury the answer in text about other things and waste tokens. Try it on a short invented passage.
The question is "How many books can I borrow?" and the answer is the 13-word sentence about borrowing. Overlap repeats the last words of each chunk at the start of the next.
Without JavaScript: with 12-word chunks and no overlap, the borrowing sentence is cut across two chunks. With 16-word chunks it sits whole in the first chunk, but is only 13 of 16 words. At 40 words per chunk it fits, but is a third of the text.
There is no universal chunk size. People tune it on their own documents, often cutting at natural boundaries such as headings and paragraphs. Finally, retrieval is also where access control belongs: if a user must not see a document, filter it out before ranking, so it can never reach the prompt.
Tools and agents: a loop with a budget
A model only produces text. A tool lets that text cause something to happen. You describe functions to the model, such as lookup_orders(customer). When it wants one, it replies with a structured request naming the function and its arguments. Your code, not the model, runs the function and sends the result back, and the model carries on. An agent is this repeated in a loop until the model says it is finished. The idea of interleaving reasoning and actions was studied in the ReAct paper (Yao and colleagues, 2022). Open standards such as the Model Context Protocol, introduced in November 2024 and since placed under the Linux Foundation, aim to let one tool description work with many models and applications.
Follow one small task through the loop and watch the context. The key thing to notice is that each call re-sends everything said so far.
Token counts are invented but proportionate: 800 tokens of instructions and tool descriptions, 30 for the question, 40 per tool request, 150 and 30 for the two tool results, 45 for the final answer. The exchange rate (0.92) is invented too.
Without JavaScript: the three model calls read 830, 1,020 and 1,090 tokens, 2,940 in all, to produce 125 tokens of output.
Two engineering facts follow. First, cost grows with the length of the loop, and faster than linearly, because the history is re-read every time (providers' prompt-caching features exist to soften this). Second, errors compound. If each step is right 95% of the time and a task needs ten steps, the chance of getting through without a mistake is 0.9510, about 60%.
Computed as pn, assuming independent steps. Real agents can notice and repair their own errors, which helps, and can also compound them, which does not.
Without JavaScript: 95% per step over 10 steps gives 59.9% success; 99% over 20 steps gives 81.8%; 90% over 10 steps gives 34.9%.
That is why designers keep agents on a short leash: fewer steps, narrow tools, a cap on the number of loops and on spending, and a human confirming anything that cannot be undone, such as sending money or deleting files. Whether to give a system more autonomy is a trade-off between usefulness and risk that depends on the cost of a wrong action.
Evaluation: a test set is the new unit test
How do you know that a new prompt, a new model or a new chunk size is better? Reading a few outputs and nodding is the usual answer and a poor one. The discipline from Software Engineering (34/31) carries over: collect cases, define what passing means, run them on every change. A test set is a list of realistic inputs, each with what a good answer must contain or avoid, built from real usage and from every failure you find. Start with a few dozen and grow it.
Some checks are exact (is the JSON valid; is the cited document the right one). Many are not, since there are many good answers to a question. A common answer is LLM-as-judge: another model grades the output against a rubric. It scales and it is useful, but it is itself a measuring instrument that needs checking. Zheng and colleagues (2023) found that a strong judge agreed with human raters over 80% of the time, about as often as humans agree with each other, and also documented biases: judges tend to favour the answer shown first (position), longer answers (verbosity) and answers written by themselves (self-enhancement). The remedies are practical: swap the order and keep only consistent verdicts, grade against specific criteria, and keep a sample labelled by humans to measure the judge itself.
Then there is the arithmetic of small samples. Suppose prompt A passes 70% of your cases and prompt B passes 85%. Is B better?
Intervals are Wilson 95% confidence intervals for the pass rate, computed live. The judge setting widens each range by that many points, a worst-case bound for a judge that is wrong on that share of cases, all in one direction.
Without JavaScript: with 20 cases, A's plausible range is 48% to 85% and B's is 64% to 95%: they overlap heavily. With 200 cases, A is 63% to 76% and B is 79% to 89%: separate. A judge that disagrees with humans on 15% of cases could erase the whole gap.
A single run on a small set mostly measures luck. The common practices are larger sets, running the comparison on the same cases both ways, and treating a small gap as no evidence. And an evaluation set is not safe from overuse: tune a prompt against it long enough and it stops being an honest test, the same trap as any held-out data (Machine Learning, 34/32). Wire the set into your delivery pipeline (36/76), so that a prompt change which lowers the score is blocked like a failing unit test.
Cost and latency: tokens are the meter
Hosted models are billed by the token, separately for what goes in and what comes out. Output is priced higher: on two providers' published price lists checked in November 2026, an output token cost five to six times an input token. Cost per request is therefore
Time behaves differently. The model reads the whole input in parallel, which takes a fraction of a second, then writes the output one token at a time. So latency is mostly output length, while cost is often dominated by input, especially once retrieval or an agent's history has pushed thousands of tokens into every call. Try a support assistant, with invented prices for a small and a large model.
All invented: small model $0.30 in and $1.50 out per million tokens, 80 tokens per second, 300 ms plus 0.05 ms per input token before the first word; large model $3 and $15, 40 tokens per second, 500 ms plus 0.1 ms per input token. Cached input is priced at a tenth of normal. Real figures differ by provider, model and load: measure yours.
Without JavaScript: at 3,000 input and 300 output tokens, the small model costs $0.00135 a request, $405 over 30 days at 10,000 requests a day, and takes about 4.2 seconds. The large model costs ten times as much, $4,050, and takes about 8.3 seconds.
The levers follow from the formula: shorter prompts and fewer retrieved chunks, caching a long unchanging prefix, capping output length, and routing easy requests to a small model and hard ones to a large one. The routing decision needs the evaluation set too, because "the small model is good enough" is a claim to test.
Security and privacy: assume the text is hostile
A model cannot reliably tell instructions from data. Everything in the prompt is text, so text from an outsider can read like a command. In prompt injection (a term Simon Willison proposed in September 2022, by analogy with SQL injection) the attacker's words steer the model. Direct injection is a user typing it. Indirect injection, described by Greshake and colleagues in 2023, hides the instructions in material the app will fetch later: a web page, an email, a shared document, a retrieved chunk. The user never sees it.
The OWASP Top 10 for LLM Applications (2025) lists prompt injection first. Its page is frank that no method has been shown to prevent it completely, so the advice is to reduce what a hijacked model can do: limit its permissions, keep secrets and powerful functions in application code, label untrusted content, validate outputs in code, and require human approval for high-risk actions. Willison's rule of thumb, the "lethal trifecta" (June 2025), names the dangerous combination: an app that has access to your private data, is exposed to untrusted content, and can communicate outward. Remove any one leg and the data-theft path closes. It is a design heuristic, not a proof of safety.
A teaching model of Willison's heuristic, with invented example apps. Fewer legs does not mean safe: an app can still be steered into wrong or harmful actions inside its permissions.
Without JavaScript: an email assistant that reads inbound mail (untrusted), can see your files (private) and can send mail (outward) has all three. If it can read mail but not send anything, there is no direct path for a hidden instruction to carry your data out.
Many teams first reach for a filter that blocks phrases like "ignore previous instructions". See how far that gets.
The filter blocks any message containing "ignore previous instructions", "ignore all previous instructions" or "disregard previous instructions", ignoring capital letters.
Without JavaScript: the filter blocks the plain English attack, lets through a reworded one, a letter-swapped one and a French one, and wrongly blocks a harmless request to translate a sentence that contains the phrase.
Willison argues that products which claim to detect most attacks are not enough, since in web security a 95% catch rate still leaves the door open to a determined attacker. Classifiers have a role as one layer; the structural defences above do the real work.
Privacy is the other half. Every prompt is data you send somewhere, and prompts and logs often contain names, health details or secrets. The principles are old ones: collect and send only what is necessary (the EU's GDPR, Article 5, asks that personal data be "adequate, relevant and limited to what is necessary"), keep it no longer than needed, and know what each provider does with it. Concretely: redact or avoid personal data before it leaves your system, read each provider's terms on retention and on use for training, treat prompt logs as personal data in your own storage, and respect where the data may lawfully be processed. Laws differ by country, and the details are for your lawyer, not this page.
What a graduate just sees
- The model is one component. Most of the quality of an AI feature comes from what surrounds it: the context it is given, the checks on what it returns and the tests that guard changes.
- A model's reply is untrusted input. Validate its shape in code; a valid reply can still be wrong.
- Retrieval is a ranking problem. If the right passage does not reach the prompt, nothing downstream can save the answer, and chunk size and ordering are design choices to test.
- Agents are loops. Every extra step costs tokens and risks error, and the model asks while your code acts.
- Small test sets give noisy comparisons, and a judge model is a measuring instrument with its own error.
- Input tokens drive cost and output tokens drive latency, so measure both before choosing a model.
- You cannot filter your way out of prompt injection. Limit what a fooled model can reach: private data, untrusted input and a way out should not meet.
- The airline was held responsible for its chatbot. Whoever deploys the system is answerable for what it says.
Where it shows up next
- AI Safety, Alignment & Governance (36/79): the evaluations, misuse limits and laws such as the EU AI Act, which treat the application, not just the model, as the thing to assess.
- Data Ethics, Privacy & Fairness (36/37): what data you may use, how bias reaches users through prompts and retrieved documents, and who is harmed when an answer is wrong.
- Shipping Software (36/76): evaluation sets as pipeline tests, staged rollouts for prompt and model changes, observability of every call.
- In the world: customer-support assistants, coding assistants, document search for companies and governments, and the AI engineer, a job title that barely existed a few years ago.
If you want to go deeper
- 📘 Textbook. Chip Huyen, AI Engineering: Building Applications with Foundation Models (O'Reilly, 2025), the broadest single book on this subject; and Dan Jurafsky and James Martin, Speech and Language Processing (3rd edition draft, free at web.stanford.edu/~jurafsky/slp3), whose Chapter 11 covers information retrieval and RAG.
- 📘 Gentler. Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models (O'Reilly, 2024), picture-led with code.
- 🌐 Free course. The Hugging Face LLM Course and its AI Agents Course, both free, code-first.
- 🌐 Security. The OWASP page on prompt injection, and Simon Willison's post on the lethal trifecta.
Building with a model is ordinary engineering around an unusual component. Specify what you want, ground it with retrieved facts, check every reply in code, keep loops short, measure with a test set big enough to mean something, watch the meter, and never let a single piece of hostile text reach private data and a way out.
Try this: next time an AI assistant answers you, ask what it was probably given besides your question, how anyone would know the answer was right, and what the worst thing is that it could do for you if it were wrong.
Sources. Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 February 2024 (summary: Wikipedia, "Moffatt v. Air Canada"); Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," NeurIPS 2020 (arXiv:2005.11401); Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," TACL 2023 (arXiv:2307.03172); Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models," ICLR 2023 (arXiv:2210.03629); Model Context Protocol, introduced 25 November 2024 and donated to the Linux Foundation's Agentic AI Foundation in December 2025 (Wikipedia, "Model Context Protocol"); Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," NeurIPS 2023 Datasets and Benchmarks (arXiv:2306.05685); Greshake et al., "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," 2023 (arXiv:2302.12173); Willison, "Prompt injection attacks against GPT-3" (12 September 2022) and "The lethal trifecta for AI agents" (16 June 2025); OWASP Top 10 for LLM Applications 2025, LLM01:2025 Prompt Injection; Regulation (EU) 2016/679 (GDPR), Article 5(1)(c) and (e); published API price lists of two providers, fetched November 2026, for the ratio of output to input price (the prices in the interactives are invented); structured-output modes described in provider documentation and engineering write-ups, 2024 onward. The Wilson interval is the standard formula; every number in the interactives was computed in code.
- 📘 Huyen, AI Engineering
- 🌐 Hugging Face LLM Course
- 🌐 OWASP: prompt injection