The Rise of AI Engineering

If one word captures post-2020 AI it is scale. Models behind products like ChatGPT, Gemini, and Midjourney are so large they consume a measurable slice of global electricity. Scale has two major consequences: models become capable of more tasks, and training them demands resources only a handful of organizations can afford. The second consequence gave rise to model as a service — powerful models exposed via APIs so anyone can build on top of them without investing in training infrastructure.

The combination of higher demand for AI applications and a lower barrier to building them turned AI engineering — building applications on top of readily available models — into one of the fastest-growing engineering disciplines. This chapter traces how we arrived here.

From Language Models to Large Language Models

Language Models

A language model encodes statistical information about one or more languages — in essence, how likely a word (or token) is to appear given a preceding context. The idea goes back to Claude Shannon's 1951 paper on predicting and measuring the entropy of English. The basic unit is the token, which can be a character, a whole word, or a sub-word fragment (e.g. -tion). GPT-4 breaks text into tokens at roughly ¾ words per token. A model's full set of tokens is its vocabulary (GPT-4 has ~100,000 tokens).

There are two fundamental types of language model:

Autoregressive models are often called generative because they can produce open-ended, infinite outputs from a finite vocabulary. Think of them as completion machines: given a prompt, they try to continue it. This framing is surprisingly powerful — translation, summarization, question-answering, and even spam detection can all be framed as completion tasks.

Self-Supervision: How Models Got Large

The critical breakthrough that allowed language models to scale into LLMs is self-supervision. In traditional supervised learning, every training example needs a human-provided label, which is expensive and slow. Self-supervision sidesteps this: the training labels are inferred from the data itself.

For a language model, each sentence is its own set of training examples. The sentence "I love street food." generates six training pairs: context <BOS> → label I, context <BOS> I → label love, and so on. Because text is everywhere — books, articles, blog posts, code repositories — it is possible to construct training datasets with trillions of tokens at essentially zero labeling cost.

A model's size is measured in parameters — variables updated during training. GPT-1 (2018) had 117M parameters; that was "large" at the time. GPT-2 (2019) had 1.5B, making GPT-1 look small. As of this writing, "large" means ~100B+ parameters. Larger models generally need more training data to fully exploit their capacity.

From LLMs to Foundation Models

Language models are limited to text, but humans perceive the world through vision, hearing, and more. Extending language models to handle multiple data modalities — images, video, audio, protein structures — creates foundation models. The word "foundation" reflects both their importance and the fact that they can be built upon for many downstream applications.

A multimodal model (or Large Multimodal Model, LMM) generates the next token conditioned on both text and image (or other) tokens. Self-supervision works for multimodal data too — OpenAI trained CLIP on 400 million (image, text) pairs scraped from the internet, 400× more than ImageNet, with no manual labeling.

Foundation models also mark the shift from task-specific to general-purpose models. A model trained for sentiment analysis previously could not do translation. Foundation models, by their scale and training approach, can do both out of the box — and can be further adapted for any specific task using one of three main techniques:

From Foundation Models to AI Engineering

AI engineering is the process of building applications on top of foundation models. Three converging factors created the conditions for its explosive growth:

The growth is staggering: within two years, four open-source AI engineering tools (AutoGPT, Stable Diffusion web UI, LangChain, Ollama) accumulated more GitHub stars than Bitcoin, on track to surpass React and Vue. A LinkedIn survey found that "Generative AI" and "Prompt Engineering" were growing in profile additions by 75% per month.

Foundation Model Use Cases

The chapter surveys applications across eight categories, drawing on interviews with 50 companies, 100+ case studies, and analysis of 205 open-source AI projects. A notable pattern: enterprises prefer deploying internal-facing applications first (knowledge management, internal search) before external-facing ones (customer chatbots) to manage risk, compliance, and privacy concerns.

Planning AI Applications

It is easy to build a compelling demo with foundation models. It is hard to build a profitable, production-ready product. Before building, it pays to think carefully about why you're building, what success looks like, and how the product will be maintained over time.

Use Case Evaluation

The motivation for building an AI application usually falls into one of three risk levels:

1. Existential threat — Competitors with AI can make your business obsolete. Industries most at risk: document processing, financial analysis, insurance, advertising, web design.
2. Missed opportunity — AI won't threaten your existence, but it can meaningfully boost profits and productivity (better copywrites, improved customer support, cheaper user acquisition, richer market research).
3. Strategic exploration — You don't yet know where AI fits, but you don't want to be caught flat-footed (as Kodak, Blockbuster, and BlackBerry were). Investing in AI R&D is reasonable if you can afford it.

Once you've identified the motivation, also ask: do I have to build this myself? If AI is an existential threat, in-house may be necessary. If it's a productivity tool, there may be off-the-shelf options that deliver better performance at lower cost.

The Role of AI and Humans in the Application

Apple's design framework identifies three key dimensions for understanding how AI fits in a product:

It's equally important to define human-in-the-loop design: does AI generate suggestions for human agents, handle only simple requests and route complex ones, or operate fully autonomously? Microsoft's Crawl-Walk-Run framework offers a graduated path:

1. Crawl — Human involvement is mandatory; AI supports humans.
2. Walk — AI can interact directly with internal employees.
3. Run — Increased automation, including direct AI interactions with external users.

AI Product Defensibility

The same low barrier that makes it easy to build also makes it easy for competitors to replicate. If something easy for you to build is equally easy for Google or Microsoft to build as a feature of their existing products, your product may be short-lived. There are three types of competitive advantages in AI:

• Technology — With foundation models, core technologies converge; this advantage is hard to maintain.
• Distribution — The ability to put your product in front of users at scale. Big companies dominate here.
• Data — Getting to market first and accumulating usage data creates a compounding moat. Even if you can't train directly on user data, behavioral insights guide product improvements. This is the most viable moat for startups.

Setting Expectations

Before building, define what success looks like in measurable terms. For a customer support chatbot, business metrics might include: what percentage of messages should be automated, how much quicker should responses be, and how much human labor is saved. Beyond business metrics, define a usefulness threshold — the minimum quality bar the product must clear before going in front of customers:

• Quality metrics — accuracy, relevance, safety of outputs
• Latency metrics — time to first token (TTFT), time per output token (TPOT), total latency
• Cost metrics — cost per inference request
• Other — interpretability, fairness, compliance

Milestone Planning: The Last-Mile Challenge

Initial results with foundation models can be misleading. Because base capabilities are already impressive, it's possible to build an exciting demo in a weekend. But demos don't equal products.

Evaluate off-the-shelf models first to understand their current capabilities. This reality check will likely revise your goals and resource estimates.

Maintenance: Riding the Bullet Train

AI is moving incredibly fast, and building on foundation models means committing to that pace. Changes are constant, and not all of them are convenient:

• Model improvements are generally welcome — but even beneficial changes (longer context windows, better outputs) require re-testing, re-prompting, and workflow updates.
• Pricing changes — a model you host in-house can suddenly become more expensive than an API that halved its price, or vice versa.
• Regulatory risk — AI compute and data are treated as national security assets in many countries. GDPR compliance cost businesses $9B. The US October 2023 Executive Order on AI changed GPU export rules overnight.
• IP uncertainty — Questions about whether models trained on copyrighted data affect downstream product ownership are still unresolved. Many IP-heavy companies (game studios, publishers) are cautious for this reason.

The AI Engineering Stack

AI engineering evolved out of ML engineering, and the stack reflects this lineage. Rather than trying to track every new tool, it helps to understand the fundamental building blocks.

Three Layers of the AI Stack

Any AI application stack has three layers. When building an application, you typically start at the top and move down only as needed:

A GitHub survey of 920 AI repositories with 500+ stars (March 2024) confirmed this: after ChatGPT's launch, applications and application development tools grew fastest, while infrastructure grew more slowly. Core infrastructure needs (serving, monitoring, resource management) remain largely unchanged from classical ML.

AI Engineering vs. ML Engineering

AI engineering differs from traditional ML engineering in three fundamental ways:

In short: AI engineering is less about model development and more about model adaptation and evaluation.

Model Adaptation: Two Paths

Prompt-based techniques (prompt engineering, RAG) adapt a model without changing its weights. They are easier to start with, require less data, and let you experiment across many models. They may fall short for complex tasks or strict performance requirements.

Finetuning changes the model weights by continuing to train on domain-specific data. More complex, more data-intensive, but can achieve improvements in quality, latency, and cost that prompt engineering cannot — and enables behaviors the base model has never seen.

Model Development Layer

Modeling & Training

Encompasses model architecture design, training from scratch, and finetuning. Tools: TensorFlow, PyTorch, Hugging Face Transformers. With foundation models available, deep ML knowledge (gradient descent, loss functions, backprop) is no longer required to build AI apps — but it remains highly valuable for debugging and advanced adaptation.

Dataset Engineering

Curating, generating, and annotating training data. With foundation models, the focus shifts from feature engineering on tabular data (classical ML) to deduplication, tokenization, context retrieval, and quality control (removing toxic or sensitive content). Open-ended annotation is much harder than close-ended — writing an essay for a training pair is harder than labeling an email as spam. Data is widely seen as the primary differentiator now that model architectures are converging.

Inference Optimization

Making models faster and cheaper to run. Especially critical for autoregressive models, which generate tokens sequentially — at 10 ms per token, a 100-token response takes a full second. Getting AI apps below the ~100 ms internet-standard latency threshold is a significant engineering challenge. Techniques include quantization, distillation, and parallelism (covered in Chapters 7–9).

Application Development Layer

With foundation models, the core model is no longer a differentiator — everyone can access the same GPT-4 or Claude API. Differentiation now comes from how well you build the application. This layer has three responsibilities:

Evaluation

Evaluation is about mitigating risk and uncovering opportunities throughout the entire model adaptation process — from selecting a model, to benchmarking progress, to detecting production issues. Evaluation is harder with foundation models than in classical ML because:

• Open-ended outputs have no single correct answer; you can't enumerate all valid responses for a chatbot.
• Different adaptation techniques produce radically different performance. When Google launched Gemini, it claimed superiority over GPT-4 on MMLU using CoT@32 prompting (32 examples). When both were given 5 examples, GPT-4 performed better. The evaluation method changed the winner.

Prompt Engineering & Context Construction

Getting a model to express desired behavior from the input alone, without modifying weights. Prompt engineering is not just about writing instructions — it includes providing necessary context, examples (few-shot prompting), format specifications, and for complex multi-step tasks, a memory management system for the model to track history. RAG (retrieval-augmented generation) is a key context construction technique.

AI Interface

Creating interfaces for end users to interact with AI applications. Before foundation models, AI was embedded invisibly into existing products (fraud detection in Stripe, recommendations in Netflix). Now, AI can be a standalone product (ChatGPT, Perplexity) or a plug-in (Copilot in VSCode, Grammarly in Google Docs). Common interface types:

Web / Desktop / Mobile apps Browser extensions Chat platform integrations (Slack, Discord, WhatsApp) Product plug-ins (VSCode, Shopify, Microsoft 365) Voice interfaces AR / VR embodied agents

Conversational interfaces also change how user feedback is collected — feedback is richer and more natural but harder to extract than traditional click-through data.

AI Engineering vs. Full-Stack Engineering

The rising importance of application development and interfaces brings AI engineering closer to full-stack development. The ecosystem has broadened from Python-only (PyTorch, TensorFlow) to include JavaScript APIs (LangChain.js, Transformers.js, Vercel AI SDK), attracting frontend engineers.

Full-stack engineers bring a key advantage: they can quickly turn ideas into demos, get feedback, and iterate. The new AI engineering workflow rewards this speed. In classical ML engineering, you gathered data and trained a model first, building the product last. With foundation models, you can start with the product first and invest in training only once the product shows promise.

Chapter Summary

Chapter 1 sets the stage for everything that follows. It explains how decades of incremental progress in language modeling — self-supervised training at scale — suddenly unlocked a general-purpose AI capability layer that anyone can build on. Foundation models are not just better at existing tasks; they enable entirely new categories of application.

The use-case survey shows that the highest-value opportunities today cluster around productivity amplification (coding, writing, information synthesis) and automation (customer support, data extraction, workflow agents). But the chapter is equally honest about limits: hallucinations are real, evaluation is hard, and the gap between a polished demo and a production product can take months to close.

The AI engineering stack inherits solid principles from ML engineering — systematic experimentation, rigorous evaluation, feedback loops for continuous improvement — while introducing new emphases: model adaptation over model training, open-ended output evaluation, and interface design as a first-class concern. Understanding this stack is the foundation for everything the rest of the book covers.

Training Data

An AI model is only as good as the data it was trained on. If there is no Vietnamese text in the training data, a model won't translate into Vietnamese. The central challenge is that collecting sufficient high-quality data is expensive, so model developers often rely on whatever is available.

The dominant source is Common Crawl, a nonprofit-maintained web crawl of ~2–3 billion pages per month. Google's curated subset, C4 (Colossal Clean Crawled Corpus), is slightly cleaner but still problematic — Common Crawl contains clickbait, misinformation, propaganda, conspiracy theories, racism, and misogyny. The 1,000 most common websites in the dataset include outlets that rank low on NewsGuard's trustworthiness scale. Yet Common Crawl (or variants of it) is used in most foundation models that disclose their training data, including GPT-3 and Gemini.

Multilingual Models

English accounts for 45.9% of Common Crawl — eight times more than Russian (5.97%), the second most represented language. This imbalance has severe consequences:

• Performance gaps: On the MMLU benchmark (57 subjects, 14,000 questions), GPT-4 performs dramatically better in English than in Telugu or Marathi. For math problems in English, GPT-4 solved them 3× more often than in Armenian or Farsi — and failed entirely in Burmese and Amharic.
• Severe under-representation: Languages like Punjabi (1.41% of world speakers, only 0.006% of Common Crawl — a 231× gap), Swahili (115× gap), and Urdu (105× gap) are structurally disadvantaged.
• Tokenization cost: Inefficient tokenization for certain languages makes inference slower and more expensive. Hindi requires a median of 32 tokens for the same content that English conveys in 7. Burmese needs 72 — making it 10× slower and 10× more expensive than English for the same content when using per-token pricing.
• Unexpected safety failures: ChatGPT-3.5 refused to produce misinformation in English 6 out of 7 times, but complied all 7 times in simplified and traditional Chinese — suggesting alignment is unevenly applied across languages.

Domain-Specific Models

General-purpose models perform well on domains represented in their training data (coding, law, business), but poorly on specialized domains with data that is rare or unavailable publicly — such as drug discovery (protein/DNA/RNA sequences) or cancer screening (X-ray and fMRI scans). Building high-performing domain models often requires curating specialized datasets:

AlphaFold — DeepMind, trained on ~100,000 known protein structures
BioNeMo — NVIDIA, biomolecular data for drug discovery
Med-PaLM2 — Google, LLM combined with medical Q&A data
phi-1 — Microsoft, 1.3B parameters trained on 7B high-quality coding tokens, outperforms much larger general models on coding benchmarks

Model Architecture

Before training, developers must decide on the model's structure. Architecture choices affect not just capability but usability — a 7B-parameter model is far easier to deploy than a 175B one. The dominant architecture for language-based foundation models is the transformer, introduced by Vaswani et al. (2017).

The Transformer Architecture

The transformer was designed to solve the limitations of the then-dominant seq2seq architecture (2014), which used RNNs (recurrent neural networks). Seq2seq was adopted by Google Translate in 2016, prompting wide interest. But it had two fundamental problems:

• Information bottleneck: The decoder generated outputs conditioned on only the encoder's final hidden state — like writing a book summary using only the last sentence of the book.
• Sequential processing: RNNs process tokens one at a time, making them very slow for long sequences. For a 200-token input, each token must finish before the next begins.

The transformer addresses both with the attention mechanism. Input tokens are processed in parallel (not sequentially), and the decoder can attend to any input token when generating each output — like writing a summary by referencing any page in the book.

Transformer inference has two distinct phases:

The Attention Mechanism

The attention mechanism computes how much focus the model should place on each previous token when generating the next one. It uses three learned vector types:

• Query (Q): Represents what the decoder is currently "looking for" — the question being asked.
• Key (K): Represents each previous token's "address" — like a page number in a book. Used to determine relevance to the query.
• Value (V): Represents the actual content of each previous token — the page's substance.

Attention is computed as: softmax(QKᵀ / √d) × V. A high dot product between Q and a K means "pay a lot of attention to this token's value." Modern transformers use multi-headed attention — multiple attention heads allow the model to attend to different groups of previous tokens simultaneously (Llama 2-7B has 32 heads, each operating on 128-dimensional vectors).

Each transformer block contains two modules: an Attention module (4 weight matrices: Q, K, V, output projection) and an MLP module (feedforward layers with non-linear activations like ReLU or GELU). The model is also wrapped with an embedding module (before) and an output layer / unembedding layer (after), which maps to token probabilities.

Alternative Architectures

The transformer has dominated since 2017, outlasting seq2seq and GANs. But it has real limitations — especially around context length (quadratic scaling of attention with sequence length). Several challengers are gaining traction:

Model Size & Scaling

Model size is measured in parameters — variables updated during training. More parameters generally means greater capacity to learn. A 13B model in the same family will typically outperform a 7B model. A model's scale is fully described by three numbers:

Sparse models complicate the parameter count: a model with 90% zero-value parameters only uses 10% of its parameters effectively. Mixture-of-Experts (MoE) models exploit this — Mixtral 8x7B has 46.7B total parameters but activates only 12.9B per token, giving it the cost/speed profile of a ~13B model while retaining the capacity of a much larger one.

The Chinchilla Scaling Law

Given a fixed compute budget, what is the optimal ratio of model size to training data? DeepMind's 2022 "Chinchilla" paper answered this by training 400 models ranging from 70M to 16B parameters on 5B to 500B tokens. Their finding:

This revealed that most large models at the time (GPT-3, Gopher) were massively undertrained — they used too many parameters for the data they were given. Chinchilla itself (70B parameters, 1.4T tokens) outperformed the 280B-parameter Gopher on most tasks, despite being 4× smaller.

Importantly, the scaling law optimizes for quality — but production also cares about usability. Meta deliberately chose smaller Llama models that could run on consumer hardware, trading some quality for wide adoption and easier fine-tuning. Sardana et al. (2023) extended Chinchilla to account for inference demand, computing the optimal parameter count when the cost of running the model at scale matters.

Scaling Bottlenecks

Every order-of-magnitude increase in model size so far has improved performance. Three more orders of magnitude would yield 100-trillion-parameter models. Two bottlenecks make this increasingly difficult:

Post-Training

Pre-training produces a capable but difficult-to-use model: it's optimized for text completion (not conversation), and trained on indiscriminate internet data that can produce racist, sexist, or simply wrong outputs. Post-training addresses both issues. It typically uses only ~2% of the compute of pre-training (InstructGPT numbers) but dramatically changes the model's usability.

A useful analogy: pre-training is like reading to acquire knowledge; post-training is like learning how to use that knowledge. Post-training consists of two steps:

Supervised Finetuning (SFT)

Without SFT, a pre-trained model given "How to make pizza?" might respond by adding more context to the question or generating follow-up questions — it has no concept of conversation. SFT uses demonstration data — (prompt, response) pairs — to teach the model appropriate conversational behavior. This is sometimes called behavior cloning.

The data must cover the full range of tasks the model is expected to handle (question-answering, summarization, translation, etc.). High-quality demonstration data requires skilled labelers — among those who labeled for InstructGPT, ~90% had at least a college degree and one-third had a master's degree. Generating one (prompt, response) pair can take 30 minutes and cost ~$10. OpenAI used 13,000 pairs for InstructGPT (~$130,000 in labeling alone, before overhead).

Preference Finetuning (RLHF & DPO)

SFT teaches the model to converse, but not what kind of conversations it should have. Preference finetuning uses RLHF (Reinforcement Learning from Human Feedback) or DPO (Direct Preference Optimization) to steer the model away from harmful, biased, or wrong outputs.

RLHF has two stages:

1. Train a reward model: Instead of asking labelers to score responses (which is noisy), ask them to compare pairs of responses and choose the better one. This produces comparison data in format (prompt, winning_response, losing_response). The reward model is trained to predict these preferences — maximizing the score difference between winning and losing responses.
2. Optimize the SFT model using PPO: Use the reward model as a guide. The SFT model generates responses to random prompts; the reward model scores them; the model is updated via PPO (Proximal Policy Optimization) to generate higher-scoring responses.

Sampling

A model generates outputs through sampling. This is the process that makes AI probabilistic — and understanding it explains many behaviors that otherwise seem mysterious.

When generating the next token, the model outputs a logit vector — one logit per vocabulary item. Logits are converted to probabilities via softmax. The model then samples from this probability distribution. Always picking the highest probability token is called greedy sampling — it's boring and repetitive. Most production systems use richer strategies.

Temperature, Top-k, and Top-p

Test Time Compute

Rather than generating one output, generate multiple and pick the best. This is test time compute — trading inference cost for output quality. Strategies include:

• Best-of-N (random): Generate N outputs independently, select the one with the highest reward model score or highest average logprob. On average costs ~2× for N=2.
• Beam search: Instead of N independent outputs, maintain a "beam" of the most promising partial outputs at each generation step.
• Self-consistency: For tasks expecting exact answers (math, multiple-choice), pick the most common answer across N outputs. This is what Google did for Gemini's MMLU evaluation — sampling 32 outputs per question.
• Parallel generation: Generate multiple responses simultaneously; show the user the first valid/completed one (useful for reducing perceived latency on chain-of-thought queries).

Structured Outputs

Production systems often need model outputs in a specific format (JSON, YAML, SQL, regex). There are five approaches, from least to most powerful:

1. Prompting: Instruct the model to use a specific format. Simple but not guaranteed. Even a few percent of invalid outputs can be unacceptable.
2. AI validation: Use a second model call to validate/correct the output. Doubles cost and latency.
3. Post-processing: Write scripts to fix predictable formatting errors. LinkedIn's defensive YAML parser improved correct outputs from 90% to 99.99%. Only works if errors are consistent and fixable.
4. Constrained sampling: Filter the logit vector at each step to allow only tokens that satisfy the format grammar. Tools: guidance, outlines, llama.cpp. Powerful but requires format-specific grammars and can increase latency.
5. Finetuning: Train the model on examples of the target format. Most reliable and generalizable. Can be combined with a classifier head for guaranteed class-restricted output.

The Probabilistic Nature of AI

Sampling makes AI outputs probabilistic. Ask the same question twice: a human gives the same answer both times; an AI model may give different answers. If the model estimates Vietnamese cuisine has a 70% chance of being "best," it will say Vietnamese 70% of the time and Italian 30% of the time. This is fundamentally different from deterministic software — and it is both AI's greatest strength (creative tasks) and its most frustrating property (reliability tasks).

Inconsistency

Inconsistency appears in two forms: (1) same input, different outputs — the same prompt gives different answers on different runs; (2) slightly different input, drastically different outputs — accidentally capitalizing a word or adding punctuation can produce a completely different response.

Mitigations for same-input inconsistency:

• Cache responses so repeated identical queries return the same answer.
• Fix sampling variables (temperature = 0, fixed top-k/top-p).
• Fix the seed for the random number generator used in sampling.

Note: even with all variables fixed, hardware differences across machines can still produce slightly different outputs. If you use a model API, you may have limited control over this.

Hallucination

Hallucination is when a model produces a response not grounded in fact. The sampling process alone doesn't explain it — how does something never seen become a probable output? Two complementary hypotheses:

Partial mitigations: Adding "if you're unsure, say so" to prompts; requesting concise responses (fewer tokens = fewer opportunities to fabricate); RAG (grounding responses in retrieved documents, covered in Chapter 6); reinforcement learning techniques that differentiate user-provided context from model-generated tokens.

Chapter 2 Summary

Chapter 2 gives you the mental model needed to work effectively with foundation models. The key insight throughout is that the model's behavior is almost entirely determined by three interacting forces: the data it was trained on, the architecture that shaped how it processes information, and the sampling strategy that determines how it generates responses.

Training data explains why models underperform in non-English languages, why medical models need specialized datasets, and why "more data" isn't always the answer — data quality and diversity matter as much as quantity. The transformer architecture's attention mechanism explains both its power (any output can reference any input) and its limitations (quadratic scaling with context length). The Chinchilla scaling law demystifies the relationship between model size, data size, and compute budget.

Post-training bridges the gap between a capable but raw pre-trained model and a useful, safe assistant — but it is approximate and far from foolproof. Human preference is diverse and hard to capture in a mathematical formulation, and the RLHF process can make some failure modes (like hallucination) worse even as it improves others.

Finally, sampling makes AI probabilistic. This probabilistic nature is the root cause of both AI's creativity and its unreliability. The rest of the book explores how to build robust AI engineering workflows that account for — and systematically tame — this probabilistic nature.

Why Evaluating Foundation Models Is So Hard

Teams rushing to deploy AI applications quickly discover that evaluation is the hardest part. Greg Brockman (OpenAI) noted that "evals are surprisingly often all you need." Yet a 2023 a16z study found that 6 out of 70 decision makers evaluate models purely by word of mouth — what the author calls a "vibe check." This section explains why rigorous evaluation is so difficult.

Four Compounding Challenges

Language Modeling Metrics: Entropy, Cross Entropy, Perplexity

Many foundation models have a language model at their core, and language model performance on a dataset strongly correlates with downstream task performance. Understanding these metrics helps you interpret model reports and use them in evaluation and data-processing workflows.

Entropy

Entropy measures how much information, on average, a single token carries — equivalently, how difficult it is to predict the next token in a sequence. A two-token language (upper/lower) has entropy 1 bit. A four-token language (upper-left/upper-right/lower-left/lower-right) has entropy 2 bits. The more possible tokens and the more uniformly they are distributed, the higher the entropy. Shannon introduced this concept in 1951 to characterize English text.

Cross Entropy

When you train a language model, you are trying to get the model's learned distribution Q to approximate the true distribution P of the training data. The model's cross entropy on that data is:

Training minimizes cross entropy. A lower cross entropy means the model's distribution is closer to the true distribution of the training corpus.

Bits-per-Character and Bits-per-Byte

Since different models use different tokenization schemes, comparing raw cross-entropy values across models is misleading. Bits-per-character (BPC) normalizes by average characters per token. Bits-per-byte (BPB) further normalizes by character encoding scheme (ASCII vs UTF-8), making it the most portable metric. Equivalently, BPB tells you how efficiently the model can compress raw text — a model with BPB of 3.43 can represent original 8-bit bytes in 3.43 bits, compressing text by more than half.

Perplexity

Perplexity (PPL) is the exponential of cross entropy: PPL(P,Q) = e^H(P,Q). While cross entropy measures difficulty in bits, perplexity measures the number of equally likely choices the model faces at each token position. A PPL of 4 means the model is choosing among roughly 4 equally probable options. Low-entropy (structured) text like HTML gives lower perplexity than high-entropy prose.

Three Key Uses of Perplexity

Exact Evaluation: Functional Correctness

Exact evaluation produces unambiguous judgments — there is no subjectivity in the score. The two families are functional correctness and similarity measurements.

Functional correctness asks: does the generated output actually do what was asked? It is the ultimate metric, but also the hardest to automate. Code generation is the primary domain where automation is tractable.

pass@k for Code

The standard protocol for code benchmarks (HumanEval, MBPP, Spider, BIRD-SQL) works as follows: for each problem in the benchmark, generate k code samples. A problem is "solved" if any of those k samples passes all test cases. The final score — pass@k — is the fraction of problems solved. By design, pass@1 ≤ pass@3 ≤ pass@10, since more samples give the model more chances to stumble on a correct solution.

Functional correctness is not limited to code. Game-playing bots (Tetris score), scheduling optimizers (energy saved), and any task with measurable objectives can all be evaluated this way. The challenge is that for complex end-to-end tasks, evaluating intermediate steps is often harder than evaluating the final outcome.

Exact Evaluation: Similarity Measurements

For tasks that can't be evaluated with functional correctness, comparing generated outputs to reference responses (also called ground truths or canonical responses) is the next option. Reference data is typically human-generated, though AI-generated references with human review are increasingly common.

Exact Match

The simplest case: the generated response must match one of the references exactly. Works for short, unambiguous answers ("What's 2+3?" → "5") and trivia queries. Fails on any open-ended response — "How are you?" and "How is it going?" are semantically equivalent but an exact match algorithm counts one as wrong.

Lexical Similarity

Lexical similarity measures token overlap without caring about meaning. Two main families:

Fuzzy matching (approximate string matching) counts the minimum edit distance — insertions, deletions, substitutions — needed to transform one string into another. "bad" → "bard" is 1 edit; "bad" → "cash" is 3 edits.

N-gram similarity measures the overlap of n-token sequences. BLEU counts the fraction of n-grams in the generated text that appear in any reference. ROUGE counts the fraction of reference n-grams that appear in the generated text. These metrics dominate NLP benchmarks like WMT (translation), COCO Captions (image captioning), and GEMv2.

Semantic Similarity

Semantic similarity measures closeness in meaning, not surface form. "What's up?" and "How are you?" are lexically dissimilar but semantically close. The approach requires converting each text to an embedding and then computing cosine similarity between the embeddings. BERTScore uses BERT embeddings; MoverScore uses a mixture of algorithms. Semantic similarity is more robust than lexical similarity but depends on the quality of the underlying embedding model.

Introduction to Embedding

An embedding is a dense numerical vector that represents the meaning of a piece of data. The sentence "the cat sits on a mat" might be encoded as [0.11, 0.02, 0.54, …]. In practice, embedding vectors have 100–10,000 dimensions. The key property is that similar inputs should have similar (close) embeddings.

Cosine Similarity

Given embeddings A and B, cosine similarity = (A · B) / (‖A‖ · ‖B‖). The value ranges from −1 (opposite) to +1 (identical). Cosine similarity is preferred over Euclidean distance because it is invariant to vector magnitude — it measures angle, not distance.

Joint (Multimodal) Embeddings

A new frontier is embedding data from different modalities into a shared space. CLIP (Radford et al., 2021) was the first major model to unify image and text embeddings: given (image, caption) pairs, CLIP trains a text encoder and an image encoder such that each image's embedding is close to its caption's embedding. This enables text-based image search and zero-shot image classification. ULIP extends this to 3D point clouds; ImageBind supports six modalities including audio.

AI as a Judge

The challenges of evaluating open-ended responses drove many teams toward human evaluation. But human evaluation is slow and expensive. The natural next question: can AI automate evaluation? The approach — AI as a judge (or LLM as a judge) — is now the dominant method for production evaluation. LangChain's 2023 State of AI report found that 58% of evaluations on their platform used AI judges.

Why AI Judges Work

AI judges are fast, cheap relative to humans, and work without reference data (making them usable in production where references are unavailable). They can evaluate any criterion you can describe in a prompt: correctness, repetitiveness, toxicity, factual consistency, harmlessness, and more. Studies show strong agreement with humans: GPT-4 achieved 85% agreement with human evaluators on MT-Bench, higher than the 81% inter-human agreement. AlpacaEval AI judges showed 0.98 correlation with the human-evaluated LMSYS Chatbot Arena leaderboard.

Three Evaluation Approaches

Prompting an AI Judge Effectively

A strong judge prompt should specify: (1) the task the judge must perform; (2) the exact criteria to evaluate against; and (3) the scoring system. Prefer text-based scoring over raw numbers — models handle classification better. Discrete 1–5 scales outperform continuous 0–1 ranges. Including worked examples (what a 1, 3, and 5 look like and why) significantly improves reliability.

Limitations of AI as a Judge

Three Biases to Watch

Self-bias: a model favors its own outputs. GPT-4 gives itself a 10% higher win rate; Claude-v1 gives itself a 25% higher win rate in pairwise comparisons.

Position bias: many AI judges favor the first answer in a pairwise comparison (opposite of humans, who tend to favor the last — recency bias). Mitigate by running comparisons in both orderings.

Verbosity bias: judges tend to favor longer answers regardless of quality. GPT-4 and Claude-1 preferred ~100-word responses with factual errors over ~50-word correct responses. GPT-4 is less susceptible than GPT-3.5, suggesting this may improve with model strength.

Types of Specialized AI Judges

Rather than using a large general-purpose model as judge, you can train or use small, specialized judges for specific tasks:

Small, specialized judges can outperform large general-purpose judges on their specific task because they are trained on the exact criteria and scoring system you care about. Preference models are especially valuable: they can generate the preference data needed for RLHF alignment without expensive human annotation.

Ranking Models with Comparative Evaluation

Often you don't need an absolute score — you just need to know which model is better. Comparative evaluation skips absolute scoring entirely: instead of rating each model independently (pointwise), you show evaluators two responses side by side and ask which they prefer, then derive a ranking from the comparison results.

How It Works: Chatbot Arena

LMSYS's Chatbot Arena (2023) crowdsources pairwise comparisons from the public. A user submits a prompt, receives two anonymous responses from two randomly-selected models, votes for the better one, and only then sees which models generated which responses. This design makes gaming the leaderboard difficult. In January 2024, 244,000 comparisons across 57 models had been collected.

From match outcomes, a rating algorithm computes a score for each model. Chatbot Arena originally used Elo (popularized by chess) and later switched to the Bradley–Terry algorithm because Elo is sensitive to the order in which matches are processed. The resulting scores predict: if model A ranks higher than model B, A should win more than 50% of pairwise comparisons.

Three Challenges of Comparative Evaluation

Why Comparative Evaluation Still Has a Future

Despite the challenges, comparative evaluation has strong arguments in its favor. As models surpass human capability on specific tasks, humans may be unable to give absolute scores — but may still detect differences between two outputs. Comparative evaluation never "saturates" (unlike benchmarks with ceiling effects) as long as newer, stronger models keep arriving. And it directly captures human preference, the quality users ultimately care about.

Chapter 3 Summary

Evaluation is uniquely difficult for foundation models because of four compounding challenges: the sophistication required to judge outputs, the open-ended nature of responses, black-box model internals, and benchmark saturation. Despite growing investment in evaluation tooling, it remains underserved relative to model development.

Language modeling metrics — cross entropy, perplexity, BPC, and BPB — measure how well a model predicts text. Perplexity has three non-obvious practical uses beyond guiding training: as a proxy for downstream capability, as a contamination detector, and as an anomaly detector. These metrics are less useful for post-trained models, where the entropy-compression relationship breaks down.

Exact evaluation includes functional correctness (best automated for code via pass@k) and similarity measurements (exact match → lexical/BLEU/ROUGE → semantic/embedding-based). Each is more powerful but more resource-intensive than the last.

Embeddings are dense vector representations of meaning, and cosine similarity between embeddings is the backbone of semantic evaluation, retrieval, and deduplication throughout the rest of the book. Joint embedding models like CLIP extend this concept across modalities.

AI as a judge is fast, cheap, and increasingly reliable — but must be used carefully. Its biases (self, position, verbosity) and its non-standardized criteria mean that AI judge scores are only interpretable in the context of the specific model and prompt used. Specialized judges (reward models, preference models) can outperform general-purpose judges for narrow tasks.

Comparative evaluation sidesteps absolute scoring and is harder to game than benchmark-based methods. It is the basis of leaderboards like Chatbot Arena but struggles with scalability and the inability to distinguish absolute quality levels.

Evaluation Criteria and Evaluation-Driven Development

Before investing time and money in building an application, understand how it will be measured. The author calls this evaluation-driven development — a mirror of test-driven development in software engineering. Define evaluation criteria before building.

At the highest level, evaluation criteria fall into four buckets — each addressed below.

Domain-Specific Capability

Domain-specific capabilities are constrained by the model's training data and architecture. If a model never saw Latin text during training, it cannot understand Latin. The evaluation approach is typically exact evaluation using domain-specific benchmarks.

For coding: functional correctness (pass@k). For math and science: multiple-choice questions (MCQs), where accuracy against the correct option is the metric. In April 2024, 75% of tasks in Eleuther's lm-evaluation-harness were multiple-choice, including MMLU, AGIEval, and ARC-C.

Generation Capability: Factual Consistency

The most pressing generation quality issue is factual inconsistency (hallucination). Two evaluation settings:

Three escalating techniques for detecting factual inconsistency:

Factual consistency can also be framed as textual entailment (NLI — natural language inference): given a premise (context) and hypothesis (output), classify the relationship as entailment, contradiction, or neutral. Small specialized models like DeBERTa-v3-base-mnli-fever-anli (184M parameters, trained on 764K examples) perform this classification efficiently.

Generation Capability: Safety

Safety is an umbrella for all harmful output categories: inappropriate language, harmful tutorials, hate speech, threats, stereotypes, and political/religious bias. Studies show models carry measurable political leanings (GPT-4 is more left-libertarian; Llama is more authoritarian, per Feng et al., 2023).

Options for safety evaluation range from general-purpose AI judges to specialized lightweight classifiers (Facebook's hate speech model, Skolkovo Institute's toxicity classifier, Perspective API, and language-specific models). Benchmarks include RealToxicityPrompts (100,000 prompts likely to elicit toxic outputs) and BOLD.

Instruction-Following Capability

Instruction-following measures whether the model produces outputs that conform to the format, style, and content constraints you specify — independently of whether the model has the underlying domain knowledge. A model can know sentiment analysis perfectly but still fail instruction-following if it outputs "HAPPY" instead of "POSITIVE/NEGATIVE/NEUTRAL."

Two key benchmarks:

IFEval (Google, Zhou et al., 2023) tests 25 automatically verifiable instruction types: keyword inclusion, forbidden words, length constraints (words, sentences, paragraphs), language, postscript, JSON format, bullet count, and more. Score = fraction of instructions correctly followed.

INFOBench (Qin et al., 2024) broadens the definition to include content constraints, linguistic style, and tone — things that can't be automatically verified. For each instruction, the authors construct a list of yes/no questions, which are answered by human or AI evaluators. GPT-4 proved to be a reliable, cost-effective evaluator for this task, more accurate than Amazon Mechanical Turk annotators.

Roleplaying Capability

Roleplaying — asking the model to assume a character or persona — is the 8th most common instruction type in LMSYS's one-million conversation dataset. It serves both entertainment (gaming NPCs, AI companions) and prompt engineering (e.g., "Act as a senior software engineer and review my code"). Evaluation is hard to automate: benchmarks include RoleLLM and CharacterEval, and AI judges with role-specific prompts are the dominant approach.

Cost and Latency

A high-quality model that is too slow or too expensive to run is not useful. Key latency metrics for foundation models: time to first token (critical for perceived responsiveness), time per output token, and total time per query. Cost is measured per token for APIs; for self-hosted models, it is engineering and compute.

At low usage, model APIs are usually cheaper. At high scale, self-hosted models amortize fixed compute costs and can become much cheaper per token. This crossover point makes the build-vs-buy question worth revisiting periodically as scale grows. The author recommends separating "must-have" from "nice-to-have" latency requirements — high latency is often annoying, rarely a dealbreaker.

Model Selection Workflow

Model selection is not a one-time decision. As you progress through prompt engineering → RAG → finetuning, you will revisit model selection at each stage. The general process has two goals: (1) find the best achievable performance, and (2) map models along the cost–performance axis to choose the best value.

Hard vs. Soft Attributes

Hard attributes are model properties you cannot or will not change: license type, training data transparency, model size constraints, whether the model runs on-device. Hard attribute filtering can dramatically reduce your candidate pool before any experimentation.

Soft attributes are improvable through adaptation: accuracy on your task, toxicity rate, factual consistency. Be realistic about how much these can be improved. The author has seen 20% accuracy jump to 70% by decomposing a task into two steps — but also spent weeks on models that never became usable.

Four-Step Workflow

Model API vs. Self-Hosting: Seven Axes

Before the open-source vs. proprietary debate, it is important to distinguish open weight (weights public, training data private) from open model (weights and training data public). As of writing, almost all "open source" models are open weight only.

Navigating Public Benchmarks

Thousands of benchmarks exist. Eleuther's lm-evaluation-harness supports over 400; OpenAI's evals has ~500; Google's BIG-bench alone has 214. Two fundamental questions for building a useful benchmark leaderboard: what to include, and how to aggregate.

Public Leaderboards and Their Limitations

Hugging Face's Open LLM Leaderboard (2023 version) averaged six benchmarks: ARC-C, MMLU, HellaSwag, TruthfulQA, WinoGrande, and GSM-8K. Stanford's HELM used ten benchmarks, with only MMLU and GSM-8K overlapping with Hugging Face. Different leaderboards reach different conclusions about model rankings — and neither explains their benchmark selection process with full rigor.

Key findings on Hugging Face's six benchmarks: WinoGrande, MMLU, and ARC-C are highly correlated (Pearson r ≥ 0.86), meaning two of the three are partly redundant. TruthfulQA is only moderately correlated with the others (r ≈ 0.48–0.55) — improving reasoning doesn't automatically improve truthfulness.

Data Contamination

Data contamination (also called data leakage or "training on the test set") happens when benchmark data appears in a model's training corpus. Models trained on internet-scraped data almost certainly absorbed publicly available benchmark questions before the benchmarks were intended to be used for evaluation.

Rylan Schaeffer demonstrated this satirically in 2023: a one-million-parameter model trained exclusively on benchmark data achieved near-perfect scores and outperformed much larger models on those benchmarks. GPT-3 analysis found 13 benchmarks with ≥40% overlap in training data.

Detection methods:

Designing Your Evaluation Pipeline

Public benchmarks filter out bad models. Your private evaluation pipeline finds the best model for your application. This section gives a four-step framework.

Step 1: Evaluate All Components in the System

Real AI applications are multi-step pipelines. Evaluate each component independently, not just the end-to-end output. Example: an application that (a) extracts text from a PDF, then (b) extracts the current employer from that text. If the final output is wrong, evaluating each step separately tells you exactly where the failure occurred.

For conversational applications, distinguish turn-based evaluation (quality of each individual response) from task-based evaluation (did the application ultimately accomplish the user's goal, and how many turns did it take?). Task-based evaluation is more important but harder to define — the boundaries between tasks in an ongoing conversation are ambiguous.

Step 2: Create an Evaluation Guideline

A clear evaluation guideline is the most important artifact in your pipeline. Define not just what good responses look like, but explicitly what bad responses look like. LinkedIn found that the first hurdle in deploying generative AI was simply defining what a "correct" response meant — for a Job Assessment application, a technically correct "You are a terrible fit" is still a bad response because it lacks actionable guidance.

For each criterion, choose a scoring system (binary 0/1, 1–5 scale, continuous 0–1) and create a rubric with concrete examples. Validate the rubric with human reviewers — if humans find it ambiguous, your AI judge will too. Tie evaluation metrics to business outcomes: "factual consistency of 80% enables us to automate 30% of customer support tickets."

Step 3: Define Evaluation Methods and Data

Match evaluation methods to criteria. Use small specialized classifiers for toxicity (faster, cheaper). Use semantic similarity for relevance. Use AI judges for factual consistency. Mix and match: a cheap classifier on 100% of data plus an expensive AI judge on a random 1% sample gives reasonable confidence at manageable cost.

For data, bootstrap your 100–1,000 evaluation examples to check stability: if different 100-example bootstraps give scores that vary wildly (e.g., 70% vs. 90%), your evaluation set is too small to be trustworthy. OpenAI's rule of thumb: detecting a 10% difference requires ~100 examples; detecting a 3% difference requires ~1,000 examples; 1% requires ~10,000.

Don't forget production data. During development, you have reference data. In production, you have actual users. Design your evaluation strategy to work in both modes — the transition from offline to online evaluation requires rethinking which metrics remain valid without references.

Step 4: Iterate on the Pipeline

Evaluation pipelines must evolve as user behavior and application requirements change. But frequent changes make metrics incomparable over time. Maintain a stable core of criteria and only add/remove when necessary. Rigorously version your evaluation configuration: which data, which rubric, which AI judge model and prompt, which sampling parameters were used in each evaluation run. Without this, a 2% score improvement might reflect a more lenient judge, not a better application.

Chapter 4 Summary

An unreliable evaluation pipeline is one of the biggest blockers to AI adoption. This chapter translates the toolkit from Chapter 3 into an operational workflow for model selection and application evaluation.

Evaluation criteria fall into four buckets — domain capability, generation quality, instruction-following, and cost/latency — each requiring different measurement techniques. The build-vs-buy decision (model API vs. self-hosted) is not a one-time choice but a recurring tradeoff across seven axes; the right answer changes as scale, privacy requirements, and performance needs evolve.

Public benchmarks are useful for narrowing the candidate model pool but cannot be trusted for final selection: they cover a narrow slice of capabilities, they use inconsistent selection criteria, and they are almost certainly contaminated for top models. The prevalence of data contamination — where benchmark questions appear verbatim in training data — is one reason why benchmark saturation happens so fast. Use public benchmarks to eliminate obviously bad models, then use your private evaluation pipeline to select the best.

A reliable private evaluation pipeline requires five things: component-level evaluation (not just end-to-end), a clear and validated scoring rubric, metrics tied to business outcomes, data slicing to detect Simpson's Paradox, and rigorous versioning so that score changes reflect application changes rather than pipeline changes.

Anatomy of a Prompt

A prompt typically contains three components: a task description (what you want the model to do and what role it should play), examples of how to do the task, and the concrete task itself (the specific question, text, or input). Not every prompt needs all three parts — a zero-shot prompt omits examples entirely — but more complete prompts generally produce more reliable outputs.

How much prompt engineering is needed depends on the model's robustness to perturbation. If a small change (e.g., "5" vs. "five", an extra newline, different capitalization) causes the model's output to change dramatically, more fiddling is required. Robustness correlates strongly with overall model capability: stronger models are more robust, which is why upgrading the underlying model often reduces prompt-engineering overhead.

Prompt Engineering Best Practices

The following techniques are distilled from prompt engineering guides published by Anthropic, OpenAI, Meta, and Google, as well as from teams that have successfully deployed generative AI applications at scale. Unlike early "prompt hacks" (write "Q:" not "Questions:"), these practices remain relevant as models improve.

Write Clear and Explicit Instructions

Clarity is the single most important attribute. Define the scoring system precisely (1–5 or 1–10?). Specify whether fractional scores are allowed. If a model produces preambles like "Based on this essay, I'd give it…", tell it explicitly not to. Ambiguity that a human might resolve with common sense often trips up models. As you experiment, update your prompt to handle each undesirable behavior you observe.

Assigning a persona helps the model adopt the right perspective. The same essay might score 2/5 from a default model but 4/5 from a model prompted to act as a first-grade teacher. Providing examples reduces ambiguity about the desired output format and tone. If you're worried about cost, use compact example formats — arrow-separated (e.g., pineapple pizza → edible) rather than verbose Input:/Output: blocks.

Output format specification is especially important for downstream applications. If you need JSON, specify the keys. Use end-of-input markers for structured outputs — without them, a model may continue appending to the input rather than generating the structured response. Markers should be strings unlikely to appear in normal inputs.

Provide Sufficient Context

Reference texts help models just as they help students. Including a paper in the context dramatically improves answers about that paper. Context also mitigates hallucination: without necessary information, a model falls back on internal knowledge, which may be unreliable. You can provide context directly or equip the model with tools to retrieve it (RAG, web search). The process of collecting the relevant context for a given query is called context construction.

When you need the model to use only provided context, clear instructions ("answer using only the provided context") plus quotes-from-source requirements help — but cannot be guaranteed through prompting alone. Finetuning on a private corpus provides stronger enforcement.

Defensive Prompt Engineering

Once deployed, an application can be used by malicious actors as well as legitimate users. Prompt attacks exploit the model's instruction-following capability — the same capability you rely on. There are three main attack classes:

Chapter 5 Summary

Prompt engineering is simultaneously the most accessible and most underestimated model adaptation technique. The first part of this chapter established the building blocks: the anatomy of a prompt (task description, examples, task), why in-context learning works, the mechanics of system and user prompts, and how context length limits are expanding faster than our ability to use them effectively.

Best practices distilled from leading AI providers converge on clarity, decomposition, and iteration: unambiguous instructions, concrete examples, explicit output formats, chain-of-thought to improve reasoning, self-critique, and systematic versioning of prompts as first-class artifacts. Prompt engineering tools can accelerate this process but introduce hidden API costs and template errors — treat their outputs with the same skepticism you'd apply to any generated code.

The second half of the chapter established the threat model for deployed applications. The same instruction-following capability that makes a model useful makes it exploitable. The three attack families — extraction, jailbreaking/injection (including automated PAIR attacks and the powerful indirect injection vector), and training data extraction (including the divergence attack) — require a layered defense: model-level instruction hierarchy, prompt-level restrictions, and system-level isolation and guardrails.

Retrieval-Augmented Generation (RAG)

Many real-world tasks require more background knowledge than fits in a context window — entire codebases, legal document repositories, medical literature, or a company's product catalog. RAG addresses this with a two-step process: (1) retrieve relevant documents from an external store, (2) generate a response conditioned on those documents. The quality of the whole system is gated primarily by the quality of the retriever.

A RAG system can be seen as a special case of an agent where the retriever is a tool. The distinction is that pure RAG uses retrieval as its only external action, while a general agent can also write data, execute code, and call arbitrary APIs.

Agents

An agent is a system characterized by three components: an environment (all information sources it can perceive), actions (everything it can do), and a planner (the AI model that decides the action sequence). RL agents learn their planner through reinforcement learning; FM agents use the foundation model itself as the planner. In practice, these two paradigms are converging.

Memory Systems

Both RAG and agents frequently encounter information volumes that exceed a model's context window. A memory system provides structured mechanisms for storing and retrieving information across the three tiers that parallel human memory:

Memory management determines what to add to and remove from short-term memory. Strategies:

FIFO (First In, First Out) — remove the oldest messages when context fills up. Simple, but dangerous: early messages often contain the most important information (task goals, user preferences). OpenAI defaults to this; LangChain supports N-last-message retention.

Summarization — compress older messages into a running summary, retaining key named entities. Bae et al. (2022) refine this with a classifier that determines, for each sentence in both the memory and the summary, which to include in the new memory.

Reflection-based management (Liu et al., 2023) — after each action, the agent reflects on new information and decides whether to insert it into memory, merge it with existing memory, or replace contradicted information. Handles evolving world states better than FIFO.

Chapter 6 Summary

This chapter covered the two most widely deployed patterns for extending a model beyond its training data and context window. RAG emerged first and has been rapidly adopted across both consumer and enterprise use cases. Its core insight is that retrieval quality determines generation quality: getting the right passages in context is more important than prompting the generator well. Term-based retrieval (BM25) provides a strong, fast baseline; embedding-based retrieval adds semantic understanding at the cost of requiring ANN infrastructure. Hybrid search + cross-encoder reranking combines both. Contextual retrieval — augmenting chunks with document-level context before indexing — addresses the most common retrieval failure mode.

Agents extend RAG by giving the model a broader action space: not just retrieval, but code execution, API calls, database writes, and more. The planner (foundation model) decomposes goals, selects tools, and executes plans that can be sequential, parallel, conditional, or looping. Reflection (ReAct's Thought-Act-Observation loop; Reflexion's evaluator-critic architecture) allows agents to catch and correct their own mistakes, dramatically improving success rates on complex tasks.

Agent failures are systematic: planning failures (wrong tool, wrong parameters, wrong goal), tool failures (bad output, absent tools, translation errors), and efficiency failures (too many steps, excessive cost). Evaluation requires dedicated planning datasets and per-failure-mode metrics. Security risks from indirect prompt injection — where malicious instructions hide in external content an agent retrieves — are especially serious for write-action-capable agents and should be addressed with the instruction hierarchy and system-level isolation.

Both patterns are prompt-based: they improve model output by changing inputs without touching model weights. The next chapter introduces model adaptation through finetuning — modifying the model itself to unlock capabilities that prompting and retrieval cannot reach.

When to Finetune

Finetuning requires significantly more resources than prompting — data annotation, ML engineering knowledge, compute, and ongoing maintenance. It is almost never the first thing you should try. The recommended progression is: (1) prompting with best practices, (2) adding more examples to the prompt, (3) adding RAG for information-based failures, (4) PEFT for behavioral failures, (5) full finetuning if resources allow, (6) combining RAG + finetuning.

Reasons to Finetune

The primary reason is improving quality — both general capability and task-specific performance. The most common use case is enforcing output format and style: JSON schemas, SQL dialects, domain-specific syntaxes, concise vs. verbose responses. A general model that handles standard SQL may fail on a company-specific dialect; finetuning on examples of that dialect fixes it.

Other strong reasons include bias mitigation (finetuning on counterfactual examples — female CEOs, nurses as men — can counteract biases in pre-training data), distillation (training a small model to imitate a larger one; Grammarly's Flan-T5 outperformed a GPT-3 variant on writing tasks despite being 60× smaller), and eliminating prompt overhead (finetuned models can work with much shorter prompts, since examples no longer need to be included in-context).

Reasons Not to Finetune

Finetuning one task can degrade performance on others — the "alignment tax." If an application spans diverse query types, you need to finetune on all of them simultaneously or accept degradation. Beyond data, finetuning demands ML training knowledge (optimizers, learning rates, overfitting/underfitting) and inference infrastructure. And as new base models are released at rapid pace, determining when a new base model outweighs your finetuned model requires ongoing evaluation.

A common pattern: someone insists prompting doesn't work and demands finetuning — but investigation reveals the prompt experiments were minimal and unsystematic. Systematic prompt engineering should always precede finetuning. The experimental pipeline you build for prompting (evaluation, versioning, annotation guidelines) is the same foundation you'll need for finetuning anyway.

Memory Bottlenecks

Foundation models are so large that memory is the limiting factor for both inference and training. Understanding memory math is essential for selecting hardware and understanding why finetuning techniques work.

Inference memory ≈ N × M × 1.2, where N is the parameter count and M is bytes per parameter. The 1.2× accounts for activations and KV cache (~20% of weights). A 13B model in FP16 (2 bytes/param): 13B × 2 × 1.2 = 31.2 GB.

Training memory = weights + activations + gradients + optimizer states. Gradients add one value per trainable parameter. The Adam optimizer adds two more (first and second moment), so each trainable parameter requires 3 additional values. For a 13B-parameter model with Adam in FP16: gradients + optimizer states alone = 13B × 3 × 2 = 78 GB, far exceeding most GPU capacity. Activations can be even larger than weights for long sequences; gradient checkpointing trades recomputation for memory by discarding and recomputing activations.

Numerical Representations

Floating point formats differ in bits allocated to range (exponent bits) and precision (significand bits). FP32 (4 bytes) is standard training precision. FP16 (2 bytes) is used for inference and low-precision training — but its range is limited (values above ~65K → infinity). BF16 (2 bytes) was designed by Google for TPUs: same total bits but more range bits and fewer precision bits than FP16. BF16 can represent large values FP16 cannot, making it safer for training, but less numerically precise. TF32 (NVIDIA, 19 bits) offers FP32-compatible range with FP16 precision.

Loading a model in the wrong format is a common silent failure. When Llama 2 shipped in BF16, many teams loaded it in FP16 and saw dramatically worse performance without understanding why.

Parameter-Efficient Finetuning (PEFT)

Full finetuning updates all model parameters — for a 7B model with Adam in FP16, that's ~56 GB for weights + gradients + optimizer states, exceeding most GPU memory. Partial finetuning freezes early layers, but is parameter-inefficient: you need to update ~25% of parameters to match full finetuning performance (Houlsby et al., 2019). PEFT achieves near full-finetuning performance with orders of magnitude fewer trainable parameters by inserting small additional modules into the frozen model.

Houlsby et al.'s original adapters inserted two modules into each transformer block, achieving within 0.4% of full finetuning performance with only 3% of trainable parameters — but adding inference latency due to the extra layers. PEFT techniques split into two families:

Model Merging

Model merging combines multiple models (often finetuned variants of the same base) into a single model, without requiring additional GPU-based training. It can improve performance, reduce memory footprint, and enable multi-task deployment. Many top models on Hugging Face's Open LLM Leaderboard are merged models.

Summing: Linear Combination and Task Arithmetic

The simplest approach: average the weights of multiple finetuned models. Conceptually, subtracting the base model from a finetuned model yields a task vector (also called delta parameters) that encodes the capability delta for that task. Task vectors support task arithmetic (Ilharco et al., 2022): adding task vectors combines capabilities; subtracting a task vector removes capabilities (e.g., removing facial recognition or pre-training biases). Model soups (Wortsman et al., 2022) showed averaging several finetuned checkpoints improves accuracy without extra inference cost.

Linear combination works best for models finetuned on the same base. For differently-sized models, projection into a common dimension space is needed. SLERP (Spherical Linear Interpolation) merges two models along the geodesic of a sphere rather than linearly — better at preserving the geometry of the parameter space. SLERP supports only two models at a time (can be applied sequentially for more).

TIES and DARE improve merging quality by first pruning redundant task vector parameters (the majority of parameter changes during finetuning are small and irrelevant to performance). Yadav et al. showed keeping only the top 20% of task vector parameters gives comparable performance to keeping all 100%. Fewer redundant parameters means less interference when multiple task vectors are merged.

Layer Stacking (Frankenmerging)

Take layers from different models and stack them. Creates novel architectures not achievable by weight averaging. Goliath-120B was frankenmerged from two Llama 2-70B variants. Layer stacking enables sparse upcycling (Komatsuzaki et al., 2022): create a mixture-of-experts (MoE) model by making multiple copies of certain layers from a dense pre-trained model and training a router on top — outperforming MoE models trained from scratch. Together AI used this to create Mixture-of-Agents comparable to GPT-4o. Depthwise scaling (SOLAR 10.7B) stacks two copies of a 7B model while summing 16 shared layers, producing a 48-layer 10.7B model ready for further finetuning.

Finetuning Tactics

Development Paths

Progression path: (1) test code with cheapest/fastest model, (2) test data quality with a middling model (if training loss doesn't go down with more data, the data is the problem), (3) push performance with the best model, (4) run all models to map the price/performance frontier.

Distillation path: (1) start small dataset + strongest model → best possible finetuned model, (2) use that model to generate more training data, (3) finetune a cheaper smaller model on this new dataset.

Frameworks and Hyperparameters

Finetuning APIs (OpenAI, cloud providers) are simplest but limit base model choice and tuning flexibility. Open frameworks (LLaMA-Factory, unsloth, PEFT, Axolotl, LitGPT) support a wide range of methods and models. For multi-machine training, distributed frameworks (DeepSpeed, PyTorch Distributed, ColossalAI) are needed.

Learning rate: typically 1e-7 to 1e-3; start from pre-training's final LR × {0.1–1}; use learning rate schedules (warm-up + decay). If loss fluctuates wildly → LR too high; if loss decreases slowly → LR too low.

Batch size: larger batches = more stable updates but more memory. With constrained memory, use gradient accumulation — accumulate gradients across several mini-batches before updating weights, effectively simulating larger batches.

Epochs: 1–2 for datasets of millions of examples; 4–10 for thousands. Monitor training vs. validation loss curves: both decreasing → need more data or epochs; training down but validation up → overfitting, reduce epochs or increase regularization.

Prompt loss weight: during instruction finetuning, training examples have both prompt and response tokens. Since at inference the model only generates responses, response tokens should drive the loss. Default prompt loss weight is ~10% — the model learns mostly from responses but slightly from prompts.

Chapter 7 Summary

Finetuning is the most technically demanding model adaptation technique, touching everything from transfer learning theory to GPU memory arithmetic. The chapter opened with the decision framework: finetuning is warranted when prompt engineering has been thoroughly tried, and the remaining failures are behavioral rather than informational. When the problem is missing or outdated facts, RAG is the right first move.

Memory is the central constraint. Full finetuning a 7B model requires ~56 GB — impractical for most practitioners. PEFT solves this by reducing trainable parameters by orders of magnitude. LoRA is the dominant PEFT technique: it decomposes weight updates into low-rank matrices, adds zero inference latency when merged, and enables modular multi-model serving. QLoRA extends LoRA to 4-bit base model weights, bringing 65B-scale finetuning to a single consumer GPU.

Model merging is an exciting complement to finetuning: combining finetuned models through linear combination, task arithmetic, SLERP, or layer stacking can create models that outperform each constituent. TIES and DARE pruning improve merge quality by removing redundant task-specific parameters before combining.

Data Quality, Coverage, and Quantity

Data Quality

High-quality data is defined by six characteristics: relevant (matches your task domain), aligned with task requirements (annotations reflect what good performance looks like, not just factual accuracy), consistent (two annotators give similar scores for similar examples — requires clear annotation guidelines), correctly formatted (no HTML tags, no trailing whitespace, right case and number formats), sufficiently unique (deduplication prevents skewed distributions and benchmark contamination), and compliant (no PII, no copyrighted material, consistent with relevant laws and policies).

Human-generated data is often more prone to errors than expected — the Llama 3 team found human annotations inconsistent for nuanced safety policies, leading them to develop AI-assisted annotation tools. The annotation guideline is often the hardest part: you must specify not just what a good response looks like but what distinguishes scores of 3 and 4, and how to handle correct-but-unhelpful responses.

Data Coverage (Diversity)

Training data should cover the full range of inputs your application will encounter: different instruction lengths, typo patterns, response formats (JSON, yes/no, open-ended), topics, programming languages, and conversation styles. Adding heterogeneous data sometimes hurts: "The Data Addition Dilemma" (Shen et al., 2024) showed that in some cases, adding more diverse data worsens performance. The right diversity depends on the application.

Llama 3's data mix across training phases is instructive: pre-training devotes ~42% to math and code (well above their share of internet content), while preference finetuning shifts to 82% general knowledge reflecting real user distribution. The code and math emphasis reflects a consistent finding: high-quality code and math data improves reasoning capabilities more per token than natural language text.

Data Quantity

The right quantity depends on finetuning technique (PEFT needs hundreds to thousands; full finetuning often needs hundreds of thousands or millions), task complexity, and how close the base model already is to the target behavior. With few examples (<100), more advanced models finetune better; with many examples (550K+), all models converge to similar performance.

Start with a small, well-crafted dataset (50 examples) to validate that finetuning improves the model at all. If no improvement appears at 50–100 examples, the problem is not data quantity — check hyperparameters, prompt format, and data quality first. Plot a scaling curve (performance at 25%, 50%, 100% of your data) to estimate returns from more collection. A steep slope suggests more data will help significantly; a plateau suggests diminishing returns.

Why Data Synthesis

Programmatic data generation addresses the five hardest data challenges: scale (when real-world data is scarce), coverage (generating targeted rare-event or adversarial examples), quality (AI can produce more consistent preference annotations than humans), privacy (synthetic medical records, synthetic financial claims), and distillation (training a small model on outputs from a large one). Nemotron-4 340B-Instruct used 98% synthetic data for its instruction and preference finetuning.

Model Distillation

Knowledge distillation (Hinton et al., 2015) trains a small student model to mimic a large teacher model. DistilBERT is 40% smaller than BERT, retains 97% of its language comprehension, and is 60% faster. Alpaca finetuned Llama-7B on 52K examples generated by GPT-3 175B (text-davinci-003), producing a model that behaves similarly at 4% the teacher's size.

The student can also exceed the teacher. Nemotron-4-340B-Instruct was trained using 98% synthetic data generated by Mixtral-8x7B — a 56B MoE model that is effectively smaller than the 340B student. The student outperformed the teacher on a variety of tasks. Key caveat: training indiscriminately on self-generated data degrades performance; verified, quality-filtered synthetic data is required.

Important note: many model licenses prohibit using their outputs to train competing models. Always check the teacher model's license before using it for distillation.

Data Processing

After collection, data must be inspected, deduplicated, cleaned, and formatted before training.

Inspect

Before processing, understand your data: plot distributions of token frequency, input/output lengths, topics, languages, scores. Identify outliers. If annotated, compute inter-annotator disagreement and resolve conflicts. Most importantly: read actual examples. Greg Brockman (OpenAI co-founder): "Manual inspection of data has probably the highest value-to-prestige ratio of any activity in machine learning." Spending 15 minutes staring at data typically uncovers insights that save hours of debugging.

Deduplicate

Duplicates skew data distributions and cause benchmark contamination. An Anthropic study showed repeating 0.1% of data 100× degrades an 800M parameter model to 400M parameter performance despite the remaining 90% being unique. Deduplication can operate at document, paragraph, sentence, or token level. Approaches: pairwise similarity (expensive for large datasets), hashing (MinHash, Bloom filter — group similar examples into buckets then compare within buckets), or dimensionality reduction + ANN search (same techniques as RAG retrieval, applied to data deduplication).

Clean and Filter

Remove extraneous formatting (HTML tags, Markdown artifacts) — Databricks found this improved model accuracy by 20% and reduced input token counts by 60%. Remove non-compliant content: PII, sensitive data, copyrighted material, toxic content. Filter low-quality examples using heuristics discovered through manual inspection (e.g., Kern et al. found annotations made in the second half of an annotation session are lower quality due to annotator fatigue). If data exceeds your compute budget, use active learning or importance sampling to prioritize the most valuable examples.

Format

Get data into the exact chat template the model expects (see Chapter 5). Wrong templates cause silent failures. For instruction finetuning, training data can be much simpler than prompting data: finetuned models learn the task behavior from training examples directly, so they don't need in-context examples in the prompt. A 3-shot prompt becomes 3 training examples, and the finetuned model operates with a much shorter prompt like just {INPUT} -->. After finetuning, inference prompts must exactly match the format of training examples — an extra space or missing arrow can degrade performance.

Chapter 8 Summary

Dataset engineering is the most labor-intensive part of AI development. The principles are simple — quality, coverage, quantity — but the execution involves countless decisions, iterations, and hard-won judgment calls. A small amount of high-quality, diverse data consistently outperforms a large amount of noisy data. Llama 3's performance is almost entirely explained by data improvements, not model architecture changes.

AI-powered data synthesis has made it possible to generate data at scales and with characteristics that manual annotation cannot match: self-play for agentic tasks, reverse instruction for high-quality long-form responses, code synthesis with unit test verification, and paraphrase/translation expansion. These techniques, combined with rigorous filtering and verification pipelines, produced 2.7M+ synthetic examples for Llama 3 alone. However, synthetic data is not a silver bullet — model collapse from recursive AI training, superficial imitation without reasoning, and obscured data lineage are real risks that require mixing synthetic with real data and maintaining careful provenance.

The processing pipeline — inspect, deduplicate, clean, format — is unglamorous but critical. Removing HTML tokens improved accuracy by 20%. Duplicate data equivalent to 0.1% of training examples (repeated 100×) halves model effective capacity. The chat template must be exact. Data toil cannot be eliminated; it can only be made more systematic.

Inference Overview and Performance Metrics

Computational Bottlenecks

Optimization begins by identifying the right bottleneck. Two categories matter for AI inference:

Prefill and Decode

Transformer LLM inference has two distinct phases. Prefill processes all input tokens in parallel — compute-bound — and populates the initial KV cache. Decode generates one token per forward pass, loading the full weight matrix each time — memory bandwidth-bound. Because they have different computational profiles, modern inference servers increasingly run them on separate machines (see "Decoupling Prefill and Decode").

Online vs. Batch APIs

Online APIs optimise for latency; requests are processed immediately. Batch APIs optimise for cost; requests may wait hours for processing at roughly 50% lower price (Google Gemini and OpenAI both offer this as of the writing date). Streaming mode returns each token as it is generated, reducing apparent TTFT but preventing pre-publication scoring of the response.

Latency: TTFT, TPOT, TBT/ITL

TTFT (time to first token) = duration of the prefill step; depends on input length. TPOT (time per output token) = decode step per token. Fast readers read at ~120 ms/token (6–8 tokens/s), so TPOT need not be much faster than that for streaming use cases. TBT/ITL (time between tokens / inter-token latency) are synonyms used by LinkedIn and NVIDIA respectively. Total latency = TTFT + TPOT × output-token count.

In agentic or CoT settings, TTFT from the model's perspective (first internal token) can differ greatly from time to publish — the first token the user actually sees after all invisible reasoning steps complete.

Throughput and Goodput

Throughput measures output tokens per second across all concurrent users. Input and output throughput should be counted separately because prefill and decode have different bottlenecks. Cost is directly proportional to throughput: if a system costs $2/hour and generates 100 tokens/s, output cost is ~$5.56 per million tokens.

Goodput (adapted from networking) measures requests per second that satisfy the SLO. A service completing 100 req/min but with only 30 meeting latency targets has a goodput of 30 req/min. Goodput prevents chasing raw throughput at the expense of user experience.

Utilisation: MFU and MBU

GPU utilisation from nvidia-smi reports the fraction of time the GPU is doing something, not how much of its compute capacity it is using. More useful metrics:

MFU (Model FLOP/s Utilisation) = observed tokens/s ÷ theoretical peak tokens/s at max FLOP/s. Training MFU above 50% is considered good; inference MFU is typically lower. PaLM-540B achieved 46.2% MFU on 6,144 TPU v4 chips — the state of the art at that time.

MBU (Model Bandwidth Utilisation) = (parameter count × bytes/param × tokens/s) ÷ peak memory bandwidth. Example: a 7B-param FP16 model at 100 tokens/s on an A100-80GB uses 7B × 2 × 100 = 700 GB/s → MBU = 700/2000 = 35%. MBU decreases as concurrent users increase, indicating a shift from bandwidth-bound to compute-bound.

AI Accelerators

CPUs have few powerful general-purpose cores; GPUs have thousands of smaller cores optimised for parallelism. Matrix multiplication — over 90% of neural-network FLOPs — is highly parallelisable, which is why GPUs dominate AI workloads. The revival of deep learning in 2012 was partly because AlexNet demonstrated GPU training, making large-scale experimentation accessible to PhD students with a few GPUs rather than thousands of CPUs.

AI accelerators beyond NVIDIA: AMD GPUs (ROCm), Google TPUs, Intel Habana Gaudi, Graphcore IPU, Groq LPU, Cerebras QPU. Inference accounts for up to 90% of deployed ML compute costs (Desislavov et al., 2023), so chips specialised for inference — Apple Neural Engine, AWS Inferentia, Meta MTIA, Google Edge TPU — optimise for lower precision and faster memory access rather than large capacity.

Key GPU Characteristics

Selecting accelerators: compute-bound workloads benefit from higher FLOP/s; memory bandwidth-bound workloads benefit from higher HBM bandwidth. Three questions: Can the hardware run the workload? How long does it take? What does it cost?

Model-Level Optimization

Model-level techniques modify the model itself — which can change its outputs — to reduce size or computational cost. Transformer LLMs face three main challenges: model size, autoregressive decoding, and the attention mechanism.

Model Compression

Quantization (covered in Chapter 7) and distillation (Chapter 8) are the two most widely deployed compression techniques. Pruning — either removing entire neurons/layers or setting the least-important weights to zero (creating sparsity) — is theoretically attractive. Frankle & Carlin (2019) showed that pruning can reduce non-zero parameter counts by over 90% without accuracy loss. In practice, pruning is less common: it requires understanding the model architecture, the gains are often smaller than quantization, and not all hardware exploits sparsity efficiently.

Overcoming Autoregressive Decoding

Output tokens cost 2–4× more than input tokens across APIs. Anyscale found one output token has the same latency impact as 100 input tokens. Several techniques attack this bottleneck:

Speculative Decoding

A small, fast draft model generates K candidate tokens. The target model verifies them in parallel (verification is parallelisable, like prefilling), accepts the longest valid prefix, then generates one bonus token. If all K drafts are accepted, K+1 tokens are produced in roughly one target-model call. Key insight: decoding is memory bandwidth-bound, meaning there are idle FLOPs available for free verification.

DeepMind's 4B draft model for Chinchilla-70B is 8× faster per token (1.8 ms vs. 14.1 ms) and cuts end-to-end latency by more than half. Acceptance rates are domain-dependent — code generation benefits most. The approach is implemented in vLLM, TensorRT-LLM, and llama.cpp in roughly 50 lines of PyTorch.

Inference with Reference

When the output is likely to overlap with the input (retrieval, code editing, multi-turn conversation), draft tokens are copied from the input rather than generated by a separate model. No extra model required. Yang et al. (2023) report 2× generation speedup in high-overlap use cases.

Parallel Decoding (Jacobi / Medusa)

Instead of strict left-to-right generation, these methods attempt to generate multiple future tokens simultaneously, then verify them. Lookahead decoding uses the Jacobi iterative method: K future tokens generated in parallel, any failures regenerated until all pass. Medusa (Cai et al., 2024) adds multiple small decoding heads — each trained to predict a token K steps ahead — and uses tree-based attention to verify and select the most promising token tree. NVIDIA reports 1.9× token throughput improvement on H200 GPUs with Medusa on Llama 3.1.

Attention Mechanism Optimization

The KV cache stores key and value vectors from all prior tokens to avoid recomputation. Without it, attention computation is O(n²) in sequence length. The KV cache itself grows linearly with sequence length and can be enormous: for a 500B+ model with batch size 512 and context 2048, the KV cache is ~3 TB — three times the model weights (Pope et al., 2022). Llama 2 13B with batch 32 and sequence 2048 requires 54 GB of KV cache alone.

Redesigning the Attention Mechanism

These changes require modifying the model architecture during training or finetuning. Local windowed attention (Beltagy et al., 2020) limits each token's attention to a fixed-size window, reducing KV cache by 10× for 10K-token sequences with a 1K window. Multi-query attention (Shazeer, 2019) shares a single set of K/V pairs across all query heads. Grouped-query attention (Ainslie et al., 2023) is a generalisation: heads are grouped and share K/V pairs within a group. Character.AI combines multi-query attention, interleaved local/global attention, and cross-layer attention (sharing K/V across adjacent layers) to reduce KV cache by over 20×, eliminating memory as a bottleneck for their typical 180-message conversation histories.

KV Cache Management

vLLM introduced PagedAttention (Kwon et al., 2023): the KV cache is divided into non-contiguous pages (like OS virtual memory), eliminating fragmentation and enabling flexible sharing. This was the primary driver of vLLM's rapid adoption. Other techniques: KV cache quantization (Hooper et al., 2024), adaptive compression (Ge et al., 2023), and selective KV cache (Liu et al., 2024).

Kernels and Compilers

A kernel is specialised code optimised for a specific hardware architecture — CUDA (NVIDIA), Triton (OpenAI), ROCm (AMD). Writing kernels was once esoteric, but growing demand has driven more engineers into this space. Four common kernel optimisation techniques:

• Vectorisation: process multiple contiguous data elements simultaneously to reduce I/O operations.
• Parallelisation: divide input into independent chunks processed on different cores.
• Loop tiling: reorder loop data access to match the memory hierarchy; CPU-optimal tiling differs from GPU-optimal tiling.
• Operator fusion: combine multiple passes over the same array into one to reduce reads/writes.

FlashAttention (Dao et al., 2022) fuses multiple transformer attention operators into a single, memory-efficient CUDA kernel, originally for A100. FlashAttention-3 (Shah et al., 2024) was re-engineered for H100. Kernels are chip-specific: each new architecture requires new kernels.

Compilers lower model code to hardware-compatible instructions, inserting kernels where possible. Key compilers: torch.compile (PyTorch), XLA/OpenXLA (TensorFlow/JAX), TensorRT (NVIDIA-specific), Apache TVM, MLIR. The PyTorch team improved Llama-7B throughput by ~10× on A100 through: torch.compile → INT8 quantization → INT4 quantization → speculative decoding.

Inference Service Optimization

Service-level techniques do not modify the model and therefore do not change output quality. They focus on resource management to maximise goodput given dynamic workloads.

Batching

Processing multiple requests together amortises the cost of loading model weights. The three batching strategies:

Decoupling Prefill and Decode

Prefill is compute-bound; decode is memory bandwidth-bound. Running both on the same GPU causes resource contention: a new arriving request introduces a compute-intensive prefill job that drains the GPU's ability to serve existing decode jobs. DistServe (Zhong et al., 2024) and Inference Without Interference (Hu et al., 2024) show that separating prefill and decode onto different machines significantly increases processed request volume while meeting latency SLOs. Communication overhead via NVLink is modest. Prefill:decode instance ratio depends on input length and latency priorities — 2:1 to 4:1 for long inputs prioritising TTFT; 1:1 to 1:2 for short inputs prioritising TPOT.

Prompt Caching

Many application prompts share a common prefix — the system prompt, a large document, or earlier conversation turns. A prompt cache (also called context cache or prefix cache) stores the KV state for these repeated segments so they are processed only once. Introduced by Gim et al. (November 2023), rapidly adopted by major providers.

Google Gemini charges 75% less for cached tokens (plus storage cost of $1.00/M tokens/hour). Anthropic claims up to 90% cost savings and 75% latency reduction. If your application makes 1 million API calls/day with a 1,000-token system prompt, caching eliminates ~1 billion redundant input tokens daily.

Parallelism Strategies

Replica parallelism: create multiple identical model replicas to handle more concurrent requests. Straightforward but requires chip allocation (a bin-packing problem with mixed model sizes). Each replica handles fewer requests, improving latency per request at the cost of more hardware.

Tensor parallelism (intra-operator): split individual tensor operations — such as matrix multiplication — across multiple devices. Reduces latency and enables serving models too large for one GPU. Extra communication overhead can partially offset the benefit.

Pipeline parallelism: assign different model layers to different machines; micro-batches flow through sequentially. Common in training; less favoured for latency-sensitive inference due to inter-stage communication overhead.

Context and sequence parallelism: split the input sequence itself (context parallelism) or the operators needed for the full sequence (sequence parallelism) across machines; developed specifically to handle long contexts.

AI Application Architecture

A practical architecture builds incrementally. Start with the simplest possible form and add components as real problems emerge.

Monitoring, Observability, and Orchestration

Monitoring vs. Observability

Monitoring tracks external outputs to detect failures. Observability assumes that a system's internal state can be inferred from external outputs — when something fails, you can diagnose it from logs and metrics without shipping new code. Three DevOps-derived quality metrics: MTTD (mean time to detection), MTTR (mean time to response), and CFR (change failure rate). A high CFR means bad changes should be caught earlier in evaluation before deployment.

Metrics

Design metrics around the failure modes you want to catch, not the other way around. Common categories:

• Format failures: invalid JSON, wrong schema — easy to detect and sometimes auto-fixable.
• Quality: factual consistency, conciseness, creativity — often computed with AI judges.
• Safety: toxicity rate, PII exposure, false refusal rate (over-restriction is also a failure).
• Cost & latency: tokens/request, TTFT, TPOT, total latency, cache hit rate.
• Conversational signals: early termination rate, average turns per conversation, output length distribution.

Break metrics down by user, release, prompt version, and time. Correlate against business north-star metrics (DAU, session duration, subscriptions).

Logs and Traces

Metrics aggregate events; logs are the append-only record of individual events. Logs help diagnose what happened after metrics signal something went wrong. Log everything: model name, prompt template, sampling settings, user query, final prompt, response, tool calls, tool outputs, intermediate outputs, timestamps. Use consistent tags and IDs. AI-powered log analysis and anomaly detection are common at scale.

Traces link related log events into a complete timeline of a single request — from query received to response returned, including retrieval steps, tool calls, latency per step, and cost. LangSmith is a common tracing tool for AI applications. If a query fails, a trace should pinpoint the exact step.

Drift Detection

Three sources of drift in AI applications:

AI Pipeline Orchestration

An orchestrator specifies how components fit together. Two phases: (1) components definition — register models, data sources, tools, evaluators; (2) chaining/pipelining — define the sequence of steps from query to response, with the orchestrator managing data flow between steps and notifying on errors. Popular options: LangChain, LlamaIndex, Flowise, Langflow, Haystack.

User Feedback

User feedback is proprietary data — the core of the data flywheel. A product that ships early collects data to continually improve models, compounding its advantage over competitors. User feedback can be used for evaluation (monitor application quality), development (train future models), and personalization (adapt to individual users). Collect it responsibly: respect privacy, obtain consent, explain how data is used.

Types of Conversational Feedback

The conversational interface enables richer feedback than a simple thumbs up/down. Explicit feedback (star ratings, thumbs up/down, downvotes) is sparse and biased toward unhappy users but easy to interpret. Implicit feedback is more abundant but noisier. Conversational feedback blends the two:

Natural Language Feedback

Signals extracted from the content of messages:

The FITS dataset (Xu et al., 2022) was automatically clustered into 8 natural language feedback types: (1) clarify demand again (26%), (2) complain model didn't answer / gave irrelevant info (16%), (3) point to specific search results (16%), (4) suggest using search results (15%), (5) factually incorrect or not grounded (11%), (6) not specific/accurate/complete (9%), (7) model was not confident / always hedged (4%), (8) repetitive or rude (1%).

Behavioural Feedback

Regeneration: requesting another response may indicate dissatisfaction or desire for comparison. Stronger signal in usage-based billing (where regenerating costs money) than subscriptions. Comparative A/B data from side-by-side regeneration is usable for preference finetuning. Conversation organisation: deleting = strong negative; renaming = the content was good but the auto-title was bad. Conversation length: context-dependent — long conversations are positive for AI companions, negative for customer support chatbots. Dialogue diversity: long conversations with low distinct-token count suggest the user is stuck in a loop.

Feedback Design

When to collect: at onboarding (calibration — make optional to reduce friction); when something bad happens (always give users a way to flag failures and continue their task); when the model has low confidence (show two summaries side-by-side and let the user pick — the choice becomes preference data). Apple's Human Interface Guidelines recommend against asking for positive feedback on good results, but many teams find positive signals reveal high-value features worth concentrating on.

How to collect: feedback should be seamless and non-intrusive. Good examples: Midjourney's upscale/vary/regenerate workflow generates implicit comparative signals with zero extra effort; GitHub Copilot's Tab-to-accept / continue-typing-to-reject flow makes feedback effortless and high-quality. Standalone chat applications (ChatGPT, Claude) struggle to know whether generated content was ultimately used — integration into workflow (like Gmail suggesting drafts) enables far richer implicit feedback. For deeper analysis, collect surrounding conversation context with explicit user consent.

Feedback Limitations