AI Engineering Guide

AI engineering is the discipline that makes AI features dependable: the practices, tooling and tests that turn a model call into a product component.

---

Contents

• About This Guide
• Chapter 1. What AI Engineering Is
• Chapter 2. Design
• Chapter 3. Evaluate
• Chapter 4. Ship
• Chapter 5. Observe
• Chapter 6. Improve
• Chapter 7. Operating Model and Governance
• Chapter 8. Putting It Together
• Appendix A. Production Readiness Checklist
• Appendix B. Mapping to Practices You May Already Run
• Appendix C. Example Platform Mappings
• Appendix D. Glossary
• Appendix E. Further Reading

---

About This Guide

A model call is easy to write and hard to trust. A prototype that answers ten demo questions well says little about how the feature will behave on the ten-thousandth user request, after a prompt edit, after the provider updates the model, or after the knowledge base changes. AI engineering closes that gap. It applies the habits of mature software engineering (version control, automated tests, staged releases, monitoring, incident response) to systems whose behavior is probabilistic and depends on data, context and third-party models.

This guide is organized around a five-stage loop. Each stage has its own chapter, and Chapters 7 and 8 show how to run the loop as a team and as an organization.

Stage	Question it answers	Core practice
Design	What exactly are we building, and what may change independently?	Prompts, context and model choice treated as versioned, tested code
Evaluate	Is this change better, and is it safe to release?	Evaluation sets and quality gates before every release
Ship	How does it reach users without taking risk with it?	A gateway and serving layer, so teams can swap models without rewrites
Observe	What is it doing now, in production?	Monitoring for quality, latency and cost
Improve	What do we fix next, and how do we know it worked?	Failures fed back into designs, datasets and gates

Who This Guide Is For

The guide is written for enginerring leaders, software engineers, AI/ML engineers, platform engineers, SREs, and technical product managers who build or operate features on top of foundation models. It assumes you are comfortable with CI/CD, service-oriented architectures and basic statistics. It does not assume deep knowledge of model training.

Using This Guide in a Large Engineering Organization

Organizations that already run mature engineering systems do not need a parallel process for AI. The guide is designed to plug into what exists:

• Every practice is expressed in terms of artifacts and checks that fit existing tooling: files in version control, tests in CI, flags in a configuration service, spans in a tracing system, SLOs in an on-call program.
• Appendix B maps each concept to the terms established teams already use (launch criteria, dark launch, experiment guardrails, error budgets, blameless postmortem), and adds a second table that shows the integration point in your current systems.
• Section 7.7 describes how to split ownership between a central platform team and product teams so that standards scale without becoming a bottleneck.
• Appendix C lists example building blocks on several cloud platforms and in portable open-source form, because many organizations run internal equivalents.

How to Use It

• Read Chapter 1 to align on vocabulary and principles.
• Read Chapters 2 to 6 in order if you are building a system, or jump to the stage where you feel the most pain.
• Use Chapter 7 to set up the team, risk tiers and review process, and to plan adoption across many teams.
• Work the exercises at the end of each chapter against a feature you own. They turn the guide into a team workshop.
• Use Chapter 8 and the appendices as a working reference: reference architecture, readiness checklist, and a table that maps this guide’s terms to practices you may already run.

Conventions

• The guide is vendor-neutral. Where a concrete service helps, Appendix C maps concepts to several cloud platforms and portable open-source options as examples.
• Code is illustrative Python and configuration. It shows the shape of a solution, not a library to copy.
• Numbers in worked examples are illustrative and are labelled as such.
• A running example, the Refund Policy Assistant, appears throughout. It answers questions about refund eligibility using the company’s policy documents.

---

Chapter 1. What AI Engineering Is

In this chapter: the definition and scope of AI engineering; how it differs from traditional software engineering, MLOps and SRE; and the seven principles that the rest of the guide applies.

1.1 A Working Definition

AI engineering is the practice of designing, evaluating, releasing, operating and improving software whose core behavior is produced by a model rather than written as explicit logic. The unit of work is the AI component: a bounded piece of functionality that combines a model, the instructions given to it, the context assembled for it, and the checks applied to its output.

The guide focuses on applications built on foundation models, such as large language models. The loop applies equally to classical machine learning models, with training pipelines replacing some of the prompt and context work.

1.2 Why It Is Different From Traditional Software

Aspect	Traditional software	AI feature
Behavior	Deterministic for a given input	Probabilistic; the same input can produce different outputs
Specification	Requirements map to code paths	Requirements are statistical: a quality level over a distribution of inputs
Testing	Assertions on exact outputs	Graded outputs over datasets, with confidence intervals
Sources of regression	Your code changes	Your code, prompts, context sources, retrieval index, model version, provider behavior and input drift
Failure mode	Errors, crashes, wrong status codes	Fluent, plausible and wrong answers that raise no exception
Cost model	Mostly fixed per request	Variable per request, driven by tokens and model choice
Dependencies	Libraries and services you can pin	Hosted models that can change or be retired on the provider’s schedule

The most important consequence is that a green build proves very little. Code can be unchanged while the system degrades, because a provider changes model behavior, a source document becomes stale, or user inputs shift. Evaluation and monitoring must therefore be part of the engineering system, not an afterthought.

1.3 How It Relates to Neighboring Disciplines

Discipline	Primary focus	Overlap with AI engineering
Software engineering	Correct, maintainable code	Versioning, review, testing, release practices
MLOps	Building, deploying and monitoring models, including models trained in-house	Data and training pipelines, evaluation, model registry, serving and drift monitoring
LLMOps or GenAIOps	Operating applications built on foundation models	Prompt management, evaluation, gateways, cost control
SRE	Reliability of production services	SLOs, error budgets, incident response, capacity planning
Data engineering	Pipelines and data quality	Ingestion, indexing, freshness and lineage of context data
Security and trust	Protecting systems and users	Prompt injection, data leakage, excessive agency, abuse

AI engineering is not a replacement for any of these. It is the integration layer that makes them work together for AI-powered features.

1.4 The AI Engineering Loop

The loop is continuous. A feature is never “finished”: models change, data changes, users change. The goal is to make each pass through the loop cheap, safe and evidence-based.

1.5 Seven Principles

1. Treat prompts, context assembly and model choice as code. They define behavior, so they are versioned, reviewed, tested and released like code.
2. No change ships without evidence. Every change to a behavior-defining artifact passes an evaluation gate.
3. Version what changes at different speeds independently. Prompts, models, retrieval indexes, tools, policies and datasets each have their own version and rollback path.
4. Design for failure. A model is a probabilistic, rate-limited, remote dependency. Plan for timeouts, refusals, malformed output and outages.
5. Make cost and latency quality dimensions. An answer that is correct but too slow or too expensive is a failed answer.
6. Observe quality, not only availability. A healthy service that gives wrong answers is an incident.
7. Keep a person accountable for every consequential decision. Autonomy is granted deliberately, in proportion to risk and evidence.

1.6 Chapter Summary

• AI engineering turns a model call into a dependable product component.
• Behavior depends on more than your code, so tests and monitoring must cover the whole system.
• The loop of Design, Evaluate, Ship, Observe and Improve, guided by seven principles, structures the rest of this guide.

Exercises

1. List three changes that could degrade the Refund Policy Assistant without any code change. For each, name the signal that would reveal it.
2. Map the five stages of the loop to the processes your organization already runs (launch review, experimentation, SLO review, postmortems). Where are the gaps?
3. Choose an AI feature you own. Which of the seven principles does it violate today, and what would it cost to fix the most important violation?

---

Chapter 2. Design

In this chapter: how to specify an AI feature, structure an AI component, version the artifacts that define its behavior, engineer its context, choose a model, and design for failure and safety.

2.1 Start With a Feature Specification

Before writing a prompt, write down what “good” means. A specification for an AI feature is short, structured and testable.

feature: refund-policy-assistant
task: Answer questions about refund eligibility using current policy documents.
users: Support agents (phase 1); customers (phase 2)
risk_tier: 2
autonomy: suggest-only        # a human reviews and sends every reply
success_criteria:
  answer_correctness: ">= 0.90"
  groundedness: ">= 0.95"      # every claim traceable to a cited passage
  p95_latency_seconds: "<= 6"
  cost_per_answer_usd: "<= 0.03"
failure_taxonomy:
  - wrong_policy_version
  - unsupported_claim
  - missed_exception
  - unsafe_disclosure
  - refusal_when_answerable
out_of_scope:
  - legal advice
  - executing payments

Four elements matter most:

• Success criteria are statistical and numeric. They become your quality gates in Chapter 3.
• Failure taxonomy names the ways the feature can go wrong. It shapes the evaluation set, the graders and the dashboards.
• Risk tier determines how much evidence, review and oversight are required (Chapter 7).
• Autonomy level states what the system may do without a human: suggest, act with approval, or act alone.

2.2 Anatomy of an AI Component

An AI component has six parts. Keeping them separate makes each one testable and replaceable.

Part	Responsibility	Typical form
Interface contract	Typed input and output, error semantics	Schema, function signature
Prompt template	Instructions, role, format, examples	Versioned template file
Context assembler	Gathers and orders the information the model sees	Retrieval, memory, tool results, user data
Model binding	Which model and which inference parameters	Config: provider and model identifier, version or deployment reference, temperature, token limits
Validators	Check and repair the output	Schema validation, policy checks, groundedness checks
Fallback	What happens when the above fails	Alternate model, cached answer, human handoff

from dataclasses import dataclass

@dataclass(frozen=True)
class ComponentSpec:
    name: str
    prompt_version: str        # e.g. "refund_answer@14"
    model: str                 # pinned model identifier
    temperature: float
    max_output_tokens: int
    output_schema: type        # validated before anything downstream sees it

def run(spec: ComponentSpec, request) -> "Result":
    context = assemble_context(request, budget_tokens=6000)
    prompt = render(spec.prompt_version, request=request, context=context)
    raw = gateway.invoke(spec.model, prompt,
                         temperature=spec.temperature,
                         max_tokens=spec.max_output_tokens)
    return validate_or_fallback(raw, spec.output_schema, request)

Two design rules follow. First, the model is behind an interface, so it can be swapped without touching callers. Second, nothing downstream consumes raw model output. It passes through validation first, because model output is untrusted input. If a repair or regeneration step is used, validate the repaired output again before passing it downstream.

2.3 Version Everything That Defines Behavior

Different artifacts change at different speeds and need different controls.

Artifact	Changes	Lives in	Rollback unit
Prompt templates	Often	Repository or prompt registry	Prompt version
Model binding	Occasionally	Config	Model and parameter set
Retrieval index and corpus	Continuously	Index store, with snapshots	Index snapshot
Tool definitions	Occasionally	Repository	Tool version
Guardrail policies	Occasionally	Policy repository	Policy pack version
Evaluation sets	Continuously	Dataset store	Dataset version
Thresholds and routing rules	Occasionally	Config	Config version

Combine the versions that make up a deployable behavior into a release bundle. The bundle is the unit you evaluate, promote and roll back. Pin an immutable model version or deployment revision where the provider supports it. If the provider exposes only a mutable model name, record that limitation and run scheduled evaluations to detect changes in behavior.

bundle: refund-assistant@2026.11.3
prompt: refund_answer@14
model: provider-model-id@pinned-version
retrieval_index: policies-index@2026-11-08
guardrails: policy-pack@7
tools:
  - crm.get_account@3
eval_set: refund-golden@9
config: thresholds@5

Every production request should record the bundle version, so any answer can be traced to the configuration that produced it. Where a provider does not expose an immutable model revision, record the model identifier and the provider metadata available at request time; do not imply that the model behavior is fully reproducible.

2.4 Repository Layout

A layout that mirrors the component anatomy keeps reviews focused.

2.5 Engineer the Context

For most applications, the information placed in the model’s context matters as much as the model itself. Treat context assembly as an engineered pipeline with its own tests.

• Prioritize. Order sources by value to the task (task instructions, retrieved evidence, conversation state, user profile) and cut from the bottom when the token budget is tight.
• Budget. Set explicit token budgets per section. Unbounded context raises cost and latency and can degrade quality.
• Retrieve deliberately. Choose chunking, embedding, filters and reranking based on measured retrieval quality (Chapter 3), not defaults. Re-check authorization at the retrieval boundary and preserve source identifiers so generated claims can be traced to their evidence.
• Separate trusted from untrusted content. Instructions come from you. Retrieved documents, user messages and tool results are data and can contain instructions that try to hijack the model. Mark them clearly, and never grant authority to text just because it appears in context.
• Enforce access control before retrieval. The model should only ever see content the requesting user is allowed to see.
• Be explicit about memory. Decide what persists across turns or sessions, for how long, and how users can inspect or delete it.

2.6 Choose the Model With Evidence

Select models by running your evaluation set, not by reading leaderboards alone.

Criterion	What to check
Quality	Score on your task, per slice, with confidence intervals
Latency	Time to first token and total time at realistic output lengths
Cost	Cost per successful outcome, including retries and long prompts
Context window	Fit for your largest realistic inputs
Output control	Support for structured outputs and tool calling
Data handling	Retention, training use, residency, contractual terms
Capacity	Rate limits and quota options for your traffic
Lifecycle	Version pinning, deprecation notices, retirement schedule

A practical procedure: shortlist three candidates, run the same evaluation set on each, plot quality against cost and latency, then pick a primary model and a fallback. Record the decision, the data behind it and the date, because you will revisit it when new models appear.

2.7 Design for Failure

Failure	Detection	Response
Provider timeout or transient error	Timeout, retryable error code	Retry with jitter within the request budget, then fall back to a second model
Rate limit	429 or quota signal	Queue or shed low-priority traffic, then fall back
Malformed output	Schema validation	One repair attempt, then fall back
Unsupported claim	Groundedness check	Remove the claim, regenerate, or hand off to a human
Guardrail trigger	Policy check	Safe refusal with a helpful message
Retrieval returns nothing relevant	Low relevance scores	Say that no answer was found; do not guess
Downstream tool failure	Tool error	Report partial results; never invent tool outputs

Decide the degraded mode in advance. For a support assistant, it might be “show the policy search results without a generated summary.”

2.8 Build Safety Into the Design

Threat model every AI component at design time. The recurring risks are:

• Prompt injection, direct from users or indirect through retrieved documents and tool results.
• Sensitive data disclosure through context, logs or generated text.
• Improper output handling, where model output is passed to a browser, shell, database or another service without validation.
• Excessive agency, where the model has more tools, permissions or autonomy than the task needs.

Controls that apply broadly: least-privilege tool access, allow-lists for tool arguments, output validation and encoding, redaction of personal data before it reaches the model or the logs, rate limits and abuse detection, and human approval for high-impact actions.

2.9 Chapter Summary

• Specify the feature with numeric success criteria, a failure taxonomy, a risk tier and an autonomy level.
• Structure each AI component into six parts behind a typed interface.
• Version prompts, models, indexes, tools, policies and datasets independently, and combine them into release bundles.
• Engineer context as a pipeline, choose models with your own evidence, and design degraded modes and safety controls up front.

Exercises

1. Write a feature specification in the format of Section 2.1 for an AI feature in your product: numeric success criteria, at least five failure types, a risk tier and an autonomy level.
2. Sketch the six parts of the component for that feature. Which part currently lives inside a single prompt string?
3. Write a release bundle manifest for it. Which artifacts can you not pin today, and what compensating control would you use?
4. For each of the four risks in Section 2.8, name one control you would apply to your feature.

---

Chapter 3. Evaluate

In this chapter: the evaluation pyramid, how to build and govern datasets, graders and metrics, statistical practice, quality gates, CI integration and online evaluation.

3.1 Evaluation Is the Specification and the Test Suite

In AI engineering, the evaluation set plays two roles. It is the executable specification of behavior you expect, and it is the regression suite that protects that behavior. A team without a good evaluation set can change prompts, models or retrieval and have no reliable way to know whether it made things better or worse.

3.2 The Evaluation Pyramid

Use several layers. Cheap, fast checks run often. Expensive, realistic checks run before release.

Layer	What it checks	Runs	Relative cost	Typical size
Component checks	Format validity, schema, deterministic rules, tool-call arguments	Every commit	Very low	Dozens of cases
Task evaluation	Quality on the core task against a golden set	Every pull request (smoke subset), nightly (full)	Low to medium	100 to several hundred cases
End-to-end scenarios	Multi-step flows including retrieval, tools and handoffs	Nightly and pre-release	Medium	Tens of scenarios
Safety and red-team	Injection, leakage, policy violations, abuse	Pre-release and on policy change	Medium	Hundreds of adversarial cases
Online evaluation	Real traffic quality and outcomes	Continuously	Varies	Sampled traffic

3.3 Build the Dataset

An evaluation dataset is a product. It needs a design, an owner and a version history.

Composition. Include four kinds of cases:

1. Typical cases that reflect real traffic.
2. Edge cases: unusual phrasing, long inputs, multiple languages, conflicting sources.
3. Adversarial cases: injection attempts, requests for out-of-scope or sensitive content.
4. Regression cases captured from past incidents and production failures.

Slices. Tag every case with slices such as customer segment, language, topic, risk level and difficulty. Averages hide failures, and per-slice results reveal them.

Record format. Keep cases in a plain, diffable format:

{
  "id": "refund-0142",
  "question": "Can an enterprise customer get a refund after 45 days?",
  "expected": {"answer_summary": "No, except under the annual-plan exception", "must_cite": ["policy-4.2"]},
  "slices": ["enterprise", "exception"],
  "risk": "high",
  "source": "support-ticket-2026-09"
}

Size. Start small and grow with evidence. Fifty to one hundred carefully curated cases are enough to begin exploratory evaluation and establish an initial regression suite; they are not automatically sufficient for high-confidence release decisions. Add cases from every incident and every disagreement between graders. Aim for several hundred cases for each critical slice before you rely on small differences.

Hygiene.

• Keep a development set for tuning prompts and a separate test set for gates. If you tune against the test set, your gate stops measuring generalization.
• Avoid contamination. Do not put gate cases in prompts, few-shot examples or fine-tuning data.
• Review synthetic data. Generated cases are useful for coverage, but have humans check labels before they count.
• Remove or protect personal data in datasets, and apply the same access control as production data.
• Version the dataset and record the version in every evaluation report.

3.4 Graders and Metrics

A grader turns an output into a score or decision. Use the cheapest grader that is reliable for the dimension you are measuring, and retain a human-review path for dimensions where automated grading is not sufficiently dependable.

Dimension	Example metrics	Typical grader
Correctness	Match to reference, rubric score	Deterministic where possible; model-based or human otherwise
Groundedness	Share of claims supported by cited passages	Model-based with human spot checks
Retrieval quality	Recall at k, precision at k, mean reciprocal rank	Deterministic against labeled relevant documents
Citation accuracy	Citations that actually support the claim	Deterministic plus model-based
Format validity	Schema pass rate	Deterministic
Refusal behavior	Correct refusals and unnecessary refusals	Deterministic labels plus model-based
Safety	Violation rate on adversarial set	Deterministic rules plus model-based plus human review
Latency	Time to first token, total time at p50, p95, p99	Measured
Cost	Tokens and cost per request	Measured
Tool use	Correct tool selected, valid arguments, task completed	Deterministic

Model-based graders (often called LLM-as-judge) scale well but need care:

• Write a rubric with anchored levels and prefer simple decisions (pass or fail, or a short ordinal scale) over fine-grained scores.
• Use a different model for judging than for generating when possible, and pin the judge model and prompt versions.
• Calibrate against humans. Have qualified reviewers label a representative sample, measure agreement with the grader, and revise the rubric until agreement is acceptable for your risk tier. Cohen’s kappa suits categorical judgments; choose a measure appropriate to the score type and account for class imbalance.
• Control known biases, such as preferring longer answers or the first option shown, by randomizing order and testing for length effects.
• Keep humans in the loop for ambiguous, high-risk and newly discovered failure types.

3.5 Statistical Practice

Model outputs vary, and datasets are samples. Treat results as estimates with uncertainty.

• Control variance. Set low temperature for evaluation where appropriate, and repeat runs when outputs vary. Report the spread. Low temperature can reduce variation, but it does not make a hosted model fully deterministic.
• Report confidence intervals. Bootstrap or analytic intervals quantify sampling uncertainty; they do not, by themselves, prove that a difference is practically important or eliminate grader bias.
• Compare paired. Evaluate the baseline and the candidate on the same cases and analyze the per-case differences.
• Mind the sample size. Consider a pass rate near 0.90 measured on 200 independent cases. The normal-approximation 95 percent half-width is 1.96 × √(0.9 × 0.1 / 200) ≈ 0.042, about four percentage points, so a two-point improvement is hard to distinguish from noise. For paired comparisons, analyze the per-case differences with a paired interval or test. Treat this calculation as a rough guide, not a universal sample-size rule.
• Look at slices. Check for regressions in important slices even when the overall number improves.

3.6 Quality Gates

A quality gate is an automated, versioned decision rule that decides whether a change may proceed.

Gate type	Example	Purpose
Absolute threshold	Correctness lower confidence bound at least 0.90	Meet the success criteria
No regression	Candidate not worse than baseline by more than a set margin	Protect existing behavior
Slice floor	No critical slice below 0.85	Prevent hidden failures
Hard fail	Zero observed safety violations on the critical adversarial set	Enforce non-negotiable constraints; evidence about tested cases, not proof of zero risk
Budget	p95 latency and cost per request within limits	Keep the feature viable

The following sketch shows gates as ordinary tests, so they run in the same CI system as the rest of your code.

import random
import pytest

from evals.loader import load_cases
from evals.graders import grade_correctness, passes_safety
from app.component import answer

CASES = load_cases("evals/datasets/refund-golden@9.jsonl")

def lower_bound(scores, alpha=0.05, resamples=2000, seed=7):
    """Approximate one-sided bootstrap lower bound on the mean score."""
    if not scores:
        raise ValueError("At least one score is required")
    if not 0 < alpha < 1 or resamples < 1:
        raise ValueError("alpha and resamples must be positive and valid")

    rng = random.Random(seed)
    means = sorted(
        sum(rng.choices(scores, k=len(scores))) / len(scores)
        for _ in range(resamples)
    )
    index = max(0, min(resamples - 1, int(alpha * resamples)))
    return means[index]

@pytest.fixture(scope="session")
def results():
    return [(case, answer(case.question)) for case in CASES]

def test_correctness_gate(results):
    scores = [grade_correctness(out, case.expected) for case, out in results]
    assert lower_bound(scores) >= 0.90, "Correctness gate failed: do not release"

@pytest.mark.parametrize("slice_name", ["enterprise", "exception", "multilingual"])
def test_slice_floor(results, slice_name):
    scores = [grade_correctness(out, c.expected)
              for c, out in results if slice_name in c.slices]
    assert scores, f"No cases for slice {slice_name}"
    assert sum(scores) / len(scores) >= 0.85, f"Slice floor failed: {slice_name}"

def test_no_critical_safety_violations(results):
    violations = [c.id for c, out in results
                  if c.risk == "high" and not passes_safety(out)]
    assert not violations, f"Safety violations: {violations}"

Treat gate thresholds as versioned configuration with an owner. Loosening a threshold is a reviewed change, not a quiet edit.

3.7 Integrate Evaluation With CI and CD

Stage	What runs	Time budget	Outcome
Pull request	Component checks plus a smoke subset of the golden set	Minutes	Block merge on failure
Nightly	Full golden set with repeats, slice reports, drift check on index	Hours	Open an issue on regression
Pre-release	Full suite, safety red-team, load test, cost projection	Hours	Gate promotion of the release bundle
Post-release	Shadow or canary comparison, online metrics	Continuous	Gate ramp-up; trigger rollback

Keep evaluation affordable: cache deterministic results, sample for fast loops, use cheaper judges for screening and reserve the strongest judge and human review for gates. Publish every evaluation report with the bundle version, dataset version, grader versions and run date.

3.8 Online Evaluation

Offline sets cannot cover everything real users do. Add online evaluation:

• Shadow mode runs the candidate on live traffic without showing its output, and compares it with the current system.
• Controlled experiments (A/B tests) measure outcome metrics such as task success, escalation rate and user feedback. Define guardrail metrics (safety, latency, cost) that must not regress, and decide in advance when to stop.
• Sampled review sends a stratified sample of production traces to model-based graders and human reviewers, which provides a continuous quality signal.
• User feedback (ratings, edits, escalations) is informative but biased. Use it to find cases to review, not as the only quality metric.

3.9 Red-Teaming

Maintain a structured adversarial set and extend it continuously. Cover direct and indirect prompt injection, attempts to extract system instructions or private data, tool misuse, jailbreak patterns, and harmful or off-policy requests relevant to your domain. Run the relevant adversarial suite before every release of a Tier 2 or Tier 3 feature, whenever guardrail policies change, and when a new attack pattern is published. Tier 1 features still need baseline safety checks proportionate to their use and exposure. A passing suite is evidence about the tested threat cases, not a guarantee that the system is secure against every attack.

3.10 Chapter Summary

• Evaluation sets are the specification and the regression suite.
• Layer cheap and expensive evaluations, and run them at the right points in CI and CD.
• Build datasets with typical, edge, adversarial and regression cases, tagged by slice, and keep development and test sets apart.
• Calibrate model-based graders against humans, report uncertainty, and encode decisions as versioned quality gates.
• Complement offline gates with shadow runs, experiments and sampled review in production.

Exercises

1. Draft 20 golden cases for your feature, covering typical, edge, adversarial and regression cases, and tag each with at least one slice.
2. A candidate raises the pass rate from 0.90 to 0.92 on 200 paired cases. What would you compute before accepting it? What else would you check?
3. Design a calibration study for a model-based grader: who labels, how many cases, which agreement measure, and what threshold you would accept for a Tier 2 feature.
4. Write the five gate types from Section 3.6 for your feature, with thresholds and owners.

---

Chapter 4. Ship

In this chapter: release bundles, the AI gateway and serving layer, rollout and rollback, resilience, latency, caching, capacity and cost controls.

4.1 Release the Bundle, Not the Pieces

Promote a release bundle (Section 2.3) through environments as one unit. Each promotion records the evaluation report that justified it. Because prompts, models, indexes and policies are versioned independently, you can often roll back one without the others, for example reverting to the previous prompt while keeping a new index.

4.2 The AI Gateway

The gateway is the serving layer between your applications and models. Centralizing model access gives teams a consistent control point and can let them swap models without rewriting applications. It does not remove the need to evaluate provider-specific behavior or differences in model capabilities.

Responsibility	What it does
Provider abstraction	One interface for multiple models and providers; model changes become configuration
Authentication and authorization	Identify callers, enforce which models and tools each team may use
Routing and cascading	Send each request to the right model by task, cost, latency or complexity
Retries and fallbacks	Handle timeouts and rate limits with a defined fallback chain
Guardrails	Apply input and output policy checks consistently
Caching	Reuse results where safe (Section 4.6)
Quotas and rate limits	Protect shared capacity and budgets
Logging with redaction	Record traces without leaking sensitive data
Cost attribution	Tag every call by team, feature, tenant and bundle version

Keep the gateway thin and policy-driven. Business logic belongs in the AI component. The gateway owns cross-cutting concerns, while application-level authorization and domain rules remain enforced by the systems that own them.

4.3 Serving Topologies

Option	Strengths	Trade-offs
Hosted model API	Fast to adopt, no capacity management	Less control over versions, data handling and cost at scale
Managed model platform	Enterprise controls, regional options, integrated guardrails	Provider-specific features can add lock-in
Self-hosted models	Control over data, latency and cost at high volume	You own capacity, scaling, upgrades and optimization

Many organizations use a mix. The gateway makes the mix manageable, because callers see one interface.

4.4 Rollout Strategy

Stage	User exposure	Purpose	Exit criteria
Offline gate	None	Prove the bundle meets thresholds	All gates pass
Shadow	None (output hidden)	Compare with current system on live traffic	No regression on quality, latency, cost
Internal	Employees	Catch usability and edge issues	Review of sampled traces
Canary	1 to 5 percent	Detect problems at small scale	Guardrail metrics stable
Ramp	25, 50, then 100 percent	Scale with monitoring	Metrics stable at each step
Hold-back	Small control group	Keep a baseline to measure long-term effect	Defined review cadence

Use feature flags for control. Define rollback triggers before you start, such as a safety violation, a quality score below threshold, or a cost spike, and rehearse the rollback.

4.5 Resilience Patterns

Treat the model as an unreliable remote dependency.

• Timeouts on every call, set from latency budgets.
• Retries with exponential backoff and jitter, only for transient failures and idempotent or otherwise safely repeatable operations. Do not blindly retry authorization failures, invalid requests or other permanent errors.
• Circuit breakers that stop calling a failing target and fail fast to a fallback.
• Bulkheads that isolate capacity per feature or tenant so one noisy workload cannot starve others.
• Fallback chains ordered by preference, ending in a degraded mode or human handoff. Each fallback must meet the same data-handling and authorization requirements as the primary path, and its quality limitations should be explicit.

def call_with_fallback(request, chain, timeout_s=8):
    last_error = None
    for target in chain:
        try:
            return gateway.invoke(target, request, timeout=timeout_s)
        except (TimeoutError, RateLimitError, ProviderError) as err:
            last_error = err
            metrics.increment("gateway.fallback", tags={"from": target.name})
    raise AllTargetsFailed() from last_error

4.6 Latency, Caching and Capacity

Latency.

• Stream tokens when the use case permits users to see partial output, and track time to first token separately from total time. For high-risk responses or outputs that must pass validation before display, buffer the response or stream only content that has passed the required checks.
• Run independent steps (such as multiple retrievals) in parallel.
• Reduce prompt size, and use provider prefix caching where available for stable prompt prefixes.
• Use smaller, faster models for sub-tasks such as classification or routing.

Caching.

• Exact-match caching is straightforward for repeated identical requests, but the cache key must include the relevant authorization and behavior context. Do not reuse a response across users or tenants when their permissions or data differ. Include the relevant release-bundle or artifact versions when a change could make a cached answer stale.
• Semantic caching returns answers for similar requests but can return a wrong answer for a subtly different question. Use it only where mistakes are cheap, scope it by tenant and authorization context, and tie cache invalidation to changes in policy, prompt, model and index versions.

Capacity.

• Know your tokens-per-minute and requests-per-minute limits and request quota increases early.
• Load-test with realistic input and output length distributions, not just request counts.
• Apply backpressure and priority queues so interactive traffic wins over batch work.

Cost controls. Set budgets per feature and tenant, cap output tokens, route simple requests to cheaper models, and alert on cost per successful outcome rather than on raw spend alone.

4.7 Launch Checklist

Before moving a feature beyond internal use, confirm each item.

• Release bundle manifest exists and every request logs the bundle version
• Quality gates passed with a current evaluation report
• Safety and red-team suite passed for the risk tier
• Fallback chain and degraded mode tested
• Rollback rehearsed for each artifact type
• Load test completed with realistic token distributions
• Quotas, budgets and cost alerts configured
• Dashboards, alerts and an on-call owner in place
• Data handling reviewed (retention, residency, redaction)
• System card and risk tier approved

4.8 Chapter Summary

• Promote and roll back release bundles, with independent rollback per artifact.
• Centralize model access in a gateway for abstraction, routing, guardrails, quotas and cost attribution.
• Roll out in stages with predefined rollback triggers, and build in timeouts, fallbacks and isolation.
• Manage latency, caching, capacity and cost as engineering concerns from the start.

Exercises

1. Define a fallback chain for your feature. Confirm that every fallback meets the data-handling and authorization requirements of the primary path.
2. Define rollback triggers for each stage of the rollout table in Section 4.4.
3. Design the cache key for an exact-match cache in a multi-tenant deployment. Which versions and which authorization attributes must it include?
4. Estimate peak tokens per minute for your feature from realistic input and output length distributions. Which quota do you hit first?

---

Chapter 5. Observe

In this chapter: what to measure in production, how to define service level objectives for quality, how to detect drift, how to control cost, and how to respond to AI-specific incidents.

5.1 Observe Quality, Not Only Uptime

Classic monitoring answers “is it up and fast?” AI monitoring must also answer “is it right, safe and affordable?” The three questions map to three signals: quality, latency and cost.

5.2 Trace Every Request

A trace connects the steps behind one answer. Instrument the pipeline with spans so you can reconstruct what happened.

Span	Key attributes
Request	Feature, bundle version, tenant, user segment, request identifier
Context assembly	Sources used, token counts per section, truncation applied
Retrieval	Index version, query, number of results, relevance scores
Model call	Model and version, parameters, input and output tokens, time to first token, total time, finish reason
Tool call	Tool, version, arguments (redacted), duration, status
Validation	Schema result, guardrail results, repairs
Response	Fallback used, final status, citations

Standard conventions such as the OpenTelemetry semantic conventions for generative AI help keep traces portable across tools, although they are still evolving, so check the current specification. Link traces to user feedback and evaluation results through shared identifiers.

5.3 Service Level Indicators and Objectives

Define SLIs for the dimensions users care about, then set SLOs and error budgets, exactly as you would for any service.

Dimension	Example SLI	Example SLO
Availability	Successful responses divided by all requests	99.9 percent over 30 days
Latency	p95 time to first token and p95 total time	Within the feature’s latency budget
Quality	Pass rate on sampled, graded production traces	At or above the gate threshold
Groundedness	Share of sampled answers with fully supported claims	At or above target
Safety	Guardrail violations per thousand requests	At or below target; zero for critical classes
Escalation	Share of conversations handed to humans	Within an expected range
Cost	Cost per successful outcome	At or below budget

Quality SLIs rely on continuous sampled evaluation: automatically grade a stratified sample of production traces and route uncertain ones to human review. Error budgets then let you balance release speed against reliability, as in conventional SRE practice.

5.4 Detect Drift

AI systems drift for reasons that do not appear in your commit history.

Drift type	Cause	Signal
Input drift	Users ask different things over time	Shifts in topic, language, length distributions
Output drift	Model or prompt behavior changes	Shifts in answer length, refusal rate, format failures
Provider drift	Hosted model updated behind a stable name	Quality changes with no deploy; pin versions and run scheduled evaluations
Retrieval drift	Corpus changes, index staleness	Lower relevance scores, falling citation accuracy, stale sources
Concept drift	The world or policy changes	Previously correct answers become wrong

Run the golden set on a schedule against production configuration, not only at release time. A sudden score change without a deploy is a signal to investigate provider, data, configuration or input-distribution drift; it is not proof of a single cause.

5.5 Make Cost Visible

• Record input and output tokens and price per call, attributed to feature, tenant, team and bundle version.
• Track cost per successful outcome, not only cost per request. Retries, long prompts and failures all raise it.
• Alert on anomalies: sudden growth in prompt length, retry storms, runaway agent loops.
• Review the top cost drivers monthly and feed them into the Improve stage.

5.6 Protect Privacy in Telemetry

Prompts and responses can contain personal and confidential data. Redact or hash sensitive fields before storage, apply the minimum retention that supports debugging and evaluation, restrict access by role, separate tenant data, and document what is logged. Sampling full payloads at a low rate while keeping metadata for all requests is a common compromise.

5.7 Alerting and Dashboards

Alert on symptoms that matter to users: error rate, latency over budget, quality SLI below threshold, safety violation, cost anomaly. Avoid paging on every internal metric. A good dashboard has one view per feature: traffic, latency, error and fallback rates, quality scores by slice, safety signals, cost per outcome, and the active bundle version with a marker for each release.

5.8 AI Incident Response

Incident type	Example	First mitigation
Quality regression	Accuracy drops after an index refresh	Roll back the index snapshot or prompt
Safety incident	Harmful or policy-violating output reaches users	Tighten guardrails, disable the affected capability, escalate
Data exposure	Sensitive data appears in output or logs	Stop the flow, contain, follow the privacy incident process
Cost spike	Retry loop multiplies token use	Rate limit, circuit break, fix the loop
Provider outage	Model endpoint errors	Fail over along the fallback chain

A reliable runbook follows these steps: detect, triage by severity, mitigate (roll back an artifact, switch route, degrade, or hand off to humans), diagnose with traces, fix, add the failing case to the evaluation set, and hold a blameless review. The step that adds the failure to the dataset makes recurrence less likely and gives the team evidence that the fix works, although the failure can still return in a different form.

5.9 Chapter Summary

• Trace each request across context, retrieval, model, tools and validation.
• Define SLIs and SLOs for quality, safety and cost, using continuous sampled evaluation.
• Watch for input, output, provider, retrieval and concept drift, and run scheduled evaluations.
• Attribute cost per outcome, protect privacy in telemetry, and use an incident process that feeds failures back into the dataset.

Exercises

1. Define SLIs and SLOs for quality, safety and cost. Specify the sampling rate and the stratification for graded production traces.
2. List the trace attributes you would need to diagnose an unsupported-claim failure. Which of them need redaction?
3. The quality SLI drops six points on a Tuesday with no deploy. List your hypotheses in the order you would test them, and name the data that separates them.

---

Chapter 6. Improve

In this chapter: how to turn production signals into prioritized fixes, which levers to pull first, how to migrate models safely, and how to manage the technical debt that AI systems accumulate.

6.1 The Improvement Pipeline

Improvement is a repeatable process.

1. Collect failures from monitoring, user feedback, sampled review and incidents.
2. Cluster them by symptom and by slice to find the most frequent and most costly patterns.
3. Label a representative set and name the root cause.
4. Fix with the cheapest effective lever.
5. Regress: add the cases to the evaluation set so the failure cannot return unnoticed.
6. Verify through the normal gates and rollout, then confirm the metric moved in production.

6.2 Root Cause Taxonomy

Root cause	Typical evidence
Retrieval miss	Relevant document exists but was not retrieved
Context overflow or truncation	Needed information was cut by the token budget
Prompt ambiguity	Different phrasings of the same request give inconsistent behavior
Stale or wrong source data	The model faithfully reproduced outdated content
Model limitation	Fails with perfect context and clear instructions
Tool failure	Wrong tool chosen, invalid arguments, or a tool error was ignored
Guardrail false positive	Valid requests blocked
Missing requirement	The behavior was never specified

Labeling the root cause avoids the common mistake of trying to fix every failure by editing the prompt.

6.3 Use the Cheapest Effective Lever First

Order	Lever	Typical effort	Try when
1	Clarify the product behavior or UX	Low	The failure is a missing or ambiguous requirement
2	Improve the prompt and examples	Low	Instructions are unclear or inconsistent
3	Improve context and retrieval	Medium	Evidence is missing, stale or poorly ranked
4	Add or fix tools and validators	Medium	The task needs exact data or checks
5	Change or route to a different model	Medium	A stronger or cheaper model scores better on your set
6	Fine-tune	High	You need consistent style, format or a smaller model for a narrow, stable task
7	Redesign the workflow	High	A single call cannot reliably do the task

Fine-tuning is worth considering when you have enough high-quality labeled examples, a stable task and a clear measured gap that prompting and retrieval cannot close. Retrieval is usually the better choice for knowledge that changes often.

6.4 Close the Feedback Loop Responsibly

Production data is the best source of evaluation cases, but it carries obligations.

• Get the required consent and legal basis before using user content to improve the system.
• Minimize and anonymize examples before they enter datasets.
• Treat feedback as a signal to review, not as ground truth. Users rate fluency as well as correctness, and they rarely rate at all.
• Track the flywheel: share of production failures converted into regression cases, and time from detection to gate.

6.5 Model Migration Playbook

Providers retire and replace models, and better models appear regularly. Treat migration as a routine, rehearsed procedure.

1. Review the provider’s deprecation schedule and set calendar owners.
2. Run the golden set and safety suite on the candidate model with the current prompt.
3. Adapt the prompt to the new model if needed, using the development set only.
4. Compare quality, slices, latency and cost against the baseline, with confidence intervals.
5. Run in shadow mode on live traffic.
6. Canary and ramp with rollback ready.
7. Retire the old model binding, and update the system card and decision record.

Because model access goes through the gateway and evaluation is automated, migration becomes a configuration and evidence exercise, not a rewrite.

6.6 Manage AI-Specific Technical Debt

Research on production machine learning systems has described several forms of debt that also apply to AI applications.

Debt	How it appears	Countermeasure
Entanglement	Changing one prompt, source or threshold changes everything else	Component boundaries, per-component evaluation, slice reports
Undeclared consumers	Other systems depend on outputs that were never contracted	Typed interfaces, versioned contracts, consumer registry
Data dependencies	Hidden reliance on unstable data sources	Lineage, freshness checks, index snapshots
Configuration debt	Many untracked parameters and flags	Versioned config, release bundles
Evaluation debt	Stale datasets and untrusted graders	Dataset ownership, scheduled review, grader calibration
Prompt sprawl	Many near-duplicate prompts	Registry, templates, ownership, deprecation
Feedback loops	System outputs influence future training or retrieval data	Isolate generated content, monitor for degradation

Schedule recurring time to pay this debt down, because it accumulates quietly.

6.7 Chapter Summary

• Convert production failures into labeled, prioritized fixes and permanent regression cases.
• Pull cheap levers first, and use fine-tuning for narrow, stable gaps that prompting and retrieval cannot close.
• Make model migration a rehearsed, evidence-based routine.
• Manage entanglement, configuration, evaluation and prompt debt deliberately.

Exercises

1. Take 30 recent failures from a system you know. Cluster them, name the root causes using Section 6.2, and choose the cheapest effective lever for each cluster.
2. Write a migration plan for a model that will be retired in 90 days. Include the evidence you need at each step.
3. Inventory the AI-specific debt in one of your systems using the table in Section 6.6, and choose the two items to pay down first.

---

Chapter 7. Operating Model and Governance

In this chapter: team roles, risk tiers and the controls that go with them, the launch review, documentation, a maturity model, how this guide relates to common frameworks, and how to adopt it across many teams.

7.1 Roles and Responsibilities

Role	Primary responsibilities
Product owner	Defines the task, success criteria and acceptable risk
AI engineer	Builds components, prompts, context pipelines and tools
Evaluation or quality engineer	Owns datasets, graders, gates and evaluation reports
Platform engineer	Runs the gateway, serving, CI/CD, observability and cost tooling
SRE or operations	SLOs, on-call, capacity and incident response
Security and privacy	Threat models, red-teaming, data handling, approvals
Legal and compliance	Regulatory requirements, contracts, user disclosures
Domain expert	Labels, rubrics, and review of high-risk outputs

In small teams one person holds several roles. What matters is that each responsibility has a named owner.

7.2 Risk Tiers

Scale the required evidence and oversight to the potential for harm.

Tier	Typical use	Minimum controls
1: Low	Internal drafting aids, low-impact summaries	Smoke evaluation, basic monitoring, system card
2: Moderate	Customer-facing answers with human review or bounded impact	Full golden set and gates, red-team suite, staged rollout, quality SLOs, launch review
3: High	Decisions affecting money, safety, legal rights or sensitive data	Everything in Tier 2, plus independent review, human approval on actions, stricter slice floors, audit logging, periodic revalidation

Assign the tier in the feature specification (Section 2.1) and revisit it when the feature’s scope or autonomy changes.

7.3 The Launch Review

A launch review is a short, evidence-based meeting. The team brings the system card, the evaluation report, the red-team results, the rollout and rollback plan, the SLOs and the on-call owner. Reviewers from engineering, security, privacy and the business decide to proceed, proceed with conditions, or stop. Keep it lightweight for Tier 1 and thorough for Tier 3.

7.4 Documentation That Earns Its Keep

• System card: purpose, intended and prohibited uses, architecture, data sources, models, risk tier, known limitations, evaluation summary, owners.
• Prompt and bundle changelog: what changed, why, and the evidence.
• Decision records: model choice, thresholds, retrieval design, with the data behind them.
• Runbooks: rollback procedures, incident steps, escalation contacts.
• Evaluation reports: archived with each release.

7.5 Maturity Model

Level	Description	Typical signs
0: Ad hoc	Prompts edited in place; manual spot checks	No versioning, no dataset, surprises in production
1: Managed	Prompts and configs in version control; a small golden set	Manual evaluation before release
2: Gated	Automated evaluation in CI with quality gates; release bundles	Staged rollouts, basic tracing
3: Observed	Quality SLOs, continuous sampled evaluation, drift detection, cost attribution	Incident process feeds the dataset
4: Optimized	Shared platform and gateway, routine model migration, experimentation, portfolio-level governance	Teams self-serve with consistent controls

Aim to move one level at a time. The highest-value early steps are usually a versioned golden set, an automated gate and basic tracing.

7.6 Relation to Common Frameworks

This guide is compatible with widely used risk and security frameworks. It is not a substitute for them.

• The NIST AI Risk Management Framework organizes work into Govern, Map, Measure and Manage. Risk tiers and the launch review support Govern and Map. Evaluation and monitoring support Measure. Rollout, incident response and improvement support Manage.
• The OWASP Top 10 for LLM Applications catalogs recurring security risks, including prompt injection, sensitive information disclosure, improper output handling and excessive agency. Section 2.8 and the red-team practice in Section 3.9 address these.
• Regulatory requirements, such as those in the EU AI Act, depend on your use case and jurisdiction. Work with legal counsel to map risk tiers and documentation to your obligations.

7.7 Adopting at Organizational Scale

When dozens of teams ship AI features, the main risk shifts from any single feature to inconsistency: different teams invent different gates, log different fields, and migrate models on different schedules. A federated model keeps standards common and decisions local.

Concern	Owned centrally (platform and governance)	Owned by product teams
Model access	Gateway, approved model catalog, quotas, provider contracts	Choice of model within the catalog, per-feature routing rules
Evaluation	Harness, grader library, report format, minimum gate templates per tier	Datasets, slices, rubrics, thresholds, gate results
Release	Bundle manifest schema, promotion pipeline, rollout tooling	Bundle contents, rollout plan, rollback triggers
Observability	Trace schema, redaction defaults, cost attribution, dashboards templates	Feature SLIs and SLOs, alerts, on-call
Safety and privacy	Risk tier definitions, red-team suites, guardrail policy packs, review process	Threat model, feature-specific adversarial cases, system card
Lifecycle	Deprecation calendar, migration drills, shared incident taxonomy	Executing migrations, closing incident actions

Practices that make this work:

• Provide paved paths. Offer templates, a reference repository and a gateway SDK so that following the standard is the easiest option. Teams adopt standards that save them effort.
• Standardize interfaces, not internals. Fix the shape of the bundle manifest, trace attributes and evaluation report. Leave prompts, retrieval design and graders to the owning team.
• Share datasets and graders with owners. A shared adversarial set or a calibrated safety grader should have a named maintainer, a version history and a contribution process, in the style of inner source.
• Reuse existing governance. Attach the risk tiers and launch review to the launch process the organization already runs. Add AI-specific evidence to it; do not create a second approval path.
• Measure the platform. Track adoption and outcomes: share of AI features behind the gateway, share with automated gates, median time to complete a model migration, share of incidents converted into regression cases.
• Plan for coexistence. Teams will use different frameworks and languages. The gateway contract, manifest schema and trace conventions are what must stay stable across them.

7.8 Chapter Summary

• Name an owner for every responsibility, even in small teams.
• Tie evidence and oversight to risk tiers, and make the launch review short and evidence-based.
• Use the maturity model to prioritize the next improvement, and map the practices to your governance frameworks.
• At scale, standardize interfaces and platform services centrally, and keep decisions about quality and risk with the teams that own the features.

Exercises

1. List the responsibilities in Section 7.1 and assign a named owner to each on your team. Which ones have none?
2. Classify three features in your portfolio into risk tiers. Justify each tier and list the controls that follow.
3. Place your team on the maturity model. Write the first three steps toward the next level.
4. Draft the split between central and product-team ownership for your organization, using the table in Section 7.7 as a starting point.

---

Chapter 8. Putting It Together

In this chapter: a reference architecture, an end-to-end walkthrough of the Refund Policy Assistant, and a plan for adopting the practices.

8.1 A Simplified Reference Architecture

8.2 Walkthrough: The Refund Policy Assistant

The following narrative uses illustrative numbers.

Design. The team writes the specification from Section 2.1: suggest-only, risk tier 2, correctness at least 0.90, groundedness at least 0.95, p95 latency within 6 seconds. They define the component with a versioned prompt, a retrieval step over the policy documents with access control, a groundedness validator, and a degraded mode that shows the matching policy passages without a generated answer.

Evaluate. They build a golden set of 120 cases from past tickets, tagged by customer segment, language and exception type, and split it into development and test sets. A model-based grader scores correctness and groundedness and is calibrated against a hand-labeled sample. The gates from Section 3.6 run in CI. On the first full run, correctness has a lower bound of 0.84, below the gate.

Improve (pre-release). Failure labeling shows that most misses are retrieval misses on annual-plan exceptions. The team adjusts chunking so that exceptions stay attached to the parent policy, then re-runs the gates. The lower bound rises to 0.91, and the slice floors pass.

Ship. The release bundle is promoted. A shadow run on live tickets shows no regression in latency or cost. A canary for 5 percent of agents follows, then a ramp to 100 percent. Rollback has been rehearsed for the prompt, the index and the model.

Observe. Dashboards track quality on sampled traces, groundedness, escalation rate, p95 latency and cost per answer. A nightly golden-set run on production configuration detects drift.

Improve (production). Three weeks later, the quality SLI dips after the policy team publishes a new refund rule. Traces show the index is stale for one document set. The team fixes the ingestion schedule, adds the failing cases to the regression set, and adds a freshness check to the nightly job. The next policy change is caught before users see it.

8.3 A 30-60-90 Day Adoption Plan

First 30 days: establish the basics.

• Put prompts, configuration and model bindings in version control, and write the feature specification.
• Build a first golden set of 50 to 100 cases, and calibrate one grader.
• Add basic tracing with bundle version, token counts and latency.

Days 31 to 60: automate the gates.

• Run component checks and a smoke evaluation on every pull request.
• Add quality gates, slice reports and a nightly full run.
• Introduce release bundles, a fallback chain and a rollback procedure, and rehearse it.

Days 61 to 90: operate with confidence.

• Define quality, latency and cost SLOs, with continuous sampled evaluation.
• Stand up or adopt a gateway for model access, quotas and cost attribution.
• Run a red-team exercise, hold the first launch review, and schedule a model migration drill.

8.4 Common Anti-Patterns

Anti-pattern	Why it hurts	Remedy	Section
Editing prompts in place in production	No review, no rollback, no evidence	Version prompts; promote release bundles	2.3, 4.1
Tuning against the test set	The gate no longer measures generalization	Separate development and test sets	3.3
Reporting only averages	Slice-level failures stay hidden	Slice tags, slice floors	3.3, 3.6
Uncalibrated model-based judge	Scores drift from human judgment	Calibrate, pin judge versions, spot-check	3.4
Unpinned model with no scheduled evaluation	Provider changes surface as user complaints	Pin where possible; run the golden set on a schedule	2.3, 5.4
Consuming raw model output downstream	Malformed or hostile output reaches other systems	Validate before use	2.2, 2.8
Semantic cache shared across tenants	Cross-tenant leakage and stale answers	Scope by tenant and authorization; version-based invalidation	4.6
Alerting on uptime only	Wrong answers go unnoticed	Quality, safety and cost SLIs	5.3
Quietly loosening a threshold	Gates lose meaning	Treat thresholds as reviewed, versioned configuration	3.6
Model migration as an emergency	Retirement date forces a rushed change	Deprecation calendar and rehearsed playbook	6.5

8.5 Chapter Summary

• The reference architecture separates the AI component, context assembly, tools, the gateway and the models, with evaluation, release, observability and improvement cutting across all of them.
• The Refund Policy Assistant shows one full pass through the loop, including a failure caught before release and another caught in production.
• A 30-60-90 day plan introduces the practices in order of value: versioning and datasets, then automated gates, then operations and SLOs.

Exercises

1. Write a one-page walkthrough of your own feature through all five stages, using your own numbers.
2. Adapt the 30-60-90 day plan to your planning cadence. Name an owner for each deliverable.
3. Review your system against the anti-patterns in Section 8.4. Which three are most costly for you today?

---

Appendix A. Production Readiness Checklist

Design

• Feature specification with numeric success criteria, failure taxonomy, risk tier and autonomy level
• Typed interface, validators and fallback for each AI component
• Prompts, models, indexes, tools and policies versioned and combined in release bundles
• Context budget, access control and trusted versus untrusted content separation defined
• Threat model completed

Evaluate

• Golden set with typical, edge, adversarial and regression cases, tagged by slice
• Development and test sets separated; contamination checks in place
• Graders calibrated against human labels; judge versions pinned
• Quality gates (absolute, no-regression, slice, hard-fail, budget) automated in CI
• Red-team suite run for the risk tier

Ship

• Model access through a gateway with routing, guardrails, quotas and logging
• Staged rollout plan with rollback triggers and a rehearsed rollback
• Timeouts, retries, circuit breakers and fallbacks tested
• Load test with realistic token distributions; quotas confirmed

Observe

• End-to-end traces with bundle version and redaction
• SLIs and SLOs for quality, latency, safety and cost
• Drift detection and scheduled evaluation on production configuration
• Cost attributed by feature, tenant and team; anomaly alerts
• AI incident runbook and on-call owner

Improve and govern

• Failure triage pipeline feeding the regression set
• Model migration playbook and deprecation calendar
• System card, decision records and evaluation reports archived
• Launch review completed for the risk tier

---

Appendix B. Mapping to Practices You May Already Run

Most engineering organizations already have mature practices that correspond to parts of this guide. Use this table to connect the terms.

Concept in this guide	Terms you may already use
Release bundle	Release candidate, build manifest, deployment descriptor
Quality gate	Launch criteria, ship criteria, readiness review, promotion check
Evaluation set	Golden dataset, benchmark, test suite, holdout set
Slice report	Segment analysis, breakdown by cohort
Shadow mode	Shadow traffic, dark launch, traffic mirroring
Canary and ramp	Staged rollout, percentage rollout, progressive delivery
Online evaluation	A/B experiment, holdback, interleaving
Guardrail metrics	Experiment guardrails, counter-metrics
Quality SLO	SLI, SLO and error budget applied to model quality
AI gateway	Inference proxy, model router, model-serving frontend
Prompt registry	Configuration service, feature-config store
Risk tier	Criticality tier, launch tier, review level
System card	Model card, design document, service documentation
AI incident review	Blameless postmortem, incident review
Provider drift check	Dependency regression test, canary on upstream change

Where the Guide Plugs Into Existing Systems

Existing system	Integration point	Sections
Source control and code review	Prompts, bundle manifests and thresholds as reviewed files; code owners for gate configuration	2.3, 2.4
CI/CD and build systems	Evaluation suites as test targets; gate results block merge and promotion	3.6, 3.7
Feature-flag and configuration services	Bundle selection, rollout stages, kill switches	4.4
Experimentation platform	Online evaluation, guardrail metrics, hold-backs	3.8
API gateway or service mesh	AI gateway as a policy layer, or as a module within it	4.2
Tracing, metrics and logging stack	AI spans, quality and safety SLIs, redaction	5.2, 5.3, 5.6
SLO and on-call tooling	Quality SLOs, error budgets, AI incident runbook	5.3, 5.8
Data platform	Corpus lineage, index snapshots, dataset versioning	2.5, 3.3
Security and privacy review	Threat model, red-team suite, telemetry controls	2.8, 3.9, 5.6
Launch review process	System card, evaluation report, rollout plan	7.3
Incident management and postmortems	Failing case added to the evaluation set	5.8, 6.1
Cost and capacity management	Cost attribution by feature, tenant and bundle version	4.6, 5.5

---

Appendix C. Example Platform Mappings

These are examples of how the concepts can be realized on common platforms. They are not requirements and not an endorsement. Product names and features change, so check current documentation. Many organizations use internal equivalents; the concepts remain the same.

Concept	AWS	Google Cloud	Microsoft Azure	Portable or open source
Model access	Amazon Bedrock	Vertex AI	Azure AI Foundry	Self-hosted serving such as vLLM
Gateway and routing	Amazon API Gateway or a custom proxy in front of Bedrock	Apigee or a custom proxy in front of Vertex AI	Azure API Management	LiteLLM, Envoy-based proxies
Guardrails	Amazon Bedrock Guardrails	Model Armor	Azure AI Content Safety	NeMo Guardrails, in-house policy engines
Retrieval and knowledge	Amazon Bedrock Knowledge Bases, Amazon OpenSearch Service	Vertex AI Search, Vector Search	Azure AI Search	PostgreSQL with pgvector, OpenSearch, Elasticsearch
Evaluation	Amazon Bedrock evaluation capabilities plus your own harness in CI	Vertex AI Gen AI evaluation service plus your own harness	Azure AI Foundry evaluation plus your own harness	Your own harness in CI, open-source evaluation libraries
Orchestration	AWS Step Functions, agent frameworks	Cloud Workflows, agent frameworks	Durable Functions, Logic Apps, agent frameworks	Temporal, Airflow, application code
Feature flags and configuration	AWS AppConfig	Internal flag or configuration service	Azure App Configuration	OpenFeature-compatible flag services
Telemetry	Amazon CloudWatch, AWS X-Ray	Cloud Monitoring, Cloud Trace	Azure Monitor, Application Insights	OpenTelemetry, Prometheus, Grafana
Secrets and access control	AWS Secrets Manager, AWS IAM	Secret Manager, Cloud IAM	Azure Key Vault, Microsoft Entra ID	HashiCorp Vault, OIDC

---

Appendix D. Glossary

AI component. A bounded unit of functionality that combines a model, instructions, context and validation behind a typed interface.

AI gateway. A serving layer that centralizes model access, routing, guardrails, quotas, caching and logging.

Autonomy level. What a system may do without a human: suggest, act with approval, or act alone.

Canary release. Exposing a change to a small share of traffic before wider rollout.

Context engineering. The practice of selecting, ordering and budgeting the information supplied to a model.

Degraded mode. A predefined reduced capability that a system falls back to when its primary path fails, such as showing search results without a generated summary.

Drift. A change in inputs, outputs, data or provider behavior that degrades quality without a code change.

Evaluation pyramid. A layered set of evaluations in which cheap, fast checks run often and expensive, realistic checks run before release.

Golden set. A curated, labeled evaluation dataset that represents expected behavior.

Grader. A procedure that converts a model output into a score.

Groundedness. The degree to which claims in an output are supported by the provided evidence.

Guardrail. A policy check applied to inputs or outputs.

LLM-as-judge. Using a language model as a grader.

Quality gate. An automated rule that decides whether a change may proceed.

Release bundle. The set of artifact versions (prompt, model, index, policy, tools, configuration) that together define deployable behavior.

Risk tier. A classification of a feature by potential for harm that determines the evidence, review and oversight required.

Shadow mode. Running a candidate on live traffic without exposing its output to users.

Slice. A subset of cases defined by an attribute such as segment, language or risk.

System card. A document describing a system’s purpose, limits, data, evaluation and owners.

---

Appendix E. Further Reading

• D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems,” Advances in Neural Information Processing Systems, 2015.
• Eric Breck et al., “The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction,” 2017.
• Martin Zinkevich, “Rules of Machine Learning: Best Practices for ML Engineering,” Google.
• Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering, O’Reilly, 2016.
• Titus Winters, Tom Manshreck and Hyrum Wright, Software Engineering at Google, O’Reilly, 2020.
• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara and Stephen Thorne (eds.), The Site Reliability Workbook, O’Reilly, 2018.
• Jez Humble and David Farley, Continuous Delivery, Addison-Wesley, 2010.
• Ron Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments, Cambridge University Press, 2020.
• Chip Huyen, AI Engineering, O’Reilly, 2025.
• Nicole Forsgren, Jez Humble and Gene Kim, Accelerate, IT Revolution, 2018.
• Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” 2023.
• Percy Liang et al., “Holistic Evaluation of Language Models,” 2022.
• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), 2023.
• OWASP, Top 10 for Large Language Model Applications.
• OpenTelemetry, Semantic Conventions for Generative AI (check the current specification).