Executing the AI Technology Roadmap

From Strategic and Architectural Intent to Production AI Capabilities

A practical guide to mobilization, engineering, release, operation, value realization, and continuous adaptation

Execution is the controlled conversion of strategic and architectural intent into production systems, operating capabilities, measurable outcomes, and reusable enterprise assets.

---

Contents

• Preface
• Executive Summary
• [Part I. Establish the Execution Mandate]
  • 1. What It Means to Execute an AI Technology Roadmap
  • 2. Define the Execution Contract
  • 3. Establish Decision Rights and Accountability
• [Part II. Mobilize the Portfolio and Initiatives]
  • 4. Translate the Roadmap into an Executable Portfolio
  • 5. Mobilize Initiatives with Evidence-Based Readiness
  • 6. Manage Dependencies as First-Class Portfolio Objects
• [Part III. Establish the Foundations]
  • 7. Build the Platform Before Scaling the Products
  • 8. Establish a Paved Road for AI Engineering
  • 9. Engineer Data and Knowledge Foundations
  • 10. Establish the Control Plane and Runtime Boundaries
• [Part IV. Deliver AI Products Through Disciplined Engineering]
  • 11. Treat AI Capabilities as Products
  • 12. Translate Requirements into Testable Specifications
  • 13. Design the AI Application as an End-to-End System
  • 14. Engineer Model, Prompt, Retrieval, and Agent Changes as Versioned Assets
  • 15. Build an Evaluation System Before Scaling Usage
  • 16. Engineer Agentic Systems with Explicit Control Boundaries
• [Part V. Integrate, Verify, and Release]
  • 17. Integrate with Enterprise Systems Without Creating Competing Authorities
  • 18. Apply a Layered Test Strategy
  • 19. Define Release Strategies Appropriate to Risk
  • 20. Establish Production Readiness
• [Part VI. Deploy and Operate Production AI]
  • 21. Treat Deployment as a Controlled State Transition
  • 22. Define Service-Level Objectives for AI Products
  • 23. Implement End-to-End Observability
  • 24. Operate AI Systems with SRE Discipline
• [Part VII. Govern Change Across the AI Lifecycle]
  • 25. Establish AI Change Management
  • 26. Manage Drift and Feedback
• [Part VIII. Manage Economics and Value Realization]
  • 27. Manage the Full Lifecycle Cost
  • 28. Measure Benefits Without Confusing Activity with Value
  • 29. Apply Benefits Realization Governance
• [Part IX. Scale Through Reuse and Organizational Capability]
  • 30. Scale Reusable Capabilities Without Creating a Central Bottleneck
  • 31. Build the Skills and Operating Model
  • 32. Manage Adoption and Process Change as Delivery Work
• [Part X. Govern Portfolio Execution]
  • 33. Establish a Multi-Level Governance Cadence
  • 34. Use Stage Gates as Evidence-Based Decisions
  • 35. Govern Exceptions and Architecture Decisions
  • 36. Maintain Portfolio Transparency
• [Part XI. Sequence Delivery and Manage Uncertainty]
  • 37. Sequence Work Around Outcomes and Dependencies
  • 38. Use Thin Vertical Slices
  • 39. Manage Uncertainty Explicitly
• [Part XII. A Worked Enterprise Example]
  • 40. Executing an AI-Assisted Customer Operations Initiative
• [Part XIII. Execution Artefacts and Templates]
  • 41. Minimum Execution Artefact Set
  • 42. Product Backlog Structure
  • 43. Architecture Decision Record Template
  • 44. AI Evaluation Plan Template
  • 45. Production Readiness Record Template
  • 46. Benefits Realization Template
• [Part XIV. Common Execution Failure Modes]
  • 47. Mistakes That Undermine AI Roadmap Execution
  • 48. Warning Signs That Require Intervention
• [Part XV. The End-to-End Execution Method]
  • 49. A Practical Execution Lifecycle
  • 50. First 90 Days: A Practical Mobilization Pattern
• [Part XVI. Measuring Execution Maturity]
  • 51. AI Roadmap Execution Maturity Model
  • 52. Execution Success Criteria
• Conclusion
• Appendix A. Mapping the Guide to Existing Engineering Practice
• Appendix B. Glossary
• Appendix C. Companion Reading

---

Preface

An AI technology roadmap is useful only when the organization can turn its decisions into working capabilities. The companion volume, Defining the AI Technology Roadmap, establishes business outcomes, the opportunity portfolio, capability gaps, target architecture, technology direction, investment logic, and the sequence of initiatives. This volume addresses the discipline that follows: execution.

Execution is not the mechanical implementation of a roadmap diagram. It requires product management, architecture, engineering, data management, security, risk, operations, finance, procurement, and business ownership to work inside one coherent delivery system.

Three levels of execution

Level	Question it answers	Core practice
Portfolio execution	Which roadmap initiatives are funded, staffed, sequenced, continued, paused, or stopped?	Mobilization, dependency management, stage gates, portfolio transparency
Initiative execution	How is a defined capability or business outcome delivered through a governed product or programme?	Execution contract, product ownership, evidence-based readiness
System execution	How is each AI-enabled product or platform engineered, deployed, operated, and improved?	Versioned assets, evaluation gates, controlled release, SRE discipline

These levels must connect. Portfolio governance without delivery evidence becomes bureaucracy. Delivery without architectural direction creates fragmentation. Production systems without value measurement become technology assets with uncertain business justification.

Who this guide is for

The guide is written for enterprise technology leaders, principal and enterprise architects, AI engineering leaders, platform teams, product owners, delivery executives, and governance functions. It assumes an approved or provisionally approved AI technology roadmap. Where the roadmap is incomplete, execution must expose and resolve the missing decisions rather than conceal them in delivery plans.

Fitting this guide to what you already run

Large engineering organizations already operate portfolio governance, architecture review, release engineering, reliability engineering, and risk processes. This guide does not propose a parallel system for AI. It is designed to plug into what exists:

• Governance. Part X describes a multi-level cadence, evidence-based stage gates, and exception handling that can run inside existing investment and review forums.
• Operating model. Part IX describes a federated split between platform teams and product teams, so that standards scale without becoming a bottleneck.
• Artefacts. Part XIII provides a minimum artefact set and templates that can be mapped onto existing documentation standards.
• Assessment. Part XVI provides a maturity model and success criteria for locating current practice and choosing what to strengthen next.
• Translation. Appendix A maps each major concept to practices most engineering organizations already know: SRE, DevOps and continuous delivery, MLOps, platform engineering, FinOps, and security and privacy review.

How to read it

Start with the Executive Summary, then follow the path for your role.

Reader	Start with
Executives and sponsors	Parts VIII, IX, X, XVI
Enterprise and AI architects	Parts III, IV, V, X; ADR template (§43)
Engineering and product leaders	Parts II, IV, V, XI
Risk, security, privacy, and compliance leaders	Parts III, VII, X; evaluation and readiness (§15, §20)
Platform, SRE, and operations teams	Parts III, VI, VII, VIII

Whatever your role, read Part XII (the worked example) to see the pieces working together on one initiative, and use Parts XIII to XVI as a working reference.

Conventions

• The guide is vendor-neutral. It describes capabilities, controls, and decisions rather than products.
• Sections are numbered continuously from 1 to 52 across Parts I to XVI; cross-references use these numbers (§).
• Diagrams are written in Mermaid so they can be maintained as text alongside the guide.
• Each Part opens with a short orientation and closes with Key Takeaways.
• Worked examples are illustrative. They do not prescribe a technology vendor or a regulatory interpretation.
• Formulas are conceptual aids, not calibrated universal metrics.
• Templates show minimum content. Adapt them to your own standards rather than copying them verbatim.

---

Executive Summary

Executing an AI technology roadmap is the disciplined process of mobilizing, building, integrating, deploying, operating, and improving the business and technology capabilities that the roadmap defines. It converts strategic choices into a managed portfolio of initiatives and production outcomes.

A complete execution system answers seven questions.

Question	Execution concern	Evidence of a good answer
What are we delivering?	Scope, outcomes, acceptance criteria	Approved charters and measurable outcome definitions
Who is accountable?	Sponsorship, product ownership, architecture, engineering, risk, operations	Named decision owners and an explicit responsibility model
What must exist first?	Data, platform, identity, integration, security, skills, operating foundations	Dependency map and readiness evidence
How will we build it?	Engineering methods, architecture controls, testing, delivery cadence	Delivery plan, backlog, architecture decisions, quality gates
How will it enter production?	Release, deployment, operational readiness, rollback	Release evidence, runbooks, service ownership, tested recovery
How will we know it works?	Technical performance, business outcomes, risk, adoption, economics	Instrumented measures and an agreed benefits baseline
How will it evolve?	Feedback, model changes, technology shifts, portfolio reprioritization	Review cadence, change controls, updated roadmap decisions

The execution lifecycle

flowchart TD
    A["Approved AI Technology Roadmap"] --> B["Portfolio Mobilization"]
    B --> C["Initiative Charter and Product Ownership"]
    C --> D["Architecture and Delivery Readiness"]
    D --> E["Platform, Data and Control Foundations"]
    E --> F["Iterative Product Engineering"]
    F --> G["Verification and Release Readiness"]
    G --> H["Production Deployment"]
    H --> I["Operate, Observe and Support"]
    I --> J["Measure Adoption, Outcomes, Risk and Cost"]
    J --> K["Learn, Optimize and Reprioritize"]
    K --> B

The lifecycle is iterative. Foundational capabilities may be built once and reused. Delivery may reveal new data constraints or invalidate architectural assumptions. Production evidence may change the economics of an initiative or overturn an earlier technology choice. Governance must therefore preserve the roadmap's strategic direction while allowing evidence-based adaptation.

The central principle

Deliver business outcomes through governed products, build reusable capabilities where reuse is justified, and promote systems to production only when their operational, security, risk, and value requirements are demonstrably satisfied.

Ten commitments of a well-executed roadmap

1. Every initiative has one accountable business outcome owner.
2. Dependencies and assumptions are visible, owned, and tracked.
3. Platform investment follows demonstrated demand, not anticipated demand.
4. Authorization and policy are enforced outside model reasoning.
5. Models, prompts, retrieval configurations, tools, and policies are versioned assets.
6. Evaluation exists from the first prototype and gates every release.
7. Pilots have hypotheses, exit criteria, and decision owners.
8. Production services have SLOs, owners, runbooks, and tested rollback.
9. Value is measured as completed business outcomes, not model activity.
10. Initiatives can be paused, redesigned, or retired on evidence.

---

Part I. Establish the Execution Mandate

Execution begins with agreement: what will be delivered, who is accountable, and who may decide what. This Part establishes the contract and decision rights that every later Part relies on.

1. What It Means to Execute an AI Technology Roadmap

Roadmap definition determines the intended direction. Roadmap execution builds the organizational and technical mechanisms to realize it.

Execution includes:

• translating roadmap initiatives into funded, owned, deliverable work;
• validating assumptions and dependencies before substantial investment;
• establishing product, platform, data, security, and operating foundations;
• engineering AI-enabled products and reusable capabilities;
• integrating AI systems with enterprise applications, workflows, and systems of record;
• verifying functional, non-functional, security, privacy, safety, and compliance requirements;
• deploying systems through controlled release mechanisms;
• operating, monitoring, supporting, and improving production capabilities;
• measuring realized business value and total lifecycle economics;
• feeding evidence and lessons back into the roadmap.

Execution is not synonymous with model development. A production AI capability often depends more on data quality, workflow integration, identity, policy enforcement, observability, and human operating procedures than on the model itself.

1.1 The execution boundary

The companion discipline, roadmap definition, answers:

• Why should the organization invest?
• Which business outcomes matter?
• Which AI opportunities are worth pursuing?
• Which capabilities and architectural direction are required?
• Which technology choices and investments are justified?
• In what broad sequence should capabilities be developed?

Execution answers:

• How will approved initiatives be mobilized and delivered?
• Which teams, products, platforms, and controls must be established?
• How will requirements become tested software and operational services?
• How will production risk be managed?
• How will benefits be measured and sustained?
• What evidence will justify continuation, expansion, redesign, or termination?

Roadmap definition	Roadmap execution
Establishes strategic intent and direction	Mobilizes and delivers against that direction
Identifies opportunities and capability gaps	Converts them into products, services, and engineering work
Defines target-state architecture and technology principles	Implements architecture through decisions, standards, and tested systems
Establishes investment rationale	Controls funding, delivery, sequencing, and realized value
Defines governance requirements	Operates governance in delivery and production
Describes expected outcomes	Measures actual outcomes and corrects deviations

Execution may require revisiting a roadmap decision, but the two disciplines are not interchangeable. A change to strategic scope, target architecture, investment envelope, or sequencing is an explicit roadmap change, supported by evidence and approved by the appropriate authority.

2. Define the Execution Contract

Before delivery begins, establish an execution contract: the shared understanding among the sponsor, business owner, product owner, architecture, engineering, risk, and operations.

An execution contract contains:

1. Outcome. The business result the initiative is intended to produce.
2. Scope. The users, processes, systems, geographies, business units, and use cases included.
3. Boundaries. What the initiative will not deliver.
4. Accountability. One accountable business outcome owner, plus named technology and control owners.
5. Constraints. Budget, dates, regulatory obligations, technology standards, data restrictions, and resource limits.
6. Dependencies. Platforms, data products, APIs, enterprise systems, vendors, and decisions required from other teams.
7. Acceptance. Measurable functional, operational, risk, and business acceptance criteria.
8. Evidence. The records required to show the system is safe, reliable, useful, and economically justified.
9. Change authority. Who can approve changes to scope, architecture, risk, funding, and release.
10. Exit conditions. What constitutes success, pause, redesign, or termination.

2.1 Initiative charter template

Field	Required content
Initiative ID and name	Stable identifier and concise name
Strategic alignment	Roadmap objective and capability
Business sponsor	Executive accountable for sponsorship
Outcome owner	Business leader accountable for measurable benefit
Product owner	Person accountable for product scope and priorities
Technology owner	Engineering or technology leader accountable for delivery
Architecture owner	Architect accountable for architectural coherence
Risk and control owners	Security, privacy, legal, compliance, model risk, and other applicable owners
Target users and process	Intended users, workflow, and operating context
In-scope capabilities	Product features and platform capabilities
Out-of-scope items	Explicit exclusions
Baseline and target	Current measure, target measure, and measurement method
Budget and capacity	Approved funding, roles, and constraints
Dependencies	Upstream and downstream dependencies
Milestones	Evidence-based decision points, not only calendar dates
Acceptance criteria	Product, architecture, operational, risk, and value criteria
Exit criteria	Conditions to scale, pause, redesign, or stop

A charter does not replace the product backlog, solution architecture, delivery plan, or risk assessment. It connects them.

3. Establish Decision Rights and Accountability

AI delivery spans organizational boundaries. Without explicit decision rights, teams meet late-stage disputes over data access, model selection, policy interpretation, deployment authority, and operational ownership.

A practical model distinguishes four roles:

• Accountable (A): owns the outcome or decision.
• Responsible (R): performs the work.
• Consulted (C): provides required expertise or approval.
• Informed (I): receives status and evidence.

3.1 Example responsibility model

Activity	Business sponsor	Product owner	Architect	Engineering	Data owner	Risk / security	Operations
Business outcome definition	A	R	C	I	C	C	I
Product backlog and priorities	I	A/R	C	C	C	C	C
Target solution architecture	I	C	A/R	R	C	C	C
Data access and quality	I	C	C	R	A	C	I
Model and retrieval evaluation	I	A	C	R	C	C	C
Security and control design	I	C	R	R	C	A	C
Production readiness	I	C	C	R	C	C	A/R
Benefit realization	A	R	I	C	C	I	C
Production incident response	I	C	C	R	C	C	A

The exact allocation varies by organization and regulatory context. What matters is that every material decision has one clear accountable owner. A committee may advise or approve, but it should not obscure individual accountability.

3.2 Decision-rights matrix

Define decision authority for at least these areas:

• business scope and outcome changes;
• product prioritization;
• data access and permitted data use;
• architecture exceptions;
• model and vendor selection;
• security and privacy risk acceptance;
• human oversight and workflow authority;
• production release;
• material model or prompt changes;
• incident severity and service restoration;
• funding changes and initiative termination.

Architecture review must not substitute for business accountability. Equally, business urgency must not override mandatory security, privacy, safety, or legal controls.

Key Takeaways: Part I

• Execution is the conversion of intent into operating capability, not the mechanical build of a diagram.
• Agree an execution contract before delivery begins.
• Name one accountable owner for every material decision.
• Treat changes to scope, architecture, investment, or sequencing as explicit roadmap changes.

---

Part II. Mobilize the Portfolio and Initiatives

A roadmap contains themes and broad initiatives. This Part turns them into a traceable, owned portfolio and shows how to start each initiative at a level of investment matched to its unresolved risk.

4. Translate the Roadmap into an Executable Portfolio

Execution needs a structure that connects strategic themes and target capabilities to accountable delivery units.

flowchart TD
    A["Enterprise AI Strategy"] --> B["AI Technology Roadmap"]
    B --> C["Strategic Themes"]
    C --> D["Roadmap Initiatives"]
    D --> E["Products and Platform Capabilities"]
    E --> F["Epics"]
    F --> G["Features and Enablers"]
    G --> H["Stories, Tasks and Engineering Work"]
    H --> I["Tested and Operated Releases"]
    I --> J["Business Outcomes"]

The hierarchy is not a rigid project taxonomy. Its purpose is traceability: each delivery item has a defensible relationship to a roadmap capability and a business outcome.

4.1 Classify the work

Work class	Purpose	Examples
Business-facing AI products	Deliver a defined user or process outcome	Claims triage assistant, customer-service copilot, forecasting product
Shared AI platform capabilities	Provide reusable technical capabilities	Model gateway, evaluation service, prompt and model registry
Data and knowledge foundations	Make trusted, governed information usable	Data products, document ingestion, metadata, lineage, retrieval indexes
Trust and control foundations	Enforce safe and accountable use	Identity, authorization, policy enforcement, audit, model risk controls
Operating and organizational capabilities	Make production delivery sustainable	SRE, FinOps, AI evaluation operations, support, training

This classification prevents the portfolio from becoming a collection of isolated use cases and makes enabling work visible. Platform and control foundations need explicit owners, consumers, service levels, and funding logic.

5. Mobilize Initiatives with Evidence-Based Readiness

An initiative should not move directly from roadmap approval to full-scale implementation. It passes through mobilization activities that test whether the organization is ready to invest at the proposed level.

5.1 Mobilization checklist

• Business sponsor and outcome owner confirmed.
• Product owner and delivery lead assigned.
• Scope, users, workflow, and exclusions agreed.
• Baseline data and measurement method identified.
• Data sources, owners, access rights, and quality constraints understood.
• Initial architecture and integration boundaries documented.
• Security, privacy, legal, regulatory, and model-risk applicability assessed.
• Platform and environment dependencies identified.
• Vendor, procurement, and licensing dependencies identified.
• Team skills and capacity assessed.
• Initial lifecycle cost and benefit assumptions documented.
• Acceptance criteria and decision gates agreed.
• Operational owner and support model identified.
• Key assumptions, risks, and unresolved decisions recorded.

Readiness is not binary. An initiative can begin with bounded discovery while critical questions are resolved, provided the investment level and permitted activity reflect the unresolved risk.

5.2 Use discovery to retire uncertainty

Discovery is time-boxed and question-driven. It is not a miniature implementation programme.

Typical discovery questions:

• Is the required data legally and operationally available?
• Does the target workflow contain a sufficiently stable decision boundary?
• Can the model meet quality and latency requirements on representative data?
• Is retrieval necessary, and can authoritative sources be identified?
• Can the solution integrate with the system of record without creating a second authority?
• What level of human review is necessary?
• Can the economics support production at realistic usage levels?
• Do vendor, residency, or model-provider constraints change the design?

A discovery output is a decision package: evidence, remaining uncertainty, updated cost and benefit estimates, a recommended next step, and explicit decision requests.

6. Manage Dependencies as First-Class Portfolio Objects

AI initiatives routinely depend on capabilities outside their team: identity services, data platforms, document repositories, enterprise APIs, model endpoints, network controls, procurement, and operational support.

6.1 Dependency register

Field	Description
Dependency ID	Unique identifier
Dependent initiative	Initiative requiring the dependency
Provider / owner	Team accountable for delivery or availability
Required capability	What must be provided
Need-by point	Latest date or milestone at which it is required
Readiness evidence	What proves the dependency is usable
Interface contract	API, event, data, service, or process contract
Risk if delayed	Impact on scope, cost, quality, or schedule
Contingency	Alternative, sequencing change, or scope reduction
Status	Proposed, committed, in progress, ready, blocked, retired

6.2 Principles

1. Make dependencies visible before they become schedule blockers.
2. Assign an owner on both the consuming and providing sides.
3. Agree interface contracts early, then evolve them through versioned change.
4. Distinguish a dependency from a collaboration request.
5. Never mark a dependency complete because the provider says so. Validate it from the consumer's perspective.
6. Escalate unresolved cross-team dependencies through portfolio governance.

Key Takeaways: Part II

• Preserve traceability from theme to initiative to backlog item to outcome.
• Classify work so that enabling platform, data, and control work is visible and funded.
• Match investment to resolved risk; use time-boxed discovery to retire uncertainty.
• Treat dependencies as owned, tracked objects with consumer-validated readiness.

---

Part III. Establish the Foundations

Product teams move quickly only when the ground beneath them is firm. This Part covers platform capabilities, the paved road, data and knowledge foundations, and the control plane that keeps intelligence separate from authority.

7. Build the Platform Before Scaling the Products

The roadmap may identify shared platform capabilities, but execution must decide how to establish them without overbuilding. A platform should solve recurring needs for several products and reduce the cost, risk, or lead time of delivery.

A platform is not valuable merely because it centralizes technology. It is valuable when it offers reliable, secure, well-documented capabilities that product teams adopt because doing so lowers their cognitive and operational burden.

7.1 Typical enterprise AI platform capability areas

Capability area	Representative capabilities	Execution considerations
Identity and access	Workforce identity, workload identity, service-to-service authentication, authorization	Enforce least privilege and tenant-aware access
Model access	Model gateway, provider adapters, routing, quotas, fallback	Avoid hard coupling to one provider or model API
Data and knowledge	Ingestion, metadata, vector and lexical retrieval, document processing	Preserve source lineage, authority, permissions, freshness, and deletion semantics
Agent execution	Tool invocation, state, task management, bounded execution	Separate planning from orchestration and authorization
Evaluation	Test datasets, quality metrics, safety tests, regression suites	Evaluate the whole application, not only the model
Observability	Traces, logs, metrics, cost attribution, feedback	Correlate model, retrieval, tool, workflow, and business events
Governance	Policy enforcement, approvals, model inventory, audit	Make controls executable and testable
Delivery engineering	CI/CD, infrastructure as code, artifact management, release controls	Support reproducible and reversible deployments
Runtime operations	Scaling, resilience, secrets, configuration, incident response	Define service ownership and operational SLOs

These are capability areas, not a mandatory product list. Implementations may use managed services, open-source components, commercial platforms, or a mix. Select components against requirements, operating capability, security posture, and lifecycle economics.

8. Establish a Paved Road for AI Engineering

A paved road is the supported, documented, secure, and observable path for building and operating AI products. It reduces repeated engineering effort while allowing justified exceptions.

A minimum paved road provides:

• approved project and repository templates;
• identity and secret-management patterns;
• environment provisioning through infrastructure as code;
• approved model and provider access patterns;
• standard retrieval and tool integration interfaces;
• reusable evaluation harnesses;
• standard telemetry and cost attribution;
• CI/CD pipelines and release controls;
• security scanning and dependency management;
• deployment, rollback, and incident runbooks;
• documented architecture patterns and reference implementations.

8.1 Paved-road adoption model

Level	Description	Governance
Supported default	Standard pattern with reusable templates and automation	Normal review and automated controls
Approved variation	Different implementation with documented rationale	Architecture and control review
Time-bound exception	Temporary deviation with compensating controls and expiry	Named risk owner and remediation plan
Prohibited pattern	Violates mandatory policy or creates unacceptable risk	Not eligible for production approval

The paved road must be easy to adopt. If the approved route is materially slower than an unsafe workaround, teams will build shadow platforms.

9. Engineer Data and Knowledge Foundations

AI products are only as dependable as the information they can reach and the controls that govern that access. Data readiness is an execution workstream, not a preliminary checkbox.

9.1 Data foundation work

• Identify authoritative sources and accountable data owners.
• Establish lawful purpose, permitted use, and access controls.
• Profile completeness, accuracy, consistency, timeliness, and representativeness.
• Define data contracts, schemas, metadata, and lineage.
• Implement ingestion, transformation, validation, and error handling.
• Define retention, deletion, archival, and data-subject request handling where applicable.
• Establish quality monitoring and remediation ownership.
• Separate training, evaluation, and production data where required.
• Protect sensitive data in transit, at rest, in logs, and in model interactions.

9.2 Retrieval and knowledge-system controls

For retrieval-augmented generation and knowledge assistants, implement controls for:

1. source authority and document ownership;
2. document-level and, where required, chunk-level access control;
3. metadata extraction and provenance;
4. chunking and indexing version management;
5. lexical, vector, and hybrid retrieval evaluation;
6. freshness and re-indexing triggers;
7. deletion propagation and index consistency;
8. citation and evidence presentation;
9. retrieval failure and no-answer behavior;
10. tenant and user isolation.

A retrieval result is not trustworthy because it is semantically similar. Retrieval must preserve authority, relevance, freshness, sufficiency, and permission boundaries.

A conceptual model of context quality:

Context Value = Relevance × Authority × Freshness × Sufficiency × Signal Quality

This is not a calibrated metric. Its purpose is to remind teams that high semantic similarity cannot compensate for stale, unauthorized, or incomplete context.

10. Establish the Control Plane and Runtime Boundaries

Enterprise AI platforms benefit from a clear separation between policy and administration, reusable services, and workload execution.

flowchart TB
    subgraph CP["Control Plane"]
      C1["Identity and Access Policy"]
      C2["Model and Agent Registry"]
      C3["Policy and Approval Management"]
      C4["Configuration and Version Management"]
      C5["Governance, Audit and Cost Policy"]
    end
    subgraph SH["Shared AI Services"]
      S1["Model Gateway and Routing"]
      S2["Retrieval and Knowledge Services"]
      S3["Evaluation Services"]
      S4["Tool and Integration Gateway"]
      S5["Observability Services"]
    end
    subgraph RT["Workload Runtime"]
      R1["Application / Agent Runtime"]
      R2["Session and State Management"]
      R3["Tool Execution"]
      R4["Workflow and Human Review"]
    end
    CP --> SH
    CP --> RT
    SH <--> RT

The model is logical, not a prescription for separate physical infrastructure. Some services may be managed centrally while execution environments remain isolated by application, business unit, tenant, or risk tier.

10.1 Separate intelligence from authority

An LLM may interpret, classify, summarize, recommend, or propose an action. It must not become the authority for identity, permissions, policy, financial limits, or regulated decisions.

sequenceDiagram
    participant U as User or System
    participant A as AI Application / Agent
    participant P as Policy and Authorization Service
    participant T as Tool / Enterprise API
    participant S as System of Record
    U->>A: Request
    A->>A: Interpret intent and prepare proposed action
    A->>P: Request authorization for action and context
    P-->>A: Permit, deny, or require approval
    alt Authorized
        A->>T: Execute bounded operation
        T->>S: Apply transaction
        S-->>T: Authoritative result
        T-->>A: Execution result
    else Approval required
        A-->>U: Request human approval
    else Denied
        A-->>U: Explain that action cannot be performed
    end
    A-->>U: Report verified outcome

The key distinction is safe-to-answer versus authorized-to-act. A system may be allowed to provide information while being prohibited from executing the associated transaction. Authorization must be deterministic, contextual, and enforced outside the model's free-form reasoning.

Key Takeaways: Part III

• Build only the platform capabilities that real products demonstrably need.
• Make the paved road the easiest path, with governed exceptions.
• Treat data readiness, retrieval authority, and permissions as engineering work.
• Enforce authorization in a deterministic policy layer, never in the prompt.

---

Part IV. Deliver AI Products Through Disciplined Engineering

This Part treats AI capabilities as products with testable requirements, end-to-end designs, versioned assets, an evaluation system, and explicit control over agentic behavior.

11. Treat AI Capabilities as Products

A production AI capability has a product owner, target users, a defined workflow, measurable outcomes, a release strategy, and a lifecycle operating model.

A model endpoint is not a product. A chatbot interface is not a product. A product is the complete system through which a user or business process receives value, including its data, models, retrieval, policies, integrations, human workflows, and support.

11.1 Product definition

For each AI product, document:

• user personas and operational roles;
• user problem and process context;
• current workflow and baseline performance;
• target outcome and measurable value;
• permitted and prohibited use;
• inputs, outputs, and data classifications;
• model, retrieval, and tool dependencies;
• human review and escalation paths;
• system-of-record boundaries;
• availability, latency, throughput, and recovery requirements;
• adoption, training, and support requirements;
• release and deprecation strategy.

11.2 Product discovery and delivery loop

flowchart LR
    A["Observe User and Process"] --> B["Define Outcome and Hypothesis"]
    B --> C["Prototype and Test Assumptions"]
    C --> D["Evaluate with Representative Evidence"]
    D --> E["Build Production Capability"]
    E --> F["Release to Bounded Users"]
    F --> G["Measure Use and Outcomes"]
    G --> H["Improve, Expand or Retire"]
    H --> A

The loop prevents a successful demonstration from being mistaken for proof of production readiness.

12. Translate Requirements into Testable Specifications

AI requirements must be more precise than "the assistant should be accurate" or "the agent should respond quickly."

Dimension	Example requirement
Functional	The assistant retrieves policy documents and presents source citations
Quality	Answers meet a defined acceptance threshold on a representative evaluation set
Retrieval	Required authoritative evidence appears in the top-k results at a target rate
Safety	The system refuses prohibited requests and does not expose unauthorized data
Authorization	Tool actions are checked against user, tenant, resource, and transaction permissions
Latency	p95 end-to-end response time remains within the agreed service objective
Availability	Service meets the agreed monthly availability target
Resilience	Provider failure triggers a tested fallback or graceful degradation
Scalability	Throughput target is met at expected concurrency and burst profile
Cost	Cost per completed business task remains within an agreed range
Auditability	Material inputs, decisions, approvals, tool calls, and outcomes are traceable
Accessibility	User experience meets applicable accessibility requirements
Maintainability	Prompts, models, tools, and policies are versioned and testable

Requirements should be measurable, testable, and traceable to a business or risk need.

13. Design the AI Application as an End-to-End System

A production AI application may include these logical components:

flowchart TB
    U["User / Enterprise Channel"] --> I["Application Interface and API"]
    I --> O["Orchestration / Workflow"]
    O --> R["Context and Retrieval"]
    O --> M["Model Gateway"]
    O --> P["Policy and Authorization"]
    O --> H["Human Review"]
    O --> X["Enterprise Tools and APIs"]
    R --> D["Authoritative Data and Knowledge Sources"]
    M --> L["Approved Model Providers / Hosted Models"]
    X --> S["Systems of Record"]
    O --> T["Telemetry, Audit and Evaluation"]
    R --> T
    M --> T
    P --> T
    X --> T

The diagram is a logical view. A given product may not need every component, and deployment topology may differ.

13.1 Design for bounded responsibility

Give each component a clear contract:

Component	Responsibility
Application interface	Authentication, user experience, request validation, response presentation
Orchestration	Workflow state, task sequencing, retries, bounded coordination
Retrieval	Authorized evidence acquisition, ranking, filtering, provenance
Model gateway	Provider abstraction, routing, quotas, timeouts, fallback
Policy service	Deterministic authorization and policy decisions
Tool gateway	Validated, constrained access to enterprise operations
Human review	Approval, exception handling, escalation
Telemetry	Traceability across application, retrieval, model, policy, and tool execution

Avoid placing all responsibilities in a single agent prompt or orchestration function. Explicit boundaries improve testability, security, and change isolation.

14. Engineer Model, Prompt, Retrieval, and Agent Changes as Versioned Assets

AI behavior changes when models, prompts, system instructions, tools, retrieval configurations, embeddings, indexes, or policies change. Treat each as a controlled artefact.

14.1 Release manifest

Asset	Version information
Application	Build identifier and source commit
Model	Provider, model identifier, version or deployment revision
Prompt	Template ID, version, hash, and owner
Retrieval	Index version, embedding model, chunking configuration, filters
Tools	Tool schema, API version, permissions, implementation revision
Policy	Policy version and approval record
Evaluation	Dataset version, test-suite version, results
Configuration	Environment-specific configuration revision
Infrastructure	Infrastructure-as-code commit and deployed resource versions

A production incident must be diagnosable against the exact combination of assets that produced the behavior. "The model was updated" is not an adequate change history.

15. Build an Evaluation System Before Scaling Usage

Evaluation is an engineering capability. It should exist from the earliest prototype and grow with the product's impact and risk.

15.1 Evaluation layers

Layer	What is evaluated	Example techniques
Component	Individual functions and interfaces	Unit tests, schema validation, deterministic tests
Retrieval	Evidence selection and ranking	Recall@k, precision@k, MRR, nDCG, source coverage
Model response	Correctness, relevance, completeness, groundedness	Expert review, rubric scoring, benchmark datasets
Safety and policy	Refusal behavior, leakage, policy compliance	Adversarial tests, red-team scenarios, policy tests
Tool execution	Correct tool selection and valid arguments	Contract tests, permission tests, simulated tools
Workflow	End-to-end task completion	Scenario tests, state-transition tests, failure injection
Operational	Latency, throughput, reliability, cost	Load tests, soak tests, chaos and resilience tests
Business	Adoption, cycle time, quality, financial impact	Controlled rollout, cohort analysis, process metrics

No single metric measures AI quality completely. Automated evaluators help scale assessment, but high-impact use cases require calibrated human review and independent validation.

15.2 Build representative evaluation datasets

An evaluation set should represent:

• common and high-volume requests;
• difficult edge cases;
• ambiguous or incomplete inputs;
• out-of-scope requests;
• adversarial and prompt-injection attempts;
• conflicting and stale source material;
• tenant and user permission boundaries;
• tool failures and partial execution;
• language, format, and accessibility variations;
• rare but high-consequence scenarios.

Govern evaluation data. Avoid leakage between development, tuning, and final validation datasets. Record provenance, version, sampling approach, known limitations, and approval.

15.3 Evaluation gates

A release must not pass merely because the average score improved. Define critical failure thresholds and non-negotiable controls.

Gate	Minimum evidence	Failure response
Functional	Required scenarios pass	Block release or reduce scope
Quality	Target metrics achieved on representative data	Improve the system or narrow supported use
Safety	No unresolved critical safety failures	Block release
Authorization	No unauthorized access or action in test suite	Block release
Resilience	Required failure modes handled	Remediate or document an approved limitation
Performance	Load and latency targets met	Optimize or adjust capacity
Cost	Unit economics within approved tolerance	Rework routing, context, or usage design
Operations	Monitoring, alerting, runbooks, and ownership ready	Block production promotion

Set thresholds per use case. Do not copy a generic score threshold across systems with different risk profiles.

16. Engineer Agentic Systems with Explicit Control Boundaries

Agentic systems add planning, state, tool selection, and multi-step execution. They introduce further failure modes: loops, excessive tool use, unauthorized actions, stale state, cascading retries, and unpredictable cost.

16.1 Agent execution controls

• Define the agent's permitted purpose and operating boundary.
• Register tools with explicit schemas, owners, versions, and permissions.
• Enforce authorization at the tool or policy boundary, not in the prompt alone.
• Set maximum steps, execution time, token budget, and tool-call budget.
• Make side-effecting operations idempotent where feasible.
• Require approval for consequential or irreversible actions.
• Record state transitions and tool results.
• Distinguish proposed actions from completed actions.
• Provide cancellation, timeout, and recovery mechanisms.
• Prevent untrusted tool output or retrieved content from silently changing system policy.
• Test tool failure, duplicate execution, partial completion, and conflicting results.

16.2 Agent workflow pattern

flowchart TD
    A["Receive Request"] --> B["Authenticate and Establish Context"]
    B --> C["Classify Intent and Risk"]
    C --> D["Select Bounded Workflow"]
    D --> E["Plan Next Step"]
    E --> F["Validate Tool and Action"]
    F --> G{"Policy Allows?"}
    G -- "No" --> H["Deny or Escalate"]
    G -- "Approval needed" --> I["Human Approval"]
    I --> J{"Approved?"}
    J -- "No" --> H
    J -- "Yes" --> K["Execute Tool"]
    G -- "Yes" --> K
    K --> L["Verify Result"]
    L --> M{"Task Complete?"}
    M -- "No, within limits" --> E
    M -- "Yes" --> N["Record Outcome and Respond"]
    M -- "Limit reached / failure" --> O["Stop Safely and Escalate"]

The model may propose the next step. The runtime and policy layer decide whether the step is valid, authorized, and within execution limits.

Key Takeaways: Part IV

• A product is the whole system, not the model or the chat interface.
• Write requirements that are measurable across quality, safety, authorization, resilience, and cost.
• Version every asset that can change behavior, and diagnose incidents against the exact combination.
• Gate releases on critical-failure thresholds, not average scores.
• Bound agents with budgets, approvals, and policy checks enforced by the runtime.

---

Part V. Integrate, Verify, and Release

Enterprise integration, layered testing, risk-matched release strategies, and a production-readiness decision turn a working build into a releasable service.

17. Integrate with Enterprise Systems Without Creating Competing Authorities

AI products interact with CRM, ERP, HR, finance, claims, contact-centre, document-management, and workflow systems. Integration design must preserve system-of-record boundaries:

• identify which system owns each authoritative entity;
• use approved APIs, events, or integration contracts;
• avoid duplicating authoritative business state in agent memory;
• validate all writes against enterprise rules;
• apply idempotency and transaction controls;
• propagate correlation IDs and audit context;
• handle retries, timeouts, rate limits, and partial failures;
• preserve tenant and user context end to end;
• define reconciliation and exception handling.

An agent's conversational memory is not a system of record. A vector index is not an authoritative customer database. A successful tool invocation is not proof that the business transaction committed. Verify the result from the authoritative system.

18. Apply a Layered Test Strategy

AI applications require conventional software testing and AI-specific evaluation. Neither replaces the other.

18.1 Test pyramid with AI evaluation overlay

flowchart TB
    A["Production Outcome and Operational Validation"] --> B["End-to-End Workflow and Scenario Tests"]
    B --> C["Integration and Contract Tests"]
    C --> D["Component and Unit Tests"]
    D --> E["Static Analysis, Security Scans and Schema Checks"]
    F["AI Evaluation Overlay: Quality, Grounding, Safety, Robustness, Cost"] -.-> A
    F -.-> B
    F -.-> C

18.2 Required test categories

Software correctness

• Unit, integration, contract, API, and workflow tests.
• Schema and data validation.
• Concurrency and state-management tests.
• Idempotency and duplicate-event tests.

AI behavior

• Answer quality and groundedness.
• Retrieval quality and source attribution.
• Prompt-injection resistance.
• Refusal and escalation behavior.
• Robustness to paraphrase, ambiguity, and malformed input.
• Consistency across model or prompt versions.

Security and privacy

• Authentication and authorization.
• Tenant isolation and cross-tenant leakage.
• Secrets and sensitive-data exposure.
• Prompt injection through user input, retrieved documents, and tools.
• Dependency and container vulnerabilities.
• Audit completeness and access-control enforcement.

Resilience and performance

• Provider timeout and outage.
• Throttling and rate-limit handling.
• Retrieval-store unavailability.
• Queue backlogs and burst traffic.
• Model fallback and degraded mode.
• Load, stress, soak, and recovery tests.

Operational and human factors

• User comprehension and trust calibration.
• Review workload and escalation effectiveness.
• Accessibility and usability.
• Support and incident procedures.
• Training and adoption readiness.

19. Define Release Strategies Appropriate to Risk

Match the release strategy to the impact of failure and the reversibility of the change.

Strategy	Appropriate use	Key controls
Internal preview	Early testing with trained internal users	Restricted data, clear limitations, feedback capture
Limited pilot	Bounded user group or process segment	Explicit scope, monitoring, human oversight
Canary	Gradual traffic shift	Automated health checks and rollback thresholds
Blue/green	Controlled environment switch	Data compatibility and rollback validation
Feature flag	Controlled activation by user, tenant, or feature	Authorization and audit
Shadow mode	Compare predictions without affecting decisions	Privacy controls and outcome comparison
Phased rollout	Expand by region, business unit, or workflow	Readiness evidence at each expansion gate

A pilot is not a production release with fewer users. It needs a defined hypothesis, scope, safeguards, measurement plan, exit criteria, and decision owner.

20. Establish Production Readiness

Production readiness is a decision supported by evidence, not a project status label.

20.1 Production-readiness checklist

Product and business

• Intended users, permitted use, and known limitations documented.
• Business owner accepts the workflow and operating impact.
• User guidance, training, and support channels ready.
• Outcome baseline and measurement instrumentation active.

Architecture and engineering

• Architecture decisions and exceptions recorded.
• APIs, event contracts, and dependencies validated.
• Capacity, latency, availability, and recovery requirements tested.
• Deployment and rollback procedures rehearsed.
• Configuration and artefacts versioned.

AI quality and risk

• Evaluation suite passes approved thresholds.
• Critical failure scenarios tested.
• Model, prompt, retrieval, and tool versions recorded.
• Human oversight and escalation paths operational.
• Risk owners have accepted residual risk through the correct process.

Security, privacy, and compliance

• Identity, authorization, and tenant isolation verified.
• Data use, retention, and deletion requirements implemented.
• Threat model and security testing completed.
• Audit and evidence retention configured.
• Applicable legal, regulatory, and contractual approvals obtained.

Operations

• Service owner and support rota identified.
• Dashboards, alerts, traces, and logs usable.
• SLOs and incident severity definitions agreed.
• Runbooks and escalation contacts tested.
• Backup, restore, disaster recovery, and provider-failure procedures validated where applicable.
• Cost budgets, quotas, and anomaly alerts configured.

A waiver must be explicit, time-bound, owned, and supported by compensating controls. Mandatory legal or security requirements cannot be waived by ordinary delivery governance.

Key Takeaways: Part V

• Verify results from the system of record; never trust agent memory or a successful call alone.
• Combine conventional testing with an AI evaluation overlay.
• Choose release strategies by blast radius and reversibility.
• Treat production readiness as an evidence-backed decision with explicit waivers.

---

Part VI. Deploy and Operate Production AI

Once released, an AI product is a production service. This Part covers deployment pipelines, service-level objectives, observability, and the operational discipline that keeps AI systems reliable.

21. Treat Deployment as a Controlled State Transition

Deployment changes the state of a live service. Manage it through reproducible pipelines and explicit promotion controls.

flowchart LR
    A["Source and Configuration"] --> B["Build and Package"]
    B --> C["Static, Dependency and Security Checks"]
    C --> D["Unit, Contract and Integration Tests"]
    D --> E["AI Evaluation and Risk Tests"]
    E --> F["Artefact Signing and Registry"]
    F --> G["Deploy to Test Environment"]
    G --> H["Acceptance and Operational Checks"]
    H --> I["Approval and Production Promotion"]
    I --> J["Canary / Phased Release"]
    J --> K["Monitor and Verify"]
    K --> L["Expand or Roll Back"]

The pipeline must enforce provenance: the artefact tested is the artefact deployed. Do not rebuild a different artefact after approval.

21.1 Environment strategy

Maintain environment separation appropriate to risk and platform:

• development;
• test and integration;
• evaluation and validation;
• pre-production or staging;
• production.

Use controlled data promotion. Production data should not be copied into lower environments without an approved purpose and appropriate protections. Environment parity should be sufficient to make test results meaningful, without unnecessary duplication of sensitive data.

22. Define Service-Level Objectives for AI Products

Infrastructure metrics are necessary but insufficient. AI products need service-level objectives (SLOs) that combine service health with AI-specific and business-relevant behavior.

22.1 Example SLO dimensions

Dimension	Example measure	Notes
Availability	Successful eligible requests / total eligible requests	Define exclusions explicitly
Latency	p50, p95, p99 end-to-end latency	Separate queue, retrieval, model, and tool time
Throughput	Requests or completed tasks per time unit	Include concurrency and burst profile
Quality	Evaluation pass rate or task success rate	Use representative, versioned test sets
Grounding	Supported claims or citation validity	Define scoring and sampling method
Safety	Critical policy violation rate	Some violations warrant zero tolerance
Tool reliability	Successful authorized tool executions	Distinguish tool failure from model failure
Cost	Cost per request and per completed task	Include retries and human review
Adoption	Active eligible users	Pair usage with outcome and repeat-usage measures
Business impact	Cycle time, error rate, conversion, or cost reduction	Establish causal attribution carefully

A single composite AI quality score can hide important regressions. Track critical dimensions independently and define non-compensating thresholds for safety and authorization, where a strong score elsewhere cannot offset a failure.

23. Implement End-to-End Observability

Observability must let teams reconstruct what happened across the application, model, retrieval system, policy service, tools, and human workflow.

23.1 Telemetry model

Capture, subject to privacy and data-minimization requirements:

• request and correlation identifiers;
• user, tenant, and workload identity references;
• application and release version;
• model provider, model version, and routing decision;
• prompt and configuration version identifiers;
• retrieval query metadata, source identifiers, ranking, and index version;
• policy decision and approval references;
• tool name, schema version, execution status, and duration;
• token usage, latency, retries, and estimated cost;
• safety and evaluation signals;
• human review outcome;
• authoritative transaction or workflow result.

Do not indiscriminately log raw prompts, documents, personal data, secrets, or model outputs. Define field-level redaction, access controls, retention, and sampling. Observability must not become an uncontrolled secondary data store.

23.2 Trace the complete request path

flowchart LR
    A["User Request"] --> B["API / Application"]
    B --> C["Orchestration"]
    C --> D["Retrieval"]
    C --> E["Model Call"]
    C --> F["Policy Check"]
    C --> G["Tool Call"]
    G --> H["System of Record"]
    D --> I["End-to-End Trace"]
    E --> I
    F --> I
    G --> I
    H --> I
    B --> I

A trace should make it possible to distinguish:

• incorrect source data;
• poor retrieval;
• model interpretation error;
• orchestration defect;
• policy denial;
• tool or downstream system failure;
• user or workflow error.

Without this separation, incident investigation is speculative.

24. Operate AI Systems with SRE Discipline

AI systems are production services. They require service ownership, incident management, capacity planning, reliability engineering, and continuous improvement.

24.1 Operational capabilities

• service catalogue and ownership;
• SLOs, service-level indicators, and error budgets;
• alerting and on-call procedures;
• incident, problem, and change management;
• capacity and quota management;
• backup, restore, and disaster recovery;
• vulnerability and patch management;
• model and provider dependency management;
• cost monitoring and optimization;
• user support and feedback management;
• runbooks and operational knowledge.

24.2 AI-specific incident categories

Incident category	Example	Initial response
Quality regression	Increased incorrect or unsupported answers	Reduce exposure; investigate model, retrieval, and configuration changes
Data leakage	Unauthorized information appears in output	Contain access, preserve evidence, invoke security and privacy process
Tool misuse	Agent performs an unintended operation	Disable or constrain the tool, verify transactions, assess impact
Provider outage	Model endpoint unavailable	Activate tested fallback or graceful degradation
Cost anomaly	Token or tool usage increases sharply	Apply quotas; investigate loops, routing, and traffic
Retrieval failure	Stale or incomplete knowledge returned	Revert index or ingestion change; restore source freshness
Latency degradation	Increased end-to-end response time	Isolate queue, model, retrieval, and downstream latency
Policy failure	Required approval or denial is bypassed	Disable the affected action path and invoke the control incident process

24.3 Safe degradation

Define what the product does when a dependency or control is unavailable. Examples:

• provide a non-AI workflow when the model is unavailable;
• answer only from a validated cache when freshness requirements permit;
• disable side-effecting tools when authorization cannot be verified;
• route to a human when confidence or evidence is insufficient;
• return a clear unavailable state rather than inventing an answer;
• stop agent execution when step, time, or cost limits are reached.

A fail-open approach is rarely appropriate for authorization, privacy, or consequential actions. Degradation behavior must be designed and tested, not improvised during an incident.

Key Takeaways: Part VI

• Deploy the artefact you tested, through a reproducible, gated pipeline.
• Define SLOs that include quality, grounding, safety, tool reliability, and cost, with non-compensating thresholds for critical controls.
• Instrument the full request path so failures can be attributed to data, retrieval, model, orchestration, policy, or tools.
• Design and rehearse safe degradation; fail closed for authorization and privacy.

---

Part VII. Govern Change Across the AI Lifecycle

AI behavior can change without a code release. This Part establishes change classification and drift management so that every material change is governed and traceable.

25. Establish AI Change Management

AI systems change through software releases, provider updates, prompt edits, retrieval changes, new tools, policy changes, data drift, and user behavior. Treat each material change as a governed lifecycle event.

25.1 Change classification

Change type	Examples	Expected control
Low impact	UI text, non-functional documentation	Standard software change
Configuration	Temperature, timeout, routing threshold	Versioning and regression tests
Prompt or orchestration	System instruction, workflow logic	Evaluation and approval based on impact
Retrieval	Chunking, embedding model, ranking, source set	Retrieval regression and permission validation
Model	Provider, model family, model version	Comparative evaluation, risk review, rollout controls
Tool	New capability, schema, side effect	Security review, contract tests, permission validation
Policy	Approval rules, access conditions, transaction limits	Formal policy approval and audit
Data	New source, changed classification	Data-owner and privacy review, retention review
Architecture	New platform boundary, shared service, tenant topology	Architecture decision and impact assessment

25.2 Model and prompt release discipline

A controlled model or prompt change includes:

1. reason for change and expected benefit;
2. affected products, workflows, users, and risk classes;
3. current and proposed versions;
4. representative evaluation results;
5. security, privacy, and policy impact;
6. performance and cost comparison;
7. deployment and rollback plan;
8. approvals and release evidence;
9. a post-release monitoring window.

A model update that materially changes behavior is not a routine dependency patch.

26. Manage Drift and Feedback

Production behavior can deteriorate even when no software changes. Data distributions, user behavior, source content, model providers, and business rules all evolve.

Monitor:

• input and workload distribution changes;
• retrieval coverage and source freshness;
• answer quality and task success;
• refusal, escalation, and override rates;
• tool selection and execution outcomes;
• model and provider behavior changes;
• cost and latency distribution;
• user feedback and adoption;
• outcome metrics and downstream error rates.

Govern feedback. A user rating is a signal, not ground truth. Human corrections may be useful for evaluation or improvement but should not automatically become training data or policy changes.

26.1 Feedback loop

flowchart TD
    A["Production Events and User Feedback"] --> B["Validate and Classify Signals"]
    B --> C["Identify Quality, Data, Policy or Workflow Issue"]
    C --> D["Create Governed Improvement Item"]
    D --> E["Change Design and Evaluation"]
    E --> F["Approve and Release"]
    F --> G["Measure Post-Change Outcome"]
    G --> A

Key Takeaways: Part VII

• Classify changes by what they can affect, and scale controls accordingly.
• Treat material model and prompt changes as releases, with comparative evaluation and rollback.
• Monitor for drift continuously, and route feedback through a governed improvement loop.

---

Part VIII. Manage Economics and Value Realization

Delivery is not the end of accountability. This Part covers lifecycle cost, credible measurement of benefits, and the governance that keeps the business outcome owner accountable after go-live.

27. Manage the Full Lifecycle Cost

AI cost is not limited to model tokens or endpoint charges. Total cost of ownership covers the whole system and operating model.

27.1 Cost categories

Cost category	Examples
Models and inference	Input and output tokens, hosted endpoints, accelerators, provider minimums
Retrieval and data	Storage, vector search, indexing, ingestion, document processing, data transfer
Application runtime	Compute, containers, serverless execution, queues, databases, networking
Tool execution	Enterprise API calls, workflow engines, external service charges
Engineering	Product, data, platform, ML/AI, security, architecture, QA
Evaluation	Human review, test infrastructure, benchmark maintenance
Governance	Risk assessment, compliance evidence, audits, approvals
Operations	SRE, monitoring, incident response, support, on-call
Change and lifecycle	Model upgrades, re-indexing, migration, retraining, decommissioning
Adoption	Training, process redesign, change management

27.2 Unit economics

Choose a unit that reflects delivered business value, not infrastructure activity. Examples: cost per completed claim, per resolved customer query, per approved invoice, per qualified lead, per validated document, or per completed employee task.

Cost per Completed Task =
    Total Attributable Lifecycle Cost
    / Number of Successfully Completed Tasks

The denominator must exclude failed or abandoned tasks if the measure claims to represent successful completion. Include human review and exception handling where they are part of the operating process.

28. Measure Benefits Without Confusing Activity with Value

AI activity metrics are not business outcomes. Token volume, prompt counts, model calls, and chatbot sessions may indicate usage; they do not prove value.

28.1 Benefits hierarchy

Level	Example measures
Adoption	Eligible users active, repeat usage, workflow coverage
Productivity	Time saved, reduced manual steps, cases handled per employee
Quality	Error rate, rework, completeness, decision consistency
Service	Resolution time, response time, customer satisfaction
Financial	Cost per transaction, operating expense, incremental revenue
Risk	Loss avoidance, control effectiveness, policy violations
Strategic	New products, improved decision speed, increased organizational capability

28.2 Establish credible attribution

Before deployment, record:

• baseline period and baseline performance;
• target population and comparison group, where feasible;
• process changes occurring alongside the AI introduction;
• seasonality and external influences;
• measurement owner and source systems;
• data quality limitations;
• expected time to benefit.

Use controlled experiments where feasible. Where randomization is impractical, use quasi-experimental approaches, matched cohorts, pre/post analysis with controls, or transparent contribution analysis. Do not attribute every post-deployment improvement to AI.

29. Apply Benefits Realization Governance

The business outcome owner remains accountable after go-live. Technology teams can report service health and usage; they cannot independently certify business value.

29.1 Benefits register

Field	Description
Benefit ID	Unique identifier
Initiative and product	Related delivery unit
Outcome	Specific business improvement
Baseline	Starting value and period
Target	Expected value and target date
Measurement source	System, report, or approved calculation
Benefit owner	Accountable business leader
Measurement owner	Person responsible for reporting
Realization status	Not started, emerging, on track, achieved, missed
Attribution confidence	High, medium, or low, with rationale
Dependencies	Process, adoption, data, or organizational changes
Corrective action	Intervention if benefits are not materializing

If adoption is high but benefits are absent, investigate workflow fit, quality, incentives, process design, and measurement. Do not respond by automatically increasing model usage.

Key Takeaways: Part VIII

• Count the full lifecycle cost, including evaluation, governance, operations, and change.
• Define unit economics around completed business tasks.
• Record baselines and attribution method before deployment.
• Keep the business outcome owner accountable for benefits after go-live.

---

Part IX. Scale Through Reuse and Organizational Capability

Scale depends on balancing shared guardrails with distributed accountability, building multidisciplinary skills, and treating adoption as delivery work.

30. Scale Reusable Capabilities Without Creating a Central Bottleneck

Centralization improves consistency and reduces duplication. Excessive centralization slows delivery and creates a queue for every product team. A balanced model centralizes enterprise guardrails and shared capabilities, and distributes product accountability and domain knowledge.

30.1 Federated operating model

Responsibility	Central platform / enablement	Domain product team
Model access and provider contracts	Shared gateway and approved patterns	Selects from supported options
Identity and security	Enterprise policy and controls	Implements standards and product-specific authorization
Evaluation framework	Shared tooling and common standards	Owns domain datasets and acceptance
Data governance	Enterprise policies and model	Owns domain data stewardship, quality, and permitted use
Product design	Patterns, coaching, reusable components	Owns user problem and workflow
Deployment	Standard pipelines and platform services	Owns release decision within controls
Operations	Shared telemetry and platform SLOs	Owns product SLOs and user support
Benefits	Portfolio measurement	Owns product outcome framework

30.2 Reuse decision criteria

Reuse a capability when it has:

• more than one credible consumer;
• a stable, understandable interface;
• a clear service owner;
• a support and lifecycle plan;
• a measurable adoption or efficiency benefit;
• appropriate isolation and configuration boundaries.

Do not generalize a one-off solution prematurely. Premature abstraction creates complexity without real reuse.

31. Build the Skills and Operating Model

Roadmap execution requires multidisciplinary capability. Hiring only prompt engineers or data scientists will not create a production AI organization.

Capability areas include:

• product management and domain process design;
• enterprise, solution, and AI architecture;
• software and distributed-systems engineering;
• data engineering and knowledge management;
• ML/AI engineering and model operations;
• security, privacy, legal, compliance, and risk;
• UX, human factors, and accessibility;
• SRE, platform engineering, and FinOps;
• change management, training, and business adoption.

31.1 Roles and accountabilities

Role	Core execution accountability
Executive sponsor	Strategic priority, funding, escalation
Business outcome owner	Benefit realization and process adoption
Product owner	Product value, backlog, acceptance
Enterprise architect	Enterprise alignment and cross-domain coherence
AI / solution architect	End-to-end AI design, boundaries, quality and risk controls
Engineering lead	Technical delivery and engineering quality
AI / ML engineer	Model integration, evaluation, optimization, lifecycle
Data engineer	Data pipelines, quality, metadata, lineage
Security / privacy / risk	Control requirements, review, assurance
SRE / platform engineer	Reliability, deployment, observability, capacity
Change lead	Training, communications, process transition

Small teams may combine roles, but accountability must remain explicit.

32. Manage Adoption and Process Change as Delivery Work

An AI product that is technically sound but ignored by users has not achieved its outcome. Adoption work includes:

• workflow redesign and role clarification;
• user research and usability testing;
• training based on actual tasks;
• clear explanation of system limitations;
• guidance on when to verify, override, or escalate;
• feedback channels and issue triage;
• management support and incentives;
• accessibility and language needs;
• measurement of sustained use and process outcomes.

For human-in-the-loop workflows, define the reviewer's authority, information, workload, response time, override mechanism, and accountability. A nominal approval step is not meaningful oversight if reviewers lack the time or evidence to decide.

Key Takeaways: Part IX

• Centralize guardrails and shared services; federate product accountability.
• Generalize only what has more than one real consumer.
• Staff for the full multidisciplinary system, not just models.
• Make human oversight real by giving reviewers authority, evidence, and time.

---

Part X. Govern Portfolio Execution

Governance should be frequent enough to resolve decisions and detect risk, and light enough to preserve delivery flow.

33. Establish a Multi-Level Governance Cadence

Forum	Typical cadence	Primary decisions
Product team review	Weekly or continuous	Backlog, quality, blockers, experiments
Architecture and engineering review	Weekly or fortnightly	Design decisions, standards, technical risk
Risk and control review	At defined gates and on material changes	Risk treatment, control evidence, approvals
Initiative steering	Monthly or milestone-based	Scope, funding, dependencies, escalation
AI portfolio review	Monthly or quarterly	Priority, capacity, investment, stop/scale decisions
Executive strategy review	Quarterly or when triggered	Strategic direction, risk appetite, material trade-offs

Cadence should reflect portfolio size, risk, and delivery velocity. High-risk systems may need more frequent operational and risk review.

34. Use Stage Gates as Evidence-Based Decisions

A stage gate is not a ceremonial approval meeting. It is a decision point at which the organization reviews evidence and chooses whether to proceed, constrain, remediate, pause, or stop.

34.1 Recommended gates

Gate	Decision	Required evidence
G0: Portfolio entry	Is the opportunity worth discovery?	Strategic alignment, initial value hypothesis, owner
G1: Mobilization	Is the initiative ready for bounded delivery?	Charter, scope, dependencies, risk applicability
G2: Architecture and data readiness	Is the proposed solution feasible and governable?	Architecture, data access, control design, cost assumptions
G3: Pilot readiness	Can the bounded pilot run safely?	Evaluation plan, safeguards, user scope, exit criteria
G4: Production readiness	Can the system be operated responsibly?	Test results, approvals, SLOs, runbooks, support
G5: Scale decision	Should exposure and investment expand?	Production evidence, benefits, risk, unit economics
G6: Lifecycle review	Continue, redesign, replace, or retire?	Value, quality, cost, risk, technical health

A gate can approve a constrained next step rather than the entire initiative. For example, an initiative may be approved for synthetic-data testing while production data access remains unresolved.

35. Govern Exceptions and Architecture Decisions

Exceptions are sometimes necessary, but they must not become invisible permanent deviations. An exception record includes:

• the requirement or standard being varied;
• business and technical rationale;
• alternatives considered;
• risk and impact assessment;
• compensating controls;
• accountable risk owner;
• approval authority;
• expiry date and review trigger;
• remediation or exit plan.

Architecture decision records (ADRs) capture material choices: context, options, decision, consequences, assumptions, and revisit conditions. Keep ADRs concise enough to remain useful and specific enough to support future change (template in §43).

36. Maintain Portfolio Transparency

A portfolio dashboard should show more than red/amber/green status.

View	Measures
Strategic alignment	Roadmap theme, outcome, capability contribution
Delivery	Milestones, scope, throughput, blockers, forecast
Dependencies	Critical path, owner, readiness, aging
Architecture	Decision status, exceptions, reuse, technical debt
Risk	Open critical risks, control evidence, residual risk
Quality	Evaluation trends, critical failures, defect escape
Operations	Availability, latency, incidents, recovery
Economics	Spend, forecast at completion, cost per task
Adoption	Eligible users, active use, workflow penetration
Benefits	Baseline, target, realized value, attribution confidence

A dashboard should expose the evidence behind status. A green indicator without a defined measure or source is not governance.

Key Takeaways: Part X

• Use a layered cadence from team to executive, scaled to risk.
• Make gates decisions on evidence, and allow constrained approvals.
• Give every exception an owner, an expiry, and a remediation plan.
• Show the evidence behind every status indicator.

---

Part XI. Sequence Delivery and Manage Uncertainty

Sequence work around value, dependency, readiness, risk, and learning rather than calendar order alone.

37. Sequence Work Around Outcomes and Dependencies

A typical sequence:

1. establish mandatory identity, data, security, and delivery controls;
2. build the minimum platform capabilities needed by the first products;
3. deliver a bounded product with measurable value;
4. validate operational and economic assumptions;
5. generalize proven components into shared capabilities;
6. expand to additional workflows and business units;
7. optimize performance, unit economics, and reliability;
8. retire redundant or superseded systems.

This is a pattern, not a universal sequence. If a mature platform already exists, the first product may proceed while selected gaps are remediated. If the data foundation is inadequate, data work may dominate the critical path.

38. Use Thin Vertical Slices

Avoid building all platform components in isolation before validating a real business workflow. Equally, avoid delivering a narrow interface that bypasses enterprise controls.

A thin vertical slice demonstrates a complete, bounded path:

• one user or system entry point;
• one workflow;
• a limited authoritative data set;
• one model or approved model route;
• a bounded retrieval or tool capability;
• deterministic authorization;
• basic observability and evaluation;
• a human escalation path;
• measurable outcome instrumentation.

The slice should be production-shaped even if it is not yet production-scaled.

38.1 Vertical slice versus horizontal build

Horizontal build	Thin vertical slice
Builds many layers independently	Proves a complete user-to-outcome path
Defers integration risk	Exposes integration risk early
Can create unused platform assets	Validates real consumer demand
Often delays feedback	Produces early evidence
May overfit to theoretical requirements	Grounds architecture in a real workflow

The best approach combines a thin vertical slice with deliberate investment in reusable foundations that are demonstrably needed.

39. Manage Uncertainty Explicitly

AI delivery contains uncertainty in model behavior, data quality, provider changes, user adoption, regulation, and economics. Classify it:

• Known constraint: an established fact, such as a data-residency requirement.
• Assumption: a plausible but unverified belief.
• Hypothesis: a proposition to test through an experiment.
• Dependency: an external capability or decision required.
• Risk: an uncertain event with potential impact.
• Decision: a choice that must be made by an accountable owner.

39.1 Assumption and risk register

Item	Type	Evidence	Owner	Trigger	Response required
Source documents are current	Assumption	Data profiling and owner confirmation	Data owner	Freshness below threshold	Remediate or constrain use
Model meets latency target	Hypothesis	Representative load test	Engineering	p95 exceeds SLO	Optimize, route, or change model
External API remains available	Dependency	SLA and resilience test	Integration owner	Repeated outage	Fallback or redesign
Users adopt the workflow	Risk	Cohort usage and workflow data	Product owner	Adoption below target	Process redesign and training

Do not convert uncertainty into false commitment. Use evidence gates, contingency plans, and explicit decision triggers.

Key Takeaways: Part XI

• Sequence by value, dependency, readiness, risk, and learning.
• Prove a production-shaped thin slice before scaling any layer.
• Name each uncertainty by type and attach an owner, trigger, and response.

---

Part XII. A Worked Enterprise Example

One initiative, end to end. The example is illustrative and prescribes no vendor or regulatory interpretation.

40. Executing an AI-Assisted Customer Operations Initiative

A financial services enterprise has approved a roadmap initiative to reduce customer-service handling time and improve response consistency. The product assists service agents by retrieving approved policy and account information, drafting responses, and initiating a limited set of authorized workflow actions.

40.1 Business outcome and scope

Item	Definition
Outcome	Reduce average handling time while maintaining or improving response quality
Users	Internal customer-service agents
Initial workflow	Policy and account inquiry support
Initial scope	Retrieval, answer drafting, source citations, selected read-only tools
Exclusions	Autonomous financial decisions, unrestricted account changes, unsupported external advice
System of record	Existing customer and account platforms
Human role	Agent verifies and sends the customer response
Success measures	Handling time, first-contact resolution, quality score, escalation rate, cost per resolved inquiry

40.2 Mobilization

The sponsor appoints a business outcome owner and product owner. The architect maps the customer-service application, identity provider, policy repository, account APIs, contact-centre workflow, and logging platform.

Discovery establishes:

• which policy repositories are authoritative;
• whether document-level permissions can be enforced;
• which account fields may be exposed to the assistant;
• what the existing agent workflow permits;
• expected concurrency and response-time requirements;
• baseline handling time and quality measures.

The initiative is authorized for a bounded pilot only after data access, user scope, and control requirements are agreed.

40.3 Target execution architecture

flowchart TB
    A["Contact Centre Desktop"] --> B["AI Assistance API"]
    B --> C["Workflow Orchestrator"]
    C --> D["Identity and Authorization"]
    C --> E["Knowledge Retrieval"]
    E --> F["Approved Policy Repository"]
    C --> G["Model Gateway"]
    G --> H["Approved Model"]
    C --> I["Read-only Account API"]
    I --> J["Customer System of Record"]
    C --> K["Human Review and Agent Workflow"]
    C --> L["Audit, Traces, Evaluation and Cost"]

The first release uses read-only account access. Any later write capability requires a separate authorization design, risk review, approval path, and release gate.

40.4 Delivery increments

Increment	Capability	Evidence
0	Data, identity, and workflow discovery	Source authority, permissions, baseline
1	Policy retrieval and citation	Retrieval evaluation and access-control tests
2	Response drafting	Quality, grounding, and safety evaluation
3	Read-only account context	API contract and authorization tests
4	Agent desktop integration	End-to-end workflow and usability validation
5	Bounded pilot	Adoption, quality, handling time, and incident evidence
6	Production release	SLOs, runbooks, support, approvals
7	Scale and optimize	Benefits, unit economics, and operational trends

40.5 Pilot decision

The pilot defines:

• a fixed user cohort and workflow boundary;
• a representative evaluation dataset;
• a human verification requirement;
• a clear fallback to the existing workflow;
• a time-bounded measurement period;
• stop conditions for critical leakage, policy violation, or unsafe behavior;
• expansion criteria based on quality, adoption, reliability, and unit economics.

Positive user reaction is not sufficient for scale. The initiative must show that the product improves the intended business outcome without unacceptable risk or operational burden.

40.6 Production and scale

After release, the team monitors:

• retrieval source coverage and freshness;
• response quality and citation validity;
• p95 end-to-end latency;
• tool availability and authorization failures;
• cost per resolved inquiry;
• agent adoption and override rates;
• handling time and first-contact resolution;
• incidents and customer-impact signals.

The portfolio review then decides whether to expand to more inquiry types, add write actions, improve retrieval, or pause expansion. Each change is evaluated separately against its risk and value.

---

Part XIII. Execution Artefacts and Templates

A mature programme maintains a coherent artefact set. Each artefact should support a decision, an engineering activity, an operational obligation, or measurable accountability. Do not produce documents for their own sake.

41. Minimum Execution Artefact Set

Artefact	Primary purpose	Owner
Initiative charter	Establish outcome, scope, ownership, and constraints	Sponsor / product owner
Product brief	Define users, workflow, value, and product boundaries	Product owner
Delivery backlog	Organize executable work and acceptance criteria	Product and engineering
Dependency register	Track cross-team and platform prerequisites	Delivery lead
Architecture overview	Communicate system context, boundaries, and deployment	Architect
ADR log	Preserve material technical decisions	Architect / engineering
Data and knowledge plan	Establish source authority, access, quality, and lifecycle	Data owner
Threat model and risk assessment	Identify threats, harms, and control requirements	Security / risk
Evaluation plan and results	Demonstrate AI quality and behavior	AI engineering / product
Test strategy and evidence	Demonstrate software and system correctness	Engineering / QA
Release plan	Control promotion, rollout, and rollback	Engineering / operations
Operational readiness pack	Establish support, SLOs, monitoring, and recovery	Service owner
Model and asset inventory	Track model, prompt, tool, index, and policy versions	Platform / governance
Benefits register	Track business outcomes and attribution	Business outcome owner
Cost model	Track lifecycle spend and unit economics	Finance / product
Risk and exception register	Track unresolved risk and deviations	Risk owner / architect
Product runbook	Support service operation and incident response	Operations

42. Product Backlog Structure

A useful backlog connects business outcomes to technical implementation:

A story is not complete merely because code has merged.

42.1 Example definition of done

• Acceptance criteria pass.
• Unit, integration, and contract tests pass.
• Relevant AI evaluation tests pass.
• Security and authorization tests pass.
• Telemetry and cost attribution are implemented.
• Failure and timeout behavior is defined.
• Configuration and assets are versioned.
• Documentation and runbooks are updated where needed.
• Product owner accepts the behavior.
• Required risk or architecture approvals are recorded.

43. Architecture Decision Record Template

# ADR-XXX: [Decision title]

## Status
Proposed | Accepted | Superseded | Rejected

## Context
What problem or constraint requires a decision?

## Decision drivers
- Business outcome
- Security and privacy
- Reliability and performance
- Cost and operability
- Interoperability and portability
- Reuse and maintainability

## Options considered
1. Option A
2. Option B
3. Option C

## Decision
What was selected and why?

## Consequences
- Positive consequences
- Negative consequences
- New dependencies
- Operational implications

## Controls and validation
How will the decision be tested and monitored?

## Revisit conditions
What evidence or event would justify revisiting the decision?

## Owners and approvals
Named decision owner, reviewers, and date.

44. AI Evaluation Plan Template

Field	Description
Product and version	Product build and release
Evaluation objective	Behavior or quality attribute being tested
Evaluation dataset	Source, version, sampling, and limitations
Test scenarios	Normal, edge, adversarial, and failure cases
Metrics	Defined measures and scoring approach
Thresholds	Pass/fail criteria and critical failure rules
Evaluator	Automated, human, or hybrid
Results	Scores, examples, and failure analysis
Known limitations	Gaps in coverage or evaluator reliability
Approval	Accountable acceptance authority
Regression trigger	Changes that require a rerun

45. Production Readiness Record Template

Field	Evidence
Service owner	Named accountable team
Business owner	Named outcome owner
Architecture	Approved overview and ADRs
Data	Sources, permissions, retention, lineage
AI assets	Model, prompt, retrieval, tool, and policy versions
Quality	Evaluation and regression results
Security	Threat model and test evidence
Risk	Assessment and residual risk acceptance
Performance	Load and latency results
Reliability	SLOs, failure handling, recovery
Operations	Dashboards, alerts, runbooks, support
Release	Deployment and rollback plan
Benefits	Baseline and instrumentation
Decision	Approve, conditionally approve, defer, or reject

46. Benefits Realization Template

Measure	Baseline	Target	Current	Source	Owner	Attribution confidence
Average handling time	TBD	TBD	TBD	Workflow system	Business owner	TBD
First-contact resolution	TBD	TBD	TBD	Contact-centre analytics	Business owner	TBD
Quality score	TBD	TBD	TBD	Quality review	Operations	TBD
Cost per completed task	TBD	TBD	TBD	Finance and usage telemetry	Finance	TBD
Adoption	TBD	TBD	TBD	Product analytics	Product owner	TBD

Populate values from verified enterprise baselines. Do not invent targets to make a business case look complete.

---

Part XIV. Common Execution Failure Modes

Most AI roadmap failures are predictable. This Part names them and pairs each with a corrective discipline.

47. Mistakes That Undermine AI Roadmap Execution

Failure mode	Why it fails	Corrective discipline
Treating the roadmap as a project plan	Strategic direction lacks executable ownership and acceptance	Create initiative charters and delivery backlogs
Starting with model selection	Ignores workflow, data, risk, and integration constraints	Start with outcome and system context
Building a large platform first	Creates cost and complexity without validated consumers	Build minimum reusable foundations around real demand
Running pilots without experiments	Pilots become permanent	Define hypotheses, exit criteria, scope, safeguards, and scale/stop gates
Measuring prompts and tokens as value	Activity does not prove business benefit	Measure completed tasks and business outcomes
Treating model evaluation as a one-time test	Behavior changes with data, models, prompts, and tools	Maintain versioned regression evaluation
Relying on prompt instructions for authorization	Model behavior is not a security boundary	Enforce policy and authorization deterministically
Logging everything	Creates privacy, security, and retention exposure	Apply minimization, redaction, and access controls
Allowing uncontrolled tool access	Creates excessive authority and side-effect risk	Use bounded tools, policy checks, and approval
Ignoring system-of-record boundaries	Creates inconsistent or unauditable business state	Keep authoritative transactions in enterprise systems
Centralizing every decision	Creates delivery bottlenecks	Centralize guardrails, federate product accountability
Decentralizing everything	Creates fragmentation and duplicated risk	Provide paved roads and governed exceptions
Scaling before operations are ready	Converts product defects into service incidents	Require operational readiness and SLO evidence
Ignoring unit economics	Usage growth may raise cost faster than value	Measure cost per completed task and lifecycle TCO
Treating adoption as training alone	Workflow and incentive problems remain	Redesign the process and measure sustained use
Failing to retire old capabilities	Creates duplicate cost and conflicting systems	Include decommissioning in lifecycle planning

48. Warning Signs That Require Intervention

Escalate when:

• no accountable business outcome owner exists;
• a critical data source has no owner or permitted-use decision;
• a system can execute consequential actions without external authorization;
• critical evaluation failures are accepted without explicit risk ownership;
• a pilot has no scale, stop, or transition decision;
• a production service has no operational owner;
• model or prompt changes are not traceable;
• costs rise without corresponding completed-task value;
• user adoption is reported without process outcome evidence;
• teams repeatedly bypass the paved road;
• architecture exceptions have no expiry or remediation plan;
• a roadmap initiative continues despite invalidated assumptions.

---

Part XV. The End-to-End Execution Method

A single method that ties the preceding Parts together, with a concrete first-90-days pattern.

49. A Practical Execution Lifecycle

Activities may overlap where dependencies and risk allow, but the decision logic stays explicit.

Step	Activity	Key outputs	Decision
1	Confirm roadmap mandate and scope	Strategic traceability and portfolio boundaries	Enter portfolio
2	Mobilize	Charter, owners, scope, baseline	Authorize discovery / mobilization
3	Validate assumptions and dependencies	Evidence, dependency register, risk log	Proceed, constrain, or stop
4	Establish architecture and data readiness	Architecture, data plan, ADRs, control design	Approve solution direction
5	Build minimum foundations	Paved road, identity, data, evaluation, telemetry	Approve bounded engineering
6	Deliver thin vertical slice	Integrated product path and initial evidence	Continue, redesign, or stop
7	Complete iterative increments	Tested product and release candidate	Approve pilot
8	Run bounded pilot	Adoption, quality, safety, cost, and workflow evidence	Scale, remediate, or stop
9	Complete production readiness	SLOs, runbooks, approvals, rollback	Approve production
10	Operate and realize value	Service health, user feedback, benefits	Continue and optimize
11	Review lifecycle and portfolio	Value, risk, cost, technical health	Expand, redesign, replace, or retire

49.1 Execution control loop

flowchart TD
    A["Roadmap Commitment"] --> B["Fund and Mobilize"]
    B --> C["Validate Readiness"]
    C --> D{"Ready for Investment?"}
    D -- "No" --> E["Resolve Gaps or Stop"]
    E --> C
    D -- "Yes" --> F["Build and Integrate"]
    F --> G["Evaluate and Verify"]
    G --> H{"Release Criteria Met?"}
    H -- "No" --> I["Remediate / Narrow Scope"]
    I --> F
    H -- "Yes" --> J["Deploy and Operate"]
    J --> K["Measure Value, Risk and Cost"]
    K --> L{"Continue, Scale or Change?"}
    L -- "Scale" --> B
    L -- "Improve" --> F
    L -- "Stop / Retire" --> M["Controlled Exit and Learning"]

50. First 90 Days: A Practical Mobilization Pattern

A 90-day plan is an example of how to establish execution discipline for an initial portfolio or first initiative. It is not a universal delivery promise.

Period	Primary focus	Expected outputs
Days 1–15	Mandate, ownership, portfolio inventory	Confirmed sponsor, outcome owners, initiative charters, governance cadence
Days 16–30	Readiness and architecture	Dependency map, data assessment, architecture outline, risk applicability
Days 31–45	Minimum platform and control foundations	Repository and CI/CD patterns, identity, evaluation harness, telemetry baseline
Days 46–60	Thin vertical slice	Integrated workflow, initial tests, user feedback, updated cost assumptions
Days 61–75	Bounded pilot preparation and execution	Pilot controls, evaluation results, operational procedures, adoption evidence
Days 76–90	Production decision and next portfolio cycle	Readiness assessment, benefit evidence, scale/remediate/stop recommendation

A complex or high-impact product may require substantially longer. The objective is not to force production deployment by day 90. It is to produce reliable evidence and make the next investment decision.

---

Part XVI. Measuring Execution Maturity

Use the maturity model to find where practice stands and what to strengthen next, and the success criteria to confirm that execution is working.

51. AI Roadmap Execution Maturity Model

Dimension	Level 1: Ad hoc	Level 2: Emerging	Level 3: Defined	Level 4: Managed	Level 5: Adaptive
Portfolio	Isolated experiments	Basic initiative tracking	Governed portfolio	Outcome and dependency management	Dynamic, evidence-based portfolio
Product	Demo-led	Named product owners	Product lifecycle defined	Continuous outcome management	Adaptive product portfolio
Architecture	Local decisions	Informal review	Reusable patterns and ADRs	Measured conformance and reuse	Architecture evolves from production evidence
Data	Unowned or fragmented	Initial source inventory	Governed data products	Quality and lineage monitored	Continuous data fitness management
AI evaluation	Manual spot checks	Basic test datasets	Versioned evaluation suites	Release gates and production feedback	Risk-calibrated continuous evaluation
Security and risk	Late review	Basic checklists	Embedded control requirements	Continuous assurance	Adaptive controls based on risk signals
Delivery	Manual deployment	Partial automation	Standard CI/CD	Reproducible, governed delivery	Optimized flow with automated controls
Operations	Reactive support	Basic monitoring	Defined SLOs and runbooks	SRE and incident learning	Resilience continuously optimized
Economics	Unclear cost	Basic usage tracking	Unit-cost model	Cost and benefit governance	Portfolio optimization by realized value
Adoption	Launch-focused	Training and feedback	Measured adoption	Workflow and behavior optimization	Continuous organizational learning

The maturity model is a diagnostic instrument, not a ranking exercise. A regulated, high-impact product may require advanced controls even if the organization is early in other dimensions.

52. Execution Success Criteria

An AI technology roadmap is being executed effectively when:

• every initiative has traceable business outcomes and accountable owners;
• dependencies and assumptions are visible and actively managed;
• shared platform capabilities have real consumers and service ownership;
• product teams follow secure, reproducible engineering practices;
• AI behavior is evaluated against representative evidence;
• authorization and policy controls are enforced outside model reasoning;
• systems are deployed with tested rollback and operational readiness;
• production behavior is observable across the complete request path;
• business value and lifecycle cost are measured;
• material changes are governed and traceable;
• lessons from production inform product and roadmap decisions;
• low-value or unsafe initiatives can be paused, redesigned, or retired.

---

Conclusion

Defining an AI technology roadmap establishes direction, capability requirements, architectural choices, investment logic, and sequence. Executing it turns those decisions into products, platforms, operating capabilities, and measurable business outcomes.

Execution must connect portfolio governance to engineering reality. Every initiative needs a clear business owner. Platform investments must solve real shared needs. AI behavior must be evaluated continuously. Model intelligence must stay separate from enterprise authority. Production systems must be operated with the rigor expected of any other critical enterprise service.

The essential sequence:

The roadmap is not complete when its initiatives are approved. It is not executed when a prototype works. It is not successful merely because a model is in production.

It is executed successfully when governed AI capabilities are adopted in real workflows, deliver measurable outcomes, operate within acceptable risk and cost, and can be sustained and improved over time.

---

Appendix A. Mapping the Guide to Existing Engineering Practice

Most engineering organizations already run mature practices that cover much of this guide. The table maps each guide concept to the established practice it extends, and states what is specific to AI. It uses generic practice names so that it applies regardless of internal tooling or terminology.

Guide concept	Established practice it extends	What is specific to AI
Execution contract and charter (§2)	Product requirements documents, design reviews, programme charters	Explicit evidence requirements, model and data risk owners, exit conditions
Decision rights (§3)	RACI and DACI models, ownership files, service ownership	Data-use, model-selection, and risk-acceptance decisions made explicit
Dependency register (§6)	Cross-team dependency tracking, platform service agreements	Dependencies on data products, model endpoints, and evaluation assets
Paved road (§8)	Platform engineering, golden paths, internal developer platforms	Model gateway, evaluation harness, and prompt and tool templates in the template set
Data foundations (§9)	Data governance, data contracts, lineage and catalog tooling	Document-level permissions, retrieval authority, deletion propagation to indexes
Control plane (§10)	Service mesh and policy engines, centralized authorization	Authorization outside the model; safe-to-answer versus authorized-to-act
Release manifest (§14)	Software bills of materials, build provenance, configuration management	Model, prompt, index, tool schema, and policy versions as release assets
Evaluation gates (§15)	Test gates in CI/CD, launch review criteria	Dataset-driven, non-deterministic evaluation; critical-failure thresholds
Agent controls (§16)	Least privilege, rate limits, circuit breakers, change approvals	Step, time, token, and tool-call budgets; proposed versus completed actions
Test strategy (§18)	Test pyramid, contract testing, chaos and load testing	AI evaluation overlay; prompt-injection and cross-tenant leakage tests
Release strategies (§19)	Canary, blue/green, feature flags, dark launches	Shadow mode for comparing model outputs; pilots with hypotheses
Production readiness (§20)	Production readiness reviews, launch checklists	Residual-risk acceptance and human-oversight validation
Deployment pipeline (§21)	Continuous delivery, artefact signing, promotion pipelines	Evaluation and risk tests as a promotion stage
SLOs (§22)	Service-level objectives, error budgets, SLIs	Quality, grounding, and safety SLOs with non-compensating thresholds
Observability (§23)	Distributed tracing, structured logging, metrics	Retrieval, model, policy, and tool spans; privacy-aware prompt logging
Incident response (§24)	Incident command, postmortems, runbooks	AI incident categories: leakage, tool misuse, quality regression, cost anomaly
Change management (§25)	Change advisory processes, risk-tiered change review	Model and prompt changes treated as behavior-changing releases
Drift management (§26)	Monitoring, anomaly detection, model monitoring in MLOps	Retrieval freshness, provider behavior change, governed feedback loops
Lifecycle cost (§27)	FinOps, cost allocation, unit economics	Token and inference cost, human-review cost, cost per completed task
Benefits realization (§29)	Experiment platforms, product analytics, benefits tracking	Attribution under concurrent process change
Federated model (§30)	Platform and product team split, inner-source models	Central guardrails and gateway; domain-owned datasets and acceptance
Governance cadence (§33–36)	Architecture review boards, investment committees, risk councils	Evidence-based gates specific to pilots, evaluation, and model risk

Alignment with external frameworks

The guide is compatible with widely used risk and management frameworks. The following are common reference points; confirm applicability and current versions with your risk and compliance functions.

• NIST AI Risk Management Framework for risk identification, measurement, and management across the AI lifecycle.
• ISO/IEC 42001 for an AI management system.
• OWASP Top 10 for LLM Applications for threat modelling of prompt injection, data leakage, and excessive agency (see §16, §18).
• DORA research metrics (deployment frequency, lead time, change failure rate, recovery time) as delivery-performance measures alongside the AI-specific measures in §22.
• Site reliability engineering practice for SLOs, error budgets, and incident learning (see §22, §24).

---

Appendix B. Glossary

Term	Definition
ADR	Architecture decision record: a concise record of a material design choice, its context, alternatives, and consequences.
Agent	A system that uses a model to plan and perform multi-step tasks, typically by invoking tools.
Authorization (policy)	A deterministic decision, made outside the model, on whether an action is permitted for a user in a context.
Benefits register	The record of expected and realized business outcomes, with baselines, owners, and attribution confidence.
Control plane	The set of services that manage policy, configuration, registries, and governance for AI workloads.
Evaluation gate	A release criterion based on evaluation evidence, including critical-failure thresholds.
Execution contract	The shared agreement on outcome, scope, accountability, constraints, dependencies, acceptance, evidence, change authority, and exit conditions.
Grounding	The degree to which a generated response is supported by authoritative evidence.
Model gateway	A service that abstracts model providers and applies routing, quotas, timeouts, and fallback.
Non-compensating threshold	A threshold that must be met independently; strength elsewhere cannot offset a failure.
Paved road	The supported, documented, secure, and observable path for building and operating AI products.
Release manifest	The record of versions of every asset that determines behavior in a release.
SLO	Service-level objective: a target for a measurable aspect of service behavior.
System of record	The system that holds the authoritative state for a business entity or transaction.
Thin vertical slice	A complete, bounded, production-shaped path from user entry to measurable outcome.
Unit economics	Cost per unit of delivered business value, such as cost per completed task.

---

Appendix C. Companion Reading

This guide should be read alongside Defining the AI Technology Roadmap.

• Defining establishes business outcomes, the opportunity portfolio, current-state assessment, target capabilities, architecture, technology direction, investment rationale, and roadmap sequence.
• Executing establishes mobilization, engineering, platform foundations, evaluation, release, operations, value realization, and continuous adaptation.

Together they form a complete management cycle:

The defining discipline determines what the organization should become and why. The execution discipline makes that future operational, measurable, and sustainable.