AI 底层逻辑 / 经典论文
AI Delivery Assurance：控制塔与发布就绪架构
AI delivery assurance 是把不确定的 AI 工作转化为阶段性证据、决策信心、残余风险归属和上线后学习的运行纪律。它不是把项目管理再包装成更多审批, 而是回答一个更硬的问题:
703 行ai-foundations/papers/155-ai-delivery-assurance-control-tower-release-readiness-architecture.md
AI Delivery Assurance / Control Tower / Release Readiness Architecture 解读
配对阅读：本篇的操作手册版（模板/RACI/门禁/runbook）是
docs/AI_DELIVERY_ASSURANCE_CONTROL_TOWER_RELEASE_READINESS_PLAYBOOK.md。第一遍读本篇建立原理与架构判断；第二遍做案例时再用 playbook 查表落地，两者不需要重复精读。
核心问题: AI initiative 如何从 discovery、pilot、release、scale 到 post-release assurance 被持续管理, 既能形成高管可信的 evidence-based control tower, 又不把治理变成低价值 bureaucracy。
重要说明: 本文是学习和内部架构训练材料, 不构成法律意见、监管解释、合规确认、审计意见、模型验证结论、风险接受决定、财务投资建议或生产上线批准。正式项目中的审批权、残余风险接受、监管沟通、审计依赖、客户影响判断和发布授权必须由机构授权角色结合司法辖区、产品、客户群、风险偏好、内部政策、模型风险、信息安全、隐私、供应商合同和运营能力确认。访问日期按 2026-06-30 记录。
Source Anchors
以下来源用于组织 AI delivery assurance、控制塔、架构描述、需求证据、工程绩效、可观测性和 SLO 语言。本文只将这些来源作为产品、架构和内部 assurance 的设计锚点, 不声称任何文档或 gate 自动构成法律、监管、审计或模型验证批准。
| Source | Official link | 本文采用的思想 |
|---|---|---|
| NIST AI Risk Management Framework | https://www.nist.gov/itl/ai-risk-management-framework | 用 Govern / Map / Measure / Manage 组织 AI 风险识别、度量、处置、监控和持续改进证据。 |
| ISO/IEC 42001 AI management system | https://www.iso.org/standard/81230.html | 用 AI management system 的 scope、policy、risk and opportunity、operation、performance evaluation、management review 和 improvement 设计 assurance operating model。 |
| ISO/IEC/IEEE 42010 Architecture Description | https://www.iso.org/standard/74393.html | 用 stakeholder、concern、viewpoint、architecture view、correspondence 和 rationale 组织 release readiness 视图与架构证据。 |
| ISO/IEC/IEEE 29148 Requirements Engineering | https://www.iso.org/standard/72089.html | 用 stakeholder need、requirement、information item、verification、validation 和 traceability 设计 evidence contract 与 acceptance criteria。 |
| DORA metrics | https://dora.dev/ | 用 deployment frequency、lead time for changes、change failure rate、failed deployment recovery time 的思想衡量 AI delivery flow 与 release quality。 |
| OpenTelemetry Documentation | https://opentelemetry.io/docs/ | 用 traces、metrics、logs、context propagation 和 semantic conventions 的思路设计 delivery telemetry、runtime evidence 和 release observability。 |
| Google SRE Service Level Objectives | https://sre.google/sre-book/service-level-objectives/ | 用 SLI / SLO / error budget 语言设计 AI 服务可靠性、质量、成本和安全运行阈值。 |
核心导读
AI delivery assurance 是把不确定的 AI 工作转化为阶段性证据、决策信心、残余风险归属和上线后学习的运行纪律。它不是把项目管理再包装成更多审批, 而是回答一个更硬的问题:
What evidence supports the next AI delivery decision,
what uncertainty remains,
who owns the residual risk,
and what production signal will prove the decision wrong?
控制塔的真正价值不是“看起来可控”, 而是让 discovery、pilot、release、launch、scale 与 post-release assurance 形成同一条证据链。对 AI 产品和架构而言, 这条链必须同时覆盖业务结果、流程采用、模型与提示词行为、RAG 来源、工具权限、运营容量、风险控制、成本、回滚和生产监控。
1. 问题定义: 从项目状态到决策可信度
AI 项目的失败经常落在两个极端之间。
| 极端 | 表现 | 后果 |
|---|---|---|
| Delivery theater | 周报、RAG status、committee、sign-off 很多, 但证据无法证明产品、架构、风险和运营准备 | 上线决策看似稳健, 实际依赖口头承诺和 slide narrative |
| Speed without assurance | 团队用 demo、offline score 或 sponsor pressure 推进 pilot / release / scale | 生产中出现客户伤害、运营队列爆炸、成本漂移、证据断裂和无法回滚 |
成熟的 AI delivery assurance 不是让所有团队填更多表, 而是建立一条 evidence-based decision chain:
Business problem
-> discovery evidence
-> pilot learning evidence
-> architecture runway evidence
-> release readiness evidence
-> launch control evidence
-> scale readiness evidence
-> post-release assurance evidence
-> portfolio learning and capability reuse
控制塔应该让管理层、交付团队、架构治理、风险控制和运营负责人同时看见:
- 每个 AI initiative 处于哪个 evidence stage, 而不是只看百分比进度。
- 哪些 readiness gate 已通过, 哪些只是条件性通过。
- 哪些 dependency 正在 burn down, 哪些仍会阻断 release。
- 哪些风险暴露在下降, 哪些只是被登记为 issue 但没有真正减少。
- 哪些 evidence object 足够支持 pilot、limited release、scale 或 stop。
- 哪些 residual risk 已由授权 owner 接受, 有 expiry、监控和补偿控制。
- 上线后的 quality、cost、safety、adoption 和 value 是否仍在可接受范围内。
低成熟度治理把 assurance 理解为更多审批、更多模板、更多 committee、更晚才让 risk / architecture / operations 参与、上线前集中补证据。结果通常是速度下降但风险没有下降, 业务方也会绕过流程。高级 assurance 的原则相反:
Evidence is generated as work happens.
Gates are decision points, not reporting ceremonies.
Risk tier determines depth.
Exceptions are visible, owned and expiring.
Telemetry replaces subjective confidence where possible.
Post-release learning improves future gates.
2. 架构模型: Control Tower 作为证据操作系统
Control tower 是一个跨产品、架构、风险、运营和价值的 evidence operating system。它不替代团队交付, 也不把所有判断集中到一个委员会; 它把关键证据对象、风险暴露、依赖关系、变更版本和决策记录连接起来。
Initiative portfolio
-> stage and decision state
-> readiness gate evidence
-> dependency / risk / issue telemetry
-> quality / cost / safety / value metrics
-> exception and residual risk registry
-> management action log
-> post-release assurance learning
2.1 Reference Architecture
Work management systems
Jira / Azure DevOps / roadmap / release calendar
|
Evidence registry
problem brief | PRD | architecture views | eval reports | runbooks | approvals
|
Risk and dependency engine
dependency graph | risk burndown | exception register | residual risk owner
|
Telemetry and observability
traces | metrics | logs | eval runs | adoption events | cost | incidents
|
Control tower analytics
stage health | readiness confidence | blocker aging | SLO | DORA | value realization
|
Decision forums
discovery council | architecture review | release readiness | scale review | post-release assurance
这个模型的关键不是 dashboard 技术栈, 而是数据对象的语义一致: 同一个 initiative、gate、evidence、risk、dependency、release bundle 和 decision record 能在不同视图中被追踪。
2.2 Core Objects
| Object | Minimum fields |
|---|---|
| Initiative | id、business capability、use case、owner、stage、risk tier、target outcome、current decision |
| Gate | gate id、stage、entry criteria、exit criteria、required evidence、decision owner、decision options |
| Evidence object | type、claim supported、source、version、owner、created date、validity period、quality rating、trace link |
| Dependency | upstream owner、delivery date、criticality、impact path、burn-down status、contingency |
| Risk | scenario、cause、impact、current exposure、treatment、target exposure、owner、burn-down evidence |
| Issue | realized problem、severity、customer/control impact、owner、resolution evidence |
| Exception | waived criterion、reason、residual risk、compensating control、owner、expiry、monitoring trigger |
| Release bundle | model、prompt、RAG index、tool contract、rules、workflow、feature flags、eval baseline、rollback path |
| Assurance metric | metric contract、definition、owner、threshold、source、decision use |
| Management action | action、owner、due date、evidence required、status、escalation route |
2.3 Decision Accountability Map
角色名称本身不重要, 重要的是每类判断都有明确 accountability。成熟控制塔至少需要覆盖这些责任面:
| Responsibility surface | Accountability |
|---|---|
| Outcome thesis | 业务目标、目标流程、采用假设、scale / stop 条件和价值叙事是否可信 |
| Requirements and workflow evidence | stakeholder need、acceptance criteria、exception path、human oversight 和 evidence traceability 是否闭合 |
| Architecture runway | 数据、模型、RAG、工具、身份、可观测、回滚和证据存储能力是否支持 release 与 scale |
| Eval and quality evidence | eval contract、regression、UAT、production sampling 和 critical failure disposition 是否充分 |
| Operations readiness | SOP、capacity、training、support、fallback、manual queue 和 incident route 是否准备好 |
| Risk and control evidence | risk tier、control evidence、exception、residual risk ownership 和 monitoring trigger 是否有效 |
| Value realization | baseline、unit economics、benefit recognition、cost leakage 和 post-release value review 是否可解释 |
3. 关键机制与生命周期
AI initiative 的 assurance lifecycle 可以分为七个阶段。每个阶段不是自动升级的里程碑, 而是为下一次决策购买证据。
| Stage | 核心问题 | 主要决策 |
|---|---|---|
| 1. Discovery assurance | 问题是否真实、值得做、适合 AI | fund discovery / stop / redirect to process or data fix |
| 2. Pilot assurance | AI 能否在受控范围内证明价值、风险和 adoption 信号 | enter pilot / extend learning / stop |
| 3. Architecture runway assurance | 支撑发布和规模化的架构能力是否存在或可交付 | build runway / limit scope / delay release |
| 4. Release readiness assurance | 产品、模型、prompt、RAG、tool、流程、运营和控制是否达到 limited go 条件 | release / conditional release / hold |
| 5. Launch assurance | 上线过程是否按批准范围、cohort、traffic、control 和 rollback 运行 | continue ramp / pause / rollback |
| 6. Scale readiness assurance | 生产证据是否支持扩大用户、场景、自动化或地区 | scale / restrict / redesign / stop |
| 7. Post-release assurance | 上线后价值、质量、成本、安全和风险是否持续成立 | continue / remediate / re-certify / retire |
3.1 Readiness Gate Taxonomy
Readiness gate 应按决策类型设计, 不是所有阶段使用同一 checklist。
| Gate | 目标 | Decision options |
|---|---|---|
| Opportunity gate | 确认问题真实、重要、适合 AI 或流程改造 | fund discovery / redirect / stop |
| Discovery gate | 确认 baseline、stakeholder、risk tier、data feasibility 和 value thesis | pilot / more discovery / stop |
| Pilot gate | 确认受控试点范围、evaluation、human oversight、runbook 和 learning plan | start pilot / shadow only / hold |
| Architecture runway gate | 确认数据、RAG、model gateway、tool gateway、identity、logging、rollback 能支撑 release | build / accept constraint / limit scope |
| Release readiness gate | 确认 release bundle、quality、safety、operations、cost、monitoring 和 rollback 具备 | go / conditional go / no-go |
| Launch gate | 确认生产 ramp 按批准范围运行, 监控正常 | continue / pause / rollback |
| Scale gate | 确认生产 value、adoption、quality、risk、cost 和 capacity 支持扩展 | scale / restrict / redesign / stop |
| Post-release assurance gate | 确认持续运行证据、incident learning、control effectiveness 和 benefits realization | continue / recertify / remediate / retire |
3.2 Discovery and Pilot Readiness
| Evidence area | Discovery strong evidence |
|---|---|
| Problem baseline | volume、cycle time、cost、quality、risk、complaint、manual effort 有数据或可解释样本 |
| Target user and workflow | 明确角色、流程步骤、case type、exception path 和 human decision rights |
| AI suitability | 比较 no-AI、process change、rules automation、AI assist、AI automation、vendor option |
| Risk tier | 按 customer impact、decision impact、data sensitivity、automation boundary 分级 |
| Learning plan | 写清最便宜可信的 pilot evidence、kill criteria 和 decision date |
| Evidence area | Pilot strong evidence |
|---|---|
| Pilot scope | cohort、channel、region、case type、risk tier、traffic cap、duration 明确 |
| Evaluation contract | golden scenarios、critical failures、acceptance criteria、reviewer calibration |
| Human oversight | 谁 review、何时 escalate、如何记录 override、怎样处理 disagreement |
| Data and privacy boundary | 数据来源、最小化、访问、日志、retention 和 redaction 明确 |
| Learning instrumentation | adoption、quality、cost、latency、risk、feedback、outcome events 已定义 |
3.3 Architecture Runway Evidence
Architecture runway 不是未来愿景, 而是 release 前必须存在或明确受限的能力。
| Runway capability | Evidence |
|---|---|
| Model gateway | route、version、fallback、cost tagging、policy enforcement、logging |
| Prompt registry | prompt version、owner、diff、approval、test linkage |
| RAG source authority | corpus manifest、ACL、freshness、lineage、citation and index version |
| Tool gateway | contract、permission tier、dry-run、idempotency、approval、action ledger、kill switch |
| Identity and entitlement | role mapping、least privilege、segregation of duties、service account control |
| Observability | trace coverage、metric contract、logs、dashboard、alert route |
| Evidence store | immutable or controlled evidence link、version、owner、retention class |
| Rollback path | artifact-level rollback for model、prompt、index、tool、rules、workflow |
3.4 Model / Prompt / RAG / Tool Change Readiness
AI 行为变化不只来自 code deploy。模型路由、prompt、RAG index、tool contract、阈值、业务规则和 workflow 都可能改变生产行为。
| Change surface | Readiness evidence |
|---|---|
| Model | intended use、limitations、eval delta、segment results、latency/cost impact、fallback |
| Prompt | prompt diff、policy boundary eval、tone and commitment review、output schema test |
| RAG | source manifest、critical document recall、citation accuracy、freshness test、ACL test |
| Tool | OpenAPI / AsyncAPI contract、permission scope、dry-run、approval flow、idempotency、audit log |
| Rules / thresholds | decision table diff、backtest、capacity impact、owner sign-off、rollback |
| Monitoring | metric definition、threshold rationale、alert test、sampling plan、runbook |
3.5 Launch and Scale Readiness
| Domain | Launch evidence |
|---|---|
| Product | release scope、user journey、feature flags、approved copy、known limitations |
| Quality | eval pass、UAT pass、critical failure zero or accepted with controls、defect disposition |
| Safety | prohibited behavior tests、red-team findings、customer harm route、escalation |
| Operations | SOP、training、support model、manual queue、fallback、incident contacts |
| Cost | cost per case、budget threshold、route optimization、p95 latency and capacity |
| Telemetry | production traces、version tags、dashboard freshness、alert routing |
| Rollback | drill outcome、decision authority、rollback sequence、customer remediation path |
Scale gate 必须比 launch gate 更严格, 因为 scale 放大了未知风险。
| Evidence | Scale question |
|---|---|
| Adoption durability | 用户是否持续在目标工作流中合格使用, 而不是 novelty effect |
| Quality stability | segment、case mix、risk tier、language、channel 是否稳定通过 |
| Value realization | benefit 是否扣除 review load、cost、rework、support 和 control overhead |
| Operational capacity | 人工复核、support、SRE、incident、manager coaching 是否能承接 |
| Control effectiveness | override、escalation、defect、complaint、incident 是否在阈值内 |
| Architecture scalability | 数据、RAG、tool、observability、vendor、cost 是否能承受更高负载 |
| Residual risk | 谁接受剩余不确定性, 到何时复核, 触发什么动作 |
4. 证据与控制: Evidence Contract、Release Bundle 和 Confidence
AI delivery confidence 不是 sponsor 信心, 也不是团队努力程度。它应来自 evidence-to-confidence chain:
Claim
-> evidence object
-> evidence quality
-> owner accountability
-> traceability
-> decision criterion
-> residual uncertainty
-> monitoring trigger
例如 "contact-center agent assist is ready for limited launch" 不是一个结论, 而是一组可检验 claim:
| Claim | Evidence |
|---|---|
| 目标 call reason 的答案 grounded | RAG retrieval eval、citation QA、policy source manifest |
| 员工能正确采用 | pilot adoption funnel、accept/edit/reject reason、QA sampling |
| 高风险话题不会越界 | prohibited advice eval、handoff trigger test、approved language review |
| 运营可以承接 | support runbook、supervisor capacity、fallback queue model |
| 成本可控 | cost per qualified interaction、latency p95、token budget |
| 出错可止损 | feature flag、model route fallback、knowledge index rollback、incident route |
4.1 Confidence Levels
| Confidence | Evidence standard | 可支持的决策 |
|---|---|---|
| Conceptual | 业务问题明确, 但证据主要来自 SME、market scan、专家判断 | discovery funding |
| Directional | 有 baseline、offline eval、prototype、small sample 或 limited user evidence | controlled pilot |
| Operational | 有 pilot telemetry、workflow evidence、control test、runbook 和 release path | limited launch |
| Production | 有真实生产 cohort、monitoring、incident response、benefit and risk evidence | scale decision |
| Declining | 上线后 adoption、quality、cost、risk 或 value 证据变差 | hold / rollback / redesign |
Directional confidence 不能包装成 Production confidence。控制塔要显示“证据足够支持什么决策”, 而不是显示“项目是否绿色”。
4.2 Assurance Scope
| Dimension | 典型问题 |
|---|---|
| Problem assurance | 问题、目标用户、流程痛点和 baseline 是否真实 |
| Value assurance | causal value logic、benefit register、unit economics 是否可信 |
| Requirements assurance | stakeholder need、acceptance criteria、human oversight 是否可追踪 |
| Architecture assurance | 数据、模型、RAG、工具、集成、可观测、回滚、证据是否具备 |
| Quality assurance | eval、UAT、regression、human review、production sampling 是否覆盖 |
| Safety and control assurance | customer harm、policy boundary、access、privacy、security、misuse 是否受控 |
| Operations assurance | SOP、training、capacity、support、fallback、incident route 是否准备 |
| Delivery assurance | dependency、risk、defect、decision、exception 是否可见并在下降 |
| Post-release assurance | 生产指标、SLO、DORA、value realization 和 corrective action 是否运行 |
4.3 Evidence Contract
Evidence object 是 control tower 的原子单元。没有 evidence contract, dashboard 会变成主观状态汇总。
| Field | Description |
|---|---|
| evidence_id | 稳定 ID, 可被 gate、decision、dashboard 引用 |
| evidence_type | baseline、eval、architecture view、risk memo、runbook、telemetry snapshot、decision record |
| claim_supported | 该证据支持哪个 readiness claim |
| source_system | Jira、Git、model registry、eval platform、observability、GRC、document store |
| owner | 对证据正确性负责的人或团队 |
| reviewer | 审阅证据的人, 不等于正式监管或审计批准 |
| version | 文档、模型、prompt、RAG index、tool contract、metric 或 dashboard version |
| creation_date | 证据生成时间 |
| validity_period | 证据在什么条件或时间内有效 |
| quality_rating | strong、adequate、limited、stale、contested |
| limitations | 适用范围、样本限制、confounder、known gap |
| trace_links | requirement、risk、control、test、release、runtime trace |
| decision_use | support discovery、pilot、release、scale、post-release review |
4.4 Evidence Object Library
| Object | Minimum content |
|---|---|
| Problem evidence brief | business problem、baseline、users、workflow、pain points、risk exposure |
| Outcome thesis | target outcome、AI role、human boundary、causal value logic |
| Option assessment | no-AI、process、rules、AI assist、automation、vendor、platform options |
| Requirements-to-eval map | stakeholder need、requirement、acceptance criteria、eval scenario、control link |
| Architecture view pack | context、data flow、model/RAG/tool、control、runtime、observability、rollback views |
| Release bundle manifest | model、prompt、index、rules、tool、workflow、feature flags、monitoring、eval baseline |
| Eval and regression report | dataset、rubric、segment result、critical failures、delta、reviewer notes |
| Operations readiness pack | SOP、training、capacity、support tier、fallback、incident route |
| Risk and exception record | risk scenario、treatment、residual risk、owner、expiry、monitoring |
| Dashboard metric contract | definition、source、calculation、threshold、owner、decision use |
| Post-release review | production metrics、incidents、complaints、adoption、cost、lessons、actions |
4.5 Evidence Quality Rubric
| Rating | Meaning |
|---|---|
| Strong | current、source-linked、versioned、reviewed、traceable to decision、limitations clear |
| Adequate | current and relevant, but sample size or review depth limited |
| Limited | useful for learning, not sufficient for release or scale decision alone |
| Stale | previous version、expired validity、changed context、or missing current production data |
| Contested | stakeholders disagree on interpretation、metric contract、source or sufficiency |
5. Dependency、Risk、Exception 的控制机制
5.1 Dependency Burn-Down
Dependency burn-down tracks conditions that must become true before release or scale.
| Dependency type | Example | Burn-down evidence |
|---|---|---|
| Data | KYC document metadata not available in onboarding workflow | data contract signed、sample validated、lineage visible |
| Architecture | tool gateway lacks write-action approval token | gateway deployed、contract test passed、audit trace verified |
| Operations | AML reviewer capacity cannot support pilot sampling | reviewer roster、queue simulation、SOP and schedule approved |
| Knowledge | policy corpus lacks current fee-waiver rules | source owner assigned、corpus manifest updated、retrieval eval passed |
| Vendor | model route lacks fallback in approved region | vendor review、route test、failover drill |
| Security | service account too broad for contact center RAG | entitlement review、least privilege evidence、access test |
| Finance | benefit baseline not agreed | baseline method、finance owner、unit economics model |
Dependency status should not be red / amber / green alone. It needs:
dependency
impact if late
owner
date needed
burn-down evidence
contingency
decision affected
5.2 Risk Burndown vs Issue Tracking
Risk burndown is not the same as issue closure.
| Concept | Definition | Example |
|---|---|---|
| Risk | A potential future harm or uncertainty | RAG may cite stale policy in customer service answers |
| Issue | A realized problem | QA found 4 stale policy citations in pilot |
| Risk treatment | Action intended to reduce likelihood or impact | source manifest、freshness monitor、citation QA、no-answer rule |
| Risk burndown evidence | Proof exposure is lower | stale citation rate falls、freshness SLO met、high-risk samples pass |
Weak dashboard:
Risk: stale policy answer
Status: amber
Action: monitor
Strong dashboard:
Risk: stale policy answer in fee-waiver customer conversations
Current exposure: 3.2% stale citation in pilot QA sample
Target exposure: below 0.5% and zero high-risk customer commitments
Treatment: source owner workflow, index freshness SLO, prohibited commitment eval
Burn-down evidence: 0 stale citations in last 150 high-risk samples, freshness p95 under 4 hours
Residual risk owner: Head of Servicing Ops
Review: next release readiness forum
5.3 Delivery Telemetry Schema
| Field | Meaning |
|---|---|
| initiative_id | AI use case or platform capability |
| stage | discovery / pilot / release / launch / scale / post-release |
| gate_id | current or next decision gate |
| risk_tier | low / controlled / material / high-impact internal classification |
| evidence_completeness | required evidence objects present and current |
| evidence_quality_mix | strong / adequate / limited / stale / contested counts |
| dependency_burn_down | open critical dependencies by age and owner |
| risk_burndown | exposure trend for top risks |
| issue_escape_rate | issues found after gate that should have been found before |
| exception_count | active exceptions, aging, expiry breach |
| quality_signal | eval and production quality trend |
| cost_signal | cost per task, budget burn, p95 latency |
| safety_signal | critical failures, policy violations, customer harm indicators |
| adoption_signal | qualified workflow adoption and override trend |
| value_signal | baseline-adjusted benefit evidence |
| decision_needed | fund / hold / release / scale / stop / remediate |
5.4 Exception and Residual Risk Ownership
Exceptions are not failure if they are explicit, owned, expiring and monitored.
| Exception | Example | Required controls |
|---|---|---|
| Evidence exception | A pilot has limited segment coverage but business wants controlled launch | scope restriction、monitoring、expiry、additional sample plan |
| Architecture exception | Tool gateway lacks full automation for one low-risk read-only action | compensating review、manual audit、target remediation date |
| Operations exception | Reviewer coverage is sufficient for pilot but not scale | traffic cap、queue dashboard、scale gate condition |
| Cost exception | Unit cost above target during learning phase | budget cap、route optimization plan、scale condition |
| Monitoring exception | New metric source delayed | interim manual sampling、reduced scope、expiry |
| Residual risk record field | Content |
|---|---|
| residual_risk_id | stable id |
| gate | pilot / release / scale / post-release |
| unmet criterion | readiness criterion not fully met |
| rationale | why proceeding is still considered acceptable internally |
| scope limit | cohort、volume、risk tier、region、time window |
| compensating control | human review、sampling、feature flag、manual reconciliation、extra monitoring |
| owner | business or risk owner accountable for residual risk |
| expiry | date or trigger when exception must be closed or reapproved internally |
| monitoring trigger | metric or event that forces pause / rollback / escalation |
| closure evidence | what will prove the exception is resolved |
Bad exception:
Proceed with risk accepted.
Good exception:
Proceed with 10% contact-center pilot only for card dispute status calls.
Residual risk: citation freshness metric is not automated.
Compensating control: daily manual source freshness sample and supervisor QA.
Owner: Servicing Operations Director.
Expiry: 14 days or before scale gate, whichever comes first.
Stop trigger: any unsupported policy claim in customer-visible response.
6. Quality、Cost、Safety、Reliability Gates
Release readiness should combine quality, cost and safety rather than optimizing one dimension.
| Gate family | Question | Example evidence |
|---|---|---|
| Quality gate | Does the AI produce acceptable outputs for intended workflows and segments? | eval score、critical failure count、human QA、segment regression |
| Cost gate | Is unit economics acceptable for the qualified value event? | cost per case、token budget、latency p95、review minutes、vendor cost |
| Safety gate | Are unacceptable harms prevented, detected, escalated and recoverable? | prohibited behavior eval、policy boundary、tool approval、complaint monitor |
| Reliability gate | Can the service meet operational expectations? | SLI/SLO、error budget、fallback test、incident route |
| Evidence gate | Can readiness claims be reconstructed? | release bundle、trace tags、decision log、evidence index |
6.1 SLO Thinking for AI
Google SRE-style SLO thinking helps avoid vague "monitor it" statements.
| SLI | Example SLO |
|---|---|
| Grounded answer rate | 99% of regulated policy answers cite an approved current source in target journeys |
| Retrieval freshness | 95% of policy documents available in RAG within 4 hours of approved source update |
| Tool write success | 99.5% of approved CRM follow-up task writes complete or fail safely with no duplicate |
| Human review timeliness | 95% of high-risk AI-assisted cases reviewed within defined operations SLA |
| Trace completeness | 99% of production AI interactions include model、prompt、RAG index、tool and release version tags |
| Cost per qualified case | p95 cost remains under agreed unit economics threshold for target workflow |
6.2 DORA Thinking for AI Delivery
DORA metrics need AI adaptation because behavior can change without code deployment.
| DORA concept | AI delivery adaptation |
|---|---|
| Deployment frequency | Count behavior releases: model route、prompt、RAG index、tool contract、threshold、workflow |
| Lead time for changes | Time from change request to production behavior under control |
| Change failure rate | Share of AI releases causing rollback、customer harm signal、critical defect、or control breach |
| Failed deployment recovery time | Time to restore acceptable behavior through artifact rollback、feature flag、route fallback or manual mode |
7. Operating Cadence and Dashboard
Control tower cadence should separate flow, readiness, risk and value conversations.
| Forum | Cadence | Main question | Decision |
|---|---|---|---|
| Delivery pulse | Daily / twice weekly | Are critical dependencies、defects or launch signals blocking work today? | unblock / escalate / reassign |
| Gate readiness review | Weekly | Which initiatives can move stage based on evidence? | pilot / release / hold / condition |
| Risk and exception review | Weekly or biweekly | Are residual risks、exceptions and KRIs inside internal appetite? | accept internally / restrict / remediate |
| Architecture runway review | Biweekly | Which shared capabilities are blocking multiple initiatives? | fund runway / sequence / de-scope |
| Value and adoption review | Monthly | Are production initiatives realizing benefits after cost and controls? | scale / stop / redesign |
| Executive control tower | Monthly | What decisions require leadership action? | fund / hold / rebalance / accept residual risk internally |
| Post-release assurance review | 24h / 72h / 14d / monthly | Did launch behave as expected? | continue / rollback / corrective action |
High-quality cadence produces actions, not meeting notes:
metric signal
-> interpretation
-> decision
-> owner
-> due date
-> closure evidence
7.1 Dashboard Sections
Control tower dashboard should support executive decision, delivery action and assurance review without mixing all details into one view.
| Section | Key visuals |
|---|---|
| Portfolio stage map | initiatives by stage、risk tier、decision needed |
| Readiness confidence | gate evidence completeness and quality heatmap |
| Dependency burn-down | critical dependencies by owner、due date、aging、impact |
| Risk burndown | top risks by exposure trend、treatment evidence、residual owner |
| Release queue | upcoming release gates、readiness score、open exceptions |
| Quality / cost / safety | eval pass、production defects、cost per task、latency、policy violations |
| Launch monitor | canary cohort、exposure、stop triggers、rollback readiness |
| Scale evidence | adoption durability、value realization、operational capacity、SLO trend |
| Exception registry | active exceptions、expiry、compensating control、owner |
| Management action log | overdue actions、escalation path、closure evidence |
7.2 Executive Confidence Narrative
Executives should not receive a traffic-light dashboard without explanation. A confidence narrative has this structure:
Decision requested:
Evidence supporting the decision:
Main uncertainty:
Residual risk owner:
Conditions:
Stop / rollback trigger:
Next evidence review:
Example:
Decision requested: approve limited release for KYC onboarding assistant in two digital onboarding queues.
Evidence: document completeness eval passed on target document types, pilot reduced rework by 14%, no unsupported final rejection recommendation, reviewer queue within capacity.
Main uncertainty: non-English document quality remains limited.
Residual risk owner: Retail Onboarding Operations Head.
Conditions: exclude non-English documents from this release, daily QA sample, no automated rejection.
Stop trigger: any customer-visible unsupported rejection or manual review queue breach.
Next review: 72-hour launch review and 14-day scale readiness review.
8. 金融零售场景
8.1 AML Triage Workbench
| Assurance area | Evidence |
|---|---|
| Discovery | alert aging、investigator workload、QA narrative defect、current escalation path |
| Pilot | shadow summaries、investigator edit rate、missed evidence rate、suspicious activity boundary |
| Release | case connector、source citations、analyst final disposition retained、reviewer SOP |
| Scale | alert aging reduction after review load、no QA regression、high-risk alert sampling |
| Post-release | SAR support quality、override reasons、typology drift、case reopen trend |
8.2 KYC Onboarding
| Assurance area | Evidence |
|---|---|
| Discovery | abandonment、manual review cycle time、document rework、customer chase reasons |
| Pilot | missing-document detection、false deficiency rate、reviewer disagreement、customer friction |
| Release | no AI final rejection、policy source version、appeal / recourse path、queue capacity |
| Scale | time-to-open improvement、first-pass completion、fraud/KYC control stability |
| Post-release | complaint tags、reviewer workload、segment quality、document distribution drift |
8.3 Payment Operations Reconciliation
| Assurance area | Evidence |
|---|---|
| Discovery | exception volume、reconciliation aging、write-off risk、manual root-cause pattern |
| Pilot | AI classification accuracy、suggested resolution quality、maker-checker workflow |
| Release | ledger write boundary、dual control、audit trail、idempotency and rollback |
| Scale | exception backlog reduction、no increase in incorrect adjustments、cost per resolved case |
| Post-release | settlement breaks、reversal rate、operational incident trend、evidence completeness |
8.4 Contact Center Agent-Assist
| Assurance area | Evidence |
|---|---|
| Discovery | call reason volume、AHT、hold time、repeat contact、QA failure themes |
| Pilot | source-grounded suggestions、accept/edit/reject reasons、policy boundary hits |
| Release | approved language、citation freshness、supervisor dashboard、fallback script |
| Scale | AHT and first-contact resolution improve without complaint or QA deterioration |
| Post-release | unsupported claim rate、source freshness、agent trust、cost and latency |
8.5 Regulatory Reporting Automation
| Assurance area | Evidence |
|---|---|
| Discovery | reporting cycle bottleneck、manual evidence gaps、maker-checker pain points |
| Pilot | variance draft quality、lineage reconstructability、reviewer correction patterns |
| Release | source-of-record mapping、metric contract、attestation boundary、evidence binder |
| Scale | close-cycle reduction、rework reduction、no unsupported calculation explanation |
| Post-release | lineage completeness、data change impact、reviewer sign-off quality |
8.6 Core Modernization AI Support
| Assurance area | Evidence |
|---|---|
| Discovery | legacy knowledge bottleneck、requirement ambiguity、defect leakage、SME scarcity |
| Pilot | code / rules explanation quality、requirement trace extraction、SME validation |
| Release | no autonomous production change、source repository boundary、architecture review |
| Scale | faster analysis cycles、lower rework、better traceability、controlled knowledge reuse |
| Post-release | hallucinated legacy rule incidents、adoption by modernization squads、evidence reuse |
9. 反模式
| Anti-pattern | Why it fails | Better practice |
|---|---|---|
| RAG status as assurance | Red / amber / green hides evidence quality and uncertainty | Gate-based evidence confidence and decision record |
| One release checklist for all AI | Low-risk internal copilot and high-impact customer decision support need different depth | Risk-tiered readiness taxonomy |
| Governance after build | Evidence is hard to reconstruct and architecture gaps appear late | Evidence generated from discovery onward |
| Issue list equals risk management | Closing tickets may not reduce risk exposure | Risk burndown with exposure and treatment evidence |
| Dependency list without impact | Teams cannot prioritize or escalate effectively | Dependency graph tied to gate decisions |
| Human review as magic control | Reviewers can be overloaded、inconsistent or unsupported | Capacity model、reviewer rubric、sampling and escalation evidence |
| Pilot success equals scale | Pilot cohort may hide cost、capacity、risk and adoption durability gaps | Separate launch and scale readiness gates |
| Exceptions without expiry | Residual risk becomes permanent | Exception owner、expiry、compensating control and trigger |
| Dashboard with no decision | Metrics become theater | Every dashboard section maps to decision or action |
| Post-release assurance ignored | Production evidence never updates gates | 24h / 72h / 14d / monthly learning loop |
10. 最终心智模型
AI delivery assurance should make four truths visible:
A working demo is not release readiness.
A successful pilot is not scale readiness.
A closed issue is not reduced risk.
A green status is not executive confidence.
最终模型可以压缩为一条运行链:
Stage decision
-> evidence contract
-> architecture runway
-> release bundle
-> dependency and risk burndown
-> quality / cost / safety / reliability gates
-> exception with residual risk owner
-> launch telemetry
-> scale or stop decision
-> post-release learning
高级 AI 产品与架构能力的关键不是把控制塔做得更漂亮, 而是让每一次推进、限制、回滚、扩展和退役都能被证据重建。控制塔要把 AI 不确定性转化为可讨论的证据、可执行的决策、可归属的残余风险和可复用的生产学习。
SOTA 状态标注 (2026-07-01)
本篇属于第二、三遍深读池（参考架构/深读笔记），未列入 12 周主线必读。时效基线为写作时点；引用前请按 CLAUDE.md 全局时效性硬规则复查最新进展。模块级 SOTA 对照见 docs/AI_SYSTEMATIC_LEARNING_ROADMAP_2026.md 各周「2026 SOTA 对照」行与文末「SOTA 检查」。