AI 扩展计划 / Playbooks

AI Incident / Postmortem / Reliability Playbook

当 AI 系统在生产中出现错误回答、错误检索、工具误用、PII 泄露、成本失控、延迟退化、模型漂移或 prompt injection 时，可靠性体系必须同时完成止血、影响界定、证据保全、复盘、纠正行动、防复发和治理升级。

1,088 行AI_INCIDENT_POSTMORTEM_RELIABILITY_PLAYBOOK.md

AI Incident / Postmortem / Reliability Playbook

核心导读：当 AI 系统在生产中出现错误回答、错误检索、工具误用、PII 泄露、成本失控、延迟退化、模型漂移或 prompt injection 时，可靠性体系必须同时完成止血、影响界定、证据保全、复盘、纠正行动、防复发和治理升级。学习主线：从 AI incident taxonomy、severity model 和 incident command 开始，逐步连接 containment / rollback、postmortem、corrective action register、reliability review board、game day、regulator / board communication 和 regression evidence。系统边界：本文不讲基础业务分析方法，也不把复盘简化成事故报告模板；重点是 AI-native failure 的识别、响应、证据、架构决策和治理闭环。证据输出：完成学习后，应能产出 incident taxonomy、severity matrix、war room runbook、rollback matrix、postmortem、corrective action register、game day decision log、board / regulator pack 和 reliability evidence binder。

重要说明：本文是学习与治理证据训练材料，不构成法律、监管、审计或正式安全意见。金融零售正式项目必须由 business owner、technology、security、privacy、legal、compliance、model risk、operational risk、internal audit 共同确认适用要求、通知义务、证据保留和客户补救。

1. 核心导读

传统事故管理问的是：

服务是否宕机，多久恢复，根因是什么，如何避免再发生。

AI 事故管理要多问五层：

AI 做错了什么
-> 哪个组件导致或放大了错误
-> 哪些客户、员工、案例、工具动作和监管义务受影响
-> 如何把系统恢复到受控状态
-> 如何把事故样本变成 eval、guardrail、架构门禁和治理证据

这份手册的核心观点：

AI reliability 不是模型准确率，不是 APM，也不是事后归档。它是一个从 detect、triage、command、contain、recover、postmortem、regression、governance 到 architecture decision 的运营闭环。

System Evidence Pack 的核心产出：

Evidence artifact	系统能力
AI Incident Taxonomy	能识别 AI-native failure，而不是只看 5xx 和 timeout。
Risk-Tiered Severity Model	能把客户伤害、监管风险、数据泄露、工具动作、成本和可用性合并分级。
Incident Command Runbook	能组织跨职能 war room，控制决策节奏和证据保全。
Containment / Rollback Matrix	能把模型、prompt、index、tool、policy、route、UI、vendor 变更恢复到受控状态。
Postmortem Template	能写出无责但有责任闭环的复盘。
Corrective Action Register	能把根因转成 owner、due date、control、regression evidence 和 residual risk。
Reliability Review Board Charter	能设计治理机制，而不是靠单个项目组临时判断。
Game Day / Tabletop Pack	能主动演练 bad retrieval、PII leak、tool misuse、policy bypass 和 cost spike。

2. Source Anchors

以下来源作为本文的外部锚点。正式项目应记录 access date、版本、内部 policy mapping 和 legal / risk sign-off。

Source	Official / primary link	本手册使用方式
NIST AI RMF	https://www.nist.gov/itl/ai-risk-management-framework	用 Govern / Map / Measure / Manage 组织 AI 风险识别、度量、处置和治理闭环；把 incident loop 接入 AI inventory、risk tier、monitoring 和 management reporting。
NIST GenAI Profile	https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence	用于识别 GenAI 独特风险：生成内容错误、数据泄露、滥用、供应商依赖、评测限制、lifecycle actor 和治理证据。
NIST CSF 2.0	https://www.nist.gov/cyberframework	用 Govern、Identify、Protect、Detect、Respond、Recover 的网络安全运营语言组织 AI incident response、恢复、证据和改进。
Google SRE Postmortem Culture	https://sre.google/sre-book/postmortem-culture/	用无责复盘、明确 postmortem 触发条件、根因与贡献因素、预防措施跟踪来设计 AI postmortem culture。
MITRE ATLAS	https://atlas.mitre.org/	用 AI attack tactics / techniques 组织 prompt injection、data poisoning、model evasion、exfiltration、tool misuse 等攻击路径和事件分析。
OECD AI Incidents Monitor	https://oecd.ai/en/incidents	用 AI incident / hazard、harm type、affected stakeholder、autonomy level、business function 等字段训练外部事件雷达和分类语言。
AI Observability / Cost / SLO Playbook	`docs/AI_OBSERVABILITY_COST_SLO_PLAYBOOK.md`	把 trace、span、quality dashboard、SLO、cost ledger 和 incident timeline 接入本文的响应流程。
AI Threat Modeling / Red Team / Agent Security Playbook	`docs/AI_THREAT_MODELING_RED_TEAM_PLAYBOOK.md`	把 threat model、red-team scenario、tool gateway、DLP、kill switch 和 tabletop 接入事故防复发。
AI Model Risk Management Playbook	`docs/AI_MODEL_RISK_MANAGEMENT_PLAYBOOK.md`	把 incident、model drift、eval regression、model change 和 validation issue 转成 MRM 证据。
AI Audit Evidence Binder Playbook	`docs/AI_AUDIT_EVIDENCE_BINDER_PLAYBOOK.md`	把事故日志、RCA、corrective action、approval、customer remediation 和 regression result 纳入审计证据包。

3. 高级框架：AI Reliability Operating Loop

AI 可靠性不是单点能力，而是一个运营循环。

1. Detect
   -> telemetry, eval regression, customer complaint, analyst override, cost alert, security alert

2. Triage
   -> classify failure type, severity, affected population, customer/regulatory impact, active blast radius

3. Command
   -> assign incident commander, scribe, domain owners, decision cadence, evidence preservation

4. Contain
   -> disable risky path, rollback prompt/model/index/tool/policy, route to safe fallback, pause automation

5. Recover
   -> validated fix, regression tests, staged restart, enhanced monitoring, customer remediation

6. Postmortem
   -> timeline, impact, root cause, contributing factors, detection gaps, response gaps, decision quality

7. Prevent recurrence
   -> eval cases, guardrails, architecture decision, release gate, SLO changes, ownership changes

8. Govern
   -> reliability review board, risk acceptance, audit evidence, board/regulator/customer communication

3.1 AI Incident 和普通 Incident 的差异

维度	普通生产事故	AI 生产事故
成功假象	HTTP 200 通常代表技术成功	HTTP 200 可能返回错误、幻觉、过期政策、越权工具建议或泄露内容
根因对象	code、infra、dependency、capacity	model、prompt、RAG、index、policy、tool、judge、human handoff、vendor route、feedback loop
影响范围	请求失败、服务不可用、数据丢失	客户误导、错误业务动作、监管陈述错误、PII 泄露、偏见、成本爆炸、模型风险
证据要求	logs、metrics、deployment history	trace、prompt version、model version、retrieved docs、tool calls、policy decisions、human review、eval result
恢复策略	rollback、reroute、scale、fix code	pin model、freeze prompt、rollback index、disable tool、force template、switch to human queue、re-run eval
防复发	tests、alerts、runbook	regression eval、golden set、adversarial cases、control mapping、architecture gate、review board

3.2 AI Reliability 的四个控制面

控制面	关注问题	关键 owner	典型证据
Product control plane	AI 被允许回答什么、建议什么、执行什么、何时升级人工	Product governance / Business Owner	use case boundary、risk tier、customer impact map、fallback policy
Architecture control plane	模型、RAG、工具、策略、日志、回滚、供应商如何被隔离和控制	AI Architect / Platform Owner	C4 diagram、component version map、kill switch design、route policy
Risk control plane	风险等级、验证、持续监控、残余风险、审计证据是否成立	Model Risk / Compliance / Audit	validation issue log、control evidence、risk acceptance、management attestation
Operations control plane	谁值班、谁指挥、如何止血、如何沟通、如何复盘	SRE / Incident Commander	incident state doc、timeline、postmortem、corrective action register

4. AI Incident Taxonomy

事故分类的目的不是贴标签，而是决定谁进 war room、先切断哪个路径、保留哪些证据、启动哪些通知和回归测试。

Category	典型症状	关键证据	立即 containment	防复发方向
Bad retrieval	RAG 找到过期、错误、低权限、跨租户或不相关文档	query、index version、retrieved doc IDs、ACL filter、citation IDs、freshness	回滚 index；禁用问题文档；切换权威搜索；强制引用校验；高风险问题转人工	source trust、ACL test、freshness SLO、retrieval eval、index release gate
Hallucination	模型生成未被证据支持的事实、政策、金额、日期、原因	prompt、model output、citation support score、judge result、human edits	限制答案类型；启用 answerability gate；高风险回复改模板；增加拒答	groundedness eval、citation checker、template locking、risk-tiered response policy
Policy bypass	模型绕过业务、合规、安全、销售、投资建议或投诉规则	policy decision log、guardrail result、prompt version、conversation	启用 blocklist / classifier；关闭自由生成路径；高风险 intent 强制升级	policy engine externalization、rule test suite、prompt conflict review
Tool misuse	Agent 调错工具、越权查询、重复执行、错误提交、错误更新 case	tool span、tool args hash、approval ID、side effect、idempotency key	关闭写工具；改 dry-run；冻结高风险 action；撤销错误动作；人工复核队列	tool permission matrix、step budget、dual control、idempotency、approval UX
PII / PCI / secrets leak	输出、日志、外发工具或供应商调用泄露敏感数据	output sample、DLP result、egress log、trace redaction、affected records	停止外发；启用 redaction；关闭日志明文；通知 privacy/legal；隔离供应商 route	data minimization、DLP regression、log masking、field-level policy、tenant isolation
Cost spike	token、retrieval、rerank、judge、tool 或 loop 导致成本异常	cost ledger、route policy、token counts、cache status、loop trace	启用 budget cap；降级模型；限制上下文；关闭 retry loop；rate limit	unit economics SLO、loop detector、quota、prompt size budget、cache policy
Latency degradation	TTFT、total latency、tool latency、queue wait、judge latency 超阈值	latency spans、queue metrics、vendor status、retrieval timing	降级模型；绕过非必要 judge；减少 top_k；启用 cached answer；切换 fallback	latency SLO、capacity test、route policy、timeout budget、dependency isolation
Model drift / eval regression	离线或线上质量分数下降，升级后行为改变	eval run ID、model alias、prompt/index version、golden set delta	pin 旧模型；停止 rollout；扩大 human review；回滚 route	canary eval、shadow test、model pinning、approval workflow、drift dashboard
Prompt injection	用户、文档、网页、邮件、tool output 诱导模型忽略规则或泄露	malicious payload、context labels、tool proposal、blocked/allowed decision	禁用受影响入口；强制 untrusted-context labeling；关闭外发和写工具	adversarial eval、instruction hierarchy、tool gateway、content isolation
Data / index poisoning	知识库、记忆、feedback、fine-tuning data 被恶意或错误内容污染	source change log、document owner、index manifest、memory write log	暂停 ingest；回滚 index；清理 memory；冻结 feedback training	source approval、data lineage、memory governance、poisoning red-team
Judge failure	LLM judge 或规则评测误判，使坏输出通过或好输出被误挡	judge prompt、rubric version、expert disagreement、score drift	暂停自动通过；改人工抽样；回滚 judge prompt；提高阈值	judge calibration、expert benchmark、rubric versioning、meta-eval
Human handoff failure	本应升级人工但未升级，或人工无法理解 AI 上下文	escalation log、queue SLA、handoff payload、review notes	强制人工队列；停止自动完成；补发 case context	warm handoff spec、queue SLO、review training、override analytics
Vendor / dependency incident	LLM、embedding、vector DB、tool API、MCP server、cloud 出现异常或变更	vendor status、model version、contract notice、route log	切换 vendor；降级本地模型；禁用第三方 connector；冻结新请求	vendor exit plan、multi-provider route、contract controls、dependency game day

4.1 事件不是事故的边界

Signal	何时只是 event	何时升级为 incident
单个差评	用户主观不满意，未违反政策或证据	命中高风险 intent、客户受损、同类模式重复、人工判定错误严重
Eval 下降	低风险指标轻微波动，未进生产	高风险 golden set fail、已上线版本退化、release gate 被绕过
成本升高	预算内季节性流量	单 use case 超预算、loop / abuse、单位经济性失效、影响服务能力
Prompt injection 被拦截	控制按预期工作	拦截率异常升高、出现未拦截样本、攻击者触发工具提议或数据外泄
客户投诉	普通服务投诉	投诉指向 AI 误导、拒绝人工、错误承诺、错误费用、歧视或隐私问题

5. Severity Model：AI 事故分级

Severity 必须按影响和控制失效，而不是按团队紧张程度。建议将金融零售 AI 事故分为 SEV0 到 SEV4。

Severity	定义	触发条件	决策权限	默认动作
SEV0 - Critical AI Harm	已造成或极可能造成重大客户伤害、监管违规、重大数据泄露、资金损失、系统性错误业务动作或董事会级声誉风险	PII / PCI 大规模泄露；客户资金或账户状态被错误改变；监管报告/AML/SAR 重大错误；高风险 agent 自动执行失控；跨租户数据暴露	Executive incident lead + Legal / Risk / CISO / Business head	立即停用相关 AI path；保全证据；启动客户补救、法律评估、监管沟通准备和董事会通知
SEV1 - High Impact	影响受监管流程、客户可见输出或高风险员工决策，但可通过快速 containment 限制范围	错误 KYC / AML 建议批量出现；客服 AI 给出错误费用/权利说明；高风险工具绕过审批但未大规模执行	Incident Commander + Business Owner + Model Risk / Compliance	冻结变更；回滚组件；扩大人工复核；24-48 小时内完成初版 postmortem
SEV2 - Material Degradation	质量、安全、延迟、成本或可用性显著退化，影响业务效率或中风险流程	groundedness 低于阈值；retrieval freshness failure；p95 延迟超 SLO；单位成本超预算；人工拒绝率异常	IC + Product / Platform Owner	降级或分流；开 corrective action；一周内完成复盘和回归证据
SEV3 - Controlled Issue	局部问题，已被 guardrail、人审或监控捕获，没有客户实质影响	bad retrieval 被 answerability gate 拦截；prompt injection 被阻断；低风险 internal copilot 错误	Product / Ops Owner	记录 issue；转 regression case；下次 review board 汇总
SEV4 - Learning Signal	早期 hazard、近失误、桌面演练发现、离线 eval 失败	tabletop 发现 runbook 缺口；red-team 样本失败；外部 incident radar 触发内部 review	Reliability owner	更新测试、runbook、training 和 backlog

5.1 Severity 判定维度

Dimension	Low	Medium	High	Critical
Customer harm	无客户影响	客户困惑或轻微不便	客户权益、费用、投诉、账户操作受影响	大规模客户损害、资金或法律权利受影响
Regulatory exposure	无监管流程	间接流程	AML/KYC/credit/complaints/adverse action 受影响	监管报告、法定义务、消费者保护严重风险
Data sensitivity	无敏感数据	内部数据	PII、交易、身份、信用、AML 数据	大规模 PII/PCI、跨租户、外部泄露
Automation level	read / summarize	recommend / draft	act with approval	act without effective approval 或不可逆动作
Blast radius	单请求	单团队或小样本	多业务线、客户群或高价值客户	系统性、跨租户、跨区域
Detectability	自动拦截	监控发现	客户/员工发现	外部媒体、监管、审计或重大投诉发现
Reversibility	可立即修正	可补救	补救成本高	难以逆转或需客户/监管通知

5.2 AI-Specific SEV Escalation Rules

Trigger	最低 SEV
客户可见 AI 输出虚构费用、资格、拒绝原因、投诉权利、投资/信贷建议	SEV1
AI 工具调用改变客户账户、交易、退款、case 状态，且审批或幂等失效	SEV1；若不可逆或批量执行则 SEV0
PII / PCI / secrets 出现在客户可见输出、外部工具、供应商 ticket、日志导出	SEV1；大规模或跨租户则 SEV0
AML / KYC / fraud / credit 高风险流程中的 AI 建议系统性错误	SEV1；若影响监管提交或客户权益则 SEV0
Prompt injection 导致工具提议、数据外泄、policy bypass 或持久化污染	SEV1
Eval regression 已进入生产且影响高风险 use case	SEV1
成本异常导致预算熔断、服务降级或其他客户流程受影响	SEV2；若影响关键服务可升 SEV1

6. Incident Command：AI War Room 设计

AI 事故不能只由工程值班处理，因为根因可能是业务政策、模型版本、知识库、工具权限、人工流程或供应商变更。War room 需要明确角色，而不是所有人同时发言。

6.1 角色与职责

Role	主要职责	关键决策
Incident Commander	控制节奏、分配任务、设定更新频率、做最终 operational decision	是否升级 SEV；是否停用 AI path；是否宣布 containment 完成
Scribe / Evidence Lead	记录 timeline、decision log、证据链接、版本和行动项	哪些证据进入 postmortem 和 audit binder
AI Product Lead	定义客户/业务影响、fallback experience、补救路径和用户沟通	是否关闭某类 intent；是否切人工；如何处理受影响用户
AI Architect / Platform Lead	判断组件边界、回滚路径、route policy、日志保全和技术恢复	rollback model / prompt / index / tool / policy / vendor 的顺序
Model Risk / Validation	判断模型风险、eval regression、验证缺口和风险接受	是否需要 re-validation；是否阻止 restart
Security Lead	判断 prompt injection、exfiltration、tool abuse、credential / connector 风险	是否启动 security incident；是否隔离 connector / secrets / MCP server
Privacy / Data Protection	判断 PII / PCI / sensitive data 暴露、通知义务和数据最小化缺口	是否需要 privacy incident workflow
Legal / Compliance	判断监管、合同、客户通知、投诉和保存义务	是否准备 regulator response、legal hold 或 board escalation
Business Owner	判断业务影响、客户补救、人工产能和 residual risk	是否接受降级服务；是否暂停流程
Customer / Operations Lead	执行客户支持、人工队列、分行/坐席指引、话术	如何识别和联系受影响客户
Vendor Owner	协调模型/云/工具供应商、SLA、变更说明和证据	是否切换 vendor；是否触发合同 incident notice

6.2 War Room 节奏

阶段	时间目标	产出
First 15 minutes	确认 SEV、IC、scribe、受影响系统、初始 containment owner	Incident state doc、bridge channel、decision cadence
First 30 minutes	锁定 blast radius、证据保全、是否关闭高风险路径	Impact hypothesis、freeze decision、initial customer / regulatory exposure
First 60 minutes	执行 containment、确认 fallback、开始受影响对象识别	Containment decision、rollback record、affected population query
First 4 hours	验证 containment 有效、准备内部更新、定义补救策略	Stable state declaration、executive update、remediation backlog
First 24 hours	初版 postmortem、customer/regulator/board communication draft、regression plan	Draft RCA、corrective action register、communication pack
First 5 business days	完整复盘、review board、risk acceptance 或 restart approval	Final postmortem、evidence pack、board / regulator-ready summary

6.3 Decision Log 最小字段

Field	示例
Decision ID	DEC-2026-AML-001
Time	2026-06-29 14:35 America/Chicago
Decision	禁用 AML copilot 的 free-form SAR narrative 生成，仅保留 evidence summary
Decision owner	Incident Commander + AML Business Owner
Rationale	发现 4 个高风险 case 的 narrative 引用了过期 typology；RAG index rollback 尚未完成
Alternatives considered	仅提高 judge 阈值；全量停用 copilot；切换旧 index
Risk accepted	分析员效率下降，但避免错误 narrative 进入提交流程
Evidence link	trace IDs、index manifest、affected case query、approval record
Revisit condition	新 index 通过 50 个 regression cases 且 Model Risk 批准 restart

7. Containment / Rollback 决策矩阵

AI containment 的原则：

先停止伤害，再恢复能力；
先切断高风险动作，再修复回答质量；
先保全证据，再清理状态；
先验证 rollback，再重新放量。

7.1 组件级 Rollback Matrix

Suspected component	典型信号	最快 containment	可靠恢复条件
Model	同一 prompt / index 下输出风格、拒答、准确性突然变化	pin previous model；切备用 provider；提高 human review	Canary eval 通过；供应商变更说明复核；高风险 sample 专家复核
Prompt	新版本后 policy bypass、格式错误、拒答下降	rollback prompt template；锁定 approved response templates	Prompt diff review；regression eval；policy owner sign-off
RAG index	错误引用、过期政策、权限错配、freshness failure	rollback index；禁用受污染 source；改权威门户 fallback	Index manifest 校验；ACL test；source owner approval；retrieval eval
Retriever / reranker	召回相关性下降、低信任文档上位	降级到 hybrid search；调整 top_k / filters；禁用 reranker	Retrieval benchmark 回归；low-trust source penalty 验证
Tool gateway	越权 tool proposal、approval bypass、重复执行	关闭写工具；只读模式；强制 manual approval	Permission matrix review；idempotency test；approval trace sample
Policy engine / guardrail	拦截突然下降或误拦截上升	回滚 rule set；启用 conservative mode；强制 escalation	Policy test suite；false positive / false negative review
Memory / feedback loop	污染记忆、错误偏好、重复错误学习	暂停 memory read/write；清理污染记录；禁用 online learning	Memory lineage review；write policy test；sampling approval
Judge / eval	坏输出通过、好输出被拒、分数漂移	暂停自动通过；转人工抽样；回滚 judge prompt	Expert calibration；judge benchmark；threshold review
UI / disclosure	客户误解 AI 能力、无法转人工、错误提示	改静态文案；强制人工入口；关闭高风险 flow	UX content approval；complaint trigger test；accessibility check
Vendor dependency	延迟、输出异常、版本漂移、SLA 告警	Route away；降级本地或备用模型；暂停 connector	Vendor RCA；contract notice；internal canary；exit plan review

7.2 Failure Type 到 Containment

Failure	First containment	Second containment	Recovery gate
Bad retrieval	回滚 index 或禁用 source	高风险问题强制人工	通过 retrieval golden set、ACL test、freshness check
Hallucination	限制自由生成；启用模板	降级为引用摘要或拒答	groundedness >= threshold；expert sample pass
Policy bypass	启用 conservative policy	暂停相关 intent	policy regression suite pass；compliance sign-off
Tool misuse	禁用写工具	撤销 / 补偿 side effect；重放 idempotency check	tool permission test；approval audit sample
PII leak	停止外发和明文日志	识别 affected records；privacy workflow	DLP regression pass；redaction evidence；legal decision
Cost spike	预算熔断和 rate limit	降级模型、缩短上下文、停 retry loop	unit cost 回到 SLO；loop detector verified
Latency degradation	route to fast path	关闭非必要 judge / rerank / long context	p95 latency stable；capacity margin confirmed
Model drift	pin previous model	增加人工复核	canary eval + shadow traffic pass
Prompt injection	隔离入口和不可信内容	关闭工具 / 外发 / memory write	adversarial regression pass；security approval

7.3 Kill Switch 分层

Switch level	作用范围	适用场景	风险
L1 Response mode switch	从生成回答切到模板、拒答或引用摘要	hallucination、policy bypass、customer-facing risk	用户体验下降，但业务不中断
L2 Intent switch	关闭某类 intent，例如投诉、退款、KYC exception	单个业务 flow 失控	人工队列压力上升
L3 Tool switch	禁用写工具、外发工具、高风险查询工具	tool misuse、prompt injection、PII leak	自动化收益下降
L4 Component switch	回滚 model、prompt、index、policy、judge	组件变更引发回归	可能回到旧缺陷，需要回归验证
L5 System switch	停用整个 AI capability	SEV0 或无法确认 blast radius	业务影响最大，但止血最确定

8. Detection and Evidence：没有证据就没有复盘

AI incident 的证据必须能回答：

谁问了什么
系统看到了什么
模型/检索/工具/策略如何决策
输出影响了谁
我们何时知道、何时止血、何时确认恢复

8.1 Detection Signals

Signal	检测来源	可触发的 incident 类型
Human reject spike	QA / reviewer decision	hallucination、bad retrieval、policy bypass、model drift
Citation support failure	groundedness judge、citation checker	hallucination、bad retrieval、index poisoning
Tool policy denied spike	tool gateway	prompt injection、excessive agency、permission issue
DLP hit	output filter、egress gateway、log scanner	PII leak、secrets leak、cross-tenant data issue
Cost per case spike	cost ledger	loop、long-context expansion、abuse、vendor price/route change
p95 / p99 latency breach	tracing / APM	vendor degradation、retrieval slowdown、judge bottleneck
Eval regression	CI / release gate / scheduled eval	model drift、prompt regression、index regression
Customer complaint	contact center、complaints system、social channel	customer-facing hallucination、handoff failure、wrong advice
Security alert	SIEM、WAF、MCP gateway、red-team monitor	prompt injection、tool abuse、exfiltration
External incident radar	OECD monitor、industry reports、vendor notices	hazard review、third-party dependency risk

8.2 Evidence Pack 最小内容

Evidence	字段
Incident state doc	SEV、status、IC、scribe、systems、affected use cases、current containment
Timeline	detection、triage、decision、rollback、recovery、communication、restart
Trace sample	trace IDs、request IDs、user role、workflow step、risk tier
Version map	model、prompt、index、retriever、reranker、judge、policy、tool schema、UI copy
Impact query	affected customers / cases / employees / transactions / outputs
Tool side effect log	tool name、arguments hash、approval ID、result、reversal / compensation
Data exposure assessment	fields exposed、records count、destination、retention、redaction status
Eval / regression result	failed cases、baseline, current score, rerun after fix
Communication record	internal update、customer script、regulator memo、board brief
Corrective action register	owner、due date、control, verification evidence, closure decision

8.3 Trace Fields for Incident Readiness

Field group	Required fields
Business context	use_case_id、risk_tier、workflow_step、customer_segment、employee_role、case_id
AI component versions	model_alias、model_version、prompt_id、prompt_version、index_version、policy_version、judge_version
Retrieval	query、filters、retrieved_doc_ids、citation_doc_ids、source_trust、freshness_days、ACL decision
Tool	tool_name、risk_level、input_hash、policy_decision、approval_id、side_effect、idempotency_key
Safety	guardrail_decision、DLP_decision、prompt_injection_score、escalation_reason
Quality	groundedness_score、citation_correctness、answerability、human_review_decision、edit_reason
Performance	ttft_ms、total_latency_ms、retrieval_latency_ms、tool_latency_ms、judge_latency_ms
Cost	input_tokens、output_tokens、reasoning_tokens、retrieval_cost、judge_cost、tool_cost、total_cost
Audit	trace_id、request_id、retention_class、redaction_status、access_control_group

9. Postmortem：无责，但不能无行动

AI postmortem 的质量标准：

Bad postmortem	Good postmortem
“模型幻觉了，已优化 prompt”	明确哪个模型 / prompt / index / policy / tool / workflow 组合失效，为什么控制没有拦住，如何验证不会复发
“用户问法太刁钻”	识别产品边界、answerability gate、fallback 和 user education 是否不足
“坐席没有认真审核”	分析人审界面是否提供证据、风险提示、差异高亮和足够时间
“供应商模型变了”	检查 vendor change notice、model pinning、canary eval、route policy 和 exit plan
“已加强监控”	说明新增指标、阈值、owner、alert route、演练和验证样本

9.1 Postmortem Template

# AI Incident Postmortem: AML Copilot Bad Retrieval, 2026-06-29

## 1. Executive Summary

- Incident ID: AI-INC-AML-2026-001
- Severity: SEV1
- Status: Contained, restart pending Reliability Review Board approval
- Use case: AML Copilot evidence summary and SAR narrative draft
- Business owner: Head of AML Operations
- Incident commander: AI Platform SRE Lead
- Detection time: 2026-06-29 09:42 America/Chicago
- Containment time: 2026-06-29 10:18 America/Chicago
- Recovery time: 2026-06-30 16:00 America/Chicago after regression approval
- Customer / business impact: 23 AML cases required analyst re-review; no customer-facing message sent
- Regulatory / privacy / security exposure: Potential regulatory narrative quality risk; no PII leakage and no SAR submission confirmed
- Current residual risk: Low after index rollback, narrative freeze and freshness regression

## 2. What Happened

- Observable symptom: Senior analyst found SAR draft citing a typology retired in 2026-Q1
- AI failure category: Bad retrieval and hallucinated regulated narrative
- Affected users / cases / customers / transactions: 9 analysts, 23 AML cases, 0 customer contacts, 0 submitted SARs
- Affected AI components: AML policy index v2026.06.28, retrieval filter v1.7, narrative prompt v3.2
- First known bad version: index-aml-policy-2026.06.28
- Last known good version: index-aml-policy-2026.06.21

## 3. Impact

| Impact area | Assessment | Evidence |
|---|---|---|
| Customer harm | No direct customer impact; cases stayed in analyst workflow | case_state_query_2026_06_29.csv |
| Operational impact | 23 cases re-reviewed; analyst queue delayed by 4.5 hours | AML queue dashboard snapshot |
| Financial impact | No direct loss; 18 analyst hours used for re-review | workforce impact estimate |
| Regulatory / compliance | Potential SAR quality risk if drafts had been submitted | SAR submission cross-check showing zero submitted |
| Data / privacy | No PII leaked outside approved systems | DLP report DLP-AML-2026-001 |
| Security | No prompt injection or external access found | SIEM query and context payload review |
| Reputation | Internal operational issue; no external communication required | Legal / Compliance decision log |

## 4. Timeline

| Time | Event | Owner | Evidence |
|---|---|---|---|
| 09:42 | Analyst reports outdated typology in SAR draft | AML QA Lead | QA ticket AML-QA-7782 |
| 09:50 | IC opens SEV1 war room and assigns scribe | AI Platform SRE Lead | incident channel export |
| 10:05 | Impact query identifies 23 affected cases | Data Engineer | impacted_case_query.sql |
| 10:18 | SAR narrative generation disabled; evidence summary remains read-only | Platform Lead | route_policy_change_2026_06_29 |
| 10:31 | Index rolled back to v2026.06.21 | RAG Owner | index manifest diff |
| 12:10 | Analysts complete first 10 case re-reviews | AML Operations | QA review sample |
| 16:40 | No submitted SARs found for affected drafts | Compliance | SAR submission reconciliation |

## 5. Detection

- Which signal detected the issue: Human QA by senior AML analyst
- Which signal should have detected it earlier: Freshness regression in the index release gate
- Detection gap: Citation checker validated source existence but not effective date
- Alert / dashboard changes: Added retired-source citation alert and freshness breach tile

## 6. Containment and Recovery

- Immediate containment: Disabled SAR narrative draft generation for AML Copilot
- Rollback / route / kill switch used: L1 response mode switch plus index rollback to v2026.06.21
- Fallback experience: Analysts receive evidence summary and link to policy portal; narrative written manually
- Recovery validation: 50 AML golden cases, 10 retired-typology adversarial cases and 23 affected cases rechecked
- Restart approval: Required from Model Risk, AML Business Owner and AI Reliability Review Board

## 7. Root Cause and Contributing Factors

### Direct root cause

The AML RAG index release included retired typology guidance without effective-date filtering, allowing outdated evidence to support generated SAR narrative text.

### Contributing factors

| Factor | Explanation | Control gap |
|---|---|---|
| Product boundary | Narrative generation was allowed for all AML typology questions | No high-risk narrative freshness gate |
| Architecture | RAG index and narrative prompt were released independently | Component release coupling missing |
| Data / RAG | Retired documents remained searchable with low metadata quality | Effective-date filter and source owner approval missing |
| Model / prompt | Prompt encouraged concise narrative even when evidence was stale | Answerability instruction did not include retired-source refusal |
| Tool / workflow | No external write tool was involved | Tool control worked as designed |
| Monitoring | Citation support did not check source validity period | Freshness metric absent |
| Human review | Analyst QA caught the issue before submission | Human review worked but late in workflow |
| Governance | Index release did not require Model Risk review for retired policies | Gate scope too narrow |

## 8. Why Existing Controls Failed

| Control | Expected behavior | Actual behavior | Fix |
|---|---|---|---|
| Citation checker | Ensure narrative claims are supported by cited evidence | Confirmed document existed but ignored retirement date | Add effective-date validation and retired-source block |
| Index release gate | Prevent stale or unauthorized policy from entering production | Checked document count and ACL only | Add source owner approval and freshness regression |
| Human review | Catch low-quality narrative before regulatory submission | Caught issue before submission | Move freshness alert earlier in workflow |

## 9. Corrective Actions

| Action ID | Action | Owner | Due date | Verification evidence | Closure approver |
|---|---|---|---|---|---|
| CA-AML-001 | Add effective-date filter to AML retrieval | RAG Owner | 2026-07-03 | 60-case retrieval regression pass | Model Risk |
| CA-AML-002 | Block narrative generation when citation freshness fails | AI Platform Lead | 2026-07-05 | 10 retired-typology cases refused or escalated | AML Business Owner |
| CA-AML-003 | Add AML index release owner approval | Data Governance Lead | 2026-07-08 | signed source manifest in evidence binder | Compliance |

## 10. Regression and Recurrence Prevention

- New eval cases: 50 AML SAR narrative golden cases with citation support and freshness assertions
- New adversarial cases: 10 retired typology and conflicting policy cases
- New monitoring: retired-source citation rate, freshness_days p95, narrative blocked by freshness gate
- Release gate changes: AML index release requires source owner approval and freshness regression pass
- Architecture decision records: ADR-AML-004 requires evidence freshness gate before regulated narrative generation
- Runbook changes: Bad retrieval containment now requires index rollback plus affected case query
- Game day scenario added: AML retired typology tabletop for quarterly reliability exercise

## 11. Communication

- Customer communication: Not required because no customer-facing output or account action occurred
- Regulator / examiner communication: Prepared evidence pack for examiner request; no proactive notice recommended by Legal / Compliance
- Board / executive communication: Included in monthly AI risk dashboard as SEV1 contained incident
- Internal operations guidance: AML analysts instructed to use manual narrative drafting until restart approval

## 12. Residual Risk Decision

- Residual risk: Low for restarted narrative generation after freshness gate; medium for same-day policy updates until source SLA improves
- Risk owner: Head of AML Operations
- Accepted / rejected / requires additional controls: Accepted with source-owner SLA and monthly freshness audit
- Next review date: 2026-08-15 AI Reliability Review Board

9.2 Root Cause Tree for AI Incidents

Layer	Questions
Business boundary	AI 是否被允许处理这个 intent、客户类型、地区、产品或风险等级？
Data source	证据是否权威、最新、可访问、按权限过滤、可追溯？
Retrieval	查询改写、filter、top_k、reranker、citation 是否把正确证据带入上下文？
Prompt / policy	指令是否冲突、过宽、未区分可信和不可信上下文？
Model	模型是否升级、route 改变、参数改变、能力或拒答行为变化？
Tool	工具权限、schema、审批、幂等、回滚、side effect 是否受控？
Judge / guardrail	拦截或评分是否覆盖该失败模式，是否被绕过或漂移？
Human review	人审是否看到足够证据，是否有时间、权力和流程去覆盖 AI？
Monitoring	质量、安全、成本、延迟、业务 outcome 是否有阈值和 owner？
Governance	变更是否经过 gate，风险是否被正式接受，owner 是否明确？

10. Corrective Action Register

Corrective action 的目标不是关闭 Jira，而是降低复发概率或影响范围。每个行动项都必须有验证证据。

Field	填写标准	合格样例
Action ID	与 incident ID 关联	CA-AML-2026-001
Failure mode	对应 taxonomy	Bad retrieval + hallucinated SAR narrative
Control objective	要降低的风险	高风险 AML narrative 必须由有效证据支持
Action	可执行变更	增加 citation support gate，低于阈值时只输出 evidence summary
Owner	单一责任人	AML AI Platform Lead
Due date	明确日期	2026-07-12
Verification evidence	可复核结果	50 个 SAR golden cases 全部通过；10 个过期 typology cases 被拒答
Risk reduction	预期效果	降低错误 narrative 进入 analyst workflow 的概率
Residual risk	仍然存在的风险	新 typology 发布当天仍可能存在 freshness gap
Closure approver	有权关闭的人	Model Risk + AML Business Owner
Next review	防止行动项假关闭	2026-08-15 review board

10.1 Action 类型

Type	例子	何时使用
Eval action	新增 golden / adversarial / regression cases	事故样本可转成可重复测试
Architecture action	增加 policy engine、tool gateway、route isolation、index manifest	根因是边界和控制不足
Product action	缩小 use case、改变 disclosure、增加人工入口、限制答案类型	AI 被放进了不适合的客户旅程
Data action	source trust、owner approval、freshness SLO、ACL、data contract	RAG / data lineage 是根因
Operations action	alert、runbook、on-call、war room、rollback drill	响应慢、检测晚、证据不足
Governance action	review board、approval gate、risk acceptance、policy update	决策链或责任边界不清
Vendor action	SLA、model pinning、change notice、exit plan、support escalation	外部依赖导致或放大事故

10.2 Recurrence Prevention Ladder

Level	防复发强度	示例
L0 Manual reminder	依赖人记住	仅培训坐席“注意检查引用”
L1 Checklist	依赖流程提醒	上线前检查 prompt / index version
L2 Monitoring	能发现	bad citation rate alert
L3 Gate	能阻止发布	eval fail 时 CI 阻止 prompt / index release
L4 Runtime control	能实时阻断	answerability gate、DLP、tool gateway、policy engine
L5 Architectural isolation	能限制 blast radius	model route isolation、read-only mode、tenant boundary、kill switch
L6 Governance system	能持续学习	review board、audit sample、game day、risk appetite update

复盘后的行动项至少应达到 L3；SEV0 / SEV1 的核心失效应优先设计到 L4 或 L5。

11. Architecture and Product Mapping

11.1 从事故类型到架构控制

Incident type	Product decision	Architecture control	Governance evidence
Bad retrieval	明确哪些答案必须引用权威来源	Source registry、ACL filter、index manifest、citation checker	Data owner approval、retrieval eval report
Hallucination	高风险答案从自由生成改为模板 / 引用摘要	Answerability classifier、groundedness gate、template composer	Prompt approval、quality dashboard
Policy bypass	将业务规则从 prompt 中外置	Policy engine、intent classifier、decision table、guardrail trace	Compliance sign-off、policy version log
Tool misuse	定义 read / draft / act 边界	Tool gateway、ABAC、dual control、idempotency、dry-run	Permission matrix、approval evidence
PII leak	限制输入、输出、日志和外发字段	DLP、field redaction、egress gateway、log masking	Privacy impact record、DLP test
Cost spike	定义每个 outcome 的 unit economics	Cost ledger、quota、route policy、cache、loop detector	Budget review、FinOps dashboard
Latency degradation	设定按风险分层的响应体验	Fast path、async review、timeout budget、fallback response	SLO document、capacity test
Model drift	明确模型升级不是透明小变更	Model alias pinning、canary、shadow eval、rollback	Change approval、eval comparison
Prompt injection	将不可信内容和工具权限隔离	Context labeling、sandbox、tool approval、memory write policy	Red-team report、ATLAS mapping

11.2 AI Reliability ADR 模板

# ADR-AML-004: Require Evidence Freshness Gate Before SAR Narrative Generation

## Context

- Incident / risk trigger: AI-INC-AML-2026-001 showed SAR narrative could cite retired typology guidance
- Affected use case: AML Copilot SAR narrative drafting
- Risk tier: Tier 1 regulated decision-support workflow
- Current weakness: Citation checker verifies support but does not verify source effective date or retirement status

## Decision

- Chosen architecture / product control: Add evidence freshness gate before narrative generation; stale evidence forces evidence-summary-only mode
- Components changed: Retrieval metadata filter, citation checker, narrative composer, release gate
- Runtime behavior: If any cited AML policy source is retired or outside effective date, the system refuses narrative generation and escalates to analyst manual drafting
- Rollback approach: Disable freshness gate through controlled feature flag only after IC and Model Risk approval; default fallback is evidence-summary-only mode

## Options Considered

| Option | Benefit | Risk | Decision |
|---|---|---|---|
| Prompt-only warning | Fast to implement | Model may still use stale evidence under pressure | Rejected |
| Freshness gate before narrative | Blocks stale regulated narrative at runtime | Some valid edge cases require manual drafting | Chosen |
| Full narrative shutdown | Maximum safety | Removes productivity benefit for low-risk summaries | Reserved for SEV1 containment |

## Consequences

- Reliability impact: Reduces recurrence of stale-source SAR narrative and creates auditable block events
- Customer impact: No direct customer-facing impact; analyst workflow may require more manual drafting during stale-source periods
- Cost / latency impact: Adds citation metadata check with negligible latency; may increase manual review hours
- Governance impact: AML source owner approval becomes mandatory for index release
- Residual risk: Same-day policy changes can still create a temporary gap until source SLA is met

## Verification

- Eval cases: 50 AML golden cases and 10 retired-source adversarial cases
- Monitoring: retired-source block rate, freshness_days p95, manual drafting escalation count
- Game day scenario: Quarterly AML retired typology tabletop
- Review board approval: Required before re-enabling narrative generation

11.3 Release Gate After Incident

Gate	Required evidence
G1 Scope gate	Use case boundary updated；forbidden uses 明确；fallback owner 确认
G2 Component gate	model / prompt / index / tool / policy version map 完整
G3 Eval gate	事故样本进入 golden set；regression 通过；expert sample 通过
G4 Security / privacy gate	DLP、prompt injection、tool misuse、egress、log masking 测试通过
G5 Operations gate	runbook、kill switch、on-call、alert、dashboard 已更新
G6 Governance gate	Model Risk / Compliance / Business Owner 对 residual risk 做出决策
G7 Restart gate	分阶段放量计划、rollback condition、monitoring window 明确

12. Customer / Regulator / Board Communication

沟通不是 PR 话术，而是控制面的一部分。越早准备结构化事实，越少在压力下做模糊承诺。

12.1 Communication Matrix

Audience	关注问题	不能说	应该准备
Customers	我是否受影响，发生了什么，如何补救，是否需要行动	“AI 偶尔会错”“我们已优化模型”	事实、影响、补救、人工渠道、时间线、后续通知
Frontline / Operations	现在怎么处理客户、案例、例外和人工队列	“先凭经验处理”	临时 SOP、话术、升级条件、受影响 case list
Executives	影响范围、风险、恢复、补救成本、下一步决策	只报技术细节	SEV、impact, containment、customer/regulatory exposure、decision needed
Board / Risk Committee	是否触及风险偏好、控制是否失效、治理是否有效	“项目团队会修好”	root cause theme、risk appetite、corrective action、accountability、timeline
Regulator / Examiner	是否影响受监管义务、客户权益、数据、模型风险、第三方	未经确认的法律结论	verified facts、controls、affected population、remediation、evidence path
Vendor	是否供应商导致、需要什么证据、SLA 是否触发	只口头描述	trace sample、timestamps、model/route IDs、contract notice、RCA request

12.2 Executive Update 模板

# AI Incident Executive Update

- Incident ID:
- Severity:
- Current status:
- Affected capability:
- What happened:
- Customer / business impact:
- Data / privacy / security exposure:
- Regulatory exposure:
- Containment completed:
- Remaining risk:
- Decisions needed:
- Next update time:

12.3 Regulator / Examiner Response Pack

Section	内容
Factual summary	已验证事实、时间线、影响范围，不包含猜测
System description	AI system、workflow、human role、model/RAG/tool components
Control design	上线前控制、监控、人工复核、policy、DLP、tool gateway
Control failure	哪些控制未生效、为什么未生效
Customer impact	受影响对象、补救、通知、投诉处理
Remediation	containment、corrective actions、completion evidence
Governance	owner、review board、risk acceptance、management oversight
Evidence index	trace、logs、evals、approval、communication、postmortem 路径

12.4 Board Brief 结构

Slide	内容
1. Incident summary	SEV、时间线、影响、当前状态
2. Business and customer impact	客户、员工、案例、资金、投诉、品牌
3. Risk and control view	失效控制、残余风险、风险偏好影响
4. Management action	containment、corrective actions、owner、due date
5. Governance ask	需要董事会知情、批准、预算、风险接受或监督的事项

13. AI Reliability Review Board

AI Reliability Review Board 不是大型委员会，而是高风险 AI 系统的运营门禁。它把 incident、eval regression、architecture change、model update、vendor change 和 risk acceptance 放在同一个决策桌上。

13.1 Charter

Item	内容
Mission	降低 AI 事故复发概率和影响范围，确保高风险 AI use case 的 release、change、restart 和 residual risk 有证据、有 owner、有审批
Scope	Tier 1 / Tier 2 AI systems；SEV0-SEV2 incidents；model/prompt/index/tool/policy 高风险变更
Decision rights	approve restart、require additional controls、block release、accept residual risk、escalate to executive risk committee
Members	AI Product、AI Platform、Architecture、SRE、Security、Privacy、Model Risk、Compliance、Business Owner、Audit observer
Cadence	SEV0/1 event-driven；SEV2 weekly；normal monthly reliability review
Inputs	incident postmortem、eval trend、SLO breach、cost trend、red-team result、vendor notice、customer complaint trend
Outputs	decision log、corrective action status、risk acceptance、architecture ADR、game day backlog

13.2 Review Agenda

Agenda item	决策问题
Open incidents	是否真正 containment，是否仍有客户/监管风险
Corrective actions	是否有逾期、假关闭、缺验证证据的行动项
Eval and drift	哪些模型 / prompt / index / tool 出现质量或安全退化
SLO / cost / latency	是否触发 error budget 或 budget guardrail
Customer harm signals	投诉、人工升级、拒绝率、错误补救是否异常
Vendor changes	是否需要 re-validation、model pinning 或 exit test
Game day findings	演练发现是否进入 release gate 和 runbook

13.3 Decision Categories

Decision	Criteria	Evidence
Restart approved	containment 完成，fix 通过 regression，monitoring window 明确	eval report、trace sample、owner sign-off
Restart with restrictions	核心风险降低，但 residual risk 仍需限制	disabled intent、human review、traffic cap
Continue shutdown	影响或根因不清，无法证明控制有效	incomplete RCA、failed regression、unresolved privacy/legal issue
Risk accepted	业务决定接受残余风险，并有补偿控制	risk memo、business sign-off、review date
Escalate	超出 board 权限或触及监管/董事会风险偏好	executive memo、legal/compliance analysis

14. Game Days and Tabletops

Game day 的目标不是表演成熟，而是暴露系统、流程和组织在压力下的真实弱点。

14.1 Game Day 设计原则

原则	解释
基于真实 trace	用生产相似的请求、知识库、工具权限和角色，不只用抽象故事
覆盖跨职能决策	Product governance、architecture、SRE、security、privacy、model risk、compliance 都要做决定
演练证据保全	检查是否能找到 prompt、index、retrieval、tool、DLP、approval 和 impacted population
演练 kill switch	不只讨论是否关闭，要实际验证配置、权限和恢复步骤
生成行动项	每次演练必须产出 corrective actions 和 owner

14.2 Tabletop Scenarios

Scenario	Inject	Expected decisions	Evidence to test
Bad retrieval in AML copilot	分析员发现 SAR narrative 引用了过期 typology	停 narrative 生成、回滚 index、识别受影响 case、Model Risk review	index manifest、retrieved docs、case list、regression eval
KYC assistant PII leak	外部供应商 ticket 中出现客户身份证件摘要	关闭外发、privacy workflow、log scan、vendor notice	egress log、DLP decision、affected records、retention
Payment dispute agent tool misuse	Agent 重复提交 provisional credit	禁用写工具、撤销重复动作、检查 idempotency、客户补救	tool span、approval ID、transaction ledger、compensation
Customer-facing AI hallucination	客户被告知不存在的 fee waiver 权利	关闭自由生成、发布坐席话术、识别客户、投诉处理	output samples、customer IDs、policy source、QA decisions
Prompt injection via uploaded PDF	投诉附件诱导 Agent 调用外部 email tool	隔离 upload path、关闭外发工具、security incident triage	malicious payload、context label、tool proposal、blocked/allowed decision
Cost spike via loop	Planner 重复检索和 judge，成本 8 倍上升	预算熔断、rate limit、关闭 retry loop、恢复 unit cost	cost ledger、trace loop、route policy、quota
Latency degradation	p95 从 8s 到 35s，人工队列积压	降级模型、异步处理、减少 rerank、业务 SLA 更新	latency spans、queue wait、fallback success
Model drift after vendor upgrade	同一 golden set fail 率从 3% 到 18%	pin 旧模型、暂停 rollout、vendor RCA、shadow eval	model alias、eval diff、vendor notice、route log

14.3 Game Day Scorecard

Dimension	1 - Weak	3 - Adequate	5 - Strong
Detection	只能靠人工发现	有部分 alert，但信号分散	自动 alert 命中 failure mode，owner 清晰
Triage	争论分类和 SEV	30 分钟内完成初判	15 分钟内明确 SEV、blast radius 和 war room
Containment	不知道关闭哪里	能手动关闭主要路径	kill switch 分层、可验证、可回滚
Evidence	找不到版本或 trace	能找到核心日志	trace、版本、impact query、tool side effect 完整
Communication	技术更新难被业务理解	有基础 executive update	客户/监管/董事会版本清晰、事实一致
Prevention	只写行动项	有 owner 和 due date	行动项有 regression evidence 和 review board closure

15. 金融零售 AI 事故案例

15.1 AML Copilot：错误检索导致 SAR narrative 风险

项目	内容
System	AML Copilot，为分析员总结告警证据并起草 SAR narrative
Failure	RAG 检索到过期 typology guidance，模型把旧规则写成当前风险理由
Severity	SEV1；若 narrative 已提交监管则升 SEV0
Detection	Senior analyst QA 发现 narrative 与最新政策冲突；citation checker 未检查 effective date
Containment	关闭 SAR narrative 自动草稿，只保留 evidence summary；回滚 index；高风险 case 全部人工复核
Root cause	Index manifest 没有 policy effective date；retriever freshness filter 未启用；release gate 未包含过期 typology regression
Corrective actions	增加 freshness SLO；policy source owner 审批；50 个 AML regression cases；narrative 输出必须带 citation support score
Evidence artifact	AML AI Incident Postmortem + Retrieval Freshness ADR + SAR Narrative Regression Pack

15.2 KYC Assistant：PII 外发到供应商系统

项目	内容
System	KYC 文档审核助手，自动总结客户身份证明和缺失项
Failure	Assistant 把客户身份证件摘要写入外部 vendor support ticket，ticket 系统不在批准处理范围
Severity	SEV1；若包含大规模证件号或跨境处理则 SEV0
Detection	DLP 在 egress gateway 命中身份证号 pattern；privacy alert 自动升级
Containment	关闭外发 ticket tool；清理 vendor ticket；启用 legal hold；识别 affected customers；暂停相关日志导出
Root cause	Tool gateway 未区分内部 case note 和外部 support ticket；DLP 只在最终回答启用，未覆盖 tool payload
Corrective actions	egress policy 按目的绑定；tool input field redaction；vendor processor matrix 更新；PII leak adversarial eval
Evidence artifact	KYC PII Incident Pack + Egress Policy Matrix + DLP Regression Evidence

15.3 Payment Dispute Agent：工具误用导致重复 provisional credit

项目	内容
System	支付争议处理 Agent，读取争议材料、建议下一步、草拟客户通知
Failure	Agent 在 retry 后重复调用 provisional credit tool，幂等 key 未绑定 dispute ID 和 amount
Severity	SEV1；若批量资金影响或无法撤销则 SEV0
Detection	账务对账发现同一 dispute 两笔 provisional credit；tool side effect alert 滞后
Containment	禁用写工具；切只读建议模式；冻结受影响 dispute；启动客户和账务补救
Root cause	Tool schema 未要求 idempotency key；approval UI 未展示已执行 action；retry policy 未识别 side effect
Corrective actions	写工具强制 idempotency；approval ledger；retry 禁止重复 side effect；payment game day 每季度演练
Evidence artifact	Payment Agent Tool Misuse Postmortem + Tool Permission / Idempotency ADR

15.4 Customer-Facing AI：错误费用减免承诺

项目	内容
System	信用卡客服 AI，客户可直接询问费用、争议、权益和投诉
Failure	AI 虚构“年费可无条件减免一次”的政策，客户认为银行已承诺
Severity	SEV1，因为客户可见且涉及费用和权利；若大规模传播则 SEV0
Detection	投诉系统中 fee waiver 关键词异常上升；QA 抽样确认 AI 输出无来源支持
Containment	关闭费用政策自由回答；改用模板 + 权威 API；受影响客户转人工补救
Root cause	Answerability gate 未覆盖费用承诺；UI 披露没有区分一般信息和正式承诺；citation optional
Corrective actions	fee / rate / eligibility 全部 template locking；complaint trigger dashboard；customer remediation script
Evidence artifact	Customer-Facing AI Incident Memo + Regulated Conversation Boundary Card

15.5 Fraud Review Copilot：模型漂移导致误报解释退化

项目	内容
System	欺诈复核 Copilot，帮助运营解释交易阻断原因并建议下一步
Failure	Vendor 模型升级后，解释变得更自信但证据更少，human override 从 6% 升到 19%
Severity	SEV2；若导致客户账户错误冻结或拒绝申诉则 SEV1
Detection	Human reject spike + golden set scheduled eval fail
Containment	Pin previous model；扩大人工复核；暂停新模型 rollout
Root cause	Model alias 未 pin；vendor change notice 未进入 AI release gate；shadow eval 样本未覆盖冻结解释
Corrective actions	model alias pinning；vendor change intake；fraud explanation golden set；review board restart approval
Evidence artifact	Model Drift Incident Review + Vendor Change Gate Checklist

16. Templates and Artifacts

16.1 AI Incident Intake Card

Field	Example
Incident ID	AI-INC-2026-001
Reporter	AML QA Lead
Detection channel	Human QA + citation support alert
Use case	AML Copilot SAR narrative
Suspected category	Bad retrieval / hallucination
Initial SEV	SEV1
Customer / regulatory exposure	Potential regulatory narrative quality issue; no submission confirmed
Active blast radius	23 cases generated since index v2026.06.28
Immediate ask	Disable SAR narrative generation and run affected case query

16.2 Affected Population Query Spec

Question	Query requirement
哪些输出可能受影响	按 prompt_version、index_version、model_route、time window、intent、risk_tier 查询
哪些客户或案例受影响	关联 case_id、customer_id、employee_id、channel、workflow_state
是否发生工具 side effect	关联 tool_call、approval_id、side_effect、ledger / case status
是否已经客户可见或监管提交	关联 message_sent、document_exported、SAR_submitted、letter_generated
是否已人工覆盖	关联 review_decision、edit_reason、override_time

16.3 Customer Remediation Worksheet

Field	填写标准
Affected segment	客户类型、产品、地区、时间范围
Harm hypothesis	错误费用、错误拒绝、错误说明、隐私暴露、延迟、错误补偿
Verification method	trace + CRM + transaction / case system cross-check
Remediation action	更正通知、费用调整、人工回访、投诉升级、数据删除请求
Approval	Legal、Compliance、Business Owner、Customer Ops
Evidence	customer list hash、message template、completion status、exception log

16.4 AI Reliability Dashboard Spec

Tile	Metric	Threshold	Owner
Incident count by category	SEV0-SEV4 per month	SEV1+ immediate review	Reliability Lead
Bad retrieval rate	citation unsupported / stale / ACL fail	High-risk use case > 1% alert	Data / RAG Owner
Hallucination / groundedness	groundedness fail rate	Tier 1 > 0.5% alert	Model Risk
Tool policy violations	blocked / attempted / allowed high-risk tool calls	Any unexpected allowed high-risk call	Security / Tool Owner
PII / DLP	DLP hit by route and destination	Any external PII hit	Privacy
Cost unit economics	cost per resolved case / analysis / dispute	> approved budget guardrail	Platform operations / FinOps
Latency SLO	p95 TTFT / total / tool latency	breach for 3 intervals	SRE
Human reject / override	rejected outputs, edit distance, override reason	spike vs baseline	Business Ops
Eval regression	scheduled eval pass rate by version	high-risk fail blocks release	Validation
Corrective action aging	overdue, no evidence, repeated failure	overdue SEV1 action escalates	Review Board

16.5 Reliability Review Board Packet

Section	内容
Open incidents	SEV、status、owner、next decision
Closed incidents	postmortem link、RCA theme、closure evidence
Corrective actions	overdue、blocked、accepted risk、closed with evidence
Eval / drift	regressions、model changes、prompt / index changes
Security / privacy	injection attempts、PII events、tool gateway exceptions
SLO / cost	latency, availability, unit cost, error budget
Customer harm	complaints、handoff failure、remediation status
Decisions requested	restart、block release、accept residual risk、fund control

17. 30-Day System Learning Path：AI Incident and Reliability Evidence Pack

目标：30 天内围绕一个金融零售 AI use case，完成一套可审查的 AI Incident / Postmortem / Reliability evidence pack。推荐主线选择 AML Copilot、KYC Assistant、Payment Dispute Agent 或 Customer-Facing AI。

Day	任务	Artifact
1	选择 use case，定义 AI 行为边界、risk tier、禁止用途	Use Case Boundary Card
2	画系统组件：model、prompt、RAG、tool、policy、judge、human handoff	Component Map
3	定义 AI incident taxonomy，选 8-10 个最相关 failure modes	Incident Taxonomy
4	设计 SEV0-SEV4 severity model	Severity Matrix
5	设计 incident command RACI 和 war room cadence	Incident Command Runbook
6	定义 trace evidence 字段	Evidence Schema
7	设计 AI reliability dashboard	Dashboard Spec
8	写 bad retrieval scenario	Scenario Card 1
9	写 hallucination / policy bypass scenario	Scenario Card 2
10	写 tool misuse scenario	Scenario Card 3
11	写 PII leak scenario	Scenario Card 4
12	写 cost spike / latency degradation scenario	Scenario Card 5
13	写 model drift / eval regression scenario	Scenario Card 6
14	设计 containment / rollback matrix	Rollback Matrix
15	设计 kill switch 分层	Kill Switch Spec
16	选择一个 scenario，写完整 incident timeline	Incident Timeline
17	写 affected population query spec	Impact Query Spec
18	写 postmortem 初版	Postmortem Draft
19	写 root cause tree 和 contributing factors	RCA Worksheet
20	写 corrective action register	Corrective Action Register
21	把事故样本转成 eval regression cases	Regression Pack
22	写 architecture ADR	Reliability ADR
23	写 customer / operations communication	Customer and Ops Comms
24	写 regulator / board brief	Governance Communication Pack
25	设计 reliability review board charter	Review Board Charter
26	运行一次 tabletop，记录 decision log	Game Day Decision Log
27	根据 tabletop 更新 runbook 和 dashboard	Runbook Revision
28	汇总 evidence binder index	Evidence Index
29	写 1500-2500 字 reliability case study	Reliability Case Study
30	准备 design review question bank 和 decision narrative	Decision Narrative Pack

17.1 30 天最终交付包

Artifact	证据价值
Executive one-pager	展示你能向高管解释 AI 事故风险和决策
System and control map	展示架构判断，而不是只做流程文档
Incident taxonomy + SEV model	展示高级风险分类能力
Postmortem + corrective action register	展示生产可靠性和治理闭环
Game day pack	展示主动验证和组织协同能力
Regression pack	展示把事故样本转成 EvalOps 的能力
Board / regulator communication	展示金融零售场景的风险表达

18. Design Review Questions

这些问题用于 reliability architecture review、postmortem review 和 restart gate，不用于背诵话术。每个问题都应落到 severity、containment、evidence、corrective action 和 governance decision。

18.1 AI Incident 与普通 SRE Incident 的边界

Review question：AI incident response 和普通 SRE incident response 最大区别是什么？

Design position：普通 SRE 主要处理 availability、latency、error rate。AI incident 还要处理语义正确性、证据支持、政策边界、工具动作、数据泄露、模型漂移和客户/监管影响。HTTP 200 也可能是事故，所以必须有 trace、eval、guardrail、human review 和治理闭环。

System reasoning：AI incident 应拆成 detect、triage、command、contain、recover、postmortem、regression、governance。关键不是“模型错了”，而是定位哪个组件失效：prompt、RAG、index、policy、tool、judge、model route、human handoff 或 vendor change。然后按风险分级决定是否 rollback model、回滚 index、禁用工具、切模板、人审、客户补救和监管沟通。复盘后必须把事故样本进入 regression eval 和 release gate，否则只是写报告。

Evidence check：incident record 是否能同时复原 technical health、semantic failure、customer impact、control failure 和 restart condition。

18.2 Severity Model

Review question：如何给 AI 事故分级？

Design position：不要只按技术故障分级，而要按客户伤害、监管暴露、数据敏感度、自动化程度、blast radius、可逆性和检测来源分级。客户可见错误承诺、PII 外泄、高风险工具误执行、AML/KYC/credit 流程系统性错误，最低就应该是 SEV1。

System reasoning：SEV0-SEV4 可以形成分层处置。SEV0 是重大客户伤害、监管违规、大规模数据泄露或不可逆工具动作；SEV1 是高影响但可 containment 的受监管流程或客户可见问题；SEV2 是质量、成本、延迟、eval 显著退化；SEV3 是被控制捕获的局部问题；SEV4 是 hazard 或演练发现。这样 severity 能直接驱动 war room、通知、postmortem 深度、review board 和 restart gate。

Evidence check：severity decision 是否记录判定维度、升级规则、通知要求、review depth 和 restart gate。

18.3 Bad Retrieval Containment

Review question：Bad retrieval 导致错误答案时如何处理？

Design position：先止血：回滚 index 或禁用错误来源，高风险问题切人工或模板。然后保留 query、retrieved doc IDs、index version、ACL、citation 和输出样本。根因通常在 source trust、freshness、ACL、chunking、reranker 或 release gate。最后把样本加入 retrieval regression。

System reasoning：先判断是否客户可见或影响受监管流程。如果是，立即关闭相关 intent 的自由生成，切换到权威搜索或人工复核。技术上做 index manifest diff、source owner check、freshness filter、ACL test 和 citation support check。治理上让 Data Owner 和 Model Risk 共同确认恢复条件。防复发不是“调 prompt”，而是 source registry、index release gate、retrieval eval、freshness SLO 和 impacted population query。

Evidence check：evidence pack 是否包含 index manifest、retrieval trace、ACL decision、citation support、affected population query 和 regression case。

18.4 Agent Tool Misuse 防线

Review question：Agent tool misuse 为什么危险，如何设计防线？

Design position：Tool misuse 把语言错误放大成真实业务动作，比如重复退款、错误关闭投诉、越权查账户。防线要在工具网关，而不是 prompt：最小权限、risk-tiered tools、approval、idempotency、dry-run、step budget、side-effect logging 和 kill switch。

System reasoning：工具应分为 read-only、draft、write-low、write-high、irreversible。模型只能 propose，系统通过 ABAC / purpose binding / policy engine 决定是否 allowed、blocked 或 requires approval。所有写动作必须有 idempotency key、approval ID、ledger、reversal path 和 trace。事故时第一步通常是把写工具切只读，保全 side effect log，识别 affected population，再做补偿和 regression。

Evidence check：tool gateway 是否记录 actor、purpose、approval、action hash、idempotency key、side effect 和 reversal path。

18.5 PII Leak Response

Review question：PII leak 事故如何处理？

Design position：先停止外发和明文日志，隔离受影响 route，保全 trace 和 egress log，识别字段、记录数、目的地和保留状态。然后由 privacy、legal、security 判断通知和补救。防复发要覆盖输入、输出、tool payload、日志和供应商调用，不只做回答过滤。

System reasoning：证据包应包含 DLP decision、output sample、egress destination、trace ID、redaction status、affected records 和 vendor handling。Containment 包括关闭外发工具、清理第三方 ticket、禁用日志导出、启用 field redaction。根因可能是目的绑定缺失、tool gateway 未做 DLP、日志策略错误或供应商处理范围不清。修复要进入 DLP regression、egress policy、privacy impact record 和 vendor control。

Evidence check：privacy incident record 是否能证明数据类别、外发路径、保留状态、补救动作和供应商控制。

18.6 Blameless Postmortem With Accountability

Review question：Postmortem 怎么做到无责但不失去 accountability？

Design position：无责不是无人负责。无责是避免追责个人，聚焦系统为什么让合理的人在当时信息下做出错误决策；accountability 是每个 corrective action 都有 owner、due date、验证证据和关闭审批。

System reasoning：好的 postmortem 会记录时间线、影响、检测缺口、控制失效、根因和贡献因素。它不会写“某人没审核好”或“模型幻觉了”，而会问人审界面有没有显示证据、release gate 为什么没覆盖样本、policy 为什么只写在 prompt、tool 为什么没有 approval。最后行动项必须能验证，例如新增 50 个 regression cases、启用 kill switch、通过 DLP test、更新 ADR，而不是“加强培训”。

Evidence check：corrective action register 是否包含 owner、due date、verification evidence、closure approver 和 residual risk。

18.7 Incident To Governance Loop

Review question：如何把 AI 事故转成长期治理能力？

Design position：每个 SEV1+ 事故都应该进入三条线：eval regression、architecture decision、governance review。也就是样本进入测试，控制进入架构，残余风险进入 review board 和证据包。

System reasoning：AI Reliability Review Board 定期看 incident、eval trend、SLO、cost、security、privacy、customer harm 和 corrective action aging。事故后的 restart 需要通过 scope、component、eval、security/privacy、operations、governance 和 phased rollout gate。这样事故不只是项目组修 bug，而是改变 release gate、tool permission、source governance、model pinning、vendor controls 和 board reporting。

Evidence check：每个重大事故是否生成 regression case、ADR、gate update、risk acceptance 或 board decision。

18.8 Cost Spike As Incident

Review question：成本 spike 算不算 AI incident？

Design position：算。AI 成本 spike 可能来自长上下文、retry loop、judge 放大、工具循环或滥用，可能导致预算熔断、服务降级和单位经济性失效。它应该按业务影响和可控性分级。

System reasoning：要看 cost per outcome，而不是只看总账单。证据包括 token、route、cache、retrieval、rerank、judge、tool 和 loop trace。Containment 包括 quota、rate limit、降级模型、缩短上下文、关闭 retry loop、启用 cache。防复发是 budget guardrail、step budget、loop detector、unit economics dashboard 和 release 前成本压力测试。

Evidence check：cost incident record 是否能把 spend spike 追溯到 route、loop、tool、judge、cache 和 outcome。

18.9 Regulator / Board Communication

Review question：面对监管或董事会，AI 事故应该怎么讲？

Design position：讲事实、影响、控制、补救和治理，不讲模型黑箱借口。要说明发生了什么、谁受影响、哪些控制失效、如何 containment、如何补救、如何防复发、谁负责和何时完成。

System reasoning：需要准备两套材料：regulator / examiner response pack 和 board brief。监管材料需要 system description、control design、control failure、affected population、remediation 和 evidence index。董事会材料需要风险偏好、客户和业务影响、管理层行动、重大残余风险和需要批准的决策。所有数字都必须来自可复核查询，未确认的内容标为 under assessment。

Evidence check：external communication pack 是否区分 confirmed facts、under assessment、management action、residual risk 和 decision request。

18.10 Reliability Review Board

Review question：为什么 AI Reliability Review Board 有必要？

Design position：AI 事故根因跨产品、模型、数据、工具、安全、隐私、合规和供应商，不能靠单个团队自己判定 restart。Review Board 把事件、eval、变更、风险接受和证据放在同一个治理机制里。

System reasoning：Review Board 的价值是 decision rights。它可以批准 restart、要求额外控制、阻止 release、接受残余风险或升级高管。输入包括 postmortem、eval trend、SLO breach、cost、red-team、vendor change 和 customer harm。输出是 decision log、corrective action、ADR、risk acceptance 和 game day backlog。这比项目组“修完上线”更适合金融零售高风险 AI。

Evidence check：review board packet 是否包含输入证据、决策类别、条件、owner、due date、risk acceptance 和后续演练 backlog。

19. 自检清单

Check	达标标准
Taxonomy	覆盖 bad retrieval、hallucination、policy bypass、tool misuse、PII leak、cost spike、latency degradation、model drift、eval regression、prompt injection
Severity	SEV 模型同时考虑客户、监管、数据、工具、成本、延迟、blast radius 和可逆性
Command	有 IC、scribe、product、architecture、model risk、security、privacy、legal/compliance、business、vendor owner
Containment	能按 model、prompt、index、retriever、tool、policy、judge、vendor 分层 rollback
Evidence	Trace 能复原 prompt、model、retrieval、tool、policy、judge、human review 和影响对象
Postmortem	包含 timeline、impact、RCA、contributing factors、control failure、corrective actions、residual risk
Corrective actions	每个行动项有 owner、due date、verification evidence、closure approver
Communications	覆盖 customer、operations、executive、board、regulator、vendor
Governance	有 reliability review board、restart gate、risk acceptance、audit evidence
Game days	至少覆盖 bad retrieval、PII leak、tool misuse、prompt injection、cost spike、model drift
Evidence pack	能形成 incident pack、ADR、dashboard spec、game day log、board brief、decision narrative

20. Final Principle

AI production reliability 的成熟度不在于事故少，而在于：

事故能早发现，
影响能快收敛，
证据能复原，
决策能追踪，
客户能补救，
风险能治理，
同类错误不会以同样方式再次发生。

成熟的 AI reliability 能力不只是解释模型行为，而是把 AI 系统做成可运营、可回滚、可审计、可复盘、可治理的生产系统。