目录
AI Incident / Postmortem / Reliability Playbook
受众：AI PM、AI Architect、Platform PM、Model Risk、AI Governance、SRE / Production Engineering、Security、Compliance、Internal Audit。
核心问题：当 AI 系统在生产中出现错误回答、错误检索、工具误用、PII 泄露、成本失控、延迟退化、模型漂移或 prompt injection 时，团队如何快速止血、可审计复盘、形成防复发闭环，并把事故经验转成架构决策、产品门禁和治理证据。
学习目标：不讲 BA 基础，不停留在“写事故报告”。目标是训练高级角色能定义 AI incident taxonomy、severity model、incident command、rollback / containment、postmortem、corrective action register、AI reliability review board、game day、regulator / board communication 和可展示作品集。
重要说明：本文是学习与作品集材料，不构成法律、监管、审计或正式安全意见。金融零售正式项目必须由 business owner、technology、security、privacy、legal、compliance、model risk、operational risk、internal audit 共同确认适用要求、通知义务、证据保留和客户补救。
1. 一句话定位
传统事故管理问的是：
服务是否宕机，多久恢复，根因是什么，如何避免再发生。
AI 事故管理要多问五层：
AI 做错了什么
-> 哪个组件导致或放大了错误
-> 哪些客户、员工、案例、工具动作和监管义务受影响
-> 如何把系统恢复到受控状态
-> 如何把事故样本变成 eval、guardrail、架构门禁和治理证据
这份手册的核心观点：
AI reliability 不是模型准确率，不是 APM，也不是事后归档。它是一个从 detect、triage、command、contain、recover、postmortem、regression、governance 到 architecture decision 的运营闭环。
适合放入作品集的最终产出：
Portfolio artifact 展示能力 AI Incident Taxonomy 能识别 AI-native failure，而不是只看 5xx 和 timeout。 Risk-Tiered Severity Model 能把客户伤害、监管风险、数据泄露、工具动作、成本和可用性合并分级。 Incident Command Runbook 能组织跨职能 war room，控制决策节奏和证据保全。 Containment / Rollback Matrix 能把模型、prompt、index、tool、policy、route、UI、vendor 变更恢复到受控状态。 Postmortem Template 能写出无责但有责任闭环的复盘。 Corrective Action Register 能把根因转成 owner、due date、control、regression evidence 和 residual risk。 Reliability Review Board Charter 能设计治理机制，而不是靠单个项目组临时判断。 Game Day / Tabletop Pack 能主动演练 bad retrieval、PII leak、tool misuse、policy bypass 和 cost spike。
2. Source Anchors
以下来源作为本文的外部锚点。正式项目应记录 access date、版本、内部 policy mapping 和 legal / risk sign-off。
Source Official / primary link 本手册使用方式 NIST AI RMF https://www.nist.gov/itl/ai-risk-management-framework 用 Govern / Map / Measure / Manage 组织 AI 风险识别、度量、处置和治理闭环；把 incident loop 接入 AI inventory、risk tier、monitoring 和 management reporting。 NIST GenAI Profile https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence 用于识别 GenAI 独特风险：生成内容错误、数据泄露、滥用、供应商依赖、评测限制、lifecycle actor 和治理证据。 NIST CSF 2.0 https://www.nist.gov/cyberframework 用 Govern、Identify、Protect、Detect、Respond、Recover 的网络安全运营语言组织 AI incident response、恢复、证据和改进。 Google SRE Postmortem Culture https://sre.google/sre-book/postmortem-culture/ 用无责复盘、明确 postmortem 触发条件、根因与贡献因素、预防措施跟踪来设计 AI postmortem culture。 MITRE ATLAS https://atlas.mitre.org/ 用 AI attack tactics / techniques 组织 prompt injection、data poisoning、model evasion、exfiltration、tool misuse 等攻击路径和事件分析。 OECD AI Incidents Monitor https://oecd.ai/en/incidents 用 AI incident / hazard、harm type、affected stakeholder、autonomy level、business function 等字段训练外部事件雷达和分类语言。 AI Observability / Cost / SLO Playbook docs/AI_OBSERVABILITY_COST_SLO_PLAYBOOK.md把 trace、span、quality dashboard、SLO、cost ledger 和 incident timeline 接入本文的响应流程。 AI Threat Modeling / Red Team / Agent Security Playbook docs/AI_THREAT_MODELING_RED_TEAM_PLAYBOOK.md把 threat model、red-team scenario、tool gateway、DLP、kill switch 和 tabletop 接入事故防复发。 AI Model Risk Management Playbook docs/AI_MODEL_RISK_MANAGEMENT_PLAYBOOK.md把 incident、model drift、eval regression、model change 和 validation issue 转成 MRM 证据。 AI Audit Evidence Binder Playbook docs/AI_AUDIT_EVIDENCE_BINDER_PLAYBOOK.md把事故日志、RCA、corrective action、approval、customer remediation 和 regression result 纳入审计证据包。
3. 高级框架：AI Reliability Operating Loop
AI 可靠性不是单点能力，而是一个运营循环。
1. Detect
-> telemetry, eval regression, customer complaint, analyst override, cost alert, security alert
2. Triage
-> classify failure type, severity, affected population, customer/regulatory impact, active blast radius
3. Command
-> assign incident commander, scribe, domain owners, decision cadence, evidence preservation
4. Contain
-> disable risky path, rollback prompt/model/index/tool/policy, route to safe fallback, pause automation
5. Recover
-> validated fix, regression tests, staged restart, enhanced monitoring, customer remediation
6. Postmortem
-> timeline, impact, root cause, contributing factors, detection gaps, response gaps, decision quality
7. Prevent recurrence
-> eval cases, guardrails, architecture decision, release gate, SLO changes, ownership changes
8. Govern
-> reliability review board, risk acceptance, audit evidence, board/regulator/customer communication
3.1 AI Incident 和普通 Incident 的差异
维度 普通生产事故 AI 生产事故 成功假象 HTTP 200 通常代表技术成功 HTTP 200 可能返回错误、幻觉、过期政策、越权工具建议或泄露内容 根因对象 code、infra、dependency、capacity model、prompt、RAG、index、policy、tool、judge、human handoff、vendor route、feedback loop 影响范围 请求失败、服务不可用、数据丢失 客户误导、错误业务动作、监管陈述错误、PII 泄露、偏见、成本爆炸、模型风险 证据要求 logs、metrics、deployment history trace、prompt version、model version、retrieved docs、tool calls、policy decisions、human review、eval result 恢复策略 rollback、reroute、scale、fix code pin model、freeze prompt、rollback index、disable tool、force template、switch to human queue、re-run eval 防复发 tests、alerts、runbook regression eval、golden set、adversarial cases、control mapping、architecture gate、review board
3.2 AI Reliability 的四个控制面
控制面 关注问题 关键 owner 典型证据 Product control plane AI 被允许回答什么、建议什么、执行什么、何时升级人工 AI PM / Business Owner use case boundary、risk tier、customer impact map、fallback policy Architecture control plane 模型、RAG、工具、策略、日志、回滚、供应商如何被隔离和控制 AI Architect / Platform Owner C4 diagram、component version map、kill switch design、route policy Risk control plane 风险等级、验证、持续监控、残余风险、审计证据是否成立 Model Risk / Compliance / Audit validation issue log、control evidence、risk acceptance、management attestation Operations control plane 谁值班、谁指挥、如何止血、如何沟通、如何复盘 SRE / Incident Commander incident state doc、timeline、postmortem、corrective action register
4. AI Incident Taxonomy
事故分类的目的不是贴标签，而是决定谁进 war room、先切断哪个路径、保留哪些证据、启动哪些通知和回归测试。
Category 典型症状 关键证据 立即 containment 防复发方向 Bad retrieval RAG 找到过期、错误、低权限、跨租户或不相关文档 query、index version、retrieved doc IDs、ACL filter、citation IDs、freshness 回滚 index；禁用问题文档；切换权威搜索；强制引用校验；高风险问题转人工 source trust、ACL test、freshness SLO、retrieval eval、index release gate Hallucination 模型生成未被证据支持的事实、政策、金额、日期、原因 prompt、model output、citation support score、judge result、human edits 限制答案类型；启用 answerability gate；高风险回复改模板；增加拒答 groundedness eval、citation checker、template locking、risk-tiered response policy Policy bypass 模型绕过业务、合规、安全、销售、投资建议或投诉规则 policy decision log、guardrail result、prompt version、conversation 启用 blocklist / classifier；关闭自由生成路径；高风险 intent 强制升级 policy engine externalization、rule test suite、prompt conflict review Tool misuse Agent 调错工具、越权查询、重复执行、错误提交、错误更新 case tool span、tool args hash、approval ID、side effect、idempotency key 关闭写工具；改 dry-run；冻结高风险 action；撤销错误动作；人工复核队列 tool permission matrix、step budget、dual control、idempotency、approval UX PII / PCI / secrets leak 输出、日志、外发工具或供应商调用泄露敏感数据 output sample、DLP result、egress log、trace redaction、affected records 停止外发；启用 redaction；关闭日志明文；通知 privacy/legal；隔离供应商 route data minimization、DLP regression、log masking、field-level policy、tenant isolation Cost spike token、retrieval、rerank、judge、tool 或 loop 导致成本异常 cost ledger、route policy、token counts、cache status、loop trace 启用 budget cap；降级模型；限制上下文；关闭 retry loop；rate limit unit economics SLO、loop detector、quota、prompt size budget、cache policy Latency degradation TTFT、total latency、tool latency、queue wait、judge latency 超阈值 latency spans、queue metrics、vendor status、retrieval timing 降级模型；绕过非必要 judge；减少 top_k；启用 cached answer；切换 fallback latency SLO、capacity test、route policy、timeout budget、dependency isolation Model drift / eval regression 离线或线上质量分数下降，升级后行为改变 eval run ID、model alias、prompt/index version、golden set delta pin 旧模型；停止 rollout；扩大 human review；回滚 route canary eval、shadow test、model pinning、approval workflow、drift dashboard Prompt injection 用户、文档、网页、邮件、tool output 诱导模型忽略规则或泄露 malicious payload、context labels、tool proposal、blocked/allowed decision 禁用受影响入口；强制 untrusted-context labeling；关闭外发和写工具 adversarial eval、instruction hierarchy、tool gateway、content isolation Data / index poisoning 知识库、记忆、feedback、fine-tuning data 被恶意或错误内容污染 source change log、document owner、index manifest、memory write log 暂停 ingest；回滚 index；清理 memory；冻结 feedback training source approval、data lineage、memory governance、poisoning red-team Judge failure LLM judge 或规则评测误判，使坏输出通过或好输出被误挡 judge prompt、rubric version、expert disagreement、score drift 暂停自动通过；改人工抽样；回滚 judge prompt；提高阈值 judge calibration、expert benchmark、rubric versioning、meta-eval Human handoff failure 本应升级人工但未升级，或人工无法理解 AI 上下文 escalation log、queue SLA、handoff payload、review notes 强制人工队列；停止自动完成；补发 case context warm handoff spec、queue SLO、review training、override analytics Vendor / dependency incident LLM、embedding、vector DB、tool API、MCP server、cloud 出现异常或变更 vendor status、model version、contract notice、route log 切换 vendor；降级本地模型；禁用第三方 connector；冻结新请求 vendor exit plan、multi-provider route、contract controls、dependency game day
4.1 事件不是事故的边界
Signal 何时只是 event 何时升级为 incident 单个差评 用户主观不满意，未违反政策或证据 命中高风险 intent、客户受损、同类模式重复、人工判定错误严重 Eval 下降 低风险指标轻微波动，未进生产 高风险 golden set fail、已上线版本退化、release gate 被绕过 成本升高 预算内季节性流量 单 use case 超预算、loop / abuse、单位经济性失效、影响服务能力 Prompt injection 被拦截 控制按预期工作 拦截率异常升高、出现未拦截样本、攻击者触发工具提议或数据外泄 客户投诉 普通服务投诉 投诉指向 AI 误导、拒绝人工、错误承诺、错误费用、歧视或隐私问题
5. Severity Model：AI 事故分级
Severity 必须按影响和控制失效，而不是按团队紧张程度。建议将金融零售 AI 事故分为 SEV0 到 SEV4。
Severity 定义 触发条件 决策权限 默认动作 SEV0 - Critical AI Harm 已造成或极可能造成重大客户伤害、监管违规、重大数据泄露、资金损失、系统性错误业务动作或董事会级声誉风险 PII / PCI 大规模泄露；客户资金或账户状态被错误改变；监管报告/AML/SAR 重大错误；高风险 agent 自动执行失控；跨租户数据暴露 Executive incident lead + Legal / Risk / CISO / Business head 立即停用相关 AI path；保全证据；启动客户补救、法律评估、监管沟通准备和董事会通知 SEV1 - High Impact 影响受监管流程、客户可见输出或高风险员工决策，但可通过快速 containment 限制范围 错误 KYC / AML 建议批量出现；客服 AI 给出错误费用/权利说明；高风险工具绕过审批但未大规模执行 Incident Commander + Business Owner + Model Risk / Compliance 冻结变更；回滚组件；扩大人工复核；24-48 小时内完成初版 postmortem SEV2 - Material Degradation 质量、安全、延迟、成本或可用性显著退化，影响业务效率或中风险流程 groundedness 低于阈值；retrieval freshness failure；p95 延迟超 SLO；单位成本超预算；人工拒绝率异常 IC + Product / Platform Owner 降级或分流；开 corrective action；一周内完成复盘和回归证据 SEV3 - Controlled Issue 局部问题，已被 guardrail、人审或监控捕获，没有客户实质影响 bad retrieval 被 answerability gate 拦截；prompt injection 被阻断；低风险 internal copilot 错误 Product / Ops Owner 记录 issue；转 regression case；下次 review board 汇总 SEV4 - Learning Signal 早期 hazard、近失误、桌面演练发现、离线 eval 失败 tabletop 发现 runbook 缺口；red-team 样本失败；外部 incident radar 触发内部 review Reliability owner 更新测试、runbook、training 和 backlog
5.1 Severity 判定维度
Dimension Low Medium High Critical Customer harm 无客户影响 客户困惑或轻微不便 客户权益、费用、投诉、账户操作受影响 大规模客户损害、资金或法律权利受影响 Regulatory exposure 无监管流程 间接流程 AML/KYC/credit/complaints/adverse action 受影响 监管报告、法定义务、消费者保护严重风险 Data sensitivity 无敏感数据 内部数据 PII、交易、身份、信用、AML 数据 大规模 PII/PCI、跨租户、外部泄露 Automation level read / summarize recommend / draft act with approval act without effective approval 或不可逆动作 Blast radius 单请求 单团队或小样本 多业务线、客户群或高价值客户 系统性、跨租户、跨区域 Detectability 自动拦截 监控发现 客户/员工发现 外部媒体、监管、审计或重大投诉发现 Reversibility 可立即修正 可补救 补救成本高 难以逆转或需客户/监管通知
5.2 AI-Specific SEV Escalation Rules
Trigger 最低 SEV 客户可见 AI 输出虚构费用、资格、拒绝原因、投诉权利、投资/信贷建议 SEV1 AI 工具调用改变客户账户、交易、退款、case 状态，且审批或幂等失效 SEV1；若不可逆或批量执行则 SEV0 PII / PCI / secrets 出现在客户可见输出、外部工具、供应商 ticket、日志导出 SEV1；大规模或跨租户则 SEV0 AML / KYC / fraud / credit 高风险流程中的 AI 建议系统性错误 SEV1；若影响监管提交或客户权益则 SEV0 Prompt injection 导致工具提议、数据外泄、policy bypass 或持久化污染 SEV1 Eval regression 已进入生产且影响高风险 use case SEV1 成本异常导致预算熔断、服务降级或其他客户流程受影响 SEV2；若影响关键服务可升 SEV1
6. Incident Command：AI War Room 设计
AI 事故不能只由工程值班处理，因为根因可能是业务政策、模型版本、知识库、工具权限、人工流程或供应商变更。War room 需要明确角色，而不是所有人同时发言。
6.1 角色与职责
Role 主要职责 关键决策 Incident Commander 控制节奏、分配任务、设定更新频率、做最终 operational decision 是否升级 SEV；是否停用 AI path；是否宣布 containment 完成 Scribe / Evidence Lead 记录 timeline、decision log、证据链接、版本和行动项 哪些证据进入 postmortem 和 audit binder AI Product Lead 定义客户/业务影响、fallback experience、补救路径和用户沟通 是否关闭某类 intent；是否切人工；如何处理受影响用户 AI Architect / Platform Lead 判断组件边界、回滚路径、route policy、日志保全和技术恢复 rollback model / prompt / index / tool / policy / vendor 的顺序 Model Risk / Validation 判断模型风险、eval regression、验证缺口和风险接受 是否需要 re-validation；是否阻止 restart Security Lead 判断 prompt injection、exfiltration、tool abuse、credential / connector 风险 是否启动 security incident；是否隔离 connector / secrets / MCP server Privacy / Data Protection 判断 PII / PCI / sensitive data 暴露、通知义务和数据最小化缺口 是否需要 privacy incident workflow Legal / Compliance 判断监管、合同、客户通知、投诉和保存义务 是否准备 regulator response、legal hold 或 board escalation Business Owner 判断业务影响、客户补救、人工产能和 residual risk 是否接受降级服务；是否暂停流程 Customer / Operations Lead 执行客户支持、人工队列、分行/坐席指引、话术 如何识别和联系受影响客户 Vendor Owner 协调模型/云/工具供应商、SLA、变更说明和证据 是否切换 vendor；是否触发合同 incident notice
6.2 War Room 节奏
阶段 时间目标 产出 First 15 minutes 确认 SEV、IC、scribe、受影响系统、初始 containment owner Incident state doc、bridge channel、decision cadence First 30 minutes 锁定 blast radius、证据保全、是否关闭高风险路径 Impact hypothesis、freeze decision、initial customer / regulatory exposure First 60 minutes 执行 containment、确认 fallback、开始受影响对象识别 Containment decision、rollback record、affected population query First 4 hours 验证 containment 有效、准备内部更新、定义补救策略 Stable state declaration、executive update、remediation backlog First 24 hours 初版 postmortem、customer/regulator/board communication draft、regression plan Draft RCA、corrective action register、communication pack First 5 business days 完整复盘、review board、risk acceptance 或 restart approval Final postmortem、evidence pack、board / regulator-ready summary
6.3 Decision Log 最小字段
Field 示例 Decision ID DEC-2026-AML-001 Time 2026-06-29 14:35 America/Chicago Decision 禁用 AML copilot 的 free-form SAR narrative 生成，仅保留 evidence summary Decision owner Incident Commander + AML Business Owner Rationale 发现 4 个高风险 case 的 narrative 引用了过期 typology；RAG index rollback 尚未完成 Alternatives considered 仅提高 judge 阈值；全量停用 copilot；切换旧 index Risk accepted 分析员效率下降，但避免错误 narrative 进入提交流程 Evidence link trace IDs、index manifest、affected case query、approval record Revisit condition 新 index 通过 50 个 regression cases 且 Model Risk 批准 restart
7. Containment / Rollback 决策矩阵
AI containment 的原则：
先停止伤害，再恢复能力；
先切断高风险动作，再修复回答质量；
先保全证据，再清理状态；
先验证 rollback，再重新放量。
7.1 组件级 Rollback Matrix
Suspected component 典型信号 最快 containment 可靠恢复条件 Model 同一 prompt / index 下输出风格、拒答、准确性突然变化 pin previous model；切备用 provider；提高 human review Canary eval 通过；供应商变更说明复核；高风险 sample 专家复核 Prompt 新版本后 policy bypass、格式错误、拒答下降 rollback prompt template；锁定 approved response templates Prompt diff review；regression eval；policy owner sign-off RAG index 错误引用、过期政策、权限错配、freshness failure rollback index；禁用受污染 source；改权威门户 fallback Index manifest 校验；ACL test；source owner approval；retrieval eval Retriever / reranker 召回相关性下降、低信任文档上位 降级到 hybrid search；调整 top_k / filters；禁用 reranker Retrieval benchmark 回归；low-trust source penalty 验证 Tool gateway 越权 tool proposal、approval bypass、重复执行 关闭写工具；只读模式；强制 manual approval Permission matrix review；idempotency test；approval trace sample Policy engine / guardrail 拦截突然下降或误拦截上升 回滚 rule set；启用 conservative mode；强制 escalation Policy test suite；false positive / false negative review Memory / feedback loop 污染记忆、错误偏好、重复错误学习 暂停 memory read/write；清理污染记录；禁用 online learning Memory lineage review；write policy test；sampling approval Judge / eval 坏输出通过、好输出被拒、分数漂移 暂停自动通过；转人工抽样；回滚 judge prompt Expert calibration；judge benchmark；threshold review UI / disclosure 客户误解 AI 能力、无法转人工、错误提示 改静态文案；强制人工入口；关闭高风险 flow UX content approval；complaint trigger test；accessibility check Vendor dependency 延迟、输出异常、版本漂移、SLA 告警 Route away；降级本地或备用模型；暂停 connector Vendor RCA；contract notice；internal canary；exit plan review
7.2 Failure Type 到 Containment
Failure First containment Second containment Recovery gate Bad retrieval 回滚 index 或禁用 source 高风险问题强制人工 通过 retrieval golden set、ACL test、freshness check Hallucination 限制自由生成；启用模板 降级为引用摘要或拒答 groundedness >= threshold；expert sample pass Policy bypass 启用 conservative policy 暂停相关 intent policy regression suite pass；compliance sign-off Tool misuse 禁用写工具 撤销 / 补偿 side effect；重放 idempotency check tool permission test；approval audit sample PII leak 停止外发和明文日志 识别 affected records；privacy workflow DLP regression pass；redaction evidence；legal decision Cost spike 预算熔断和 rate limit 降级模型、缩短上下文、停 retry loop unit cost 回到 SLO；loop detector verified Latency degradation route to fast path 关闭非必要 judge / rerank / long context p95 latency stable；capacity margin confirmed Model drift pin previous model 增加人工复核 canary eval + shadow traffic pass Prompt injection 隔离入口和不可信内容 关闭工具 / 外发 / memory write adversarial regression pass；security approval
7.3 Kill Switch 分层
Switch level 作用范围 适用场景 风险 L1 Response mode switch 从生成回答切到模板、拒答或引用摘要 hallucination、policy bypass、customer-facing risk 用户体验下降，但业务不中断 L2 Intent switch 关闭某类 intent，例如投诉、退款、KYC exception 单个业务 flow 失控 人工队列压力上升 L3 Tool switch 禁用写工具、外发工具、高风险查询工具 tool misuse、prompt injection、PII leak 自动化收益下降 L4 Component switch 回滚 model、prompt、index、policy、judge 组件变更引发回归 可能回到旧缺陷，需要回归验证 L5 System switch 停用整个 AI capability SEV0 或无法确认 blast radius 业务影响最大，但止血最确定
8. Detection and Evidence：没有证据就没有复盘
AI incident 的证据必须能回答：
谁问了什么
系统看到了什么
模型/检索/工具/策略如何决策
输出影响了谁
我们何时知道、何时止血、何时确认恢复
8.1 Detection Signals
Signal 检测来源 可触发的 incident 类型 Human reject spike QA / reviewer decision hallucination、bad retrieval、policy bypass、model drift Citation support failure groundedness judge、citation checker hallucination、bad retrieval、index poisoning Tool policy denied spike tool gateway prompt injection、excessive agency、permission issue DLP hit output filter、egress gateway、log scanner PII leak、secrets leak、cross-tenant data issue Cost per case spike cost ledger loop、long-context expansion、abuse、vendor price/route change p95 / p99 latency breach tracing / APM vendor degradation、retrieval slowdown、judge bottleneck Eval regression CI / release gate / scheduled eval model drift、prompt regression、index regression Customer complaint contact center、complaints system、social channel customer-facing hallucination、handoff failure、wrong advice Security alert SIEM、WAF、MCP gateway、red-team monitor prompt injection、tool abuse、exfiltration External incident radar OECD monitor、industry reports、vendor notices hazard review、third-party dependency risk
8.2 Evidence Pack 最小内容
Evidence 字段 Incident state doc SEV、status、IC、scribe、systems、affected use cases、current containment Timeline detection、triage、decision、rollback、recovery、communication、restart Trace sample trace IDs、request IDs、user role、workflow step、risk tier Version map model、prompt、index、retriever、reranker、judge、policy、tool schema、UI copy Impact query affected customers / cases / employees / transactions / outputs Tool side effect log tool name、arguments hash、approval ID、result、reversal / compensation Data exposure assessment fields exposed、records count、destination、retention、redaction status Eval / regression result failed cases、baseline, current score, rerun after fix Communication record internal update、customer script、regulator memo、board brief Corrective action register owner、due date、control, verification evidence, closure decision
8.3 Trace Fields for Incident Readiness
Field group Required fields Business context use_case_id、risk_tier、workflow_step、customer_segment、employee_role、case_id AI component versions model_alias、model_version、prompt_id、prompt_version、index_version、policy_version、judge_version Retrieval query、filters、retrieved_doc_ids、citation_doc_ids、source_trust、freshness_days、ACL decision Tool tool_name、risk_level、input_hash、policy_decision、approval_id、side_effect、idempotency_key Safety guardrail_decision、DLP_decision、prompt_injection_score、escalation_reason Quality groundedness_score、citation_correctness、answerability、human_review_decision、edit_reason Performance ttft_ms、total_latency_ms、retrieval_latency_ms、tool_latency_ms、judge_latency_ms Cost input_tokens、output_tokens、reasoning_tokens、retrieval_cost、judge_cost、tool_cost、total_cost Audit trace_id、request_id、retention_class、redaction_status、access_control_group
9. Postmortem：无责，但不能无行动
AI postmortem 的质量标准：
Bad postmortem Good postmortem “模型幻觉了，已优化 prompt” 明确哪个模型 / prompt / index / policy / tool / workflow 组合失效，为什么控制没有拦住，如何验证不会复发 “用户问法太刁钻” 识别产品边界、answerability gate、fallback 和 user education 是否不足 “坐席没有认真审核” 分析人审界面是否提供证据、风险提示、差异高亮和足够时间 “供应商模型变了” 检查 vendor change notice、model pinning、canary eval、route policy 和 exit plan “已加强监控” 说明新增指标、阈值、owner、alert route、演练和验证样本
9.1 Postmortem Template
# AI Incident Postmortem: AML Copilot Bad Retrieval, 2026-06-29
## 1. Executive Summary
- Incident ID: AI-INC-AML-2026-001
- Severity: SEV1
- Status: Contained, restart pending Reliability Review Board approval
- Use case: AML Copilot evidence summary and SAR narrative draft
- Business owner: Head of AML Operations
- Incident commander: AI Platform SRE Lead
- Detection time: 2026-06-29 09:42 America/Chicago
- Containment time: 2026-06-29 10:18 America/Chicago
- Recovery time: 2026-06-30 16:00 America/Chicago after regression approval
- Customer / business impact: 23 AML cases required analyst re-review; no customer-facing message sent
- Regulatory / privacy / security exposure: Potential regulatory narrative quality risk; no PII leakage and no SAR submission confirmed
- Current residual risk: Low after index rollback, narrative freeze and freshness regression
## 2. What Happened
- Observable symptom: Senior analyst found SAR draft citing a typology retired in 2026-Q1
- AI failure category: Bad retrieval and hallucinated regulated narrative
- Affected users / cases / customers / transactions: 9 analysts, 23 AML cases, 0 customer contacts, 0 submitted SARs
- Affected AI components: AML policy index v2026.06.28, retrieval filter v1.7, narrative prompt v3.2
- First known bad version: index-aml-policy-2026.06.28
- Last known good version: index-aml-policy-2026.06.21
## 3. Impact
| Impact area | Assessment | Evidence |
|---|---|---|
| Customer harm | No direct customer impact; cases stayed in analyst workflow | case_state_query_2026_06_29.csv |
| Operational impact | 23 cases re-reviewed; analyst queue delayed by 4.5 hours | AML queue dashboard snapshot |
| Financial impact | No direct loss; 18 analyst hours used for re-review | workforce impact estimate |
| Regulatory / compliance | Potential SAR quality risk if drafts had been submitted | SAR submission cross-check showing zero submitted |
| Data / privacy | No PII leaked outside approved systems | DLP report DLP-AML-2026-001 |
| Security | No prompt injection or external access found | SIEM query and context payload review |
| Reputation | Internal operational issue; no external communication required | Legal / Compliance decision log |
## 4. Timeline
| Time | Event | Owner | Evidence |
|---|---|---|---|
| 09:42 | Analyst reports outdated typology in SAR draft | AML QA Lead | QA ticket AML-QA-7782 |
| 09:50 | IC opens SEV1 war room and assigns scribe | AI Platform SRE Lead | incident channel export |
| 10:05 | Impact query identifies 23 affected cases | Data Engineer | impacted_case_query.sql |
| 10:18 | SAR narrative generation disabled; evidence summary remains read-only | Platform Lead | route_policy_change_2026_06_29 |
| 10:31 | Index rolled back to v2026.06.21 | RAG Owner | index manifest diff |
| 12:10 | Analysts complete first 10 case re-reviews | AML Operations | QA review sample |
| 16:40 | No submitted SARs found for affected drafts | Compliance | SAR submission reconciliation |
## 5. Detection
- Which signal detected the issue: Human QA by senior AML analyst
- Which signal should have detected it earlier: Freshness regression in the index release gate
- Detection gap: Citation checker validated source existence but not effective date
- Alert / dashboard changes: Added retired-source citation alert and freshness breach tile
## 6. Containment and Recovery
- Immediate containment: Disabled SAR narrative draft generation for AML Copilot
- Rollback / route / kill switch used: L1 response mode switch plus index rollback to v2026.06.21
- Fallback experience: Analysts receive evidence summary and link to policy portal; narrative written manually
- Recovery validation: 50 AML golden cases, 10 retired-typology adversarial cases and 23 affected cases rechecked
- Restart approval: Required from Model Risk, AML Business Owner and AI Reliability Review Board
## 7. Root Cause and Contributing Factors
### Direct root cause
The AML RAG index release included retired typology guidance without effective-date filtering, allowing outdated evidence to support generated SAR narrative text.
### Contributing factors
| Factor | Explanation | Control gap |
|---|---|---|
| Product boundary | Narrative generation was allowed for all AML typology questions | No high-risk narrative freshness gate |
| Architecture | RAG index and narrative prompt were released independently | Component release coupling missing |
| Data / RAG | Retired documents remained searchable with low metadata quality | Effective-date filter and source owner approval missing |
| Model / prompt | Prompt encouraged concise narrative even when evidence was stale | Answerability instruction did not include retired-source refusal |
| Tool / workflow | No external write tool was involved | Tool control worked as designed |
| Monitoring | Citation support did not check source validity period | Freshness metric absent |
| Human review | Analyst QA caught the issue before submission | Human review worked but late in workflow |
| Governance | Index release did not require Model Risk review for retired policies | Gate scope too narrow |
## 8. Why Existing Controls Failed
| Control | Expected behavior | Actual behavior | Fix |
|---|---|---|---|
| Citation checker | Ensure narrative claims are supported by cited evidence | Confirmed document existed but ignored retirement date | Add effective-date validation and retired-source block |
| Index release gate | Prevent stale or unauthorized policy from entering production | Checked document count and ACL only | Add source owner approval and freshness regression |
| Human review | Catch low-quality narrative before regulatory submission | Caught issue before submission | Move freshness alert earlier in workflow |
## 9. Corrective Actions
| Action ID | Action | Owner | Due date | Verification evidence | Closure approver |
|---|---|---|---|---|---|
| CA-AML-001 | Add effective-date filter to AML retrieval | RAG Owner | 2026-07-03 | 60-case retrieval regression pass | Model Risk |
| CA-AML-002 | Block narrative generation when citation freshness fails | AI Platform Lead | 2026-07-05 | 10 retired-typology cases refused or escalated | AML Business Owner |
| CA-AML-003 | Add AML index release owner approval | Data Governance Lead | 2026-07-08 | signed source manifest in evidence binder | Compliance |
## 10. Regression and Recurrence Prevention
- New eval cases: 50 AML SAR narrative golden cases with citation support and freshness assertions
- New adversarial cases: 10 retired typology and conflicting policy cases
- New monitoring: retired-source citation rate, freshness_days p95, narrative blocked by freshness gate
- Release gate changes: AML index release requires source owner approval and freshness regression pass
- Architecture decision records: ADR-AML-004 requires evidence freshness gate before regulated narrative generation
- Runbook changes: Bad retrieval containment now requires index rollback plus affected case query
- Game day scenario added: AML retired typology tabletop for quarterly reliability exercise
## 11. Communication
- Customer communication: Not required because no customer-facing output or account action occurred
- Regulator / examiner communication: Prepared evidence pack for examiner request; no proactive notice recommended by Legal / Compliance
- Board / executive communication: Included in monthly AI risk dashboard as SEV1 contained incident
- Internal operations guidance: AML analysts instructed to use manual narrative drafting until restart approval
## 12. Residual Risk Decision
- Residual risk: Low for restarted narrative generation after freshness gate; medium for same-day policy updates until source SLA improves
- Risk owner: Head of AML Operations
- Accepted / rejected / requires additional controls: Accepted with source-owner SLA and monthly freshness audit
- Next review date: 2026-08-15 AI Reliability Review Board
9.2 Root Cause Tree for AI Incidents
Layer Questions Business boundary AI 是否被允许处理这个 intent、客户类型、地区、产品或风险等级？ Data source 证据是否权威、最新、可访问、按权限过滤、可追溯？ Retrieval 查询改写、filter、top_k、reranker、citation 是否把正确证据带入上下文？ Prompt / policy 指令是否冲突、过宽、未区分可信和不可信上下文？ Model 模型是否升级、route 改变、参数改变、能力或拒答行为变化？ Tool 工具权限、schema、审批、幂等、回滚、side effect 是否受控？ Judge / guardrail 拦截或评分是否覆盖该失败模式，是否被绕过或漂移？ Human review 人审是否看到足够证据，是否有时间、权力和流程去覆盖 AI？ Monitoring 质量、安全、成本、延迟、业务 outcome 是否有阈值和 owner？ Governance 变更是否经过 gate，风险是否被正式接受，owner 是否明确？
10. Corrective Action Register
Corrective action 的目标不是关闭 Jira，而是降低复发概率或影响范围。每个行动项都必须有验证证据。
Field 填写标准 合格样例 Action ID 与 incident ID 关联 CA-AML-2026-001 Failure mode 对应 taxonomy Bad retrieval + hallucinated SAR narrative Control objective 要降低的风险 高风险 AML narrative 必须由有效证据支持 Action 可执行变更 增加 citation support gate，低于阈值时只输出 evidence summary Owner 单一责任人 AML AI Platform Lead Due date 明确日期 2026-07-12 Verification evidence 可复核结果 50 个 SAR golden cases 全部通过；10 个过期 typology cases 被拒答 Risk reduction 预期效果 降低错误 narrative 进入 analyst workflow 的概率 Residual risk 仍然存在的风险 新 typology 发布当天仍可能存在 freshness gap Closure approver 有权关闭的人 Model Risk + AML Business Owner Next review 防止行动项假关闭 2026-08-15 review board
10.1 Action 类型
Type 例子 何时使用 Eval action 新增 golden / adversarial / regression cases 事故样本可转成可重复测试 Architecture action 增加 policy engine、tool gateway、route isolation、index manifest 根因是边界和控制不足 Product action 缩小 use case、改变 disclosure、增加人工入口、限制答案类型 AI 被放进了不适合的客户旅程 Data action source trust、owner approval、freshness SLO、ACL、data contract RAG / data lineage 是根因 Operations action alert、runbook、on-call、war room、rollback drill 响应慢、检测晚、证据不足 Governance action review board、approval gate、risk acceptance、policy update 决策链或责任边界不清 Vendor action SLA、model pinning、change notice、exit plan、support escalation 外部依赖导致或放大事故
10.2 Recurrence Prevention Ladder
Level 防复发强度 示例 L0 Manual reminder 依赖人记住 仅培训坐席“注意检查引用” L1 Checklist 依赖流程提醒 上线前检查 prompt / index version L2 Monitoring 能发现 bad citation rate alert L3 Gate 能阻止发布 eval fail 时 CI 阻止 prompt / index release L4 Runtime control 能实时阻断 answerability gate、DLP、tool gateway、policy engine L5 Architectural isolation 能限制 blast radius model route isolation、read-only mode、tenant boundary、kill switch L6 Governance system 能持续学习 review board、audit sample、game day、risk appetite update
复盘后的行动项至少应达到 L3；SEV0 / SEV1 的核心失效应优先设计到 L4 或 L5。
11. Architecture and Product Mapping
11.1 从事故类型到架构控制
Incident type Product decision Architecture control Governance evidence Bad retrieval 明确哪些答案必须引用权威来源 Source registry、ACL filter、index manifest、citation checker Data owner approval、retrieval eval report Hallucination 高风险答案从自由生成改为模板 / 引用摘要 Answerability classifier、groundedness gate、template composer Prompt approval、quality dashboard Policy bypass 将业务规则从 prompt 中外置 Policy engine、intent classifier、decision table、guardrail trace Compliance sign-off、policy version log Tool misuse 定义 read / draft / act 边界 Tool gateway、ABAC、dual control、idempotency、dry-run Permission matrix、approval evidence PII leak 限制输入、输出、日志和外发字段 DLP、field redaction、egress gateway、log masking Privacy impact record、DLP test Cost spike 定义每个 outcome 的 unit economics Cost ledger、quota、route policy、cache、loop detector Budget review、FinOps dashboard Latency degradation 设定按风险分层的响应体验 Fast path、async review、timeout budget、fallback response SLO document、capacity test Model drift 明确模型升级不是透明小变更 Model alias pinning、canary、shadow eval、rollback Change approval、eval comparison Prompt injection 将不可信内容和工具权限隔离 Context labeling、sandbox、tool approval、memory write policy Red-team report、ATLAS mapping
11.2 AI Reliability ADR 模板
# ADR-AML-004: Require Evidence Freshness Gate Before SAR Narrative Generation
## Context
- Incident / risk trigger: AI-INC-AML-2026-001 showed SAR narrative could cite retired typology guidance
- Affected use case: AML Copilot SAR narrative drafting
- Risk tier: Tier 1 regulated decision-support workflow
- Current weakness: Citation checker verifies support but does not verify source effective date or retirement status
## Decision
- Chosen architecture / product control: Add evidence freshness gate before narrative generation; stale evidence forces evidence-summary-only mode
- Components changed: Retrieval metadata filter, citation checker, narrative composer, release gate
- Runtime behavior: If any cited AML policy source is retired or outside effective date, the system refuses narrative generation and escalates to analyst manual drafting
- Rollback approach: Disable freshness gate through controlled feature flag only after IC and Model Risk approval; default fallback is evidence-summary-only mode
## Options Considered
| Option | Benefit | Risk | Decision |
|---|---|---|---|
| Prompt-only warning | Fast to implement | Model may still use stale evidence under pressure | Rejected |
| Freshness gate before narrative | Blocks stale regulated narrative at runtime | Some valid edge cases require manual drafting | Chosen |
| Full narrative shutdown | Maximum safety | Removes productivity benefit for low-risk summaries | Reserved for SEV1 containment |
## Consequences
- Reliability impact: Reduces recurrence of stale-source SAR narrative and creates auditable block events
- Customer impact: No direct customer-facing impact; analyst workflow may require more manual drafting during stale-source periods
- Cost / latency impact: Adds citation metadata check with negligible latency; may increase manual review hours
- Governance impact: AML source owner approval becomes mandatory for index release
- Residual risk: Same-day policy changes can still create a temporary gap until source SLA is met
## Verification
- Eval cases: 50 AML golden cases and 10 retired-source adversarial cases
- Monitoring: retired-source block rate, freshness_days p95, manual drafting escalation count
- Game day scenario: Quarterly AML retired typology tabletop
- Review board approval: Required before re-enabling narrative generation
11.3 Release Gate After Incident
Gate Required evidence G1 Scope gate Use case boundary updated；forbidden uses 明确；fallback owner 确认 G2 Component gate model / prompt / index / tool / policy version map 完整 G3 Eval gate 事故样本进入 golden set；regression 通过；expert sample 通过 G4 Security / privacy gate DLP、prompt injection、tool misuse、egress、log masking 测试通过 G5 Operations gate runbook、kill switch、on-call、alert、dashboard 已更新 G6 Governance gate Model Risk / Compliance / Business Owner 对 residual risk 做出决策 G7 Restart gate 分阶段放量计划、rollback condition、monitoring window 明确
12. Customer / Regulator / Board Communication
沟通不是 PR 话术，而是控制面的一部分。越早准备结构化事实，越少在压力下做模糊承诺。
12.1 Communication Matrix
Audience 关注问题 不能说 应该准备 Customers 我是否受影响，发生了什么，如何补救，是否需要行动 “AI 偶尔会错”“我们已优化模型” 事实、影响、补救、人工渠道、时间线、后续通知 Frontline / Operations 现在怎么处理客户、案例、例外和人工队列 “先凭经验处理” 临时 SOP、话术、升级条件、受影响 case list Executives 影响范围、风险、恢复、补救成本、下一步决策 只报技术细节 SEV、impact, containment、customer/regulatory exposure、decision needed Board / Risk Committee 是否触及风险偏好、控制是否失效、治理是否有效 “项目团队会修好” root cause theme、risk appetite、corrective action、accountability、timeline Regulator / Examiner 是否影响受监管义务、客户权益、数据、模型风险、第三方 未经确认的法律结论 verified facts、controls、affected population、remediation、evidence path Vendor 是否供应商导致、需要什么证据、SLA 是否触发 只口头描述 trace sample、timestamps、model/route IDs、contract notice、RCA request
12.2 Executive Update 模板
# AI Incident Executive Update
- Incident ID:
- Severity:
- Current status:
- Affected capability:
- What happened:
- Customer / business impact:
- Data / privacy / security exposure:
- Regulatory exposure:
- Containment completed:
- Remaining risk:
- Decisions needed:
- Next update time:
12.3 Regulator / Examiner Response Pack
Section 内容 Factual summary 已验证事实、时间线、影响范围，不包含猜测 System description AI system、workflow、human role、model/RAG/tool components Control design 上线前控制、监控、人工复核、policy、DLP、tool gateway Control failure 哪些控制未生效、为什么未生效 Customer impact 受影响对象、补救、通知、投诉处理 Remediation containment、corrective actions、completion evidence Governance owner、review board、risk acceptance、management oversight Evidence index trace、logs、evals、approval、communication、postmortem 路径
12.4 Board Brief 结构
Slide 内容 1. Incident summary SEV、时间线、影响、当前状态 2. Business and customer impact 客户、员工、案例、资金、投诉、品牌 3. Risk and control view 失效控制、残余风险、风险偏好影响 4. Management action containment、corrective actions、owner、due date 5. Governance ask 需要董事会知情、批准、预算、风险接受或监督的事项
13. AI Reliability Review Board
AI Reliability Review Board 不是大型委员会，而是高风险 AI 系统的运营门禁。它把 incident、eval regression、architecture change、model update、vendor change 和 risk acceptance 放在同一个决策桌上。
13.1 Charter
Item 内容 Mission 降低 AI 事故复发概率和影响范围，确保高风险 AI use case 的 release、change、restart 和 residual risk 有证据、有 owner、有审批 Scope Tier 1 / Tier 2 AI systems；SEV0-SEV2 incidents；model/prompt/index/tool/policy 高风险变更 Decision rights approve restart、require additional controls、block release、accept residual risk、escalate to executive risk committee Members AI Product、AI Platform、Architecture、SRE、Security、Privacy、Model Risk、Compliance、Business Owner、Audit observer Cadence SEV0/1 event-driven；SEV2 weekly；normal monthly reliability review Inputs incident postmortem、eval trend、SLO breach、cost trend、red-team result、vendor notice、customer complaint trend Outputs decision log、corrective action status、risk acceptance、architecture ADR、game day backlog
13.2 Review Agenda
Agenda item 决策问题 Open incidents 是否真正 containment，是否仍有客户/监管风险 Corrective actions 是否有逾期、假关闭、缺验证证据的行动项 Eval and drift 哪些模型 / prompt / index / tool 出现质量或安全退化 SLO / cost / latency 是否触发 error budget 或 budget guardrail Customer harm signals 投诉、人工升级、拒绝率、错误补救是否异常 Vendor changes 是否需要 re-validation、model pinning 或 exit test Game day findings 演练发现是否进入 release gate 和 runbook
13.3 Decision Categories
Decision Criteria Evidence Restart approved containment 完成，fix 通过 regression，monitoring window 明确 eval report、trace sample、owner sign-off Restart with restrictions 核心风险降低，但 residual risk 仍需限制 disabled intent、human review、traffic cap Continue shutdown 影响或根因不清，无法证明控制有效 incomplete RCA、failed regression、unresolved privacy/legal issue Risk accepted 业务决定接受残余风险，并有补偿控制 risk memo、business sign-off、review date Escalate 超出 board 权限或触及监管/董事会风险偏好 executive memo、legal/compliance analysis
14. Game Days and Tabletops
Game day 的目标不是表演成熟，而是暴露系统、流程和组织在压力下的真实弱点。
14.1 Game Day 设计原则
原则 解释 基于真实 trace 用生产相似的请求、知识库、工具权限和角色，不只用抽象故事 覆盖跨职能决策 PM、Architect、SRE、Security、Privacy、Model Risk、Compliance 都要做决定 演练证据保全 检查是否能找到 prompt、index、retrieval、tool、DLP、approval 和 impacted population 演练 kill switch 不只讨论是否关闭，要实际验证配置、权限和恢复步骤 生成行动项 每次演练必须产出 corrective actions 和 owner
14.2 Tabletop Scenarios
Scenario Inject Expected decisions Evidence to test Bad retrieval in AML copilot 分析员发现 SAR narrative 引用了过期 typology 停 narrative 生成、回滚 index、识别受影响 case、Model Risk review index manifest、retrieved docs、case list、regression eval KYC assistant PII leak 外部供应商 ticket 中出现客户身份证件摘要 关闭外发、privacy workflow、log scan、vendor notice egress log、DLP decision、affected records、retention Payment dispute agent tool misuse Agent 重复提交 provisional credit 禁用写工具、撤销重复动作、检查 idempotency、客户补救 tool span、approval ID、transaction ledger、compensation Customer-facing AI hallucination 客户被告知不存在的 fee waiver 权利 关闭自由生成、发布坐席话术、识别客户、投诉处理 output samples、customer IDs、policy source、QA decisions Prompt injection via uploaded PDF 投诉附件诱导 Agent 调用外部 email tool 隔离 upload path、关闭外发工具、security incident triage malicious payload、context label、tool proposal、blocked/allowed decision Cost spike via loop Planner 重复检索和 judge，成本 8 倍上升 预算熔断、rate limit、关闭 retry loop、恢复 unit cost cost ledger、trace loop、route policy、quota Latency degradation p95 从 8s 到 35s，人工队列积压 降级模型、异步处理、减少 rerank、业务 SLA 更新 latency spans、queue wait、fallback success Model drift after vendor upgrade 同一 golden set fail 率从 3% 到 18% pin 旧模型、暂停 rollout、vendor RCA、shadow eval model alias、eval diff、vendor notice、route log
14.3 Game Day Scorecard
Dimension 1 - Weak 3 - Adequate 5 - Strong Detection 只能靠人工发现 有部分 alert，但信号分散 自动 alert 命中 failure mode，owner 清晰 Triage 争论分类和 SEV 30 分钟内完成初判 15 分钟内明确 SEV、blast radius 和 war room Containment 不知道关闭哪里 能手动关闭主要路径 kill switch 分层、可验证、可回滚 Evidence 找不到版本或 trace 能找到核心日志 trace、版本、impact query、tool side effect 完整 Communication 技术更新难被业务理解 有基础 executive update 客户/监管/董事会版本清晰、事实一致 Prevention 只写行动项 有 owner 和 due date 行动项有 regression evidence 和 review board closure
15. 金融零售 AI 事故案例
15.1 AML Copilot：错误检索导致 SAR narrative 风险
项目 内容 System AML Copilot，为分析员总结告警证据并起草 SAR narrative Failure RAG 检索到过期 typology guidance，模型把旧规则写成当前风险理由 Severity SEV1；若 narrative 已提交监管则升 SEV0 Detection Senior analyst QA 发现 narrative 与最新政策冲突；citation checker 未检查 effective date Containment 关闭 SAR narrative 自动草稿，只保留 evidence summary；回滚 index；高风险 case 全部人工复核 Root cause Index manifest 没有 policy effective date；retriever freshness filter 未启用；release gate 未包含过期 typology regression Corrective actions 增加 freshness SLO；policy source owner 审批；50 个 AML regression cases；narrative 输出必须带 citation support score Portfolio artifact AML AI Incident Postmortem + Retrieval Freshness ADR + SAR Narrative Regression Pack
15.2 KYC Assistant：PII 外发到供应商系统
项目 内容 System KYC 文档审核助手，自动总结客户身份证明和缺失项 Failure Assistant 把客户身份证件摘要写入外部 vendor support ticket，ticket 系统不在批准处理范围 Severity SEV1；若包含大规模证件号或跨境处理则 SEV0 Detection DLP 在 egress gateway 命中身份证号 pattern；privacy alert 自动升级 Containment 关闭外发 ticket tool；清理 vendor ticket；启用 legal hold；识别 affected customers；暂停相关日志导出 Root cause Tool gateway 未区分内部 case note 和外部 support ticket；DLP 只在最终回答启用，未覆盖 tool payload Corrective actions egress policy 按目的绑定；tool input field redaction；vendor processor matrix 更新；PII leak adversarial eval Portfolio artifact KYC PII Incident Pack + Egress Policy Matrix + DLP Regression Evidence
15.3 Payment Dispute Agent：工具误用导致重复 provisional credit
项目 内容 System 支付争议处理 Agent，读取争议材料、建议下一步、草拟客户通知 Failure Agent 在 retry 后重复调用 provisional credit tool，幂等 key 未绑定 dispute ID 和 amount Severity SEV1；若批量资金影响或无法撤销则 SEV0 Detection 账务对账发现同一 dispute 两笔 provisional credit；tool side effect alert 滞后 Containment 禁用写工具；切只读建议模式；冻结受影响 dispute；启动客户和账务补救 Root cause Tool schema 未要求 idempotency key；approval UI 未展示已执行 action；retry policy 未识别 side effect Corrective actions 写工具强制 idempotency；approval ledger；retry 禁止重复 side effect；payment game day 每季度演练 Portfolio artifact Payment Agent Tool Misuse Postmortem + Tool Permission / Idempotency ADR
15.4 Customer-Facing AI：错误费用减免承诺
项目 内容 System 信用卡客服 AI，客户可直接询问费用、争议、权益和投诉 Failure AI 虚构“年费可无条件减免一次”的政策，客户认为银行已承诺 Severity SEV1，因为客户可见且涉及费用和权利；若大规模传播则 SEV0 Detection 投诉系统中 fee waiver 关键词异常上升；QA 抽样确认 AI 输出无来源支持 Containment 关闭费用政策自由回答；改用模板 + 权威 API；受影响客户转人工补救 Root cause Answerability gate 未覆盖费用承诺；UI 披露没有区分一般信息和正式承诺；citation optional Corrective actions fee / rate / eligibility 全部 template locking；complaint trigger dashboard；customer remediation script Portfolio artifact Customer-Facing AI Incident Memo + Regulated Conversation Boundary Card
15.5 Fraud Review Copilot：模型漂移导致误报解释退化
项目 内容 System 欺诈复核 Copilot，帮助运营解释交易阻断原因并建议下一步 Failure Vendor 模型升级后，解释变得更自信但证据更少，human override 从 6% 升到 19% Severity SEV2；若导致客户账户错误冻结或拒绝申诉则 SEV1 Detection Human reject spike + golden set scheduled eval fail Containment Pin previous model；扩大人工复核；暂停新模型 rollout Root cause Model alias 未 pin；vendor change notice 未进入 AI release gate；shadow eval 样本未覆盖冻结解释 Corrective actions model alias pinning；vendor change intake；fraud explanation golden set；review board restart approval Portfolio artifact Model Drift Incident Review + Vendor Change Gate Checklist
16. Templates and Artifacts
16.1 AI Incident Intake Card
Field Example Incident ID AI-INC-2026-001 Reporter AML QA Lead Detection channel Human QA + citation support alert Use case AML Copilot SAR narrative Suspected category Bad retrieval / hallucination Initial SEV SEV1 Customer / regulatory exposure Potential regulatory narrative quality issue; no submission confirmed Active blast radius 23 cases generated since index v2026.06.28 Immediate ask Disable SAR narrative generation and run affected case query
16.2 Affected Population Query Spec
Question Query requirement 哪些输出可能受影响 按 prompt_version、index_version、model_route、time window、intent、risk_tier 查询 哪些客户或案例受影响 关联 case_id、customer_id、employee_id、channel、workflow_state 是否发生工具 side effect 关联 tool_call、approval_id、side_effect、ledger / case status 是否已经客户可见或监管提交 关联 message_sent、document_exported、SAR_submitted、letter_generated 是否已人工覆盖 关联 review_decision、edit_reason、override_time
Field 填写标准 Affected segment 客户类型、产品、地区、时间范围 Harm hypothesis 错误费用、错误拒绝、错误说明、隐私暴露、延迟、错误补偿 Verification method trace + CRM + transaction / case system cross-check Remediation action 更正通知、费用调整、人工回访、投诉升级、数据删除请求 Approval Legal、Compliance、Business Owner、Customer Ops Evidence customer list hash、message template、completion status、exception log
16.4 AI Reliability Dashboard Spec
Tile Metric Threshold Owner Incident count by category SEV0-SEV4 per month SEV1+ immediate review Reliability Lead Bad retrieval rate citation unsupported / stale / ACL fail High-risk use case > 1% alert Data / RAG Owner Hallucination / groundedness groundedness fail rate Tier 1 > 0.5% alert Model Risk Tool policy violations blocked / attempted / allowed high-risk tool calls Any unexpected allowed high-risk call Security / Tool Owner PII / DLP DLP hit by route and destination Any external PII hit Privacy Cost unit economics cost per resolved case / analysis / dispute > approved budget guardrail Platform PM / FinOps Latency SLO p95 TTFT / total / tool latency breach for 3 intervals SRE Human reject / override rejected outputs, edit distance, override reason spike vs baseline Business Ops Eval regression scheduled eval pass rate by version high-risk fail blocks release Validation Corrective action aging overdue, no evidence, repeated failure overdue SEV1 action escalates Review Board
16.5 Reliability Review Board Packet
Section 内容 Open incidents SEV、status、owner、next decision Closed incidents postmortem link、RCA theme、closure evidence Corrective actions overdue、blocked、accepted risk、closed with evidence Eval / drift regressions、model changes、prompt / index changes Security / privacy injection attempts、PII events、tool gateway exceptions SLO / cost latency, availability, unit cost, error budget Customer harm complaints、handoff failure、remediation status Decisions requested restart、block release、accept residual risk、fund control
17. 30-Day Lab：AI Incident and Reliability Portfolio Pack
目标：30 天内围绕一个金融零售 AI use case，完成一套可展示的 AI Incident / Postmortem / Reliability 作品集。推荐主线选择 AML Copilot、KYC Assistant、Payment Dispute Agent 或 Customer-Facing AI。
Day 任务 Artifact 1 选择 use case，定义 AI 行为边界、risk tier、禁止用途 Use Case Boundary Card 2 画系统组件：model、prompt、RAG、tool、policy、judge、human handoff Component Map 3 定义 AI incident taxonomy，选 8-10 个最相关 failure modes Incident Taxonomy 4 设计 SEV0-SEV4 severity model Severity Matrix 5 设计 incident command RACI 和 war room cadence Incident Command Runbook 6 定义 trace evidence 字段 Evidence Schema 7 设计 AI reliability dashboard Dashboard Spec 8 写 bad retrieval scenario Scenario Card 1 9 写 hallucination / policy bypass scenario Scenario Card 2 10 写 tool misuse scenario Scenario Card 3 11 写 PII leak scenario Scenario Card 4 12 写 cost spike / latency degradation scenario Scenario Card 5 13 写 model drift / eval regression scenario Scenario Card 6 14 设计 containment / rollback matrix Rollback Matrix 15 设计 kill switch 分层 Kill Switch Spec 16 选择一个 scenario，写完整 incident timeline Incident Timeline 17 写 affected population query spec Impact Query Spec 18 写 postmortem 初版 Postmortem Draft 19 写 root cause tree 和 contributing factors RCA Worksheet 20 写 corrective action register Corrective Action Register 21 把事故样本转成 eval regression cases Regression Pack 22 写 architecture ADR Reliability ADR 23 写 customer / operations communication Customer and Ops Comms 24 写 regulator / board brief Governance Communication Pack 25 设计 reliability review board charter Review Board Charter 26 运行一次 tabletop，记录 decision log Game Day Decision Log 27 根据 tabletop 更新 runbook 和 dashboard Runbook Revision 28 汇总 evidence binder index Evidence Index 29 写 1500-2500 字 portfolio case study Portfolio Case Study 30 准备 8-10 个面试问答和 5 分钟讲述 Interview Story Pack
17.1 30 天最终交付包
Artifact 面试价值 Executive one-pager 展示你能向高管解释 AI 事故风险和决策 System and control map 展示架构判断，而不是只做流程文档 Incident taxonomy + SEV model 展示高级风险分类能力 Postmortem + corrective action register 展示生产可靠性和治理闭环 Game day pack 展示主动验证和组织协同能力 Regression pack 展示把事故样本转成 EvalOps 的能力 Board / regulator communication 展示金融零售场景的风险表达
18. Interview Answers
Q1：AI incident response 和普通 SRE incident response 最大区别是什么？
版本 回答 30 秒 普通 SRE 主要处理 availability、latency、error rate。AI incident 还要处理语义正确性、证据支持、政策边界、工具动作、数据泄露、模型漂移和客户/监管影响。HTTP 200 也可能是事故，所以必须有 trace、eval、guardrail、human review 和治理闭环。 2 分钟 我会把 AI incident 拆成 detect、triage、command、contain、recover、postmortem、regression、governance。关键不是“模型错了”，而是定位哪个组件失效：prompt、RAG、index、policy、tool、judge、model route、human handoff 或 vendor change。然后按风险分级决定是否 rollback model、回滚 index、禁用工具、切模板、人审、客户补救和监管沟通。复盘后必须把事故样本进入 regression eval 和 release gate，否则只是写报告。
Q2：如何给 AI 事故分级？
版本 回答 30 秒 我不会只按技术故障分级，而会按客户伤害、监管暴露、数据敏感度、自动化程度、blast radius、可逆性和检测来源分级。比如客户可见错误承诺、PII 外泄、高风险工具误执行、AML/KYC/credit 流程系统性错误，最低就应该是 SEV1。 2 分钟 我会建立 SEV0-SEV4。SEV0 是重大客户伤害、监管违规、大规模数据泄露或不可逆工具动作；SEV1 是高影响但可 containment 的受监管流程或客户可见问题；SEV2 是质量、成本、延迟、eval 显著退化；SEV3 是被控制捕获的局部问题；SEV4 是 hazard 或演练发现。这样 severity 能直接驱动 war room、通知、postmortem 深度、review board 和 restart gate。
Q3：Bad retrieval 导致错误答案，你会怎么处理？
版本 回答 30 秒 先止血：回滚 index 或禁用错误来源，高风险问题切人工或模板。然后保留 query、retrieved doc IDs、index version、ACL、citation 和输出样本。根因通常在 source trust、freshness、ACL、chunking、reranker 或 release gate。最后把样本加入 retrieval regression。 2 分钟 我会先判断是否客户可见或影响受监管流程。如果是，立即关闭相关 intent 的自由生成，切换到权威搜索或人工复核。技术上做 index manifest diff、source owner check、freshness filter、ACL test 和 citation support check。治理上让 Data Owner 和 Model Risk 共同确认恢复条件。防复发不是“调 prompt”，而是 source registry、index release gate、retrieval eval、freshness SLO 和 impacted population query。
版本 回答 30 秒 Tool misuse 把语言错误放大成真实业务动作，比如重复退款、错误关闭投诉、越权查账户。防线要在工具网关，而不是 prompt：最小权限、risk-tiered tools、approval、idempotency、dry-run、step budget、side-effect logging 和 kill switch。 2 分钟 我会把工具分为 read-only、draft、write-low、write-high、irreversible。模型只能 propose，系统通过 ABAC / purpose binding / policy engine 决定是否 allowed、blocked 或 requires approval。所有写动作必须有 idempotency key、approval ID、ledger、reversal path 和 trace。事故时第一步通常是把写工具切只读，保全 side effect log，识别 affected population，再做补偿和 regression。
Q5：PII leak 事故如何处理？
版本 回答 30 秒 先停止外发和明文日志，隔离受影响 route，保全 trace 和 egress log，识别字段、记录数、目的地和保留状态。然后 privacy、legal、security 判断通知和补救。防复发要覆盖输入、输出、tool payload、日志和供应商调用，不只做回答过滤。 2 分钟 我会要求证据包包含 DLP decision、output sample、egress destination、trace ID、redaction status、affected records 和 vendor handling。Containment 包括关闭外发工具、清理第三方 ticket、禁用日志导出、启用 field redaction。根因可能是目的绑定缺失、tool gateway 未做 DLP、日志策略错误或供应商处理范围不清。修复要进入 DLP regression、egress policy、privacy impact record 和 vendor control。
Q6：Postmortem 怎么做到无责但不失去 accountability？
版本 回答 30 秒 无责不是无人负责。无责是避免追责个人，聚焦系统为什么让合理的人在当时信息下做出错误决策；accountability 是每个 corrective action 都有 owner、due date、验证证据和关闭审批。 2 分钟 好的 postmortem 会记录时间线、影响、检测缺口、控制失效、根因和贡献因素。它不会写“某人没审核好”或“模型幻觉了”，而会问人审界面有没有显示证据、release gate 为什么没覆盖样本、policy 为什么只写在 prompt、tool 为什么没有 approval。最后行动项必须能验证，例如新增 50 个 regression cases、启用 kill switch、通过 DLP test、更新 ADR，而不是“加强培训”。
Q7：如何把 AI 事故转成长期治理能力？
版本 回答 30 秒 每个 SEV1+ 事故都应该进入三条线：eval regression、architecture decision、governance review。也就是样本进入测试，控制进入架构，残余风险进入 review board 和证据包。 2 分钟 我会设 AI Reliability Review Board，定期看 incident、eval trend、SLO、cost、security、privacy、customer harm 和 corrective action aging。事故后的 restart 需要通过 scope、component、eval、security/privacy、operations、governance 和 phased rollout gate。这样事故不只是项目组修 bug，而是改变 release gate、tool permission、source governance、model pinning、vendor controls 和 board reporting。
Q8：成本 spike 算不算 AI incident？
版本 回答 30 秒 算。AI 成本 spike 可能来自长上下文、retry loop、judge 放大、工具循环或滥用，可能导致预算熔断、服务降级和单位经济性失效。它应该按业务影响和可控性分级。 2 分钟 我会看 cost per outcome，而不是只看总账单。证据包括 token、route、cache、retrieval、rerank、judge、tool 和 loop trace。Containment 包括 quota、rate limit、降级模型、缩短上下文、关闭 retry loop、启用 cache。防复发是 budget guardrail、step budget、loop detector、unit economics dashboard 和 release 前成本压力测试。
Q9：面对监管或董事会，AI 事故应该怎么讲？
版本 回答 30 秒 讲事实、影响、控制、补救和治理，不讲模型黑箱借口。要说明发生了什么、谁受影响、哪些控制失效、如何 containment、如何补救、如何防复发、谁负责和何时完成。 2 分钟 我会准备两套材料：regulator / examiner response pack 和 board brief。监管材料需要 system description、control design、control failure、affected population、remediation 和 evidence index。董事会材料需要风险偏好、客户和业务影响、管理层行动、重大残余风险和需要批准的决策。所有数字都必须来自可复核查询，未确认的内容标为 under assessment。
Q10：为什么 AI Reliability Review Board 有必要？
版本 回答 30 秒 因为 AI 事故根因跨产品、模型、数据、工具、安全、隐私、合规和供应商，不能靠单个团队自己判定 restart。Review Board 把事件、eval、变更、风险接受和证据放在同一个治理机制里。 2 分钟 Review Board 的价值是 decision rights。它可以批准 restart、要求额外控制、阻止 release、接受残余风险或升级高管。输入包括 postmortem、eval trend、SLO breach、cost、red-team、vendor change 和 customer harm。输出是 decision log、corrective action、ADR、risk acceptance 和 game day backlog。这比项目组“修完上线”更适合金融零售高风险 AI。
19. 自检清单
Check 达标标准 Taxonomy 覆盖 bad retrieval、hallucination、policy bypass、tool misuse、PII leak、cost spike、latency degradation、model drift、eval regression、prompt injection Severity SEV 模型同时考虑客户、监管、数据、工具、成本、延迟、blast radius 和可逆性 Command 有 IC、scribe、product、architecture、model risk、security、privacy、legal/compliance、business、vendor owner Containment 能按 model、prompt、index、retriever、tool、policy、judge、vendor 分层 rollback Evidence Trace 能复原 prompt、model、retrieval、tool、policy、judge、human review 和影响对象 Postmortem 包含 timeline、impact、RCA、contributing factors、control failure、corrective actions、residual risk Corrective actions 每个行动项有 owner、due date、verification evidence、closure approver Communications 覆盖 customer、operations、executive、board、regulator、vendor Governance 有 reliability review board、restart gate、risk acceptance、audit evidence Game days 至少覆盖 bad retrieval、PII leak、tool misuse、prompt injection、cost spike、model drift Portfolio 能形成 incident pack、ADR、dashboard spec、game day log、board brief、interview story
20. Final Principle
AI production reliability 的成熟度不在于事故少，而在于：
事故能早发现，
影响能快收敛，
证据能复原，
决策能追踪，
客户能补救，
风险能治理，
同类错误不会以同样方式再次发生。
真正高级的 AI PM / AI Architect / Platform PM / Model Risk / SRE-like 角色，不只是会解释模型能力，而是能把 AI 系统做成可运营、可回滚、可审计、可复盘、可治理的生产系统。