AI 监管检查 / 审计问询模拟包

定位: 面向 AI PM / AI BA / Solution Architect / Enterprise Architect 的检查室训练材料。目标不是背法规, 而是训练你在金融零售 AI 项目中面对监管、内审、模型风险、法务、合规、第三方风险和审计委员会追问时, 能用证据链证明系统边界清楚、控制有效、责任明确、风险可运营。

重要说明: 本文是学习与作品集训练材料, 不是法律意见、合规结论、监管解释或审计意见。正式项目必须由 Legal、Compliance、Risk、Model Risk、Internal Audit、Data Protection、Security 和业务管理层按适用司法辖区确认。

---

1. 定位与连接关系

本模拟包连接三份既有学习资产:

本文的角色定位:

• AI PM: 说明 AI use case 的业务目标、边界、客户影响、上线门槛、监控指标和停止条件。
• AI BA: 说明业务流程、政策映射、人工复核点、例外路径、证据字段和问询口径。
• Architect: 说明数据流、RAG / model / vendor 边界、日志、权限、模型网关、安全控制、变更与回滚。
• Risk / Compliance 合作者: 说明适用性判断、控制矩阵、监管问询响应、整改和管理层报告。

检查室中的核心能力不是“知道所有答案”, 而是:

Question
-> factual answer
-> control owner
-> evidence artifact
-> test result
-> residual risk
-> remediation if gap exists

---

2. Exam Room Scenario

2.1 背景设定

一家金融机构准备或已经上线三个 AI 辅助系统:

检查类型:

• 监管检查: 关注金融机构是否理解 AI 系统风险、是否有合规治理、是否能证明客户保护和监管义务没有被 AI 稀释。
• 内审检查: 关注治理设计与实际执行是否一致, 控制是否可测试, 证据是否完整, 管理层 attestation 是否有依据。
• 模型风险审查: 关注模型 / RAG / prompt / eval / monitoring 是否受控, 是否能复现输出, 是否有漂移、偏差、幻觉和解释风险。
• 第三方风险审查: 关注 LLM provider、KYC vendor、cloud、embedding model、case management integration 的 due diligence、合同、SLA、subprocessor 和退出计划。

2.2 检查室角色

2.3 检查目标

检查方通常不是只问“AI 准确率多少”, 而是看四条证据链:

1. Governance chain: AI inventory、risk tier、approval、owner、RACI、policy mapping 是否一致。
2. Data and model chain: 数据来源、权限、模型 / vendor、prompt、RAG index、版本、日志是否可追溯。
3. Control chain: human oversight、eval gate、access control、monitoring、incident response、change control 是否被测试。
4. Outcome chain: 客户影响、错误纠正、投诉、adverse action、AML / KYC 质量、运营指标是否被监控并进入整改。

2.4 检查室基本回答纪律

• 先确认范围: “本回答覆盖 AML Copilot limited production, 不覆盖尚未上线的 credit RAG pilot。”
• 先讲事实: “AI drafts, analysts approve. It cannot close alerts or file SARs.”
• 不夸大控制: 不说 “AI 不会出错”, 改说 “错误通过 eval、人工复核、抽样监控和 stop rule 管理。”
• 不把供应商当挡箭牌: “Vendor provides model service, but business owner and technology owner retain accountability.”
• 不给口头承诺替代证据: 每个回答都落到 artifact、owner、date、sample、test result。
• 主动披露缺口: 有缺口时说明 interim control、risk acceptance、target date 和 owner。

---

3. Evidence Pack Checklist

检查室证据包要能在 30 分钟内定位材料, 在 2 小时内完成 walkthrough, 在 5 个工作日内补交抽样明细。建议为每个 material AI use case 建立独立 evidence folder, 并用 evidence index 维护版本。

证据包质量标准:

• 每份材料有 version / owner / approval date / evidence location / retention rule。
• 每个控制能回答 control objective / design / operation / test method / result / exception。
• 每个高风险输出能重建 input / retrieved source / model version / prompt / output / reviewer action / final decision。
• 每个缺口有 risk acceptance or remediation plan, 不能只写“已知问题”。

---

4. 50 个 Examiner Questions

使用方法:

• 每个问题先用 30 秒给 factual answer。
• 然后提交 evidence index 中的对应材料。
• 如果证据不完整, 立即给 interim control 和 remediation plan。

4.1 Strategy Questions

4.2 Governance Questions

4.3 Data Questions

4.4 Model Questions

4.5 Eval Questions

4.6 Security Questions

4.7 Human Oversight Questions

4.8 Vendor Questions

4.9 Incident Questions

4.10 Customer Impact Questions

---

5. Red Flag List

以下回答会引发更深检查。训练目标不是避免承认问题, 而是避免暴露“无 owner、无证据、无控制、无整改”的治理失效。

---

6. 10 天演练计划

节奏: 每天准备一个材料, 进行一次 20-30 分钟问答模拟, 写一个改进动作。第 10 天完成完整检查室 dry run。

演练评分标准:

---

7. Templates

以下模板按 “可直接用于演练” 设计。示例统一使用 AML Investigation Copilot。实际项目使用时替换 use case、owner、日期、指标和证据路径。

7.1 Exam Briefing Memo

# Exam Briefing Memo - AML Investigation Copilot

## Purpose
AML Investigation Copilot assists analysts by summarizing alerts, organizing transaction evidence, retrieving approved AML policy references, and drafting investigation narratives. It does not close alerts, approve case disposition, freeze accounts, or file SARs.

## Scope
- Business process: AML alert investigation and case narrative drafting
- Users: trained AML analysts and AML supervisors
- Stage: limited production
- Data: transaction alerts, KYC profile references, historical case notes, approved AML policy documents
- AI components: model gateway, prompt template, RAG index, evidence citation service, case management integration

## Risk Classification
- Inherent risk: High
- Residual risk: Medium with controls
- Rationale: impacts AML regulatory workflow, uses sensitive customer and transaction data, and may influence analyst judgment

## Key Controls
- Human approval required for final case disposition and SAR decision
- Approved source-only RAG with document version and citation logging
- Monthly eval for red-flag recall, unsupported claim, evidence omission, and narrative quality
- RBAC inherited from AML case management entitlement
- Prompt, model, index, and tool changes require change approval and regression test
- Incident runbook defines containment, escalation, customer / regulatory impact assessment, and retest

## Current Metrics
- Citation completeness: 98.8%
- Unsupported claim rate: 1.4%
- Analyst override rate: 17%
- Severe incidents this quarter: 0
- Low / near miss events this quarter: 2, both remediated and retested

## Known Gaps and Remediation
- Gap: reviewer calibration sample size is below target for complex trade finance alerts
- Interim control: supervisor review required for trade finance cases
- Owner: Head of Financial Crime Operations
- Target date: 2026-07-31
- Retest: 200-case calibration sample after new examples are added to eval set

7.2 Evidence Index

7.3 Control Effectiveness Narrative

# Control Effectiveness Narrative - AML Copilot Human Oversight

## Control Objective
Ensure AI-generated investigation summaries and SAR draft language do not replace required analyst and supervisor judgment.

## Control Design
The AML Copilot interface presents AI output as draft-only. The case disposition workflow requires a trained analyst to review evidence citations, edit or reject the AI draft, select disposition reason codes, and submit the case for supervisor approval when required by AML policy.

## Control Operation
For every assisted case, the system logs the AI draft, retrieved evidence references, analyst edits, final disposition, reviewer ID, timestamp, and override reason where the final decision differs from AI recommendation. Supervisors review a weekly sample of assisted cases and all high-risk typology cases.

## Test Method
Internal control testing sampled 150 assisted cases from May 2026. The tester verified that each case had analyst review, citation check, final disposition, timestamp, and reviewer identity. The tester separately reviewed 30 high-risk typology cases for supervisor approval.

## Test Result
147 of 150 cases passed all required fields. Three cases had incomplete override reason text but still had analyst review and final disposition evidence. All 30 high-risk typology cases had supervisor approval.

## Exceptions and Remediation
The incomplete override reason issue was remediated by making override reason a required structured field. Retest on 2026-06-21 found 50 of 50 cases complete.

## Residual Risk
Residual risk remains medium because automation bias can still influence analyst judgment. The control is supported by monthly reviewer calibration, override trend monitoring, and targeted training for low-override analyst groups.

7.4 Regulator Q&A Sheet

7.5 Remediation Plan

---

8. 面试表达

8.1 30 秒版本

我会把金融 AI 监管检查准备成一条证据链, 而不是一份口头说明。以 AML Copilot、KYC Policy Assistant、Credit Policy RAG 为例, 我会先明确 AI 只是 draft / retrieve / recommend, 不做最终客户或合规决定; 然后用 AI inventory、risk tier、data lineage、model/vendor record、eval report、human oversight、incident log、change log、access control 和 third-party due diligence 证明控制设计和运行有效。面对问询时, 每个回答都落到 owner、artifact、测试结果、残余风险和整改计划。

8.2 2 分钟版本

在金融零售 AI 项目中, 监管或内审真正关心的是三件事: 第一, 机构是否知道自己上线了哪些 AI, 这些 AI 是否影响客户权益、合规义务或关键运营; 第二, 控制是否只是文档, 还是已经进入流程、系统、日志、评测、人工复核和变更管理; 第三, 出现错误时能否发现、止损、复盘和整改。

我的准备方式是把 use case 拆成四条证据链。Governance chain 包括 AI inventory、risk tier、RACI 和审批记录。Data and model chain 包括数据血缘、RAG source、模型版本、vendor record 和日志。Control chain 包括 eval gate、human oversight、access control、change control、security test 和 incident runbook。Outcome chain 包括客户影响、投诉、override、adverse action reason、AML / KYC 质量和整改记录。

如果是 Credit Policy RAG, 我会特别强调不能因为模型复杂就无法提供 specific and accurate adverse action reason; 如果是 AML Copilot, 我会强调 AI 不自动关闭告警或提交 SAR, 并用人工复核和 case evidence trail 证明责任没有转移给模型; 如果是 vendor LLM, 我会按 third-party risk lifecycle 准备 due diligence、合同、SLA、model update notification 和 exit plan。这样回答既能体现产品和架构能力, 也能体现合规与审计意识。

8.3 Chief Risk Officer 深挖

CRO 可能问: “你如何证明 residual risk 在 appetite 内?”

回答口径:

• 先说明固有风险: AML / credit 属于 high impact, 因为影响监管义务、客户权益或重要运营。
• 再说明控制组合: risk tier approval、human oversight、eval thresholds、access control、change management、incident response、vendor due diligence。
• 再说明控制测试结果: 抽样数量、通过率、异常、复测和趋势。
• 再说明 residual risk: 哪些风险仍存在, 为什么可以接受, review date 是什么。
• 最后说明 stop rule: 哪些指标触发暂停、降级、回滚或升级到 risk committee。

强回答示例:

The residual risk is medium and within the approved appetite for limited production, not for full-scale rollout. The basis is not a subjective assessment. We have release-gate eval results, human review samples, RBAC testing, incident trend, and vendor controls. The main remaining risk is automation bias in complex cases, so we added supervisor review for high-risk typologies and a stop rule if red-flag recall or override anomalies breach threshold for two review cycles.

8.4 Audit Committee 深挖

Audit Committee 可能问: “How does management know controls are operating, not just designed?”

回答口径:

• 每个关键控制都有 owner、频率、证据和测试方法。
• 管理层 dashboard 不只显示 AI adoption, 也显示 control pass rate、incident、override、complaint、log completeness。
• 二线风险 / 合规做 challenge, 三线内审可抽样验证。
• 对异常不回避, 要展示 finding、root cause、remediation、retest。

强回答示例:

Management receives monthly control evidence, not just project status. For AML Copilot, the evidence includes case-level review logs, citation completeness, unsupported claim sampling, analyst override trends, access review, and change tickets. Internal control testing found three override reason exceptions, the workflow was remediated, and retesting showed 50 of 50 sampled cases complete. That gives audit committee a basis to challenge both design and operating effectiveness.

---

9. Source Anchors

以下官方来源用于建立学习锚点和问询方向。所有法规适用性、义务范围和监管解释均需在正式项目中由 legal / compliance 复核。Access date: 2026-06-29。

学习时可把 source anchors 转成四类检查问题:

1. Risk classification: 这个 AI use case 是否属于高影响或监管敏感场景?
2. Control expectation: 是否有 human oversight、logging、documentation、robustness、security、eval 和 monitoring?
3. Third-party accountability: vendor 是否受控, 机构是否保留 accountability?
4. Customer explanation: 当 AI 影响信贷等客户权益时, 是否能提供具体、准确、可审计的理由?

---

10. 自检清单

用于完成本文后自测:

• 覆盖 AML Investigation Copilot、KYC Policy Assistant、Credit Policy RAG 三个检查场景。
• 覆盖 AI inventory、risk tier、data lineage、model/vendor record、eval report、human oversight、incident log、change log、access control、third-party due diligence。
• 提供 50 个 examiner questions, 并按 strategy、governance、data、model、eval、security、human oversight、vendor、incident、customer impact 分类。
• 每个问题都有回答要点和应提交证据。
• 提供 red flag list, 说明哪些回答会引发更深检查。
• 提供 10 天演练计划, 每天包含材料准备、问答模拟和改进动作。
• 提供 Exam Briefing Memo、Evidence Index、Control Effectiveness Narrative、Regulator Q&A Sheet、Remediation Plan 模板。
• 提供 30 秒、2 分钟、CRO 深挖、Audit Committee 深挖面试表达。
• 包含 EU AI Act、NIST GenAI Profile、OCC third-party risk、CFPB adverse action 四个官方 source anchors。