AI 扩展计划 / Playbooks
AI Requirements-to-Eval Cookbook
AI 需求一旦进入生产环境，就不再只是“系统应该怎样响应”的描述，而是一个可执行的质量和风险契约。Requirements-to-Eval 的工作，是把业务需求、AI 行为、失败模式、测试数据、评分方法、阈值、owner、release gate、monitoring signal 和 incident loop 连成一条链。本文偏向金融零售场景，重点处理政策问答、调查辅助、信贷材料、支付异常、财
414 行AI_REQUIREMENTS_TO_EVAL_COOKBOOK.md
AI Requirements-to-Eval Cookbook
AI 需求一旦进入生产环境，就不再只是“系统应该怎样响应”的描述，而是一个可执行的质量和风险契约。Requirements-to-Eval 的工作，是把业务需求、AI 行为、失败模式、测试数据、评分方法、阈值、owner、release gate、monitoring signal 和 incident loop 连成一条链。本文偏向金融零售场景，重点处理政策问答、调查辅助、信贷材料、支付异常、财富合规和监管变化影响分析。
1. Source Anchors
| Anchor | Link | 用法 |
|---|---|---|
| NIST AI RMF | https://www.nist.gov/itl/ai-risk-management-framework | 将 Govern / Map / Measure / Manage 转成 eval、控制和监控证据。 |
| NIST AI RMF Generative AI Profile | https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence | 为生成式 AI 的幻觉、泄露、误用和过度信任设计样本与监控。 |
| EU AI Act | https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng | 用 risk-based lens 识别高风险场景、透明度、人类监督和文档需求。 |
| ISO/IEC 42001 | https://www.iso.org/standard/81230.html | 用 AI management system 思路管理 eval 生命周期。 |
| OWASP LLM Top 10 | https://owasp.org/www-project-top-10-for-large-language-model-applications/ | 将提示注入、敏感信息泄露、过度代理等转成 red-team cases。 |
| G-Eval | https://arxiv.org/abs/2303.16634 | 参考 rubric-based evaluation 的结构化评分思路。 |
| MT-Bench / LLM-as-Judge | https://arxiv.org/abs/2306.05685 | 理解自动化 judge 的价值、偏差和校准要求。 |
2. 为什么 Acceptance Criteria 不够
传统软件验收常用确定性断言：点击提交后创建工单、金额必须大于 0、查询结果按时间倒序。AI 输出则是开放式、上下文相关、概率性并且会随模型、prompt、检索语料和用户行为变化。写“回答要准确、安全、专业、有用”并不能让系统可验收。
合格 AI requirement 必须被转成下面的执行链：
business requirement
-> expected AI behavior
-> unacceptable behavior
-> representative data
-> evaluation method
-> threshold and severity
-> release gate
-> monitoring signal
-> owner and incident response
这条链同时服务三类决策：
| 决策 | 需要的 eval 证据 |
|---|---|
| 是否继续建设 | use case 是否有可测价值、数据和控制边界 |
| 是否可以上线 | offline eval、专家复核、红队、critical failure 和回滚方案 |
| 是否可以扩大 | production monitoring、adoption、成本、质量趋势和风险事件 |
3. Requirements-to-Eval 主流程
flowchart TB
B[Business outcome] --> W[Workflow insertion point]
W --> A[Allowed AI behavior]
A --> F[Failure modes]
F --> D[Data and evidence]
D --> R[Rubric and checks]
R --> T[Test cases]
T --> M[Metrics and thresholds]
M --> G[Release gate]
G --> O[Production monitoring]
O --> I[Incident and improvement loop]
3.1 Business outcome
不要从模型能力开始，而是从流程结果和风险结果开始。
| 场景 | 业务结果 | 风险结果 |
|---|---|---|
| AML | 降低 evidence gathering 时间，提高 narrative 完整性 | 不遗漏关键 red flag，不做 final filing decision |
| KYC | 缩短 remediation cycle time，降低重复联系客户 | 不请求未授权材料，不跳过 jurisdiction control |
| 客服 | 提升 first-contact resolution，降低错误政策回答 | 不承诺无授权减免，不绕过投诉升级 |
| 支付 | 缩短 exception resolution time | 不执行未经批准的修复动作，不造成重复账务影响 |
| 信贷 | 提升 memo 一致性和材料完整性 | 不输出最终授信决定，不使用敏感或代理歧视变量 |
3.2 Workflow insertion point
AI 插入流程的位置决定评估强度。
| Insert point | Example | Evaluation implication |
|---|---|---|
| Read | 查政策、查交易、查历史 case | 重点测权限、source freshness、retrieval precision |
| Summarize | 摘要证据、客户资料、投诉 | 重点测 completeness、omission、evidence citation |
| Recommend | 推荐下一步、风险标记 | 重点测 policy boundary、escalation、human approval |
| Draft | 草拟回复、memo、narrative | 重点测 tone、facts、forbidden commitment、review flow |
| Decide | 做最终决定 | 金融高影响场景通常不允许大模型单独承担 |
| Act | 调用工具执行动作 | 重点测 tool allowlist、approval、idempotency、rollback |
3.3 Expected behavior
把“好回答”拆成可评估行为：
| Behavior dimension | 判断方式 |
|---|---|
| Evidence grounding | 关键事实是否由有效来源支持 |
| Completeness | 是否覆盖必要字段、例外、下一步和缺失信息 |
| Policy compliance | 是否遵守当前政策、授权边界和客户沟通限制 |
| Escalation | 高风险或不确定场景是否正确转人工 |
| Format | 是否符合下游工单、memo、case note 或 API schema |
| Human control | 用户是否能接受、编辑、拒绝、覆盖和解释原因 |
| Uncertainty | 不知道时是否说明无法确认，而不是编造 |
3.4 Failure modes
常见失败要被命名，否则无法被评估和修复。
| Failure | Description |
|---|---|
| Unsupported claim | 没有证据支持的事实性陈述 |
| Wrong citation | 引用不支持结论或引用过期来源 |
| Missing evidence | 没指出关键缺失信息 |
| Policy violation | 违反政策、脚本、监管或内部控制边界 |
| Unauthorized action | 建议或执行未授权动作 |
| Hallucinated rationale | 编造看似合理的解释 |
| Over-refusal | 应该回答但过度拒答 |
| Under-escalation | 高风险场景未升级 |
| Bad tone | 不适合客户或监管沟通的语气 |
| Data leakage | 泄露不该展示的数据 |
| Prompt injection follow | 遵循了文档或用户中的恶意指令 |
4. Evaluation Methods
不同问题需要不同 evaluation method。成熟系统通常混合 deterministic checks、LLM-as-Judge、专家复核、shadow mode 和 production monitoring。
4.1 Deterministic checks
适合稳定、结构化、可精确判断的要求。
| Check | 例子 |
|---|---|
| JSON schema | 输出字段、类型、必填项、枚举值 |
| Forbidden phrase | 禁止承诺费用减免、保证收益、最终拒贷 |
| Citation exists | 每个关键事实都有 source ID |
| Tool action allowed | 工具调用在 allowlist、限额和权限内 |
| Required fields present | case note、memo、客户回复所需字段齐全 |
| Policy version current | 使用当前有效政策版本 |
优势是稳定、便宜、可重复；限制是无法充分判断开放式质量。
4.2 LLM-as-Judge
适合初筛 completeness、tone、explanation quality、groundedness 和 policy compliance，但必须控制偏差。
| Risk | Control |
|---|---|
| position bias | 随机化 A/B 顺序 |
| verbosity bias | rubric 明确“长不等于好” |
| judge drift | 固定 judge version，变更后回归 |
| weak calibration | 与人工样本对齐，记录 disagreement |
| high-risk overuse | 高风险场景不能只靠 judge 作为唯一证据 |
4.3 Expert review
高风险金融零售样本需要专家复核，尤其是 AML typology、信贷与公平借贷、财富适当性、监管文本和客户权益影响。专家复核成本高，因此应当使用结构化 rubric、校准样本和分层抽样，而不是自由文本点评。
4.4 Shadow mode
当模型影响高风险流程时，先在 shadow mode 中与人工决策对照。它不改变真实结果，但可以观察 false positive、false negative、review burden、override 和 adoption friction。
4.5 Production monitoring
上线后必须继续评估。offline eval 证明“已知样本上可接受”，monitoring 证明“真实流程中仍受控”。
5. Severity 与 Threshold
阈值不能只有平均分。金融零售场景必须把 critical failure 从普通质量分中分离出来。
| Severity | Meaning | Example | Gate |
|---|---|---|---|
| S0 Critical | 可能造成严重客户、合规或财务风险 | 大模型给出最终拒贷决定；泄露客户信息 | release blocked |
| S1 High | 高风险错误，需要立即修复 | AML narrative 无证据指控客户 | release blocked or limited |
| S2 Medium | 影响质量或效率 | 漏掉一个非关键字段 | fix before scale |
| S3 Low | 轻微表达或格式问题 | 语气不够简洁 | backlog |
常见 hard stop：
critical violation = 0
unauthorized action = 0
unsupported high-risk claim = 0
regression critical failures = 0
PII leakage = 0
stale policy critical use = 0
普通质量指标可以使用均值、分位数或通过率；高风险失败应作为独立阻断项。
6. Requirements-to-Eval Matrix
Matrix 不是表格练习，而是需求、评估和上线门禁之间的最短路径。
| Requirement | Expected behavior | Failure mode | Eval method | Test data | Threshold | Severity | Owner | Gate |
|---|---|---|---|---|---|---|---|---|
| Policy-grounded answer | 回答引用当前有效政策并说明适用条件 | wrong citation, stale policy, unsupported claim | deterministic + judge + QA sample | policy cases with versions | critical stale policy = 0 | S0/S1 | knowledge owner | release |
| AML narrative draft | 覆盖交易时间线、red flags、缺失证据和来源 | omission, unsupported accusation, final decision | expert review + checklist | historical alert cases | red flag omission critical = 0 | S0/S1 | investigation owner | pilot/release |
| Payment repair recommendation | 推荐允许动作但不执行未经批准动作 | unauthorized action, wrong return code | deterministic + expert sample | return code scenarios | unauthorized repair = 0 | S0 | payments owner | release |
| Lending memo support | 草拟材料摘要和政策引用，不做最终决定 | protected factor use, wrong reason code | deterministic + expert review | lending packages | final decision output = 0 | S0 | credit risk owner | pilot |
Minimum fields：
| Field | Why it matters |
|---|---|
| requirement id | 支持 traceability |
| business owner | 对业务结果负责 |
| risk owner | 对风险接受和控制负责 |
| data source | 支持可复现和权限审查 |
| eval owner | 对评估方法和数据质量负责 |
| release threshold | 让 gate 可执行 |
| monitoring signal | 让生产反馈可回流 |
7. Bad Requirements 到 Eval-Ready Requirements
| Weak requirement | Why weak | Eval-ready rewrite |
|---|---|---|
| AI should answer accurately | 没有样本、来源、风险边界 | For policy questions, answer must cite current approved policy section; unsupported factual claims in high-risk answers must be 0. |
| AI should summarize AML cases | 未定义完整性 | Summary must include customer profile, transaction timeline, red flags, missing evidence and evidence IDs. |
| AI should recommend next best action | 权限和责任边界不明 | For payment exceptions, AI may recommend allowed next actions but cannot execute repair without approval. |
| AI should be compliant | 过于抽象 | Output must not provide personalized investment advice unless advisor review path is triggered. |
| AI should understand KYC | 不可验收 | Detect missing required KYC fields by jurisdiction and product with defined recall, and do not request unauthorized documents. |
| AI should reduce manual work | 价值不可测 | Reduce average evidence gathering time by target percentage in pilot without increasing QA defect rate or review burden. |
8. Financial Retail Eval Patterns
8.1 AML Copilot
| Requirement | Eval design |
|---|---|
| Narrative must cite evidence | citation precision check + expert sample |
| Must not decide SAR filing | forbidden final decision check |
| Must cover red flags | typology checklist recall |
| Must show missing evidence | missing-data cases |
| Must resist injected adverse-media instructions | red-team prompt injection |
Release gate:
critical unsupported claim = 0
final filing decision suggestion = 0
evidence citation threshold met by slice
expert QA accepts pilot sample
8.2 KYC Remediation
| Requirement | Eval design |
|---|---|
| Detect missing fields | historical remediation cases |
| Draft approved outreach | policy and tone judge, QA sample |
| Respect jurisdiction | jurisdiction-specific gold cases |
| No unauthorized document request | deterministic + expert review |
| No golden source update without approval | workflow state check |
8.3 Customer Service RAG
| Requirement | Eval design |
|---|---|
| Answer with current policy | policy version check |
| Cite source | citation coverage and support |
| No unsupported fee waiver | forbidden commitment check |
| Escalate complaint or risk | escalation test cases |
| Say unknown when evidence missing | missing evidence cases |
8.4 Payments Exception Agent
| Requirement | Eval design |
|---|---|
| Interpret return code correctly | deterministic code set |
| Explain root cause | expert review |
| Recommend allowed actions | action allowlist check |
| Require approval for write actions | workflow gate |
| Handle tool failure | tool failure tests |
8.5 Lending Assistant
| Requirement | Eval design |
|---|---|
| Separate calculations from prose | deterministic calculation check |
| Cite policy | citation check |
| Suggest reason codes safely | compliance expert review |
| Trigger human decision | workflow gate |
| Avoid protected or proxy factors | fairness review |
8.6 Wealth Compliance Guardrail
| Requirement | Eval design |
|---|---|
| Detect personalized advice | classifier + expert review |
| Escalate to licensed advisor | workflow gate |
| Use approved product facts | citation check |
| Block prohibited language | deterministic check |
| Provide compliant rewrite | judge + compliance sample |
8.7 Regulatory Change Impact
| Requirement | Eval design |
|---|---|
| Extract obligations | legal/compliance review |
| Map to capability, process and system | architecture review |
| Cite regulation section | citation check |
| Generate owner backlog | reviewer acceptance |
| Flag uncertainty | missing/conflict cases |
9. Risk-Tiered Release Gates
| Risk tier | Example | Required eval |
|---|---|---|
| Low | internal product knowledge retrieval | deterministic + judge sample |
| Medium | customer service draft | deterministic + judge + QA sample |
| High | AML, lending, wealth decision support | expert review + human oversight + audit + strict gate |
| Critical | autonomous customer-impacting decisions | generally no-go for large-model-only systems |
Release gate 应输出以下内容：
| Gate field | Purpose |
|---|---|
| pass / conditional / fail | 明确决策 |
| failed cases | 保留证据 |
| severity | 区分阻断和可修复问题 |
| owner | 每类问题有人负责 |
| mitigation | 说明修复、限制或补偿控制 |
| next review | 防止临时例外永久化 |
10. Red-Team Case Library
至少包括：
| Case type | Example |
|---|---|
| Prompt injection in retrieved docs | 文档中指示模型忽略系统政策 |
| Conflicting policy versions | 新旧政策都能被检索到 |
| Missing evidence | 用户要求模型确认没有来源支持的事实 |
| Unauthorized user | 无权限用户要求摘要限制文档 |
| Sensitive data request | 请求输出客户敏感资料 |
| High-risk advice request | 要求买卖建议、拒贷理由或法律承诺 |
| Tool unavailable | 工具失败后模型是否编造结果 |
| Bypass approval | 用户要求跳过审批或控制 |
| Wrong historical label | 历史 case 标注错误 |
| Ambiguous complaint | 客户表达模糊但可能涉及监管投诉 |
11. Ownership and RACI
| Activity | Business owner | Requirements owner | Architect | EvalOps | Risk / Compliance | Operations |
|---|---|---|---|---|---|---|
| Define business outcome | A | R | C | C | C | C |
| Map workflow | C | A/R | C | I | C | R |
| Define requirements | A | R | C | C | C | C |
| Build eval dataset | C | R | C | A/R | C | C |
| Set release threshold | A | C | C | R | A/R | C |
| Approve high-risk cases | C | C | C | C | A/R | C |
| Monitor production | A | C | C | R | C | R |
| Incident review | A | R | C | R | A/R | R |
Ownership 的关键是避免“技术团队负责模型，业务团队只收结果”的断裂。业务 owner 负责 outcome，risk owner 负责风险接受，EvalOps 负责评估可信度，运营 owner 负责工作流和用户行为。
12. System Exercises
这些练习不是为了产出模板，而是训练从需求到评估的判断力。
| Exercise | Work product | Evaluation focus |
|---|---|---|
| Rewrite weak requirements | 将五条弱需求改写成 expected behavior、unacceptable behavior、eval method、threshold、severity | 能否把抽象形容词变成可评估契约 |
| Build a 20-case golden set | 为客服政策检索设计 common、missing evidence、policy conflict、prompt injection、escalation cases | 数据集是否覆盖真实风险 |
| Release memo | 写 use case、eval result、critical failures、risk acceptance、decision、conditions、owner、next review | gate decision 是否有证据 |
| Production monitoring design | 设计 citation failure、override、complaint、latency、cost、drift 信号 | release 后是否持续受控 |
13. Connections
| Existing asset | Use |
|---|---|
docs/abpa/templates/04-requirements-to-eval-matrix.md | 将需求映射到评估、阈值、owner 和 gate |
docs/ai-foundations/papers/08-llm-as-judge-evaluation.md | 设计 judge、rubric 和人工校准 |
docs/AI_ARCHITECTURE_REVIEW_GATE_CHECKLISTS.md | 将 eval evidence 接入架构 gate |
docs/AI_CONTEXT_ENGINEERING_PLAYBOOK.md | 将上下文要求转成 eval cases |
docs/AI_GOVERNANCE_EVALOPS_RISK_90_PLAN.md | 深化治理和 EvalOps 实践 |
docs/FINANCIAL_RETAIL_AI_CASE_PORTFOLIO.md | 查找金融零售案例 |
14. Operating Rule
An AI requirement is not ready until it can answer:
What should happen?
What must never happen?
What data represents the behavior?
How will it be scored?
What threshold gates release?
Who owns failures?
How will production drift be detected?
How will failed traces improve the next version?