AI UAT / Regression Certification：业务验收架构
AI 系统的 UAT 不是上线前让业务用户点几遍页面并签字，而是证明某个业务能力在特定模型、prompt、RAG、工具、数据、workflow、policy 和控制版本下可接受、可运营、可回滚、可监控。验收产物应是一套可查询的 acceptance evidence architecture，而不是一封 sign-off 邮件。
AI UAT / Regression Certification / Business Acceptance Architecture 解读
配对阅读：本篇的操作手册版（模板/RACI/门禁/runbook）是
docs/AI_UAT_REGRESSION_CERTIFICATION_BUSINESS_ACCEPTANCE_PLAYBOOK.md。第一遍读本篇建立原理与架构判断；第二遍做案例时再用 playbook 查表落地，两者不需要重复精读。
核心导读
AI 系统的 UAT 不是上线前让业务用户点几遍页面并签字，而是证明某个业务能力在特定模型、prompt、RAG、工具、数据、workflow、policy 和控制版本下可接受、可运营、可回滚、可监控。验收产物应是一套可查询的 acceptance evidence architecture，而不是一封 sign-off 邮件。
重要说明：本文是学习、作品集和内部架构训练材料，不构成法律意见、合规结论、审计意见、模型验证结论、信息安全认证、无障碍合规结论或监管解释。具体适用范围、控制要求、客户影响判断、风险接受权限、上线批准、回滚触发和对外沟通，应由 Legal / Compliance / Risk / Model Risk / Information Security / Internal Audit / Business Owner / authorized management 根据机构政策和监管关系确认。访问日期按 2026-06-30 记录。
Source Anchors
| Source | Official link | 本文使用方式 |
|---|---|---|
| FFIEC Development, Acquisition, and Maintenance IT Handbook | https://ithandbook.ffiec.gov/it-booklets/development-acquisition-and-maintenance/ | 用 SDLC、testing、implementation and assessment、maintenance、change management、rollback/back-out、testing data controls 和 documentation language 组织 UAT / regression / release certification。 |
| FFIEC DA&M - V.B Testing | https://ithandbook.ffiec.gov/it-booklets/development-acquisition-and-maintenance/v-development/vb-testing/ | 用 testing scope、test results、corrective actions、UAT、regression testing、stress testing、production-data-in-testing controls 校准测试证据。 |
| FFIEC Management IT Handbook | https://ithandbook.ffiec.gov/it-booklets/management | 用 IT governance、risk management、enterprise architecture、project management、information systems reporting 组织 ownership、management reporting 和 release oversight。 |
| FFIEC Business Continuity Management IT Handbook | https://ithandbook.ffiec.gov/it-booklets/business-continuity-management | 用 business impact analysis、interdependency analysis、resilience、event management、continuity/recovery、exercises/tests、maintenance/improvement 支撑 operational readiness、degraded mode 和 rollback。 |
| NIST SP 800-218 Secure Software Development Framework | https://csrc.nist.gov/pubs/sp/800/218/final | 用 Prepare、Protect、Produce、Respond 的生命周期语言设计 secure release evidence。 |
| NIST SP 800-53 Rev. 5 | https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final | 用 control catalog、assessment、audit/logging、configuration/change、contingency、risk assessment、system integrity 等控制语言表达证据对象。 |
| NIST AI Risk Management Framework | https://www.nist.gov/itl/ai-risk-management-framework | 用 Govern / Map / Measure / Manage 组织 AI UAT 的上下文、风险、度量、处置和持续改进。 |
| NIST AI RMF Core | https://airc.nist.gov/airmf-resources/airmf/ | 用 AI RMF Core 的 function/category 思维设计 risk-to-test-to-evidence traceability。 |
| ISO/IEC 42001 AI management systems | https://www.iso.org/standard/81230.html | 用 AI management system、risk/opportunity、operational control、performance evaluation、improvement 组织验收治理和管理体系证据。 |
1. 核心问题：AI UAT 证明的是业务可接受
传统 UAT 经常被误用成上线前最后一轮业务试点：
requirements document
-> QA says tests passed
-> business users click through screens
-> defects discussed in daily call
-> UAT sign-off email
-> release
这个模式在 AI 系统里不够。AI 可能改变判断、推荐、解释、检索、分流、例外处理、人工工作负载、客户沟通、运营风险和监控方式。只证明页面能走通，无法证明业务能力可以进入生产。
AI UAT 必须回答的问题包括：
| 验收问题 | AI 场景的风险 |
|---|---|
| 业务结果是否可接受 | 模型输出概率化，不能只看 happy path |
| 覆盖了哪些客户、渠道、产品、语言和脆弱客户场景 | segment gap 会导致局部客户伤害 |
| 哪些控制真实生效 | prompt guardrail、retrieval filter、tool gateway、HITL 需要运行证据 |
| 回归影响在哪里 | model、prompt、RAG、tool、policy、workflow 任一变更都可能改变行为 |
| 例外如何接受 | AI residual risk 不能靠口头“业务同意” |
| 上线后如何发现偏移 | 离线 UAT 通过不代表生产分布持续稳定 |
| 出事如何回滚 | 模型、prompt、retriever、feature flag、workflow 和人工流程都要有回退路径 |
成熟主线应写成：
business objective
-> acceptance claim
-> acceptance criteria
-> golden journeys and synthetic packs
-> persona / segment / channel coverage
-> risk/control coverage and AI regression
-> workflow replay / shadow / parallel evidence
-> defect triage and residual risk
-> release certification
-> operational readiness
-> post-release monitoring and rollback criteria
2. 方法：Claim-Based Acceptance
UAT 应先定义要证明的 claim：
For release R,
under model M / prompt P / RAG corpus K / tool set T / workflow W / policy C,
the business capability is acceptable for population S,
because evidence E demonstrates criteria A,
with known exceptions X,
accepted by owners O,
and monitored by controls N.
这句话把 UAT 从主观认可转成证据结构。它也让 QA、业务验收、模型验证、安全评估、审计和 release management 各自边界清楚：QA 证明功能和集成，模型验证证明模型适用性和限制，安全评估证明访问和数据保护，业务验收证明业务流程、客户/员工旅程、政策边界、例外处理和运营准备可接受。
Acceptance evidence stack 至少包括：
| Stack layer | 关键对象 | 必须回答 |
|---|---|---|
| Business claim | 业务能力可接受声明 | 这次上线证明哪个业务能力可以进入生产 |
| Scope | product、journey、persona、channel、region、system、release | 哪些客户、员工、流程和技术版本在范围内 |
| Criteria | business、risk、control、operational、rollback criteria | 什么叫可接受，阈值是什么，谁有权判断 |
| Test assets | golden journey、synthetic pack、replay、shadow、parallel sample | 证据来自哪些可复用资产 |
| Regression matrix | model、prompt、RAG、tool、policy、workflow、data、UI/API | 本次变更可能影响哪些行为 |
| Control evidence | logs、eval report、defect、approval、exception、monitoring config | 控制是否真实运行并留下证据 |
| Decision record | release、hold、limited pilot、exception release、rollback | 最终决策是什么，剩余风险是什么 |
3. Reference Architecture
Business goals / policies / risk appetite / customer impact
|
v
Acceptance criteria registry
business outcome | risk/control criteria | ops readiness | rollback
|
v
Test asset library
golden journeys | persona packs | synthetic transactions | replay datasets
accessibility cases | exception scenarios | adversarial prompts
|
v
AI regression certification plane
model | prompt | retriever/corpus | tool schema | policy engine
workflow state machine | feature flags | UI/API | data pipeline
|
v
Execution and evidence capture
automated tests | business UAT sessions | workflow replay | shadow mode
parallel run | runbook drills | monitoring dry run
|
v
Defect, exception and release governance
severity | root cause | customer impact | control impact | owner | disposition
|
v
Release certification and production feedback
evidence index | risk acceptance | operational readiness | rollback criteria
quality sampling | drift | complaint/appeal | incidents | eval expansion
架构原则：
No business acceptance without explicit acceptance criteria.
No acceptance criteria without coverage evidence.
No AI release certification without model/prompt/RAG/tool/workflow regression.
No production data in testing without documented need, controls and approval.
No exception without owner, expiry, compensating control and monitoring.
No release without operational readiness and rollback criteria.
No AI-generated evidence summary without human accountable acceptance.
4. Business Acceptance Criteria Contract
业务验收标准不是 user story 的 acceptance criteria 简写。AI 系统的业务接受要覆盖结果、边界、控制、运营和证据。
| Contract field | 必须写清楚 |
|---|---|
| Capability | 被验收的业务能力，不只是“AI 已集成” |
| Population | 客户/员工/账户/交易/地区/语言/渠道/产品范围 |
| Decision impact | AI 是建议、排序、摘要、自动决策、分流还是工具执行 |
| Success criteria | 正确率、完成率、人工覆盖率、错误容忍、延迟、成本、客户影响阈值 |
| Risk criteria | 禁止输出、升级规则、脆弱客户保护、异常处置、可解释性 |
| Control criteria | HITL、dual control、policy guardrail、tool approval、logging、access |
| Regression scope | model、prompt、RAG、tool、workflow、policy、data、UI/API |
| Evidence required | eval report、test run、session record、defect summary、approval、monitoring config |
| Decision owner | business owner、risk owner、operations owner、technology owner |
| Re-cert trigger | model、prompt、policy、corpus、tool、workflow 或监控信号触发重新认证的条件 |
示例：
acceptance_id: ACC-KYC-DOC-AI-ONBOARDING-2026Q3
capability: AI-assisted document review for retail account opening
population: mobile channel, English/Spanish, domestic ID + utility bill journeys
decision_impact: AI extracts fields and recommends pass/review; final reject remains human-controlled
success_criteria: golden journey pass rate >= 98%; unsupported rejection recommendation = 0; manual review queue increase <= 12%
risk_criteria: vulnerable customer and accessibility scenarios routed without loss of recourse; uncertainty triggers human review
control_criteria: model/prompt/version logged; document hash retained; override reason required
recertification_trigger: model/prompt/policy/OCR/document taxonomy change, complaint spike, monitoring breach
5. Test Assets：Golden Journey 与 Synthetic Pack
Golden journey 是端到端业务事实、AI 行为、控制点和证据路径的可复用案例。它不是截图脚本。
| Journey type | 例子 | 验收重点 |
|---|---|---|
| Happy path | 正常开户、正常支付纠纷、正常客服政策问答 | 完成率、延迟、证据完整 |
| High-risk | 大额转账、信贷额度调整、AML 高风险告警 | 控制升级、人工复核、日志和审批 |
| Exception path | 资料不全、系统超时、知识库无答案、工具调用失败 | 降级、拒答、补救、运营队列 |
| Customer harm prevention | 误拒、误导性建议、脆弱客户、语言障碍 | recourse、accessibility、clear explanation |
| Operational stress | close period、峰值交易、供应商降级、人工队列拥塞 | BCM、SLO、degraded mode |
| Compliance-sensitive | KYC、AML、投诉、信贷解释、费用争议 | 来源引用、禁止结论、权限边界 |
Golden journey card 至少记录 journey_id、objective、persona/segment、preconditions、business steps、AI touchpoints、expected behavior、controls、evidence 和 pass/fail rule。它的价值在于每个 release 可重跑关键 journey，生产缺陷可以回写，模型或流程变更可以反查影响范围。
Synthetic transaction packs 用来覆盖真实历史数据中罕见、敏感、边界和高风险情境。
| Pack type | 覆盖内容 | 设计要求 |
|---|---|---|
| Boundary | 阈值附近、临界额度、临界日期、临界规则 | 证明规则边界和模型边界稳定 |
| Rare event | 欺诈模式、AML typology、投诉升级、灾备场景 | 不等待生产自然出现 |
| Protected / sensitive segment | 语言、年龄段、残障辅助、低数字能力、脆弱客户 | 测试公平、可及性和客户伤害控制 |
| Privacy | PII、masked data、tokenized customer、data minimization | 证据有用但不暴露真实敏感信息 |
| Adversarial | prompt injection、policy bypass、jailbreak、malicious document | 验证 guardrail 和 tool gateway |
| Operational | 超时、重复提交、第三方不可用、队列拥塞 | 验证降级、恢复和 runbook |
Synthetic pack 要记录生成方法、业务依据、隐私分类、覆盖 criteria、预期输出、reviewer、version 和 retention class，不能只由 AI 随机生成。
6. Coverage 与 AI Regression
AI UAT 的成熟度取决于是否能证明覆盖真实业务分布、高风险边缘人群和关键控制。
| Coverage axis | 金融零售例子 | 接受证据 |
|---|---|---|
| Customer segment | 新客、老客、高净值、薄信用档案、学生、退休、SMB owner | segment matrix、sample count、pass/fail |
| Vulnerability | 老年客户、残障客户、低英语熟练度、经济困难、被诈骗风险 | inclusive journey、recourse test、accessibility evidence |
| Product / Channel | checking、card、mortgage、loan；mobile、web、branch、contact center | product/channel journey result、handoff evidence |
| Geography / jurisdiction | 州、国家、跨境、data residency | routing、language、policy source version |
| Employee role | front line、supervisor、analyst、operations QA、admin | RBAC、workflow permission、training evidence |
| Risk segment | AML risk tier、fraud score、credit band、complaint severity | escalation and override evidence |
覆盖方法不应是暴力笛卡尔积，而是 risk-based pairwise coverage + material scenario coverage。关键组合来自 material segments、high-risk journey、policy-sensitive output、known defect history、accessibility 和 recourse path。
Risk/control coverage 要把测试与控制绑定：
| Risk | Acceptance control | Evidence |
|---|---|---|
| AI 输出错误导致客户错误行动 | grounding、confidence threshold、HITL、language review | golden Q&A、fact-check、review approval |
| AI 自动化扩大错误规模 | feature flag、rate limit、kill switch、shadow/parallel run | release config、kill switch drill、threshold |
| 业务规则绕过 | policy engine、tool gateway、allow/deny、dual approval | adversarial pack、tool trajectory test |
| 隐私泄露 | data minimization、masking、log redaction | privacy test、access review、sample log inspection |
| segment harm | segment eval、threshold review、recourse path | segment matrix、complaint pathway replay |
| 运营队列不可承受 | capacity model、queue monitor、manual fallback | parallel run workload comparison |
| 无法审计或回滚 | trace id、version capture、evidence vault、rollback plan | evidence query、rollback rehearsal |
AI regression matrix 声明哪些行为资产被改变：
| Change object | 可能影响 | 必要回归 |
|---|---|---|
| Foundation / vendor model | 输出风格、推理、拒答、工具调用、延迟、成本 | golden eval、red-team pack、latency/cost、high-risk journeys |
| Fine-tuned model | 分群表现、阈值、错误类型 | segment eval、calibration、backtest、parallel run |
| Prompt / system instruction | policy boundary、tone、citation、tool choice | prompt regression、adversarial prompt pack |
| RAG corpus / retriever | 答案事实、引用准确性、recall、ranking、source drift | retrieval eval、known-answer set、source freshness |
| Tool schema / API | action accuracy、side effect、authorization | trajectory test、contract test、negative cases |
| Policy / workflow / feature flag | allow/deny、handoff、queue、retry、exposure、rollback speed | decision table、workflow replay、kill switch verification |
| Data pipeline / UI | model input distribution、operator error、accessibility | DQ、drift baseline、UAT journey、training validation |
AI regression certificate 应声明 changed_objects、impacted_objects、test_assets_run、criteria_passed、defects_open、exceptions_accepted、monitoring_updated、rollback_ready 和 recertification_triggers。
7. Workflow Replay、Shadow Testing 与 Parallel Run
三种执行模式的作用不同。
| Pattern | 目的 | 证据 |
|---|---|---|
| Workflow replay | 用历史或 synthetic 事件重放端到端状态转换 | state transition result、tool call trace、exception path |
| Shadow testing | 生产流量旁路运行 AI，不影响真实决策 | output comparison、risk signal、latency/cost、no-customer-impact proof |
| Parallel run | 新旧流程同时运行并比较业务结果 | decision delta、workload delta、customer impact assessment、reconciliation |
选择方式：高影响客户决策适合 parallel run + human review；生成式客服辅助适合 shadow test + sampling QA；AML/fraud analyst copilot 适合 workflow replay + analyst benchmark；RAG 政策问答适合 golden set + retrieval replay；tool-using agent 需要 sandbox tool trajectory。
Parallel run 的 decision signals 要提前定义。
| Signal | Green | Amber | Red |
|---|---|---|---|
| Decision delta | 差异符合预期且解释充分 | 差异集中于特定 segment | 未解释差异影响高风险客户或关键控制 |
| Manual workload | 在容量模型内 | 短期增加但有计划 | 队列超过 SLA 或影响客户 |
| Error taxonomy | 错误类型可接受且已监控 | 有重复错误但补偿控制有效 | 出现禁止错误或不可解释错误 |
| Control adherence | HITL / logging / policy gate 完整 | 少量证据缺口有修复计划 | 关键控制缺证据 |
| Customer impact | 无新增 harm signal | 存在可管理投诉/申诉信号 | 客户补救或监管敏感场景受影响 |
没有阈值的 shadow/parallel run 只是观察活动，不能支持 release certification。
8. Defect Triage、Exception 与 Release Certification
AI 缺陷不能只按技术 severity 分类。应按 business impact、control impact、recurrence 和 release decision 组织。
| Severity | 标准 | 处置 |
|---|---|---|
| Critical | 禁止输出、客户重大影响、关键控制失效、隐私泄露、提交/交易错误、无法回滚 | block release、evidence freeze、executive escalation |
| High | 高风险 journey 失败、segment harm、重大手工绕行、监控缺失、重复缺陷 | release hold or exception committee approval |
| Medium | 局部 journey 失败但有补偿控制，或非关键 segment 表现低于阈值 | fix before scale or limited release with monitoring |
| Low | 文案、低风险 UI、非关键测试数据问题 | normal backlog、no silent exclusion |
Defect record 至少包括 defect_id、impacted criteria/journeys/segments、AI object version、root cause（model/prompt/RAG/tool/data/workflow/UI/control/training）、customer impact、control impact、fix/exception decision、retest evidence 和 release disposition。
AI release exception 不是“业务知道了”。它必须记录 linked release、criteria、defect、residual risk、impacted population、compensating control、expiry、owner、monitoring 和 closure criteria。不可接受的例外包括 critical control missing 但要求 full release、没有 owner 的 known issue、没有 expiry 的长期 exception、用 AI summary 替代风险接受理由，以及不披露客户/员工影响范围。
Release certification 是 evidence bundle。
| Section | 内容 |
|---|---|
| Release identity and scope | release id、model/prompt/RAG/tool/workflow/config versions、products、channels、segments、roles、regions、feature flags |
| Acceptance result | criteria、threshold、actual result、owner decision、golden journey coverage、synthetic/replay/shadow/parallel evidence |
| Regression and control | changed objects、impacted tests、risk/control evidence、AI guardrails、logging、privacy、accessibility |
| Defects and exceptions | open defects、severity、disposition、retest、accepted risk、compensating controls、expiry、monitoring |
| Readiness and decision | runbook、training、support model、BCP/degraded mode、capacity、rollback criteria、release/hold/pilot/rollback decision |
Evidence object 要有 evidence_id、type、release_id、AI object versions、linked criteria、linked controls、producer、timestamp、reviewer、checksum/immutable store、retention class 和 access log。它应支持查询 high-risk criteria without evidence、model/prompt change without regression、repeated failed journeys、expiring exceptions 和 incidents not mapped to regression pack。
9. Test Data、Privacy、Accessibility、Readiness 与 Rollback
AI UAT 的测试数据治理不能被放在附录。
| Concern | Guardrail |
|---|---|
| Production data use | 记录必要性、授权、脱敏/匿名化、访问控制、环境保护、保留期限 |
| Synthetic data | 记录生成逻辑、敏感属性、预期输出、reviewer、重识别风险 |
| Prompt / output logs | 避免保留不必要明文 PII，使用 hash、redaction、structured metadata |
| Evidence retention | evidence 有 owner、retention class、legal hold path、access log |
| Third-party model test | 避免把 restricted data 发往未批准模型或供应商环境 |
Privacy-by-UAT 要证明 AI 不需要过量上下文也能完成任务，测试环境和证据库不泄露敏感客户信息，人工 reviewer 只看到完成任务所需信息，日志可复盘但不制造新的数据暴露面。
Accessibility 是 business acceptance 的一部分。需要覆盖 screen reader、keyboard、contrast、focus order、plain language、translation consistency、cognitive load、recourse、转人工和员工复核工具可用性。
Operational readiness 要证明 runbook、培训、L1/L2/L3 support、AI governance escalation、vendor support、quality/latency/cost/drift/tool error/complaint/queue monitoring、capacity、communication、BCP/degraded mode 和 fallback 都可运行。UAT 通过但运营未准备好，仍不应 release。
Post-release monitoring 必须覆盖：
| Metric family | Purpose |
|---|---|
| Business outcome | completion rate、cycle time、conversion、case resolution |
| Quality | accuracy、citation correctness、unsupported claim、tool action success |
| Risk / segment | policy violation、escalation miss、privacy event、complaint、segment performance |
| Operations / reliability | manual review volume、queue aging、override rate、latency、timeout、vendor degradation |
| Cost / adoption | cost per successful task、token spend、human review cost、usage、override pattern、misuse signal |
Rollback criteria 应提前约定：critical customer harm signal、prohibited output、control logging failure、manual queue exceeds capacity、model/retrieval drift breach、vendor outage、privacy incident。响应可以是关闭 feature flag、route to manual、freeze version、restore prior corpus/prompt/model、expand QA sample、preserve evidence 和启动授权 incident workflow。
10. AI 在 UAT 中的边界
AI 可以加速验收工作，但不能拥有最终接受。
| AI assist use case | Value | Boundary |
|---|---|---|
| Generate candidate test cases | expands journeys and edge cases | human owner approves expected outcome and risk relevance |
| Find coverage gaps | maps requirements to tests、controls and evidence | deterministic traceability remains source of truth |
| Cluster defects | identifies recurring themes and root causes | severity/disposition remains accountable human decision |
| Map requirements to tests | builds traceability graph | cannot invent missing approvals or criteria |
| Summarize evidence | creates release memo draft | summary must cite evidence ids and be reviewed |
| Generate synthetic scenarios | improves rare-event coverage | sensitive attributes and expected outcomes need governance |
| Review UAT transcripts | finds repeated confusion and training gaps | privacy and access controls required |
| Suggest regression impact | links changed object to tests | architect/release owner confirms final scope |
AI 不能签署 UAT，不能接受残余风险，不能决定缺陷不重要，不能批准 release certification，不能替代独立验证或内审，不能在事后补造证据来填控制缺口。
11. 为什么有效、哪里容易误用
这套架构有效，是因为它把验收从“测试通过率”提升到“业务能力可接受性”。Acceptance criteria contract 让业务、风险、运营和技术对“可接受”有同一套定义；golden journey 和 synthetic packs 把生产事故和高风险边界沉淀成长期回归资产；AI regression matrix 防止团队只回归 UI/API，却漏掉 model、prompt、RAG、tool、policy、workflow 和 data pipeline；exception acceptance 把残余风险结构化；monitoring 和 rollback 承认 AI 行为会随生产分布、知识库、供应商和用户使用方式改变。
常见误用包括：
| Anti-pattern | 风险 | 改法 |
|---|---|---|
| UAT as email sign-off | 签署与 criteria、版本、证据脱节 | structured release certification packet |
| Happy-path-only UAT | exception 和 high-risk segment 没覆盖 | golden journey + synthetic + risk-based coverage |
| UI click scripts for AI behavior | 只证明页面能点 | AI regression matrix and trace evidence |
| Production data copied into UAT uncontrolled | 敏感信息扩散 | sanitized/synthetic data and documented controls |
| Defects closed by meeting consensus | 无 root cause、retest、customer impact | defect taxonomy and retest evidence |
| Prompt/RAG change treated as minor edit | 行为边界和事实答案变化不可见 | prompt/corpus versioning and regression trigger |
| Shadow test without comparison criteria | 旁路跑了但不能决策 | predefined comparison metrics and thresholds |
| Parallel run with no workload analysis | 新流程压垮人工队列 | capacity and queue readiness metrics |
| Exception with no expiry | residual risk 永久化 | owner、expiry、compensating control、monitoring |
| AI writes certification memo unsourced | 证据摘要可能幻觉或夸大 | evidence id citation and human approval |
不是所有低风险 AI 辅助都需要同等深度的 parallel run。合理分级应看客户影响、自动化程度、监管敏感性、可逆性、数据敏感度和历史缺陷。
12. 金融零售系统案例：AI 辅助开户文件审核
场景：零售银行在 mobile onboarding 中引入 AI 辅助文件审核。AI 读取身份证件和地址证明，抽取字段，建议 pass/review，但最终 reject 仍由人工控制。
验收 claim：
For release R2,
under model M3 / prompt P7 / document taxonomy D4 / OCR provider O2 / workflow W5,
AI-assisted document review is acceptable for domestic mobile onboarding
because golden journeys, synthetic packs, segment coverage, parallel-run results,
control logs, exception records and monitoring thresholds demonstrate criteria.
关键 acceptance criteria：
| Area | Criteria |
|---|---|
| Business outcome | 正常开户 journey 完成率不低于基线，manual review queue increase <= 12% |
| Customer harm | protected scenarios 中 unsupported rejection recommendation = 0 |
| Control | 每次推荐记录 model/prompt/document hash/source evidence/reviewer |
| Risk | policy uncertainty 必须进入 human review，不允许自动拒绝 |
| Accessibility | 低质量图片、语言差异、辅助路径和转人工不丢失 recourse |
| Operations | parallel run 中队列容量、客服话术、override reason 和 escalation 可运行 |
| Rollback | feature flag 可按 cohort 关闭，回退到纯人工审核 |
测试资产包括正常开户、资料补传、人工复核、拒绝建议、申诉路径、系统超时；synthetic packs 覆盖低清晰度图片、边界地址格式、不同语言文件、疑似伪造、脆弱客户、隐私字段；regression matrix 覆盖 model、prompt、OCR provider、workflow、policy rule 和 mobile UI；parallel run 比较新旧流程的 decision delta、manual workload、error taxonomy、control adherence 和 customer impact。
上线前 certification packet 应能回答：哪些 segment 已覆盖，哪些 defects 已修复，哪些例外被授权接受，监控阈值是什么，投诉或 reject spike 如何触发 rollback，证据保留如何避免 PII 扩散。
13. 学习验证
完成本文后，应能用架构对象解释以下问题。
| 验证问题 | 合格回答应包含 |
|---|---|
| AI 系统的 UAT 和传统 UAT 最大区别是什么 | AI UAT 证明特定模型、prompt、RAG、工具、workflow 和控制版本下的业务能力可接受，而不是证明页面能走通 |
| 如何判断一个 AI release 可以认证上线 | criteria 达标、high-risk journey/segment 覆盖、AI regression 完成、critical/high defects 关闭或授权接受、readiness 和 rollback 就绪 |
| Golden journey 和 synthetic pack 的区别 | golden journey 是端到端业务流程资产；synthetic pack 是构造输入样本，用于覆盖稀有、边界、敏感和高风险情境 |
| AI 在 UAT 中能做什么、不能做什么 | 能生成候选测试、找覆盖缺口、聚类缺陷、总结证据；不能签署、接受风险、降低 severity、批准 release 或补造证据 |
| Shadow testing 和 parallel run 如何选择 | shadow 用于旁路观察且不影响生产决策；parallel run 用于高影响业务结果对比和运营负载验证 |
最低可运行能力可以从 acceptance criteria registry、top golden journeys、high-risk synthetic packs、AI regression matrix、defect taxonomy、certification packet 和 rollback criteria 开始。规模化阶段再建设 evidence vault、coverage analytics、closed-loop eval expansion 和 automated recertification triggers。
最终判断标准：
Can the team reconstruct why this AI release was accepted,
which business criteria it satisfied,
which journeys and segments were tested,
which model/prompt/RAG/tool versions were certified,
which risks were accepted,
who accepted them,
and what would trigger rollback after release?
如果答案是否定的，问题不是 UAT 人员不努力，而是 business acceptance architecture 还没有被产品化。
SOTA 状态标注 (2026-07-01)
本篇属于第二、三遍深读池（参考架构/深读笔记），未列入 12 周主线必读。时效基线为写作时点；引用前请按 CLAUDE.md 全局时效性硬规则复查最新进展。模块级 SOTA 对照见 docs/AI_SYSTEMATIC_LEARNING_ROADMAP_2026.md 各周「2026 SOTA 对照」行与文末「SOTA 检查」。