AI 评估数据集生命周期架构：Eval Dataset Lifecycle / Golden Set / Test Data Factory

Date: 2026-06-30
Status: evergreen
Audience: experienced CBAP / financial retail PM / product architect / solution architect / AI governance lead / model risk partner
Output: advanced architecture note for portfolio, interview discussion, release governance design and AI platform roadmap

---

Why it matters for AI product/architecture

在金融零售 AI 项目里，eval dataset 不是一张测试表，也不是数据科学团队私有的实验文件。它是一类生产控制资产：决定系统如何证明可用、如何发现回归、如何解释上线决策、如何响应监管和内审问题。

成熟团队会把 AI eval dataset 当成生命周期对象管理：

business risk
  -> dataset portfolio
  -> golden / challenge / adversarial / regression sets
  -> real and synthetic case factory
  -> label governance
  -> coverage and drift control
  -> release promotion gates
  -> evidence packet
  -> production failure mining
  -> dataset retirement and retention

这件事对 PM / BA / 架构师重要，是因为 AI 产品上线的核心问题不是“模型分数高不高”，而是：

一句话：

AI Eval Dataset Lifecycle Architecture = 把测试数据从一次性样本升级为可版本化、可治理、可度量、可审计、可退役的质量资产供应链。

---

Concept diagram

flowchart TB
  A[Business use case and risk tier] --> B[Dataset intake]
  B --> C{Source type}
  C --> D[Real production or historical cases]
  C --> E[Synthetic and mutated cases]
  C --> F[Policy and scenario authored cases]

  D --> G[Privacy, consent, retention and de-identification gate]
  E --> H[Realism and oracle gate]
  F --> I[Business policy and SME gate]

  G --> J[Candidate case registry]
  H --> J
  I --> J

  J --> K[Label and expected-behavior governance]
  K --> L[Coverage model]
  L --> M{Promotion decision}

  M --> N[Golden set]
  M --> O[Challenge set]
  M --> P[Adversarial set]
  M --> Q[Regression set]
  M --> R[Monitor-only sample]

  N --> S[Eval runner and release gates]
  O --> S
  P --> S
  Q --> S
  S --> T[Evidence packet and approval ledger]
  T --> U[Production monitoring and failure mining]
  U --> B

  V[Dataset lineage graph] --- J
  V --- K
  V --- S
  W[Retention and legal hold controls] --- D
  W --- T

---

Core architecture model

1. Dataset portfolio, not one dataset

不同数据集服务不同决策，不能混成一个“测试集”。

架构原则：

Golden protects stable core behavior.
Challenge exposes business edge cases.
Adversarial protects trust and security boundaries.
Regression prevents known failures from returning.
Synthetic expands controlled coverage.
Production sample detects distribution change.

2. Case object as the atomic unit

每个 eval case 应被设计成可追溯对象，而不是一行 prompt。

3. Test data factory

Test Data Factory 不是“生成更多测试样本”的脚本，而是受控的案例供应链。

4. Lineage graph

Dataset lineage 要回答：

这个 case 从哪里来?
经过哪些隐私和质量处理?
谁给了什么标签和裁决?
何时进入哪个 dataset version?
哪些 release 使用过它?
哪些结果和审批依赖它?
何时需要退役或重标?

推荐把 lineage 看成图，而不是文件夹：

source event
  -> candidate case
  -> de-identified case
  -> label decision
  -> dataset version
  -> eval run
  -> release gate
  -> evidence packet
  -> production outcome

---

Lifecycle states and gates

Lifecycle states

Promotion gates

---

Financial retail scenarios

1. KYC and onboarding

PM / 架构决策：

• AI 是建议 pass/review，还是参与最终拒绝？不同决策边界需要不同 dataset gate。
• 文档图像是否可进入测试环境？如果不能，需要字段级合成或脱敏。
• 客户重提材料和人工 override 结果是否会反哺 regression set？

2. AML alert investigation

关键控制：高风险 AML case 的 AI 输出应是 analyst assistant evidence，不是最终调查结论。

3. Credit and lending

关键控制：credit AI 的 dataset 必须区分“内部 memo 辅助”与“客户可见解释”。

4. Payments and fraud

5. Contact center and complaints

---

Metrics/control/evidence model

Metrics

Controls

Evidence packet

一次高风险 AI release 至少应能归档：

dataset inventory
coverage matrix
dataset version manifests and hashes
case lineage report
label authority and conflict log
privacy and retention decision
eval run report
slice failure report
promotion gate memo
open exceptions and accepted residual risk
release approval ledger
monitoring plan and failure-mining trigger

---

Anti-patterns and failure modes

---

Architecture mapping to RAG / Agent / Copilot / Eval / Governance

---

ADR draft

ADR: Establish Governed Eval Dataset Lifecycle and Test Data Factory

Status: proposed
Date: 2026-06-30
Decision owners: AI Product Owner, AI Platform Architect, Data Governance, Model Risk, Privacy, Business Operations

Context

AI systems in KYC, AML, credit, payments, contact center and complaints depend on eval datasets for release decisions, regression protection, monitoring and audit evidence. Current practice often treats eval samples as ad hoc spreadsheets. This creates weak lineage, unclear label authority, poor coverage visibility, privacy risk and unreliable release comparisons.

Decision

We will establish a governed eval dataset lifecycle and test data factory with the following capabilities:

1. Dataset portfolio: golden, challenge, adversarial, regression, synthetic and production sample sets.
2. Case registry: each eval case has source lineage, privacy state, labels, expected behavior, severity, dataset membership and retention policy.
3. Promotion gates: intake, privacy, label authority, coverage, promotion, change impact and retirement.
4. Test data factory: controlled real-case mining, synthetic authoring, mutation, privacy transformation, oracle construction and version packaging.
5. Evidence automation: every release run stores dataset manifest, version hash, run result, slice failures, approval, exception and retention decision.

Decision drivers

• Release decisions must be defensible by slice, not only by aggregate score.
• High-impact financial retail use cases require traceable evidence for customers, operations, risk, compliance and audit.
• Production failures and complaints must become regression assets.
• Synthetic data is useful but must be governed for realism, oracle quality and lineage.
• Privacy and retention decisions must be made before data enters AI evaluation workflows.

Consequences

Positive:

• Stronger release gates and regression control.
• Better portfolio visibility across KYC, AML, credit, payments and customer operations.
• Faster response to incidents through reusable regression cases.
• Clearer evidence for audit, model risk and management review.

Trade-offs:

• Upfront effort to define case schema, owners, gates and access boundaries.
• Slower promotion of new cases until label authority and privacy checks are complete.
• Need for tooling to avoid manual evidence assembly.

Alternatives considered

Implementation notes

• Start with one high-value use case such as contact center RAG or AML copilot.
• Define the case object schema and dataset inventory before building automation.
• Make promotion gate output part of release evidence, not a separate governance attachment.
• Treat dataset retirement as a first-class workflow to prevent stale gates.

---

Interview answer: 30秒, 2分钟, CTO版本

30秒版本

AI eval dataset 不能只是一张测试表。我会把它设计成生命周期资产：golden set 保护核心能力，challenge set 覆盖复杂业务边界，adversarial set 测试安全和越权，regression set 防止已知缺陷回归，synthetic set 扩展稀有场景，production sample set 发现分布漂移。每个 case 都要有来源、标签、预期行为、覆盖切片、隐私保留、版本和证据链。这样上线决策才能回答“我们用什么数据证明这个版本可放行，以及哪些风险还没覆盖”。

2分钟版本

在金融零售 AI 里，我不会把 eval dataset 当成 QA 附件，而会当成一个质量控制产品。架构上先定义 dataset portfolio：golden、challenge、adversarial、regression、synthetic、production sample。然后定义 case object：source lineage、workflow context、expected behavior、unacceptable behavior、severity、label authority、dataset membership、retention policy 和 evidence links。

核心治理是 lifecycle gate。候选 case 先过 intake gate，确认属于批准用途；再过 privacy and retention gate，确认生产数据是否能用于 eval；然后过 label authority gate，确保 SME、risk 或 policy owner 对预期行为有裁决；最后过 coverage 和 promotion gate，决定它进入 golden、challenge、adversarial 还是 regression set。上线时，eval run 必须绑定 dataset version、component version、threshold、slice result 和 release approval。

这套架构的产品价值是把 release discussion 从“分数升了没有”变成“关键客户、关键流程、关键失败模式有没有覆盖；高风险场景有没有 hard stop；生产投诉和事故有没有进入 regression set；数据和标签变更有没有 impact analysis”。对 KYC、AML、credit、payments、contact center 和 complaints，这比单纯调模型更接近真实的产品和风险管理。

CTO版本

我会把 eval dataset lifecycle 放进 AI platform control plane。底层是 case registry 和 lineage graph，上面是 test data factory，负责真实案例挖掘、合成案例生成、变体构造、脱敏、oracle 管理和 version packaging。再上面是 dataset promotion engine，把 case 推入 golden、challenge、adversarial、regression 或 monitor-only 集合，并把每次 promotion 的 privacy、label、coverage、severity 和 owner 作为可审计事件。

对 release pipeline，我会要求任何 model / prompt / RAG / tool / workflow / policy 变更都触发 dataset impact analysis，并自动选择必须重跑的数据集。高风险 slice 失败、越权 tool action、PII 泄露、错误拒绝原因这类问题设置 hard stop；普通质量变化用趋势和置信区间辅助决策。上线证据包不靠人工整理，而由平台生成：dataset manifest、hash、run result、slice diff、exception、approval、retention decision 和 monitoring trigger。

战略上，这让组织避免两个极端：一边是 ad hoc eval 无法审计，另一边是治理流程拖慢交付。平台化后，团队可以更快做模型迁移、RAG 语料刷新和 agent 权限调整，同时保留管理层、内审和监管能理解的证据链。

---

7-day practice plan

---

Source anchors with links

这些来源用于校准风险管理、管理体系、需求与架构描述、安全威胁、可观测性和证据语言。本文把它们转成 AI eval dataset lifecycle 的产品架构，不构成法律、合规、审计或认证结论。