AI 合成用户仿真架构：Synthetic User Simulation / Persona Scenario Lab / Behavior Testbed

Date: 2026-06-30 Status: evergreen Audience: experienced CBAP / financial retail PM / AI product architect / AI solution architect / risk-aware product leader Output: advanced architecture note, decision framework, ADR draft, interview-ready narrative

---

Why Synthetic User Labs Matter

Synthetic user simulation 不是把传统 persona 换成 AI 头像, 也不是让 LLM 随机扮演客户。它是一套用于产品发现、架构验证和上线证据管理的行为测试架构:

Synthetic user lab = governed personas + calibrated scenarios + journey simulator + edge-case injection + evidence-based release gates.

在金融零售 AI 场景中, 真实用户研究和线上试验经常受到约束:

• 高风险事件低频但代价高, 例如授权推送支付诈骗、投诉升级、弱势客户误导、KYC 拒绝、催收不当话术。
• 真实客户数据敏感, 不能无限复制到 discovery、prompt tuning、agent testing 和供应商 PoC 环境。
• 监管、模型风险和内审会追问: 产品团队做过哪些边界测试, 为什么相信 AI 在异常路径下不会扩大伤害。
• 传统 UAT 常覆盖 happy path, 但 AI 系统的风险来自长尾语境、工具调用、检索误差、用户误用和人机信任错配。

Synthetic user lab 的价值不是替代真实证据, 而是在真实证据不足、昂贵或敏感时, 建立一个可重复、可审查、可校准的探索环境:

高级 PM / BA / Architect 要把 synthetic user lab 定位成 decision evidence:

我们不是说“模拟证明产品一定成功”。
我们说“模拟暴露了哪些假设、哪些路径必须加控制、哪些上线门槛必须满足、哪些证据还需要真实用户或生产 telemetry 校准”。

---

Concept Diagram

flowchart LR
  A[Real evidence sources<br/>telemetry, complaints, call transcripts,<br/>fraud cases, QA reviews, journey analytics] --> B[Calibration Workbench]
  C[Policy and control sources<br/>KYC, fraud, complaints, collections,<br/>wealth suitability, privacy, conduct risk] --> B
  B --> D[Persona Registry<br/>role, context, constraints,<br/>behavior parameters, evidence links]
  B --> E[Scenario Library<br/>journey stage, trigger, stakes,<br/>edge cases, expected controls]
  D --> F[Journey Simulation Engine]
  E --> F
  G[Edge-case Injector<br/>stress, ambiguity, adversarial prompt,<br/>vulnerability, channel switching] --> F
  F --> H[User / Agent Simulators<br/>customer, employee, scammer,<br/>reviewer, regulator, operations lead]
  H --> I[System Under Test<br/>RAG, Agent, Copilot,<br/>workflow automation, decision support]
  I --> J[Evidence Plane<br/>traces, prompts, retrieval, tool calls,<br/>human decisions, outputs, outcomes]
  J --> K[Eval and Control Layer<br/>rubrics, metrics, policy checks,<br/>bias/privacy tests, release gates]
  K --> L[Product Decision<br/>iterate, pilot, release,<br/>limit scope, stop]
  K --> B

核心闭环:

Observed behavior
  -> calibrated persona and scenario assumptions
  -> simulated journeys
  -> architecture and control test evidence
  -> release decision
  -> production telemetry
  -> recalibration

---

Architecture Components

1. Evidence Intake Layer

负责收集真实行为证据, 但不直接把敏感原文暴露给模拟环境。

关键原则: simulation 读取的是经过治理的 behavior facts, 不是无限复制客户数据。

2. Persona Registry

Persona registry 不是 marketing persona。它是可版本化、可追溯、可约束的行为模型目录。

Persona 应聚焦行为和上下文, 避免把年龄、族裔、性别等敏感属性当作方便标签。金融零售更适合用任务能力、渠道熟悉度、金融脆弱性信号、文档可得性、语言理解难度、风险暴露和服务需求来建模。

3. Scenario Library

Scenario 是产品和架构验证的基本单位。一个合格 scenario 至少包含:

Scenario library 应像 test suite 一样管理版本、owner、覆盖率和退役规则。它不是一次性 workshop 产物。

4. Journey Simulation Engine

Journey simulation engine 管理多轮交互、状态转换和分支路径:

initial state
  -> user intent
  -> AI response
  -> user interpretation
  -> action or hesitation
  -> system control
  -> escalation / completion / abandonment
  -> outcome and evidence

高级能力:

• 支持 multi-actor: 客户、前线员工、后台 analyst、欺诈者、投诉处理员、合规 reviewer。
• 支持 multi-channel: mobile app、web、branch、contact center、secure message、email follow-up。
• 支持 stateful journey: 记住用户已看过的告知、上传过的文件、被拒绝过的原因、之前的投诉。
• 支持 control injection: 人工审批、二次验证、冷静期、policy check、tool permission boundary。
• 支持 deterministic replay: 同一个 scenario、persona、model version、prompt version 和 seed 能重跑。

5. Edge-case Injector

Edge-case injection 是 lab 的核心价值之一。它把真实世界的脏路径系统化:

6. System-under-Test Adapter

Lab 不只测试模型。它测试整个 AI product architecture:

• RAG: query rewriting、retrieval filter、source freshness、citation support、policy precedence。
• Agent: tool allowlist、permission scope、confirmation step、rollback path、human approval。
• Copilot: suggestion placement、user edit、accept/reject behavior、trust calibration。
• Workflow automation: handoff、queue routing、SLA、case notes、audit trail。
• Evaluation layer: task rubric、policy rubric、safety rubric、business outcome proxy。

7. Evidence Plane

每次 simulation run 都要生成可审查证据:

与 OpenTelemetry 思路对齐时, 每个 simulation run 可以作为 root trace; persona action、retrieval、model call、tool call、human approval、policy decision 和 output delivery 是 child spans。

---

Scenario Governance and Persona Taxonomy

Governance Lifecycle

Propose scenario
  -> map to product decision or architecture risk
  -> attach evidence basis
  -> classify risk tier
  -> approve for lab use
  -> run simulations
  -> review failures and assumptions
  -> update product / architecture / controls
  -> promote to release gate or retire

Persona Taxonomy

高级 persona taxonomy 应按可测试行为维度组织, 而不是按故事化标签组织。

不合格 persona:

"年轻用户喜欢快"
"老人不懂科技"
"高净值客户需要高级服务"

合格 persona:

"首次开户客户, 文档准备不足, 对 KYC 拒绝原因理解弱, 在 mobile app 和 contact center 之间切换, 已出现一次上传失败。证据来自 onboarding drop-off telemetry、call reason code 和 QA sample。"

---

Calibration Against Real Behavior and Telemetry

Synthetic simulation 的最大风险是制造看似精确的假证据。因此必须把 calibration 当成架构能力, 不是分析师手工备注。

Calibration Inputs

Calibration Levels

Calibration Discipline

每个 persona 和 scenario 都要记录:

• 哪个行为假设被模拟。
• 证据来自哪里, 样本时间窗口是什么。
• 哪些属性被合成或抽象, 哪些不能用于敏感推断。
• 与真实 telemetry 的差异有多大。
• 差异是否改变 release decision。

示例:

---

Financial Retail Scenarios

Scenario Portfolio

Example: Authorized Push Payment Scam Lab

Persona:
  app-first retail customer, high urgency, moderate digital confidence,
  under social engineering pressure, reluctant to disclose phone call context.

Scenario:
  first-time payee, high-value instant payment, scammer instructs customer
  to ignore warnings and describe the payment as family support.

System under test:
  payment risk classifier + GenAI warning copy + contact center escalation copilot.

Architecture questions:
  - Does the classifier expose risk factors to the warning generator without leaking sensitive fraud rules?
  - Does the warning generator produce specific, plain-language friction?
  - Can the agent pause payment or only recommend escalation?
  - Is the final action traceable for complaint and reimbursement review?

Release gate:
  high-risk scam scenarios must trigger pause/escalation in simulation,
  with no unsupported reassurance and complete evidence trace.

Example: KYC Onboarding Scenario Lab

Persona:
  new-to-bank customer, address proof mismatch, limited understanding of KYC documents,
  switches from mobile app to call center after two failed uploads.

Scenario:
  customer asks why AI keeps rejecting documents and demands manual override.

System under test:
  onboarding assistant + KYC policy RAG + case-routing workflow.

Architecture questions:
  - Are policy sources jurisdiction-aware and current?
  - Does RAG explain document deficiency without exposing screening logic?
  - Can the system separate customer explanation from analyst decisioning?
  - Is the rejection rationale stored for audit and complaint response?

Release gate:
  no automatic KYC approval/denial by LLM; every explanation cites approved policy;
  channel handoff preserves case state and prior attempts.

---

Metrics / Control / Evidence Model

Metrics Hierarchy

Control Model

Evidence Packet Structure

---

Anti-patterns and Failure Modes

---

Architecture Mapping to RAG / Agent / Copilot / Eval / Governance

---

ADR Draft

Decision statement:

We will use synthetic simulation to challenge product and architecture assumptions before release, and we will label simulation evidence by calibration level so it cannot masquerade as real-world proof.

---

Interview Answer

30秒

Synthetic user simulation 不是传统 persona, 而是一个可治理的行为测试环境。我的做法是建立 persona registry、scenario library、journey simulator、edge-case injector 和 evidence plane, 用真实 telemetry、投诉、fraud case、QA 样本来校准。它的价值是上线前验证 AI RAG、Copilot、Agent 在高风险金融零售路径里的行为边界, 例如 KYC 拒绝解释、诈骗支付警告、催收话术和 wealth suitability。它不能替代真实用户研究, 但能把产品假设、架构控制和 release gate 变成证据。

2分钟

我会把 synthetic user lab 定位成 product discovery 和 architecture validation 的中间层。第一步不是让 LLM 随便扮演用户, 而是把真实证据转成 governed assets: persona registry 记录 actor、行为参数、约束、证据链接和不确定性; scenario library 记录 journey stage、触发条件、风险、预期控制和 release gate。

第二步是 journey simulation。金融零售的风险不在单轮回答, 而在多轮路径: 客户开户文件失败后转人工、诈骗者指导客户绕过警告、员工过度采纳 Copilot、agent 调错工具。模拟引擎要记录 persona action、AI response、retrieval、tool call、human approval、escalation 和 outcome proxy。

第三步是 calibration 和 governance。每个 persona 和 scenario 要标注证据等级: 是专家假设、访谈支持、telemetry 支持, 还是生产结果回流支持。高风险 release gate 不能只靠 LLM 生成的漂亮对话, 必须有可复现 run、失败分析、bias/privacy 控制、trace completeness 和 residual risk。

在架构上, 它连接 RAG 的 source grounding、Agent 的权限边界、Copilot 的 human adoption、Eval 的 journey rubric 和 Governance 的证据包。我的核心观点是: synthetic simulation 不证明产品一定成功, 它证明团队是否系统地挑战了自己的假设, 是否知道哪些路径仍不能上线。

CTO版本

我会把它作为 AI platform 的 pre-production behavior testbed, 接入 observability、eval、policy、identity 和 release governance。技术上需要五个关键设计:

1. Simulator 与 system under test 分离, 避免同一个模型生成用户、执行系统、评价结果。
2. 每次 run 都 versioned and replayable: scenario_id、persona_id、seed、model_id、prompt_version、policy_pack_version、retrieval index version 和 tool permission scope 必须入 trace。
3. Persona 和 scenario 必须有 calibration metadata, 用生产 telemetry、QA、complaints、fraud outcomes 逐步提高 confidence, 并明确不能用 simulation 替代真实 validation。
4. Edge-case injection 要覆盖业务伤害, 不只覆盖 prompt attack: KYC false rejection、APP scam warning bypass、collections conduct、complaint escalation、payment dispute misclassification、wealth suitability。
5. Release gate 不看 demo, 看 evidence packet: scenario coverage、failure severity、control effectiveness、trace completeness、privacy/bias review、residual risk and monitoring plan。

我会要求每个 high-impact AI use case 在 architecture review 前通过 synthetic lab, 但我也会把它放在治理边界内: 它是 evidence generator, 不是 compliance conclusion; 它发现风险, 不自动批准上线。

---

7-day Practice Plan

高级练习标准:

• 每个 scenario 都能回答“这个模拟支持哪个产品或架构决策”。
• 每个 persona 都有 evidence confidence, 不是凭空故事。
• 每个 release recommendation 都明确 remaining uncertainty。
• 每个高风险 failure 都转成 architecture backlog、control backlog 或 product backlog。

---

Source Anchors

---

Portfolio Positioning

这篇笔记的作品集价值在于展示三种能力:

面试中的一句话定位:

I use synthetic user simulation as a governed behavior testbed: it stress-tests AI product and architecture assumptions before release, then recalibrates against production evidence after release.