AI Product Operating Model / Empowered Teams Playbook
这些来源作为学习锚点, 不构成法律、合规、人力组织或监管咨询意见。
AI Product Operating Model / Empowered Teams Playbook
定位: 面向 AI PM / AI BA / AI Solutions Architect / Enterprise Architect / 金融零售转型负责人的 AI 产品运营模型手册。 目标: 把 AI 产品从“项目制交付”升级为 empowered teams 驱动的持续发现、持续交付、持续治理能力。 核心观点: AI product operating model 不是会议制度, 而是把业务结果、团队授权、风险责任、平台能力和学习闭环放到同一套 operating system 里。
Source Anchors
这些来源作为学习锚点, 不构成法律、合规、人力组织或监管咨询意见。
| Anchor | Link | 在本 playbook 中的用法 |
|---|---|---|
| SVPG Product Operating Model / Marty Cagan official | https://www.svpg.com/product-operating-model/ | 用 product model 语言组织 empowered teams、outcome orientation、discovery 和 delivery |
| SVPG Empowered Product Teams | https://www.svpg.com/empowered-product-teams/ | 用“给团队问题和上下文, 而不是功能清单”的原则定义 AI team autonomy |
| NIST AI RMF | https://www.nist.gov/itl/ai-risk-management-framework | 用 Govern / Map / Measure / Manage 组织 AI 风险、评估、监控和持续治理 |
1. One-Sentence Positioning
AI Product Operating Model 是一套让业务、产品、技术、数据、风险和运营围绕可验证业务结果协同工作的机制: empowered AI product teams 负责解决高价值问题, 平台团队提供可复用能力, 风险团队设定边界和门禁, 三者通过 discovery-delivery-governance cadence 持续学习、交付、评估和治理。
2. 为什么 AI 产品需要新的 Operating Model
传统数字化产品通常围绕需求、排期、上线和运营指标运转。AI 产品多了几个持续变量:
- 模型能力和价格持续变化。
- Prompt、知识库、工具调用和 policy 会持续变化。
- 输出质量不是确定性功能, 需要 eval 和线上反馈闭环。
- 风险不是上线前一次性审批, 而是随数据、模型、流程和用户行为持续变化。
- 业务价值取决于 workflow adoption, 不取决于 demo 是否惊艳。
- 金融零售场景涉及客户权益、信贷公平性、AML 合规、隐私保护、审计证据和监管解释。
因此 AI 产品不能只用项目管理语言管理。它需要回答:
Which business outcomes are teams empowered to own?
Which AI decisions can the team make independently?
Which decisions require platform, risk, data, legal or executive review?
How does discovery evidence become delivery scope?
How does delivery evidence become governance evidence?
How do we scale patterns without killing team autonomy?
好的 AI operating model 不把团队变成审批机器, 也不把风险治理变成上线末端的否决权。它把授权边界提前说清楚, 让团队在清晰 guardrails 内快速试验和交付。
3. Product Operating Model 的核心转译
| 传统项目制语言 | AI product operating model 语言 | 行为变化 |
|---|---|---|
| 交付需求清单 | 解决业务问题 | 团队对 outcome 负责, 不只对 output 负责 |
| Sponsor 提功能 | Sponsor 提问题和约束 | 团队有权定义方案路径 |
| 上线即完成 | 上线后持续学习 | eval、adoption、risk signal 进入常规节奏 |
| PM 写 PRD, 技术实现 | Product trio 共同发现和交付 | 业务、设计、技术、数据、风险一起定义可验证假设 |
| 风险部门最后审批 | 风险作为 operating partner | 风险参与 tiering、controls、release gate 和 post-launch review |
| 平台团队接需求 | 平台团队产品化横向能力 | model gateway、eval、RAG、observability、policy guardrails 可复用 |
| 成功看 project milestone | 成功看业务结果和学习速度 | cycle time、quality、adoption、cost、risk 都可度量 |
4. AI Product Trio
4.1 Trio 的基础形态
AI product trio 不是把三个人放进同一个会议, 而是明确三个互补视角共同拥有 discovery 和 delivery 质量。
| Trio role | 核心责任 | AI 场景下必须补强的问题 |
|---|---|---|
| AI Product Lead | business outcome、scope、roadmap、adoption、trade-off | 哪个 workflow 先做, 如何证明价值, 如何处理风险与速度的取舍 |
| AI Tech / Architecture Lead | system design、integration、NFR、tooling、delivery feasibility | RAG / agent / workflow / rules / vendor 如何取舍, 如何监控和回滚 |
| AI Design / BA / Process Lead | user workflow、decision journey、requirements evidence、change impact | 用户何时信任 AI, 哪些判断必须人工保留, 哪些异常路径影响运营 |
在金融零售组织中, 第三个角色常由高级 BA、服务设计师、业务架构师或运营流程 owner 承担。用户已具备 CBAP 背景时, 重点不是补基础需求技巧, 而是把业务规则、流程证据、控制点、异常路径和 eval criteria 连接起来。
4.2 Extended Product Trio
AI team 不可能只靠三个人做完所有判断。trio 应该拥有决策节奏, 同时拉入关键 partner。
| Extended partner | 什么时候必须进入 | 主要贡献 |
|---|---|---|
| Data Owner | 涉及客户数据、交易数据、知识源、标签、训练或检索 | data quality、lineage、access、retention、freshness |
| Risk / Compliance | 涉及客户权益、AML、信贷、投资建议、监管解释 | risk tier、control requirement、human oversight、audit evidence |
| Security / Privacy | 涉及 PII、敏感数据、外部模型、工具调用 | IAM、DLP、encryption、vendor security、data boundary |
| Platform Owner | 需要 model gateway、RAG、eval、observability、tool gateway | reusable capability、cost、SLO、release guardrails |
| Ops Lead | 影响一线流程、案件队列、客户服务、运营绩效 | rollout、training、feedback、exception handling |
| Legal / Model Risk | 涉及模型风险管理、披露、合同、监管承诺 | model documentation、review evidence、approved language |
4.3 Trio 的工作契约
| 契约 | 具体要求 |
|---|---|
| Problem ownership | trio 被授权解决一个业务问题, 而不是接收固定功能清单 |
| Evidence standard | 每个重要决定必须有 discovery evidence、eval evidence 或 risk evidence |
| Decision log | AI pattern、model choice、data boundary、human oversight、release gate 要有 ADR 或 decision note |
| Learning cadence | 每周看用户、质量、风险和成本信号, 不只看交付进度 |
| Escalation path | 超出授权边界时升级到 platform / risk / executive forum |
5. Decision Rights
5.1 授权原则
AI empowered team 的关键不是“团队什么都能决定”, 而是哪些决定在 guardrails 内由团队负责, 哪些决定必须共治。
| Principle | 说明 |
|---|---|
| Outcome over output | 团队被授权选择解决路径, 但必须对业务结果、质量和 adoption 负责 |
| Guardrails before speed | 先定义 risk tier、data boundary、human oversight, 再讨论快速试验 |
| Reversible decisions stay with team | 可快速回滚、影响范围小、风险低的决策留给团队 |
| Irreversible or regulated decisions require review | 涉及客户权益、合规申报、信贷拒绝、投资建议等决策必须进入门禁 |
| Platform standards are shared constraints | 日志、eval、成本、审计、模型接入、权限不能各团队自定义 |
| Risk owns policy, product owns value, platform owns enablement | 三者边界清晰, 共同负责可运营结果 |
5.2 Decision Rights Matrix
| Decision | AI Product Trio | Platform | Risk / Compliance | Data / Security | Executive / Sponsor |
|---|---|---|---|---|---|
| Business outcome and target metric | A/R | C | C | C | A for strategic priority |
| Workflow insertion point | A/R | C | C | C | I |
| AI pattern: RAG / workflow / agent / rules / vendor | A/R | C/R | C | C | I |
| Model provider within approved allowlist | R | A/R for gateway | C | C | I |
| New model outside allowlist | C | R | C | A/R for security review | A for strategic exception |
| Knowledge source selection | R | C | C | A/R by data or knowledge owner | I |
| Prompt change for low-risk assistant | A/R | C | I | I | I |
| Prompt change affecting regulated output | R | C | A/R | C | I |
| Human oversight policy | R | C | A/R | C | C |
| Eval threshold for release | R | C/R | A/R for high risk | C | I |
| Production release | A/R within approved tier | R | A/R for high risk | C | I |
| Rollback after incident | A/R | R | C/A by severity | C | I |
| Scale to new product line or geography | R | C | A/R | C | A |
Legend:
- A = accountable for final decision.
- R = responsible for preparing and executing.
- C = consulted before decision.
- I = informed after decision.
5.3 Decision Records
每个 AI capability 至少保留以下 decision records:
- Opportunity decision: 为什么这个问题值得做, 为什么不是纯流程改造或规则系统。
- AI pattern decision: 为什么选择 RAG、agent、workflow automation、classification、optimization、vendor 或 no-AI。
- Risk tier decision: 依据哪些客户、财务、合规、运营影响定级。
- Data boundary decision: 哪些数据可用、不可用、可外发、可保留。
- Human oversight decision: 哪些步骤允许建议, 哪些步骤允许自动执行, 哪些步骤必须人工批准。
- Eval release decision: 哪些 eval 指标达到上线门槛, 哪些风险被接受。
- Scale decision: 为什么从 pilot 扩展到更多团队、渠道、产品或地区。
6. Discovery-Delivery-Governance Cadence
6.1 三条节奏线
AI team 需要同时运行三条节奏线。它们不是三个部门的孤岛, 而是同一套产品学习系统。
| Cadence | 主要问题 | 主要证据 | 决策 |
|---|---|---|---|
| Discovery | 这个问题是否值得做, 用户和流程是否清楚, AI 是否是合适解法 | workflow evidence、user interview、baseline metric、risk hypothesis、prototype result | proceed / refine / stop |
| Delivery | 方案是否可交付、可集成、可评估、可运营 | architecture、eval set、trace、latency、cost、security evidence、pilot result | build / release / rollback |
| Governance | 风险是否可接受, 责任是否清晰, 上线后是否可监控和审计 | risk register、control evidence、model/prompt/index version、incident log、audit trail | approve / condition / block / retire |
6.2 推荐节奏
| Frequency | Meeting / Forum | Owner | Inputs | Outputs |
|---|---|---|---|---|
| Twice weekly during discovery | Product trio discovery working session | AI Product Lead | opportunity canvas、workflow map、user evidence、risk hypothesis | next assumption to test、prototype scope、decision notes |
| Weekly | AI quality and eval review | EvalOps / Tech Lead | golden set result、failure taxonomy、user feedback、trace samples | prompt/index/model/tool fixes、release confidence |
| Weekly | Delivery review | Tech Lead | delivery board、architecture risks、integration blockers、cost/latency data | scope trade-off、technical decision、pilot readiness |
| Biweekly | Product/risk checkpoint | Product + Risk | risk tier、control design、HITL policy、customer impact | risk acceptance path、additional controls、go/no-go conditions |
| Monthly | Platform capability review | Platform Owner | reusable requests、adoption, SLO, cost, incidents | platform roadmap, standard updates, enablement needs |
| Monthly | Business adoption review | Product + Ops | workflow KPI、adoption cohort、training feedback、exception cases | rollout adjustments、process change、retire/scale decision |
| Quarterly | AI portfolio governance | Executive sponsor | portfolio value、risk trend、cost、incidents、capability health | fund / scale / pause / retire |
6.3 Gate Flow
G0 Idea Intake
-> G1 Opportunity Discovery
-> G2 Risk and Data Readiness
-> G3 AI Pattern Decision
-> G4 Pilot Build
-> G5 Pilot Review
-> G6 Production Release
-> G7 Scale Review
-> G8 Quarterly Capability Review
| Gate | Exit criteria |
|---|---|
| G0 Idea Intake | 有业务 owner、业务问题、初步价值、初步风险边界 |
| G1 Opportunity Discovery | 有 baseline、workflow insertion point、用户证据、AI fit / no-AI alternative |
| G2 Risk and Data Readiness | risk tier、data/knowledge owner、access、retention、human oversight 初稿明确 |
| G3 AI Pattern Decision | 选择 AI pattern, 记录 trade-off, 定义 eval strategy |
| G4 Pilot Build | pilot scope、eval set、observability、rollback、support path 准备就绪 |
| G5 Pilot Review | 质量、adoption、成本、风险信号达到 pilot success criteria |
| G6 Production Release | release gate 通过, RACI、runbook、monitoring、incident path 生效 |
| G7 Scale Review | 多团队 adoption、unit economics、risk trend、platform capacity 可支撑 |
| G8 Quarterly Capability Review | 决定 continue、refresh、scale、restrict 或 retire |
7. Product / Platform / Risk Operating Model
7.1 三层模型
Business Product Teams
own workflow outcomes, adoption, user learning, use-case roadmap
AI Platform Teams
own reusable AI capabilities, standards, developer experience, cost and reliability
Risk / Governance Teams
own policy boundaries, oversight model, audit evidence, regulatory readiness
7.2 责任边界
| Area | Product team owns | Platform team owns | Risk / governance owns |
|---|---|---|---|
| Problem framing | outcome、workflow、user、value hypothesis | reusable pattern signal | risk lens、regulated boundaries |
| AI pattern | use-case fit、user trust、business trade-off | technical options、reference architecture | risk suitability、human oversight requirement |
| Model access | required capability and quality target | model gateway、allowlist、routing、fallback、logging | model use policy、restricted categories |
| Knowledge | source relevance、business semantics | ingestion, retrieval, metadata, access filter | data classification、retention、audit needs |
| Eval | business acceptance criteria、failure impact | eval harness、automation、traceability | threshold policy、risk-specific tests |
| Release | rollout scope、adoption plan、support model | deployment, SLO, monitoring, rollback | approval gate、control evidence |
| Operations | workflow KPI、training、feedback loop | telemetry, reliability, cost dashboard | incident severity, review cadence |
| Scale | product-line expansion, change management | capacity, reusable components, standards | cross-market or regulated approval |
7.3 Interaction Model
| Interaction | 触发条件 | 结果 |
|---|---|---|
| Product to Platform intake | 业务团队发现重复 AI capability need | 平台判断 build / buy / standardize / defer |
| Platform standard change | model gateway、RAG、eval、policy、logging 规则变化 | 产品团队评估影响并更新 release evidence |
| Product to Risk checkpoint | 新 use case、risk tier change、regulated workflow、customer-impacting output | 风险确认 controls、oversight、evidence |
| Risk to Product intervention | incident、quality drift、policy breach、complaint、audit finding | containment、fix plan、release restriction |
| Quarterly portfolio review | 多个 AI use case 的价值、风险、成本需要比较 | 投资、扩展、暂停、退役决策 |
8. 金融零售 AI 组织能力模型
金融零售 AI 能力不是“谁会 prompt”或“谁会接模型”。它需要产品、平台、风控、运营和业务架构共同成熟。
| Capability | Level 1: Project | Level 2: Repeatable | Level 3: Operating Model |
|---|---|---|---|
| Opportunity shaping | 各部门提 AI 点子 | 有 intake 和 scoring | 按业务能力、风险和价值管理 AI portfolio |
| Workflow design | demo 独立于真实流程 | pilot 嵌入局部流程 | AI 变成可运营流程能力, 有 adoption 和 exception loop |
| EvalOps | 主观试用 | golden set 和 release threshold | eval、online monitoring、incident、quarterly review 闭环 |
| Knowledge governance | 文档临时上传 | 有 owner 和更新周期 | 知识源、权限、版本、freshness 与审计证据统一治理 |
| Human oversight | “人工最终确认”口号 | 明确 approve / review / override | 人工节点与风险等级、队列、SLA、问责绑定 |
| AI platform | 各团队接不同模型 | gateway、RAG、eval 复用 | 平台以产品方式运营 adoption、SLO、cost、developer experience |
| Risk partnership | 风险最后审批 | 风险参与 gate | 风险规则被产品化为 policy-as-code、review cadence、evidence pack |
| Organization learning | 项目复盘零散 | case library | 用案例、模板和社区实践提升全组织 AI fluency |
9. 金融零售案例
9.1 AML Case Investigation Copilot
| Dimension | Operating model decision |
|---|---|
| Business outcome | 降低调查准备时间, 提高 case narrative 一致性, 不自动决定是否提交 SAR |
| Empowered team | AML product trio 对 workflow、case summary、analyst experience、pilot adoption 负责 |
| Platform dependency | RAG over policy/manual/case history, evidence citation, audit log, role-based retrieval |
| Risk boundary | AI 可汇总证据和草拟 narrative, 不可独立作出可疑活动结论 |
| Human oversight | AML analyst 必须确认 red flags、证据和 final filing rationale |
| Eval focus | citation accuracy、missing red flags、false rationale、policy freshness、analyst override reason |
| Governance cadence | 每周 quality review, 每月 AML risk review, 重大误导输出立即 incident triage |
关键 decision rights:
- Product trio 决定 case workspace、summary format、workflow insertion point。
- Risk/Compliance 决定 SAR 相关 language、必须人工确认的判断点。
- Data owner 决定 case history、transaction data、watchlist data 的访问和保留边界。
9.2 Customer Service RAG Assistant
| Dimension | Operating model decision |
|---|---|
| Business outcome | 提高一次解决率、降低平均处理时长、提升话术合规一致性 |
| Empowered team | 客服 product trio 对 agent experience、knowledge feedback、adoption cohort 负责 |
| Platform dependency | knowledge ingestion、permission filter、approved answer style、feedback taxonomy |
| Risk boundary | AI 提供答案建议和引用, 不自动承诺赔付、费用减免或账户处理 |
| Human oversight | 高风险客户投诉、费用争议、欺诈相关问题进入人工升级 |
| Eval focus | answer groundedness、policy compliance、citation validity、handoff quality |
| Governance cadence | 每周 knowledge gap review, 每月 compliance sampling, 季度 channel expansion review |
关键 decision rights:
- Product trio 决定先进入哪些客服队列和问题类型。
- Knowledge owner 决定政策文档来源、版本和失效机制。
- Risk/Legal 决定禁止表达、强制披露和升级规则。
9.3 Credit Decision Support
| Dimension | Operating model decision |
|---|---|
| Business outcome | 提高 underwriter 效率和解释一致性, 降低遗漏材料和返工 |
| Empowered team | 信贷 product trio 负责 decision support workflow, 不拥有自动拒绝权 |
| Platform dependency | document extraction、case summarization、policy retrieval、decision trace |
| Risk boundary | AI 可做材料检查、风险因素汇总、policy match, 不直接作出授信批准或拒绝 |
| Human oversight | underwriter 保留 final decision, override reason 必须结构化记录 |
| Eval focus | missing critical document、incorrect policy match、bias signal、explanation consistency |
| Governance cadence | 每周 pilot review, 每月 credit risk review, 季度 fair lending / model risk evidence review |
关键 decision rights:
- Product trio 决定 assistant 放在申请预审、材料补齐还是 underwriter review。
- Credit risk 决定哪些建议会影响客户权益、哪些输出必须限制。
- Model risk / compliance 决定 fairness、explainability、documentation expectations。
9.4 Enterprise AI Platform
| Dimension | Operating model decision |
|---|---|
| Business outcome | 缩短 AI use case 从 idea 到 pilot 的周期, 降低重复建设和治理成本 |
| Empowered team | Platform product trio 对 developer experience、reuse、SLO、cost、standard adoption 负责 |
| Platform dependency | 自身就是平台: model gateway、prompt registry、eval harness、RAG service、tool gateway |
| Risk boundary | 平台不替业务团队承担 use-case value, 但强制执行最低审计、日志、权限和 release gate |
| Human oversight | 高风险 tool/action 由平台提供 approval workflow 和 audit trail |
| Eval focus | gateway reliability、cost attribution、eval run coverage、policy guardrail effectiveness |
| Governance cadence | 每月 platform review, 季度 standard refresh, 重大 provider change 触发 impact review |
关键 decision rights:
- Platform team 决定平台 API、标准、reference architecture。
- Product teams 决定具体 use case 的 workflow 和 value hypothesis。
- Risk/Security 决定模型 allowlist、数据边界、最低证据要求。
9.5 Wealth Advisor Copilot
| Dimension | Operating model decision |
|---|---|
| Business outcome | 提高顾问准备效率、客户沟通一致性和下一步行动质量 |
| Empowered team | 财富顾问 product trio 对 advisor workflow、client meeting prep、adoption 负责 |
| Platform dependency | client profile summarization、product knowledge retrieval、meeting note drafting、CRM integration |
| Risk boundary | AI 可辅助准备和草拟, 不独立生成个性化投资建议或承诺收益 |
| Human oversight | 顾问确认 suitability、disclosure、客户目标和最终建议 |
| Eval focus | suitability guardrail、outdated product info、missing disclosure、hallucinated performance claim |
| Governance cadence | 每周 advisor feedback review, 每月 compliance sampling, 季度 product catalog freshness review |
关键 decision rights:
- Product trio 决定先支持哪些顾问任务, 如 meeting prep、follow-up draft、portfolio review checklist。
- Compliance 决定建议边界、披露文本和禁止表达。
- Product catalog owner 决定基金、保险、理财产品知识的版本和可见范围。
10. Templates
10.1 AI Product Operating Model Canvas
# AI Product Operating Model Canvas
## 1. Business Capability
- Capability:
- Business owner:
- Operational owner:
- Target users:
- Customer or employee impact:
## 2. Outcome
- North-star outcome:
- Baseline metric:
- Target improvement:
- Leading indicators:
- Guardrail metrics:
## 3. Workflow
- AS-IS workflow:
- AI insertion point:
- Human decision points:
- Exception paths:
- Process changes required:
## 4. AI Pattern
- Pattern selected:
- Alternatives considered:
- Why this pattern:
- No-AI alternative:
- Reversibility:
## 5. Product Trio
- AI Product Lead:
- AI Tech / Architecture Lead:
- AI Design / BA / Process Lead:
- Extended partners:
## 6. Decision Rights
- Decisions owned by team:
- Decisions requiring platform review:
- Decisions requiring risk/compliance approval:
- Decisions requiring executive approval:
- Escalation triggers:
## 7. Risk and Controls
- Risk tier:
- Data boundary:
- Human oversight:
- Policy guardrails:
- Audit evidence:
- Incident path:
## 8. Eval and Monitoring
- Golden set:
- Business acceptance criteria:
- Risk-specific eval:
- Online quality signals:
- Adoption signals:
- Cost and latency signals:
## 9. Release and Scale
- Pilot cohort:
- Release gate:
- Rollback condition:
- Scale condition:
- Quarterly review question:
10.2 Product Trio Working Agreement
| Agreement area | Concrete agreement |
|---|---|
| Outcome | 我们共同负责一个可度量业务结果, 不共同维护一个功能清单 |
| Discovery | 每周至少验证一个关键假设或失败模式 |
| Delivery | 每个 release 都有 eval、trace、rollback、support path |
| Risk | 高风险边界提前讨论, 不在上线前突袭 |
| Evidence | 重要判断进入 decision log, 不靠口头记忆 |
| User learning | 用户反馈按 failure taxonomy 分类, 进入 backlog |
| Platform use | 默认使用 approved gateway、eval、logging、audit capabilities |
| Escalation | 触发客户权益、监管、PII、自动化行动时升级 |
10.3 Decision Rights Charter
# Decision Rights Charter
## Team Scope
- Team:
- Business capability:
- Outcome:
- Risk tier:
## Team Can Decide
- Workflow UI and interaction details within approved scope.
- Prompt copy for low-risk internal guidance.
- Pilot cohort and learning plan within approved customer/data boundary.
- Backlog priority when outcome and risk guardrails are unchanged.
## Team Must Consult
- Platform: new model, new tool, new integration, new observability need.
- Data owner: new data source, changed retention, changed access pattern.
- Ops: workflow change, staffing impact, training plan.
## Team Must Get Approval
- Customer-impacting automated action.
- Regulated advice, credit, AML, complaint, fraud or suitability output.
- External model or vendor outside approved boundary.
- Production release for high-risk use case.
- Scale to new product, region, channel or customer segment.
## Escalation Triggers
- Material quality drop.
- High-severity hallucination or unsupported recommendation.
- Policy or compliance breach.
- Data leakage or unauthorized access.
- Cost spike beyond budget threshold.
- User adoption failure that invalidates business case.
10.4 Discovery Brief
# AI Discovery Brief
## Problem
- Business problem:
- Affected users:
- Current workflow pain:
- Baseline:
## Why Now
- Trigger:
- Strategic relevance:
- Operational urgency:
## AI Fit
- Task type:
- Why AI:
- Why not rules/process/reporting/training:
- Expected uncertainty:
## Risk Hypothesis
- Customer impact:
- Financial impact:
- Compliance impact:
- Privacy/security impact:
- Human oversight needed:
## Learning Plan
- Assumption 1:
- Evidence to collect:
- Prototype or experiment:
- Stop condition:
- Proceed condition:
10.5 Release Gate Evidence Pack
# AI Release Gate Evidence Pack
## Use Case
- Name:
- Owner:
- Risk tier:
- Release scope:
## Value Evidence
- Baseline:
- Pilot result:
- Adoption signal:
- Business owner sign-off:
## Quality Evidence
- Eval dataset:
- Eval metrics:
- Failure taxonomy:
- Known limitations:
- Regression result:
## Risk Evidence
- Controls:
- Human oversight:
- Prohibited outputs/actions:
- Audit logs:
- Incident runbook:
## Platform Evidence
- Model gateway:
- Prompt/config version:
- Knowledge/index version:
- Observability:
- Cost and latency:
- Rollback:
## Decision
- Release decision:
- Conditions:
- Review date:
- Decision owner:
11. Review Checklists
11.1 Operating Model Readiness
- 是否明确业务 capability 和 outcome owner?
- 是否明确 AI product trio 和 extended partners?
- 团队是否被授权解决问题, 而不是只接收功能清单?
- 是否有 decision rights charter?
- 是否区分 product、platform、risk 的责任边界?
- 是否有 discovery、delivery、governance 三条节奏?
- 是否有 release gate 和 quarterly review?
- 是否有 team-level metrics 和 portfolio-level metrics?
- 是否有 incident、rollback、retire 机制?
- 是否把用户 adoption 纳入核心节奏?
11.2 Empowered Team Review
- Team 是否拥有明确 business outcome?
- Team 是否有权限改变 workflow、方案和 backlog?
- Sponsor 是否提供 context、constraints 和 strategic priority, 而不是功能指令?
- Product trio 是否共同参加用户研究、原型验证和风险讨论?
- Tech lead 是否在 discovery 阶段参与 AI pattern 选择?
- BA / process lead 是否把流程、控制点、异常路径转化为 eval criteria?
- Risk partner 是否在早期定义 guardrails, 而不是最后审批?
- Team 是否能解释哪些决定可自主、哪些必须升级?
- Team 是否拥有线上学习数据和改进节奏?
11.3 AI Governance Review
- 是否完成 NIST AI RMF 风格的 Govern / Map / Measure / Manage 映射?
- 是否识别客户权益、金融损失、合规、隐私和运营风险?
- 是否有 risk tier 与控制强度的映射?
- 是否定义 human oversight 类型: approve、review、override、appeal?
- 是否记录模型、prompt、knowledge、tool 的版本?
- 是否有 eval threshold 和 risk-specific test?
- 是否有 audit evidence pack?
- 是否有 incident severity 和 response owner?
- 是否有 post-launch monitoring 和 review cadence?
11.4 Platform Fit Review
- 是否默认使用 approved model gateway?
- 是否使用统一 prompt/config registry?
- 是否使用统一 eval runner 或至少输出可比较 eval report?
- 是否记录 trace、retrieval、tool call、cost、latency?
- 是否接入 IAM、RBAC、DLP、audit log?
- 是否有 rollback 和 fallback?
- 是否识别可复用 platform capability?
- 是否避免为单个 use case 过度定制不可复用平台能力?
12. Metrics
12.1 Team Metrics
| Dimension | Metric | 解释 |
|---|---|---|
| Outcome | cycle time reduction, case throughput, first contact resolution, advisor prep time | 证明业务流程被改善 |
| Adoption | active users, repeat usage, workflow completion, override rate | 证明用户把 AI 纳入工作方式 |
| Quality | groundedness, citation accuracy, false positive/negative, policy compliance | 证明输出可信 |
| Risk | incident count, high-risk override, control breach, complaint signal | 证明风险被识别和管理 |
| Cost | cost per successful task, cost per case, token/cache efficiency | 证明经济性可持续 |
| Learning | assumptions tested, failure modes closed, release regression trend | 证明团队在持续学习 |
12.2 Platform Metrics
| Dimension | Metric | 解释 |
|---|---|---|
| Reuse | % AI apps using gateway/eval/RAG/tool catalog | 平台是否成为默认路径 |
| Speed | time from intake to pilot, time to first eval run | 平台是否降低重复工作 |
| Reliability | gateway SLO, retrieval latency, incident MTTR | 平台是否可运营 |
| Governance | % releases with evidence pack, % prompt/model changes versioned | 标准是否落地 |
| Cost | cost by model/team/use case, cache hit rate | 成本是否可解释 |
| Developer experience | onboarding time, support tickets, reference app adoption | 团队是否愿意用平台 |
12.3 Portfolio Metrics
| Dimension | Metric | 解释 |
|---|---|---|
| Value | realized benefit by capability, benefit confidence | 区分真实收益和估算收益 |
| Risk | risk distribution by use case, unresolved high-risk issues | 看组织风险暴露 |
| Investment | platform vs use-case spend, build/buy/vendor ratio | 看投资组合是否平衡 |
| Scale | pilots scaled, pilots stopped, capabilities retired | 证明组织能扩展也能停止 |
| Capability | team maturity, eval coverage, governance coverage | 证明 operating model 成熟度 |
13. Anti-Patterns
| Anti-pattern | 表现 | 后果 | 更好的做法 |
|---|---|---|---|
| Feature factory with AI branding | 业务方给功能清单, 团队只排期交付 | 没有 outcome ownership, demo 多、价值少 | 给团队问题、约束和指标 |
| Risk theater | 文档很多, 但没有 eval、monitoring、incident loop | 审批通过但线上不可控 | 把风险证据接入 release 和 quarterly review |
| Platform empire building | 平台先做大而全能力, use case 还没验证 | 成本高、adoption 低 | 从重复 use case pattern 中产品化平台能力 |
| POC graveyard | POC 成功后没有 owner、流程、SLO、budget | 试点无法生产化 | pilot 前定义 release gate 和 operating owner |
| Human-in-the-loop slogan | 只写“人工复核”, 没有队列、SLA、责任和 override 记录 | 风险没有实际控制 | 定义 approve/review/override/appeal 机制 |
| Eval afterthought | 上线前临时做测试集 | 质量无法复现, 回归不可控 | discovery 阶段就定义 eval strategy |
| Shadow AI tools | 团队绕过平台直接用外部模型 | 数据和审计风险 | 提供可用 gateway 和清晰 exception process |
| Centralized approval bottleneck | 所有小决策都等委员会 | 团队失去授权和速度 | 用 risk tier 和 reversibility 分层授权 |
| Adoption vanity metrics | 只看登录、调用量、生成次数 | 无法证明流程改善 | 看 workflow completion、override、cycle time、business KPI |
| One-size-fits-all governance | AML、客服、营销文案同一审批强度 | 低风险过慢, 高风险不足 | 按风险等级和客户影响设门禁 |
14. 30 天训练计划
Week 1: Operating Model Foundation
| Day | Focus | Output |
|---|---|---|
| 1 | 研读 SVPG product operating model 和 empowered teams, 提炼 AI 转译 | 1 页 AI empowered team principles |
| 2 | 选择一个金融零售 AI use case, 写 business outcome 和 no-AI alternative | AI Discovery Brief v1 |
| 3 | 画 AS-IS / TO-BE workflow, 标出 AI insertion point 和 human decision point | Workflow map |
| 4 | 定义 AI product trio 和 extended partners | Product Trio Working Agreement |
| 5 | 写 decision rights charter | Decision Rights Charter |
| 6 | 用 NIST AI RMF Govern / Map / Measure / Manage 检查风险生命周期 | Risk lifecycle map |
| 7 | 复盘: 这个团队到底被授权解决什么问题 | 1 页 operating model critique |
Week 2: Discovery and Decision Rights
| Day | Focus | Output |
|---|---|---|
| 8 | 选择 AML、客服、信贷、AI 平台或财富顾问中的一个 case 深挖 | Case scope brief |
| 9 | 定义 baseline、target metric、guardrail metric | Metrics tree |
| 10 | 写 AI pattern decision: RAG / workflow / agent / rules / vendor | ADR v1 |
| 11 | 定义 risk tier、data boundary、human oversight | Risk and control note |
| 12 | 设计 eval strategy 和 failure taxonomy | Eval plan |
| 13 | 组织一次 simulated product/risk checkpoint | Meeting notes and decisions |
| 14 | 复盘 decision rights 是否过宽或过窄 | Decision rights revision |
Week 3: Delivery and Governance
| Day | Focus | Output |
|---|---|---|
| 15 | 设计 pilot scope、cohort、support path | Pilot plan |
| 16 | 设计 release gate evidence pack | Evidence pack draft |
| 17 | 定义 observability: trace、quality、cost、latency、feedback | Monitoring plan |
| 18 | 定义 incident severity、rollback condition、communication path | Incident mini-runbook |
| 19 | 将平台依赖映射到 model gateway、RAG、eval、policy、audit | Platform dependency map |
| 20 | 设计 monthly platform review 和 business adoption review | Cadence calendar |
| 21 | 复盘是否能从 pilot 扩展到 production | Production readiness review |
Week 4: Portfolio and Interview Conversion
| Day | Focus | Output |
|---|---|---|
| 22 | 把 3 个 AI use cases 放入 portfolio matrix | AI portfolio board |
| 23 | 比较 product team、platform team、risk team 的责任边界 | Operating model map |
| 24 | 写一个 executive memo: 为什么需要 empowered AI teams | Executive memo |
| 25 | 写一个 risk memo: 为什么授权团队不等于放松治理 | Risk memo |
| 26 | 准备 5 个面试故事: AML、客服、信贷、平台、财富顾问 | STAR-T answer set |
| 27 | 整理作品集目录和证据 | Portfolio package |
| 28 | 做一次 mock interview, 强调 decision rights 和 cadence | Mock interview notes |
| 29 | 修订所有模板, 删除低价值内容 | Final artifact pack |
| 30 | 写 1 页总结: 我的 AI Product Operating Model 观点 | Position paper |
15. Interview Answers
Q1: 你如何定义 AI Product Operating Model?
30 秒版本
AI Product Operating Model 是让 AI 团队围绕业务结果持续发现、交付和治理的工作系统。它明确 empowered product teams 的授权边界, 定义 product、platform、risk 的责任关系, 并用 eval、监控、human oversight 和 governance cadence 保证 AI 能上线、能运营、能审计、能持续改进。
2 分钟版本
我不会把它理解成会议制度或 RACI 表。AI 产品的变量比传统软件更多: 模型、prompt、知识库、工具、用户行为和监管边界都在变。所以 operating model 必须覆盖四件事。第一是 outcome ownership, 团队被授权解决业务问题而不是交付功能清单。第二是 product trio, 产品、技术/架构、设计/BA/流程共同做 discovery 和 delivery。第三是 decision rights, 低风险可逆决策留给团队, 涉及客户权益、信贷、AML、投资建议、数据外发的决策进入风险和平台门禁。第四是 cadence, discovery、delivery、governance 必须持续连接, 用 eval、adoption、risk signal 和 incident evidence 决定 release、scale 或 retire。
Q2: Empowered AI product team 和传统项目团队最大的区别是什么?
30 秒版本
传统项目团队通常被要求按需求清单交付。empowered AI product team 被给予业务问题、上下文和约束, 有权选择解决路径, 同时对业务结果、质量、adoption 和风险证据负责。
2 分钟版本
在 AI 场景里这个差异更重要。因为一开始没人能完全确定最佳方案是 RAG、规则、workflow automation、agent、vendor 还是非 AI 流程改造。如果团队只能接功能需求, 很容易做出漂亮 demo 但不进入真实流程。empowered team 应该拥有发现权、方案权和 backlog trade-off 权, 但不是无限授权。它必须在 guardrails 内工作, 比如使用 approved model gateway、eval harness、audit logging, 并在高风险决策上接受 risk/compliance review。授权和治理不是对立的, 清晰 decision rights 让团队更快也更安全。
Q3: AI product trio 应该怎么组成?
30 秒版本
基础 trio 是 AI Product Lead、AI Tech/Architecture Lead、AI Design/BA/Process Lead。金融零售场景还需要 data owner、risk/compliance、security/privacy、platform owner 和 ops lead 作为 extended partners。
2 分钟版本
AI Product Lead 负责业务结果、路线图、scope 和 adoption。Tech/Architecture Lead 负责 AI pattern、系统集成、NFR、eval 和 rollback。Design/BA/Process Lead 负责用户流程、决策旅程、异常路径、控制点和需求证据。高级 BA 在这里非常关键, 因为 AI 输出是否可用往往取决于流程语义、业务规则、人工判断点和异常处理是否被清楚表达。对 AML、信贷、财富顾问这些场景, risk/compliance 不是外围审批人, 而是早期共创 guardrails 的 partner。
Q4: 如何设计 AI 产品的 decision rights?
30 秒版本
我会按风险等级、可逆性和客户影响分层。团队保留低风险、可回滚、局部流程优化的决策权; 平台拥有模型接入、日志、eval、审计等标准; 风险拥有政策边界和高风险 release gate; 高战略或跨区域扩展进入 executive review。
2 分钟版本
具体做法是先列出关键决策: workflow insertion point、AI pattern、model provider、knowledge source、prompt change、human oversight、release、rollback、scale。然后给每类决策定义 accountable owner。比如客服内部助手的低风险 prompt 优化可由 product trio 决定; 但如果输出影响费用减免、投诉处理或监管话术, 就需要 compliance review。信贷 assistant 可以帮助 underwriter 汇总材料和匹配政策, 但不能由团队单独决定自动拒绝客户。好的 decision rights 能避免两种失败: 所有决策都集中审批导致团队失速, 或所有决策都下放导致风险失控。
Q5: Discovery、Delivery、Governance cadence 如何连接?
30 秒版本
Discovery 负责证明问题和方案方向, Delivery 负责把方案做成可评估、可监控、可回滚的能力, Governance 负责确保风险、责任和证据持续有效。三者通过 gate、eval review、risk checkpoint 和 quarterly review 连接。
2 分钟版本
我会让 cadence 围绕证据流动, 而不是围绕会议。Discovery 产出 workflow evidence、baseline、AI fit、risk hypothesis 和 prototype learning。Delivery 把这些转成 architecture、eval set、monitoring、pilot plan 和 release package。Governance 使用这些证据做 risk tiering、control review、release approval 和 post-launch monitoring。上线后, 用户反馈、质量 drift、incident、成本和 adoption 信号会回到 discovery backlog 和 platform roadmap。这样 AI 产品不会停在 demo 或 launch, 而是持续运营。
Q6: 金融零售 AI operating model 最容易失败在哪里?
30 秒版本
最容易失败在三个地方: demo 脱离真实流程, 风险治理停留在文档审批, 平台能力和业务 use case 脱节。
2 分钟版本
例如 AML copilot 如果只会生成总结, 但不能引用证据、不能进入 case workflow、不能记录 analyst override, 就无法运营。客服 RAG 如果知识 owner 不明确, 文档过期会直接造成错误话术。信贷场景如果没有明确 human oversight 和 fair lending evidence, 很容易把效率工具误变成高风险自动决策。AI 平台如果只做技术组件, 不看 developer experience、eval coverage、cost attribution 和 adoption, 会变成低复用基础设施。解决方法是把 product、platform、risk 三层 operating model 同时设计。
16. Portfolio Deliverables
一个高级 AI Product Operating Model 作品集不应该只展示 PRD, 应该展示你能把组织机制、团队授权、平台能力和风险治理连接起来。
| Deliverable | 内容 | 展示能力 |
|---|---|---|
| AI Product Operating Model Canvas | 业务 capability、outcome、workflow、AI pattern、team、risk、eval、release | 端到端 operating model 设计 |
| Empowered Team Charter | product trio、extended partners、decision rights、工作契约 | empowered teams 和组织设计 |
| Decision Rights Matrix | product/platform/risk/executive 的 A/R/C/I | 高级治理和授权边界 |
| Discovery Brief | 问题、AI fit、no-AI alternative、风险假设、学习计划 | 高质量 discovery |
| AI Pattern ADR | RAG / agent / workflow / rules / vendor 的取舍 | 产品架构判断 |
| Release Gate Evidence Pack | value、quality、risk、platform、rollback、decision | 上线治理和审计证据 |
| Cadence Calendar | weekly eval、risk checkpoint、platform review、portfolio review | operating cadence 设计 |
| Financial Retail Case Pack | AML、客服、信贷、AI 平台、财富顾问 5 个案例 | 行业深度和可迁移能力 |
| Executive Memo | 为什么需要 empowered AI teams, 投资和治理如何平衡 | 高层沟通 |
| Risk Memo | 授权团队如何不降低风险控制 | 风险合作和监管思维 |
推荐作品集结构:
01-positioning.md
02-operating-model-canvas.md
03-empowered-team-charter.md
04-decision-rights-matrix.md
05-discovery-brief.md
06-ai-pattern-adr.md
07-release-gate-evidence-pack.md
08-cadence-calendar.md
09-financial-retail-case-pack.md
10-executive-and-risk-memos.md
17. 自我评估 Rubric
| Level | 能力表现 |
|---|---|
| L1 | 能解释 AI 产品需要 discovery、eval 和 governance, 但仍以项目交付为主 |
| L2 | 能为单个 use case 设计 product trio、decision rights、release gate |
| L3 | 能把 product、platform、risk 三层 operating model 连接起来 |
| L4 | 能管理金融零售 AI portfolio, 平衡价值、风险、成本和组织 adoption |
| L5 | 能推动组织从 project funding 转向 empowered teams + platform leverage + continuous governance |
高级面试中要证明自己达到 L3 以上, 不要停留在“我会写 PRD、排优先级、协调研发”。更有区分度的表达是:
- 我如何让团队从 feature team 变成 outcome team。
- 我如何定义 AI decision rights, 既授权又可审计。
- 我如何把 BA 的流程和规则能力转化为 eval criteria 和 risk controls。
- 我如何把平台能力产品化, 让多个业务团队复用。
- 我如何在 AML、客服、信贷、财富顾问等场景中区分建议、决策和行动。
18. Final Operating Principles
- AI team 要被授权解决业务问题, 不是被动接收功能清单。
- Product trio 必须共同做 discovery, 否则 AI pattern 和 workflow fit 会脱节。
- Decision rights 必须按风险、可逆性和客户影响分层。
- 平台标准要提供速度, 不是只增加审批。
- 风险团队越早参与, 团队越能快速交付。
- Eval 是产品需求的一部分, 不是测试阶段的附属品。
- Human oversight 要有队列、SLA、override 和责任记录。
- Adoption 是 AI 产品价值的核心证据。
- Quarterly review 要敢于 refresh、restrict 或 retire AI capability。
- 金融零售 AI 的成熟度, 体现在业务结果、组织授权、平台复用和风险证据能否同时成立。