AI Model Portfolio Benchmarking / Capability Scorecard / Selection Governance Playbook

配对阅读：本手册的原理/架构解读版是 docs/ai-foundations/papers/168-ai-model-portfolio-benchmarking-capability-scorecard-selection-governance-architecture.md。先读 paper 建立机制与取舍，再用本手册落地为模板、RACI 与门禁，两者不需要重复精读。

定位：面向金融零售模型组合治理、能力基准、选型证据、champion / challenger 生命周期和退役决策的系统学习手册。 边界：本 playbook 不替代模型路由策略、采购沙盒、董事会投资叙事或 AI FinOps。它定义模型组合如何被登记、评测、评分、批准、挑战、监控和退役。

---

Purpose and when to use

Purpose

使用本 playbook 建立一套可执行的模型组合治理机制：

business capability
  -> AI task taxonomy
  -> model portfolio inventory
  -> model card and approved-use boundary
  -> benchmark pack
  -> capability scorecard
  -> model selection record
  -> champion/challenger lifecycle
  -> release evidence packet
  -> retirement trigger

目标不是选出“全公司最强模型”，而是让团队能回答：

• 哪些模型家族被批准用于哪些任务？
• 哪些模型只能做 draft、summary、classification、retrieval answer 或 extraction，不能做 decision？
• 哪些 benchmark pack 支撑本次选型？
• red-team、domain、latency、cost、security 和 governance 分数如何影响批准范围？
• 新模型进入时如何挑战 champion？
• 旧模型在什么情况下必须 watchlist、限制、替换或退役？
• 审计、模型风险和产品决策如何复核当时的证据？

When to use

---

Operating model

Governance bodies

RACI

A = accountable, R = responsible, C = consulted, Info = informed.

Cadence

---

Template: model portfolio inventory

---

Template: capability scorecard

Use 1-5 scoring with written rationale. For high-risk workflows, hard blockers override weighted totals.

Hard blocker review

---

Template: benchmark pack

Benchmark result summary

---

Template: model selection record

Decision title:
Use case:
Date:
Decision owner:
Architecture owner:
Risk owner:

Decision:
Approve / reject / constrain / watchlist / retire / keep as challenger.

Selected champion:
Model id, version boundary, model family, provider mode.

Approved use:
Tasks, channels, users, risk tier, workflow steps and human role.

Prohibited use:
Decisions, customer-facing commitments, restricted data or autonomous actions not permitted.

Options considered:
1. Current champion.
2. Challenger A.
3. Challenger B.
4. No-AI or legacy baseline.

Evidence reviewed:
- model cards:
- benchmark pack:
- capability scorecard:
- red-team report:
- domain SME review:
- latency/reliability test:
- security/privacy review:
- model risk challenge:
- production monitoring signals:

Decision rationale:
Explain why this option is fit for the approved use, what tradeoffs are accepted and which scorecard dimensions drove the decision.

Conditions:
List controls, limits, monitoring, human review, fallback and expiry.

Retirement / revalidation triggers:
List exact events that invalidate approval.

Residual risk:
Describe remaining known risks and why the business owner accepts or rejects them.

Approval:
Business owner:
Product owner:
Architecture:
Security/privacy:
Model risk / governance:
Operations:

---

Template: retirement trigger

Retirement decision table

---

Template: evidence packet

---

Selection Governance Analysis

Use-case and adoption questions

Workflow and evidence questions

Architecture questions

---

Release checklist

Release outcomes:

---

Executive narrative

Use this narrative for a COO, CIO, CTO, CRO or AI governance committee:

The decision is not a search for one best enterprise model. It is governance of a model portfolio. Each model is approved for specific tasks, risk tiers and data boundaries based on repeatable evidence. The scorecard combines task quality, domain performance, safety, latency, cost fitness, security and audit readiness. Public benchmarks are only background signals; production approval depends on internal benchmark packs and red-team results. Champion / challenger reviews allow better models to enter without uncontrolled drift, while retirement triggers prevent outdated or unsafe models from remaining in regulated workflows.

For financial retail, translate it this way:

---

Scenario Studies

Scenario 1: Benchmark vs Portfolio Governance

A benchmark compares model behavior under a defined test. Portfolio governance decides which models are approved for which business tasks over time. It adds model inventory, capability taxonomy, scorecards, champion / challenger cadence, risk-tiered approval, retirement triggers and audit evidence.

Common benchmark traps:

Scenario 2: Contact Center Small-Model Challenger

Scenario: a contact center wants to replace its current copilot model with a cheaper small model.

Governance path:

1. Confirm approved use: draft guidance for agents, not direct customer promises.
2. Run current champion, small challenger and no-AI baseline against the same contact policy benchmark.
3. Score grounding, vulnerable-customer escalation, unsupported fee reversal, tone, p95 latency, cost and trace export.
4. If the small model passes routine FAQ but fails complex policy, approve it only for low-risk FAQ and route complex cases to champion.
5. Record conditions, monitoring and retirement / revalidation triggers.

Scenario 3: AML Triage Challenger

Scenario: a domain-tuned challenger improves average AML triage accuracy but misses rare high-risk typologies.

Governance path:

• Average accuracy cannot drive promotion in AML. Severity-weighted scoring and false-negative controls dominate.
• Rare high-risk typology misses are hard blockers or scope constraints.
• The missed typologies should enter the regression and challenge set.
• The current champion remains for high-risk scope until recall evidence improves.
• The challenger may be allowed only in a lower-risk assistive role if it creates value without weakening analyst review.

Scenario 4: Open vs Proprietary Model

Scenario: open-weight hosting appears safer because the model can run inside the institution.

Governance path:

• Hosting control is useful but not sufficient.
• Open and proprietary models must be compared through the same scorecard: task quality, domain score, red-team, data route, patching, license, trace, latency, cost fitness and governance readiness.
• Open-weight may win for data residency and control, but it needs patch, safety, eval and operations lifecycle.
• Proprietary may win on quality, but it needs version boundaries, logging, supplier change notice and data retention evidence.
• The selection decision should state whether control, quality, cost or evidence gaps drive the approved scope.

Scenario 5: Retirement Trigger

Scenario: a champion model still works, but the supplier changes retention and logging terms.

Governance path:

• Supplier term change is a governance trigger even if task quality has not changed.
• Expansion should freeze until data boundary and audit evidence impact are reviewed.
• The model card, security/privacy tests and evidence packet need updates.
• The selection council decides whether to keep, constrain, replace or retire the model.
• A model can be retired because evidence and control no longer support approved use, not only because quality dropped.