AI Model Portfolio Benchmarking / Capability Scorecard / Selection Governance Playbook

定位：面向 experienced CBAP / financial retail PM / product architect / solution architect / AI governance lead，把模型选型从一次性比较升级为持续性的 model portfolio governance operating model。
边界：本 playbook 不替代模型路由策略、采购沙盒、董事会投资叙事或 AI FinOps。它定义模型组合如何被登记、评测、评分、批准、挑战、监控和退役。

---

Purpose and when to use

Purpose

使用本 playbook 建立一套可执行的模型组合治理机制：

business capability
  -> AI task taxonomy
  -> model portfolio inventory
  -> model card and approved-use boundary
  -> benchmark pack
  -> capability scorecard
  -> model selection record
  -> champion/challenger lifecycle
  -> release evidence packet
  -> retirement trigger

目标不是选出“全公司最强模型”，而是让团队能回答：

• 哪些模型家族被批准用于哪些任务？
• 哪些模型只能做 draft、summary、classification、retrieval answer 或 extraction，不能做 decision？
• 哪些 benchmark pack 支撑本次选型？
• red-team、domain、latency、cost、security 和 governance 分数如何影响批准范围？
• 新模型进入时如何挑战 champion？
• 旧模型在什么情况下必须 watchlist、限制、替换或退役？
• 审计、模型风险和产品决策如何复核当时的证据？

When to use

---

Operating model

Governance bodies

RACI

A = accountable, R = responsible, C = consulted, I = informed.

Cadence

---

Template: model portfolio inventory

---

Template: capability scorecard

Use 1-5 scoring with written rationale. For high-risk workflows, hard blockers override weighted totals.

Hard blocker review

---

Template: benchmark pack

Benchmark result summary

---

Template: model selection record

Decision title:
Use case:
Date:
Decision owner:
Architecture owner:
Risk owner:

Decision:
Approve / reject / constrain / watchlist / retire / keep as challenger.

Selected champion:
Model id, version boundary, model family, provider mode.

Approved use:
Tasks, channels, users, risk tier, workflow steps and human role.

Prohibited use:
Decisions, customer-facing commitments, restricted data or autonomous actions not permitted.

Options considered:
1. Current champion.
2. Challenger A.
3. Challenger B.
4. No-AI or legacy baseline.

Evidence reviewed:
- model cards:
- benchmark pack:
- capability scorecard:
- red-team report:
- domain SME review:
- latency/reliability test:
- security/privacy review:
- model risk challenge:
- production monitoring signals:

Decision rationale:
Explain why this option is fit for the approved use, what tradeoffs are accepted and which scorecard dimensions drove the decision.

Conditions:
List controls, limits, monitoring, human review, fallback and expiry.

Retirement / revalidation triggers:
List exact events that invalidate approval.

Residual risk:
Describe remaining known risks and why the business owner accepts or rejects them.

Approval:
Business owner:
Product owner:
Architecture:
Security/privacy:
Model risk / governance:
Operations:

---

Template: retirement trigger

Retirement decision table

---

Template: evidence packet

---

PM/BA/architecture questions

PM questions

BA / CBAP questions

Architecture questions

---

Release checklist

Release outcomes:

---

Executive narrative

Use this narrative when speaking with a COO, CIO, CTO, CRO or AI governance committee:

We are not choosing a single best AI model for the enterprise. We are governing a model portfolio. Each model is approved for specific tasks, risk tiers and data boundaries based on repeatable evidence. Our scorecard combines task quality, domain performance, safety, latency, cost fitness, security and audit readiness. Public benchmarks inform us, but production approval depends on our own benchmark packs and red-team results. We run champion/challenger reviews so we can adopt better models without uncontrolled drift, and we define retirement triggers so outdated or unsafe models do not remain in regulated workflows. This gives product teams speed, architecture teams control, and risk teams evidence.

For financial retail, translate it this way:

---

Interview drills

Drill 1: Explain the difference between benchmark and portfolio governance

30-second answer:

A benchmark compares model behavior under a defined test. Portfolio governance decides which models are approved for which business tasks over time. It adds model inventory, capability taxonomy, scorecards, champion/challenger cadence, risk-tiered approval, retirement triggers and audit evidence.

Follow-up:

Drill 2: Contact center model selection

Prompt:

A contact center wants to replace its current copilot model with a cheaper small model.

Answer structure:

1. Confirm approved use: draft guidance for agents, not direct customer promises.
2. Run current champion, small challenger and no-AI baseline against the same contact policy benchmark.
3. Score grounding, vulnerable-customer escalation, unsupported fee reversal, tone, p95 latency, cost and trace export.
4. If small model passes routine FAQ but fails complex policy, approve it only for low-risk FAQ and route complex cases to champion.
5. Record conditions, monitoring and retirement/revalidation triggers.

Drill 3: AML triage challenger

Prompt:

A new domain-tuned model improves average AML triage accuracy but misses rare high-risk typologies.

Strong answer:

I would not promote it based on average accuracy. AML needs severity-weighted scoring and false negative controls. Rare high-risk typology misses are hard blockers or at least scope constraints. I would keep the current champion, add the missed typologies to the regression/challenge set, and allow the challenger only in a lower-risk assistive role if it provides value without weakening analyst review.

Drill 4: Open vs proprietary model

Prompt:

The CTO asks whether open-weight models are safer because we can host them internally.

Strong answer:

Hosting control is useful but not sufficient. I would compare open and proprietary models through the same scorecard: task quality, domain score, red-team, data route, patching, license, trace, latency, cost fitness and governance readiness. Open-weight may win for data residency and control, but it needs a patch, safety, eval and operations lifecycle. Proprietary may win on quality, but it needs version boundaries, logging, change notice and data retention evidence.

Drill 5: Retirement trigger

Prompt:

A champion model still works, but the supplier changes retention and logging terms.

Strong answer:

That is a governance trigger even if task quality has not changed. I would freeze expansion, review the data boundary and audit evidence impact, update the model card, rerun any affected security/privacy tests, and require the selection council to keep, constrain, replace or retire the model. A model can be retired because evidence and control no longer support approved use, not only because quality dropped.