AI 模型组合基准评测 / 能力评分卡 / 选型治理架构：Model Portfolio Benchmarking / Capability Scorecard / Selection Governance Architecture

Date: 2026-06-30
Status: evergreen
Audience: experienced CBAP / financial retail PM / product architect / solution architect / AI governance lead
Output: advanced architecture note for website-visible portfolio, model selection council design, audit evidence and interview discussion

---

Why model portfolio governance matters

金融零售企业不会长期只用一个 AI 模型。实际环境通常同时存在 frontier model、small model、embedding model、reranker、domain-tuned classifier、document extraction model、speech model、judge model、open-weight model、managed proprietary model 和 legacy ML model。问题不再是“哪个模型最好”，而是：

对每个业务任务、风险等级、数据边界、延迟约束、成本约束、安全要求和审计要求，哪个模型家族在当前证据下被批准使用，什么时候需要 challenger，什么时候必须退役？

这是一套 portfolio governance 问题，而不是单次 benchmark 或 vendor demo。

边界说明：本文不做模型路由策略设计，不做采购沙盒，不做董事会投资叙事，也不做 AI FinOps 成本管理。这里聚焦持续性的模型组合治理：模型能力分类、基准评测、能力评分卡、选型委员会、champion/challenger、风险分层、退役触发和证据链。

---

Concept diagram

flowchart TB
  A[Business capability map] --> B[AI task taxonomy]
  B --> C[Model portfolio inventory]
  C --> D[Model cards and approved-use boundaries]

  B --> E[Benchmark packs]
  E --> E1[Quality and task metrics]
  E --> E2[Safety and red-team tests]
  E --> E3[Domain and policy tests]
  E --> E4[Cost, latency and resilience tests]

  D --> F[Capability scorecard]
  E1 --> F
  E2 --> F
  E3 --> F
  E4 --> F

  F --> G{Model selection council}
  G -->|Approve| H[Champion model for approved use]
  G -->|Constrain| I[Conditional use with controls]
  G -->|Reject| J[Do not use or return to lab]
  G -->|Challenge| K[Challenger backlog]

  H --> L[Release and production monitoring]
  I --> L
  L --> M[Evidence packet]
  L --> N[Drift, incident and policy-change signals]
  N --> K
  N --> O[Retirement trigger]
  O --> P[Replacement / fallback / decommission]

  M --> Q[Audit, model risk, product decision record]
  Q --> C

核心思想：

model portfolio governance
  = inventory + capability taxonomy + benchmark pack + scorecard
  + selection council + champion/challenger + retirement trigger
  + auditable evidence

---

Core architecture model

1. Portfolio layers

2. Model portfolio inventory as a governed asset

模型清单不是技术团队的 spreadsheet。它要支持业务、架构、风险、审计和运营共同判断。

3. Capability scorecard as decision interface

Scorecard 的目标不是替代判断，而是把不同角色的判断放到同一张证据表里。

public benchmark tells us generic capability.
domain benchmark tells us task fit.
red-team benchmark tells us unacceptable behavior.
architecture score tells us operability.
governance score tells us whether evidence can survive audit.
selection council turns all of that into approved-use boundaries.

---

Capability taxonomy and scorecard

Capability taxonomy

Scorecard dimensions

Score each dimension 1-5, but apply hard blockers for high-risk use cases. A strong average cannot compensate for a critical failure.

Hard blockers:

• Any critical customer harm, privacy, security, regulated advice or unauthorized action failure in approved high-risk scope.
• No reproducible benchmark run for the task.
• No model/version boundary or provider configuration boundary.
• No trace export for regulated workflows requiring review.
• Model behavior materially changed without change notice or re-benchmark.
• Open model cannot be patched, scanned, hosted or access-controlled to enterprise policy.

Model family comparison

---

Benchmark and challenger lifecycle

Lifecycle states

Benchmark pack design

Champion/challenger cadence

---

Financial retail scenarios

1. Contact center agent assist

Evidence:

• contact policy RAG pack with current and stale policy conflicts.
• red-team set for customer pressure, prompt injection, and account privacy.
• trace showing retrieved sources, model output, agent edits, escalation and final disposition.

2. AML triage and investigation narrative

Evidence:

• AML typology benchmark by structuring, mule activity, funnel account, rapid movement and benign lookalikes.
• narrative rubric for material omission and evidence citation.
• human review log proving analyst retains final decision.

3. KYC document extraction

Evidence:

• document pack with passports, IDs, utility bills, bank statements, low-quality scans and non-English layouts.
• field-level confusion matrix.
• exception evidence for missing address, expired document and name mismatch.

4. Credit policy RAG

Evidence:

• policy question pack across product, state, effective date and exception rules.
• stale policy and conflicting source challenge set.
• model selection record that separates advice support from credit decisioning.

5. Payments fraud intervention

Evidence:

• fraud typology pack for APP scam, account takeover, mule transfer, false-positive customer friction.
• p95 latency test for operational queue.
• red-team tests for social engineering and disclosure of fraud controls.

6. Enterprise knowledge assistant

Evidence:

• knowledge coverage map by HR, operations, product, risk, technology and policy domains.
• entitlement test with users from branch, contact center, risk and engineering.
• benchmark that separates retriever failure from generator failure.

---

Metrics/control/evidence model

Metrics

Controls

Evidence packet

model card
  + approved-use boundary
  + benchmark pack manifest
  + run configuration
  + scorecard
  + slice and failure analysis
  + red-team report
  + latency/cost/security results
  + model selection record
  + exception or risk acceptance
  + monitoring and retirement triggers

---

Anti-patterns and failure modes

---

Architecture mapping to RAG / Agent / Copilot / Eval / Governance

This architecture also clarifies component-level selection. A single use case can have separate champions for:

• generation model.
• embedding model.
• reranker.
• extractor.
• classifier.
• safety model.
• evaluator/judge model.
• fallback model.

The governance object is not “the chatbot model”; it is the AI system model portfolio that produces behavior in a specific workflow.

---

ADR draft

Title: Establish AI model portfolio benchmarking and selection governance
Date: 2026-06-30
Status: Proposed

Context:
Financial retail AI systems use multiple model families across contact center, AML, KYC, credit, payments and enterprise knowledge workflows. Current model decisions are fragmented across product teams, vendor evaluations, platform experiments and project-specific benchmarks. Public benchmarks and one-off pilots do not provide sufficient evidence for regulated workflow decisions.

Decision:
Create a governed model portfolio architecture with:
1. model portfolio inventory and model cards;
2. AI task capability taxonomy;
3. benchmark packs by use case and risk tier;
4. weighted capability scorecards with hard blockers;
5. champion/challenger lifecycle;
6. model selection council decision records;
7. retirement triggers and evidence packets.

Approved-use decisions will be made at the task/use-case boundary, not at the generic model brand level. High-risk workflows require domain benchmark, red-team score, trace evidence, human oversight fit and explicit retirement triggers.

Alternatives considered:
1. Let each product team choose models independently. Rejected because evidence is not comparable and auditability is weak.
2. Choose one enterprise-standard model. Rejected because task, risk, latency, cost, data residency and control needs differ.
3. Use public leaderboards as primary selection evidence. Rejected because they do not represent financial retail context of use.

Consequences:
Positive:
- Decisions become comparable, auditable and reusable across use cases.
- Model changes can be assessed through challenger runs rather than preference debates.
- Product, architecture, risk and audit share the same evidence language.

Tradeoffs:
- Benchmark packs and model cards require ongoing ownership.
- Some fast experiments will be slowed by evidence gates.
- Scorecards must be maintained as models, policies and business workflows change.

Review triggers:
- new model family or provider enters portfolio;
- champion shows material regression or critical failure;
- business policy, regulation, data boundary or workflow changes;
- supplier changes model version, logging, retention or service behavior;
- quarterly portfolio review.

---

Interview answer: 30秒, 2分钟, CTO版本

30秒

我不会用一个 leaderboard 分数决定企业模型选型。我会建立 model portfolio governance：先按业务任务和风险等级定义 capability taxonomy，再为每个 use case 建 benchmark pack 和 scorecard，评估质量、domain fit、red-team、安全、延迟、成本、可审计性和人工监督适配。最后由 model selection council 决定 champion、challenger、限制条件和退役触发，并留下 model card、run result 和 selection record。

2分钟

在金融零售里，contact center、AML、KYC、credit policy RAG、payments fraud 和 enterprise knowledge assistant 对模型的要求完全不同。KYC 可能更看重 extraction accuracy 和 low-confidence routing，AML 更看重 false negative 和 typology coverage，credit policy RAG 更看重 citation support 和禁止 unsupported adverse action，contact center 更看重低延迟、话术和升级边界。

所以我会把模型治理做成持续组合管理。第一层是 model portfolio inventory，记录模型家族、版本、供应模式、数据边界、approved use 和 prohibited use。第二层是 capability taxonomy，把任务拆成抽取、分类、检索回答、摘要、建议、安全、domain policy、operability 和 governance。第三层是 benchmark pack，用真实和合成案例、domain edge cases、red-team cases 和运营测试比较 champion 与 challenger。第四层是 scorecard，用权重和 hard blockers 防止平均分掩盖高风险失败。第五层是 decision record，记录为什么批准、限制、拒绝或退役。

这样模型选择就从“哪个模型最好”变成“在这个业务任务和风险边界内，哪个模型有足够证据被批准使用”。

CTO版本

我会把模型组合治理放在 AI platform control plane 与 AI governance operating model 之间。平台负责模型网关、配置版本、eval runner、trace、policy enforcement、fallback 和 monitoring；治理负责 capability taxonomy、risk-tiered benchmark、scorecard、selection council、exception 和 retirement。关键设计是：approved use 绑定 task、workflow、risk tier、data boundary 和 benchmark evidence，而不是绑定模型品牌。

对 CTO 来说，这解决四个架构风险：第一，避免每个团队重复比较模型但证据不可复用；第二，避免供应商或模型版本变化造成 silent regression；第三，避免一刀切模型标准牺牲 latency、cost、security 或 domain fit；第四，为审计、模型风险和生产事故复盘保留可追溯证据。最终输出不是一个“最佳模型列表”，而是一套可运行的 champion/challenger portfolio，能持续吸收新模型，同时能及时退役不再适合的模型。

---

7-day practice plan

---

Source anchors with links