AI 模型组合基准评测 / 能力评分卡 / 选型治理架构：Model Portfolio Benchmarking / Capability Scorecard / Selection Governance Architecture

配对阅读：本篇的操作手册版（模板/RACI/门禁/runbook）是 docs/AI_MODEL_PORTFOLIO_BENCHMARKING_CAPABILITY_SCORECARD_SELECTION_GOVERNANCE_PLAYBOOK.md。第一遍读本篇建立原理与架构判断；第二遍做案例时再用 playbook 查表落地，两者不需要重复精读。

Date: 2026-06-30
Status: evergreen
Audience: experienced CBAP / 金融零售 AI 产品与架构从业者 / AI governance lead

---

核心导读

AI model portfolio governance 解决的不是“哪个模型最好”，而是对每个业务任务、风险等级、数据边界、延迟约束、成本约束、安全要求和审计要求，哪个模型家族在当前证据下被批准使用，什么时候需要 challenger，什么时候必须退役。它把模型选择从 vendor demo 和 leaderboard 偏好，升级为持续的组合治理。

金融零售企业的 AI 系统通常不是一个模型，而是 generation、embedding、reranker、extractor、classifier、speech、judge、safety、fallback、fine-tuned 和 legacy ML 的组合。模型选择会改变数据驻留、客户影响、自动化边界、工具权限、可解释性、供应集中、成本/延迟、事故响应和证据可得性；因此 approved use 必须绑定具体 task、workflow、risk tier、data boundary 和 benchmark evidence，而不是绑定某个模型品牌或公开排行榜。

系统价值来自 capability scorecard：把通用能力、领域任务、政策遵循、安全/滥用、稳健性、公平性、隐私、成本、延迟、可观测性、供应风险和退出能力放进同一张可挑战的证据表。Public benchmark 只能作为弱先验；真正的选择证据来自 context-of-use benchmark pack、红队样本、生产 shadow run、人工复核、失败分析、fallback 演练、model card、decision record 和 exception 记录。

治理边界要能阻止“平均分更高”掩盖关键失败。某模型可以适合内部摘要，但不适合 regulated advice；可以适合 document extraction，但不适合客户拒绝原因；可以作为 challenger shadow run，但不能越过数据、权限或供应商条件进入生产。Portfolio governance 的核心证据，是 challenger promotion、review expiry、drift trigger、incident trigger、policy change trigger、vendor change trigger 和 retirement path 能否在模型能力变化时及时重新打开决策。

问题定义

金融零售企业不会长期只用一个 AI 模型。实际环境通常同时存在 frontier model、small model、embedding model、reranker、domain-tuned classifier、document extraction model、speech model、judge model、open-weight model、managed proprietary model 和 legacy ML model。问题从单点选型变成 portfolio governance：每个模型必须绑定 approved use、prohibited use、benchmark evidence、risk tier、change trigger 和 retirement path。

边界说明：本文不做模型路由策略设计，不做采购沙盒，不做董事会投资叙事，也不做 AI FinOps 成本管理。这里聚焦持续性的模型组合治理：模型能力分类、基准评测、能力评分卡、选型委员会、champion/challenger、风险分层、退役触发和证据链。

架构模型/核心原理

flowchart TB
  A[Business capability map] --> B[AI task taxonomy]
  B --> C[Model portfolio inventory]
  C --> D[Model cards and approved-use boundaries]

  B --> E[Benchmark packs]
  E --> E1[Quality and task metrics]
  E --> E2[Safety and red-team tests]
  E --> E3[Domain and policy tests]
  E --> E4[Cost, latency and resilience tests]

  D --> F[Capability scorecard]
  E1 --> F
  E2 --> F
  E3 --> F
  E4 --> F

  F --> G{Model selection council}
  G -->|Approve| H[Champion model for approved use]
  G -->|Constrain| I[Conditional use with controls]
  G -->|Reject| J[Do not use or return to lab]
  G -->|Challenge| K[Challenger backlog]

  H --> L[Release and production monitoring]
  I --> L
  L --> M[Evidence packet]
  L --> N[Drift, incident and policy-change signals]
  N --> K
  N --> O[Retirement trigger]
  O --> P[Replacement / fallback / decommission]

  M --> Q[Audit, model risk, product decision record]
  Q --> C

核心思想：

model portfolio governance
  = inventory + capability taxonomy + benchmark pack + scorecard
  + selection council + champion/challenger + retirement trigger
  + auditable evidence

---

核心架构模型

1. Portfolio layers

2. Model portfolio inventory as a governed asset

模型清单不是技术团队的 spreadsheet。它要支持业务、架构、风险、审计和运营共同判断。

3. Capability scorecard as decision interface

Scorecard 的目标不是替代判断，而是把不同角色的判断放到同一张证据表里。

public benchmark tells us generic capability.
domain benchmark tells us task fit.
red-team benchmark tells us unacceptable behavior.
architecture score tells us operability.
governance score tells us whether evidence can survive audit.
selection council turns all of that into approved-use boundaries.

---

关键机制

Capability taxonomy

Scorecard dimensions

Score each dimension 1-5, but apply hard blockers for high-risk use cases. A strong average cannot compensate for a critical failure.

Hard blockers:

• Any critical customer harm, privacy, security, regulated advice or unauthorized action failure in approved high-risk scope.
• No reproducible benchmark run for the task.
• No model/version boundary or provider configuration boundary.
• No trace export for regulated workflows requiring review.
• Model behavior materially changed without change notice or re-benchmark.
• Open model cannot be patched, scanned, hosted or access-controlled to enterprise policy.

Model family comparison

---

Benchmark and challenger lifecycle

Lifecycle states

Benchmark pack design

Champion/challenger cadence

---

证据与控制

Metrics

Controls

Evidence packet

model card
  + approved-use boundary
  + benchmark pack manifest
  + run configuration
  + scorecard
  + slice and failure analysis
  + red-team report
  + latency/cost/security results
  + model selection record
  + exception or risk acceptance
  + monitoring and retirement triggers

---

金融零售/AI产品场景

1. Contact center agent assist

Evidence:

• contact policy RAG pack with current and stale policy conflicts.
• red-team set for customer pressure, prompt injection, and account privacy.
• trace showing retrieved sources, model output, agent edits, escalation and final disposition.

2. AML triage and investigation narrative

Evidence:

• AML typology benchmark by structuring, mule activity, funnel account, rapid movement and benign lookalikes.
• narrative rubric for material omission and evidence citation.
• human review log proving analyst retains final decision.

3. KYC document extraction

Evidence:

• document pack with passports, IDs, utility bills, bank statements, low-quality scans and non-English layouts.
• field-level confusion matrix.
• exception evidence for missing address, expired document and name mismatch.

4. Credit policy RAG

Evidence:

• policy question pack across product, state, effective date and exception rules.
• stale policy and conflicting source challenge set.
• model selection record that separates advice support from credit decisioning.

5. Payments fraud intervention

Evidence:

• fraud typology pack for APP scam, account takeover, mule transfer, false-positive customer friction.
• p95 latency test for operational queue.
• red-team tests for social engineering and disclosure of fraud controls.

6. Enterprise knowledge assistant

Evidence:

• knowledge coverage map by HR, operations, product, risk, technology and policy domains.
• entitlement test with users from branch, contact center, risk and engineering.
• benchmark that separates retriever failure from generator failure.

---

反模式

---

最终心智模型

模型治理的对象不是“一个 chatbot model”，而是能在特定工作流中产生行为的 AI system model portfolio。一个用例可以同时有 generation model、embedding model、reranker、extractor、classifier、safety model、evaluator/judge model 和 fallback model；approved use 必须绑定 task、workflow、risk tier、data boundary 和 benchmark evidence，而不是绑定模型品牌。

最终判断标准：public benchmark 只能作为弱先验，不能替代 context-of-use evidence。强平均分不能抵消 critical privacy、security、regulated advice 或 unauthorized action failure。Champion/challenger 机制让组织持续吸收新模型；retirement trigger 防止旧模型在政策、供应、成本、延迟或安全证据失效后继续服务高风险流程。

Source Anchors

---

SOTA 检查 (2026-07-01)

• HELM 本身已退居维护线：Stanford CRFM 的 HELM 框架/榜单于 2026-06-01 进入 maintenance mode（见 stanford-crfm/helm README 的 Maintenance Mode Policy）。这不推翻本篇结论，反而强化它——连最权威的"living benchmark"也会退役，说明选型证据绝不能长期挂在任何单一公共榜单上；本篇"public benchmark 只能作弱先验，context-of-use benchmark pack 才是选择证据"的主线在 2026-07 仍然成立。
• 安全基准现役主线是 MLCommons AILuminate：v1.0 论文发布于 2025-03（arXiv:2503.05731，12 个 hazard 类别、每语言 2.4 万条 prompt），GitHub 套件已迭代到 v1.1；2026-02 MLCommons 宣布 AILuminate Global Assurance Program（KPMG、Google、Microsoft、Qualcomm 等参与），并有 Jailbreak Benchmark v0.5 草案覆盖多模态越狱。本篇的 red-team gate / safety hard blocker 机制可直接挂接这条现役标准线。
• 美国银行业模型风险监管锚点已换代：Fed/OCC/FDIC 于 2026-04-17 发布修订版 interagency model risk management guidance，取代 2011 年 SR 11-7 框架。本篇的 champion/challenger、drift/incident trigger、retirement path 与 evidence packet 设计与新指引的持续监督期望方向一致；金融机构落地时引用锚点应更新为 2026 新指引而非 SR 11-7 原文。另注意 EU AI Act Annex III 高风险义务已推迟至 2027-12-02（Omnibus，2026-05-07），涉及 EU 场景的 review expiry 与合规排期应按新时间线设置。
• 2026 年评测行业叙事与本篇一致：主流综述（Kili Technology、LXT 的 2026 benchmark 盘点）指出当前约 15 个活跃主流 benchmark 中仅少数能可靠预测生产表现，评测重心已从 static leaderboard 转向 dynamic、multi-dimensional、context-of-use 评测；AI model risk management 市场 2026 年约 83 亿美元规模且监管审查加强。本篇"平均分不能掩盖 critical failure、approved use 绑定 task/risk tier/data boundary"的判断是当前主流实践方向。
• 框架性结论不随模型版本过时：capability taxonomy、scorecard hard blockers、model portfolio inventory 字段、champion/challenger lifecycle states、retirement trigger、evidence packet 是治理结构，不绑定任何具体模型或榜单版本；换掉 HELM/AILuminate 这层"证据供给"，结构照常运转。
• 库内落地参考：eval gate 的 CI 阻断实现见 docs/aipa/day19-blocking-ci-eval-gate.md（2026-07-03），失败分类到评测类型的映射见 docs/aipa/day13-taxonomy-eval-mapping.md（2026-06-27），可作为本篇 benchmark gate / failure taxonomy 机制的工程化样例。