AI Human Review Operations / Capacity Architecture 解读

面向对象: AI PM / AI BA / AI Architect / Operations Lead / Model Risk / Compliance / Workforce Planning。 核心问题: 很多 AI 项目把 human review 写成控制, 但没有设计队列、容量、技能路由、校准、升级、独立性和证据。结果是人工复核变成瓶颈、橡皮图章或控制剧场。 学习目标: 把 human review 从 generic HITL 升级为生产运营系统: queue economics、capacity planning、reviewer quality、SLA/OLA、疲劳控制、override governance、sampling、training、surge mode 和可审计 evidence。

---

Source Anchors

一句话:

Human review operations 不是“让人看一下 AI 输出”, 而是把有限的人类判断力设计成可调度、可校准、可审计、可扩容、可恢复的生产控制系统。

---

1. Thesis

Generic HITL 关注“是否有人在回路中”。Human Review Operations / Capacity Architecture 关注更难的问题:

Which cases should humans review?
Which humans are qualified?
How much review capacity is needed?
What evidence must reviewers see?
How do we know review quality improves the outcome?
Who can override, escalate, pause or stop AI?

核心观点:

• Human review 是运营能力, 不是 UI 控件。
• Review capacity 是控制容量, 不是可无限挤压的人力池。
• Reviewer calibration 是模型治理的一部分, 不是培训附件。
• Override quality 比 override count 更重要。
• 如果没有容量、校准、独立性、审计证据和升级权, human review 会变成瓶颈或控制剧场。

---

2. Why It Matters

金融零售 AI 项目常见断点:

risk tier -> queue policy -> skill routing -> capacity model
-> reviewer workspace -> calibration and QA
-> escalation / override governance -> evidence ledger
-> dashboard and workforce planning

---

3. Core Concepts

3.1 Review Unit

Review unit 是队列和容量模型的基本单位, 不一定等于一个客户、一个会话或一个 case。

3.2 Queue Economics

Human review 的经济学不是“多加几个人”, 而是 balancing case arrival rate、average handling time、risk mix、skill availability、SLA/OLA、rework、escalation、fatigue 和 surge reserve。

required_review_hours =
  case_volume * average_handling_minutes / 60
  * complexity_multiplier
  * double_review_multiplier
  * rework_multiplier
  / productivity_factor

PM 必须把 review hours 计入 AI ROI。AI 节省的一线时间如果转移为更高成本的专家复核, 价值会被吞掉。

3.3 Calibration

Calibration 是让 reviewer 对相同证据、相同 rubric、相同风险边界形成稳定判断。

---

4. Architecture Diagram

AI output / action candidate
  -> risk and impact classifier
  -> review policy engine
       - 100% review, risk-based review, stratified sample
       - exception review, blind / double review
  -> queue orchestrator
       - priority, SLA / OLA, skill routing
       - capacity throttle, conflict-of-interest rule
  -> reviewer workspace
       - AI output, evidence bundle, policy / rubric
       - missing evidence, allowed actions
  -> human decision
       - accept, edit, reject, override, escalate, stop route
  -> quality and calibration layer
       - gold cases, second review, adjudication, reviewer drift
  -> evidence ledger and dashboard
       - trace, reason code, timing, downstream action, audit replay

关键原则:

• Review policy engine 必须在 workflow 层, 不能只靠页面提示。
• Queue orchestrator 必须知道技能、SLA、容量和冲突规则。
• Evidence bundle 必须版本化, 包含 AI / model / prompt / source / policy / tool trace。
• Reviewer action 必须结构化记录, 方便 QA、审计、模型改进和事故响应。

---

5. Financial Retail Case: Payment Dispute AI Review Operations

场景:

AI 汇总支付争议材料, 推荐下一步处理和客户沟通草稿。
部分 case 涉及临时入账、拒绝争议、欺诈信号、监管时限和客户投诉。

Risk routing:

Daily AI-assisted dispute candidates: 4,000
Pre-action review rate: 18%
Average handling time: 7 minutes
Double review share: 12%
Rework multiplier: 1.10
Productive reviewer hours per day: 5.8

Required reviewers =
  4000 * 0.18 * 7 / 60 * 1.12 * 1.10 / 5.8
  = 17.9 reviewers

如果上线计划只准备 8 名 reviewer, 该控制在生产中必然变成 backlog 或 rubber stamp。 Reviewer 必须看到交易记录、客户声明、争议规则时限、AI 推荐、置信边界、政策版本、历史 dispute 和任何 tool action 参数。

---

6. PM / BA / Architect Checklist

• 每个 review queue 有 owner、SLA、OLA、skill requirement 和 backup。
• 每个 reviewer action 有 reason code 和 evidence reference。
• 高风险 override 有 authority matrix。
• 低 override rate 会触发质量检查, 不被自动视为好结果。
• Capacity model 覆盖 normal、peak、incident 和 degraded mode。
• Calibration 使用 gold cases、blind review 和 adjudication。

---

7. Code-Lite Experiment

用一个小表格模拟 capacity planning:

Inputs:
  volume_per_day = 2500
  review_rate = 0.22
  avg_minutes = 6.5
  double_review_rate = 0.15
  rework_rate = 0.08
  productive_hours = 5.5

Calculation:
  reviewed_cases = volume_per_day * review_rate
  adjusted_minutes = reviewed_cases * avg_minutes * (1 + double_review_rate) * (1 + rework_rate)
  required_fte = adjusted_minutes / 60 / productive_hours

Output:
  reviewed_cases = 550
  adjusted_minutes = 4440.15
  required_fte = 13.46

Sensitivity:

---

8. Interview Questions

Q1: Human review 和 generic HITL 有什么区别?

30 秒:

HITL 说明有人参与, 但 human review operations 要证明这个人有技能、时间、证据、独立性、权限和升级路径。否则人工复核只是按钮。

Q2: 如何防止 reviewer 变成 rubber stamp?

30 秒:

我会同时设计 evidence workspace、blind / double review、gold cases、reason code、override quality metrics、review duration monitoring 和 capacity threshold。长期 0 override 或极短 review time 不是好消息。

Q3: 如何做 reviewer capacity planning?

30 秒:

用到达量、review rate、平均处理时间、复杂度、双审比例、返工率、可用生产小时和 surge reserve 建模。高风险 AI 上线必须证明 review queue 在 normal、peak 和 incident 模式下都能守住 SLA。

Q4: Reviewer calibration 为什么是 AI governance?

30 秒:

因为人工标签、override 和审核决定会影响客户结果、模型改进和审计证据。如果 reviewer 判断不一致, 组织无法证明 AI 控制有效。

---

9. Pitfalls

Human review is production capacity.
If it is not routed, staffed, calibrated, measured, evidenced and governed,
it is not a reliable control.