AI Shadow Mode / Counterfactual Evaluation / Silent Launch Playbook

定位: 面向 experienced CBAP / financial retail PM / product architect / solution architect / AI governance lead 的高级落地手册。
核心问题: 如何在不影响客户、员工决策或系统状态的前提下, 用真实业务上下文验证 AI 决策是否值得进入 assisted mode、limited rollout 或正式治理审批。
适用范围: credit line management、AML alert triage、KYC onboarding、payment fraud intervention、collections contact strategy、contact center agent assist。
边界: 本手册不替代法律意见、模型验证报告、合规审批、UAT certification、online experimentation、release governance 或 adoption analytics。

---

Purpose And When To Use

Purpose

Shadow mode / counterfactual evaluation / silent launch 的目的, 是在 AI 真正影响客户或一线员工之前, 生成可审计、可比较、可决策的 evidence:

When To Use

Use Case Fit

---

Operating Model

1. Roles And Decision Rights

2. End-To-End Flow

intake
  -> decision boundary
  -> population and sampling plan
  -> feature/context snapshot design
  -> read-only challenger integration
  -> counterfactual event logging
  -> leakage and evidence checks
  -> human comparison
  -> delayed outcome join
  -> fairness and segment scorecard
  -> gate decision
  -> assisted mode / continue shadow / no-go

3. Cadence

---

Shadow Mode Intake Template

---

Counterfactual Event Schema Template

Design rule: every event must reconstruct what the challenger knew, what it would have done, what the champion did, and what later happened.

---

Label/Outcome Plan Template

Outcome rules:

• Each gate memo must state which outcomes are mature, proxy-only, or unavailable.
• Proxy outcomes can support readiness but cannot prove full business impact.
• Outcome join failures are evidence quality issues, not missing details to ignore.
• Label definitions must be frozen before gate analysis to avoid decision-driven relabeling.

---

Leakage Controls Template

---

Fairness/Segment Scorecard Template

Scorecard rules:

• Show champion rate beside AI rate so the review is about decision change, not raw model output.
• Separate false positive harm and false negative harm; different use cases value them differently.
• Segment thresholds must be approved before reviewing results.
• Any critical segment regression blocks broad rollout even when aggregate metrics improve.

---

Rollout Gate Template

Gate outputs:

• No-go: material leakage, prohibited action, unfair segment harm, unreviewed critical disagreement or unreconstructable evidence.
• Continue shadow: evidence incomplete, labels immature, operations not ready, or value unclear.
• Limited go: narrow segment, human approval, explicit rollback triggers, daily monitoring.
• Rollout go: mature evidence supports controlled expansion and risk owners approve.

---

Rollback Trigger Template

Rollback rule: a rollback trigger must specify who can stop shadow exposure, how quickly the path is disabled, what evidence is preserved, and how re-entry is approved.

---

Evidence Packet Template

---

PM/BA/Architecture Questions

PM Questions

BA / CBAP Questions

Architecture Questions

---

Release Checklist

This checklist is named “release” because it decides whether the AI can be released from hidden shadow into a more exposed mode. It is not a general software release checklist.

---

Executive Narrative

1-Minute Steering Committee Version

We are not asking to let the AI change customer outcomes yet. We are asking to continue or expand a controlled shadow mode where the AI sees real business events, produces counterfactual recommendations, and records what it would have done while the existing champion process remains fully in control. This gives us evidence on decision quality, delayed outcomes, fairness, human review differences, operational readiness and audit traceability before any customer-impacting rollout.

3-Minute Risk Committee Version

The control objective is to avoid moving from offline testing directly into customer-impacting AI decisions. In shadow mode, the challenger AI is read-only. It cannot write to production systems, send customer communications, change account status, alter credit lines, close AML alerts, decline payments or direct collections actions. Every recommendation is stored as a counterfactual event with decision-time features, policy and model versions, the actual champion decision and later outcome labels.

We will use this evidence to answer five questions: does AI add decision value, does it create concentrated harm, does it disagree with human experts in high-risk cases, are outcome labels mature enough to support the claim, and can operations and governance reconstruct the decision. If any material leakage, fairness breach, unauthorized action, reviewer overload or evidence gap occurs, the rollout path stops and returns to remediation.

Portfolio Storyline

I designed shadow mode as a pre-decision architecture pattern for regulated financial AI. It combines champion/challenger comparison, counterfactual logging, delayed outcome evaluation, leakage controls, fairness scorecards, human review calibration, rollout gates and rollback triggers. The value is that product, architecture and governance teams can make a disciplined go / no-go decision before the AI is allowed to affect customers.

---

Interview Drills

Drill 1: Credit Line Management

Question: “How would you validate an AI credit line recommender before launch?”

Strong answer:

I would run it in read-only shadow mode against real line-management events. The current credit policy remains champion, and the AI challenger records increase/decrease/hold/review recommendations with reason codes and decision-time feature snapshots. I would compare against actual decisions, wait for delinquency, utilization, complaint and attrition labels to mature, and run fair lending segment analysis. It can only move to assisted mode if reason-code consistency, high-severity disagreement, fairness and evidence gates pass.

Drill 2: AML Alert Triage

Question: “What makes shadow mode hard for AML?”

Strong answer:

AML labels are delayed and noisy. Analyst disposition, QA review and SAR decision are related but not identical labels. I would separate immediate analyst agreement from mature outcomes, run blind SME comparison on sampled alerts, monitor typology-specific false negatives, and preserve narrative evidence. A model that reduces queue volume but under-escalates high-risk typologies should fail the gate even if aggregate agreement looks good.

Drill 3: KYC Onboarding

Question: “How do you avoid shadow evaluation becoming biased by the current process?”

Strong answer:

I would define the population before sampling, include cases the current process sends to different paths, and keep challenger output separate from champion decision capture. For human comparison, reviewers must be blind to AI output. I would also control policy version and document availability at decision time, because onboarding results can look better if future fraud flags or corrected documents leak into evaluation.

Drill 4: Payment Fraud Intervention

Question: “Why not just A/B test the fraud model?”

Strong answer:

For payment fraud, customer harm from false declines and missed fraud can be immediate. Before any customer-impacting experiment, I want counterfactual evidence using real transaction context. The AI proposes allow, step-up, hold or decline, but the champion path still decides. We then join confirmed fraud, chargebacks, customer confirmations and complaints. Only if false positive and false negative tradeoffs are acceptable by segment would I move to a limited, reversible rollout.

Drill 5: Collections Contact Strategy

Question: “What would block rollout even if cure rate improves?”

Strong answer:

I would block rollout if the AI increases contact pressure on vulnerable customers, violates consent or contact frequency rules, under-routes hardship cases, or concentrates complaints in a protected/proxy segment. Collections AI must be judged on conduct risk and customer treatment, not only cure rate.

Drill 6: Contact Center Agent Assist

Question: “How do you test agent assist without creating automation bias?”

Strong answer:

Start with hidden shadow mode and compare AI suggested answers against actual agent responses and QA outcomes. For sampled cases, run independent review where reviewers do not see the AI output. If we later expose suggestions to agents, that becomes assisted silent launch, and we need acceptance/override tracking, citation quality, repeat contact, complaint and QA monitoring to detect automation bias.

---

Source Anchors