AI Procurement Intake / Vendor Evaluation Sandbox / Build-Buy Decision Architecture Playbook

定位: 面向 experienced CBAP / AI BA / AI PM / Solution Architect / Enterprise Architect / financial retail transformation lead 的实战手册。
目标: 把 AI idea, vendor pitch 和 business pain 转成可评估、可门禁、可审计、可决策的 procurement intake, sandbox benchmark 和 build-buy-partner evidence pack。
边界: 本手册位于合同谈判、供应商退出策略和董事会投资叙事之前。它提供上游证据, 不替代 legal, compliance, procurement, privacy, security, model risk, finance 或 internal audit 的正式审查。

---

Purpose And When To Use

Purpose

使用本 playbook 解决五类问题:

1. 业务方或高管提出 AI idea, 需要判断是否进入 AI portfolio。
2. 供应商主动 pitch, 团队需要避免 demo-led procurement。
3. 多个方案在 build, buy, partner, hybrid 之间摇摆。
4. PoC 已经出现, 但缺少证据、风险边界、成本模型和生产门禁。
5. 需要把 AI vendor evaluation 做成可展示的高级 PM / BA / 架构作品集资产。

When To Use

What Good Looks Like

AI request
  -> intake card
  -> risk-tiered triage
  -> sandbox charter
  -> benchmark plan
  -> vendor/internal/no-AI comparison
  -> build-buy-partner decision record
  -> risk acceptance if needed
  -> production gate evidence packet
  -> downstream contract, TPRM, model validation and release process

---

Operating Model

Stage Gates

RACI

A = accountable, R = responsible, C = consulted, I = informed.

Work Products

---

Template: AI Procurement Intake

Intake Triage Rules

---

Template: Vendor Sandbox Scorecard

Score each dimension from 1 to 5, then apply the weight. Adjust weights by use case risk. For AML, KYC, credit and payments, increase governance, audit, data and control weights.

Decision bands:

Hard blockers override score:

• Critical customer harm, regulatory, privacy or security failure unresolved。
• Vendor cannot explain data route, retention or support access。
• No evidence export for high-risk use case。
• Tool actions cannot be constrained or audited。
• Cost or latency cannot be modeled for production volume。

---

Template: Benchmark Plan

Benchmark Result Table

---

Template: Build-Buy Decision Record

Decision title:
Use case:
Date:
Decision owner:
Architecture owner:
Risk owner:

Decision:
Choose build / buy / partner / hybrid / stop.

Context:
Describe business outcome, workflow, AI role, data boundary, risk tier and production constraints.

Options considered:
1. Buy vendor capability.
2. Build internal capability.
3. Partner/co-build.
4. Hybrid component architecture.
5. No-AI/process option.

Evidence reviewed:
- Intake card:
- Sandbox benchmark:
- Vendor scorecard:
- Cost and latency:
- Security/privacy review:
- Architecture fit:
- Concentration and exit constraints:
- User feedback:

Rationale:
Explain why this option best balances value, control, speed, risk, cost, architecture fit and reversibility.

Component boundary:
List what is bought, built, partnered and retained internally.

Conditions:
List blockers that must close before pilot or production.

Residual risk:
List risk accepted, owner and expiry.

Reversal triggers:
List events that require redesign, re-benchmark, exit or stop.

Component Boundary Table

---

Template: Risk Acceptance

Risk acceptance rules:

• No permanent acceptance for unresolved critical AI risks。
• No acceptance without named owner and expiry。
• No acceptance for missing evidence; missing evidence is a blocker。
• No use of "human in the loop" unless review behavior is logged and measured。

---

Template: Production Gate Evidence Packet

Gate outcomes:

---

PM / BA / Architecture Questions

PM Questions

BA Questions

Architecture Questions

---

Release Checklist

Use this checklist before moving from sandbox to pilot or production.

---

Executive Narrative

One-Page Narrative

Decision requested:
Approve a controlled pilot / reject / continue sandbox / proceed to procurement diligence for [use case].

Why now:
[Business workflow] has [measured pain]. AI is plausible because the work is [unstructured, evidence-heavy, repetitive, time-sensitive], but production use requires controls over data, model behavior, human authority and audit evidence.

Options evaluated:
We compared [vendor A], [vendor B], [internal option] and [no-AI baseline] using the same sandbox dataset, tasks, rubric, cost and latency measurements.

Evidence:
The leading option achieved [quality result], [critical failure result], [cost], [latency] and [architecture fit]. Key gaps are [gap list].

Recommendation:
Proceed with [build/buy/partner/hybrid/stop] because it best balances value, risk, speed, control, cost and reversibility.

Conditions:
Before production, close [security/privacy/eval/logging/vendor] gaps, approve risk acceptance, and pass the production gate.

Stop rule:
Stop or roll back if [critical failure, data leakage, unauthorized action, cost breach, latency breach, adoption failure] occurs.

CFO / COO / CTO / CRO Lens

---

Interview Drills

Drill 1: Contact Center Vendor Pitch

Prompt:

A vendor shows a strong GenAI contact center demo. The business wants to buy quickly. How do you respond?

Strong answer:

I would not block learning, but I would move the vendor into a controlled intake and sandbox path. First I define the target workflows, such as credit card fee questions and dispute status. Then I set the AI role as agent assist, not direct customer commitment. The sandbox uses approved policy documents, synthetic customer cases, complaint and vulnerable customer edge cases, and prompt-injection tests. I compare the vendor against current knowledge search and any internal RAG option. The scorecard includes groundedness, policy compliance, escalation, p95 latency, cost per interaction, trace export and architecture fit. If it passes, I recommend a limited pilot with human review and production gate conditions.

Drill 2: AML Workbench Build vs Buy

Prompt:

Should the bank buy an AML investigation copilot or build one?

Strong answer:

I would avoid a single answer at product level. AML is high-risk and evidence-heavy, so I would likely choose hybrid. We can buy commodity document summarization or UX components, but keep internal control over source registry, entitlement, case-system integration, eval datasets, red-flag taxonomy, model gateway, audit trace and final disposition boundary. The sandbox must prove no critical red-flag omission, grounded narrative, reviewer agreement, trace reconstruction and acceptable cost/latency. If a vendor cannot export evidence or respect source permissions, it cannot be a production candidate even if the demo is impressive.

Drill 3: Credit Decision Support Risk Challenge

Prompt:

The pilot improves underwriter productivity by 30 percent but has two unsupported adverse-action draft examples. What do you do?

Strong answer:

I would treat unsupported adverse-action language as a critical failure, not an average-quality issue. The gate outcome is no-go or limited rework, depending on whether the feature can be constrained. I would remove adverse-action drafting from scope, strengthen policy retrieval and citation checks, add fair-lending and protected-class proxy cases to the eval set, require human approval logging, and rerun regression. Productivity improvement does not override customer harm and regulatory evidence requirements.

Drill 4: Payments Fraud Real-Time Constraint

Prompt:

A fraud intervention AI is accurate but p95 latency exceeds the intervention window. Is it still acceptable?

Strong answer:

Not for real-time intervention. The benchmark must be interpreted against operational SLO, not just model quality. I would either constrain the AI to post-event case enrichment, precompute certain risk narratives, use a smaller model route, or keep deterministic real-time controls while AI supports analyst review. Architecture fit includes latency, fallback and queue impact; a high-quality answer that arrives too late fails the workflow.

Drill 5: CTO Challenge On Lock-In

Prompt:

The CTO asks how you prevent AI vendor lock-in during procurement intake.

Strong answer:

I separate commodity capability from control points. During intake and sandbox, I require component mapping for model, RAG, prompt, eval, tool, workflow and logs. I prefer a model gateway, internal eval datasets, source registry, tool gateway and telemetry export so we can compare or replace vendors. I also score exportability of prompts, logs, eval results, indexes, configs and data. I do not claim perfect portability; I make exit constraints explicit before pilot and price them into the build-buy decision.

---

Source Anchors