AI Document Intelligence / Unstructured Data / Evidence Quality Architecture Playbook

定位: 面向 CBAP+ Senior BA、高级 AI PM、Product Architect、Enterprise Architect、Operations Architect、Records / Information Governance、Fraud Risk、Model Risk、KYC/KYB、Claims、Disputes、Loan and Insurance Servicing 负责人, 把 document intelligence 从“自动读文档”落地为 evidence-grade, workflow-ready, records-aware, auditable operating system。 适用范围: bank statements、paystubs、claims packages、payment disputes、KYC/KYB documents、insurance and loan servicing documents、complaints、operational correspondence、mailroom automation、agent assist and workflow automation。 核心产出: executive framing、taxonomy、decision gates、target architecture、required artifacts、RACI/operating model、implementation roadmap、evidence pack、release checklists、metrics、anti-patterns、tabletop scenarios and portfolio deliverables。

核心判断:

Financial document AI succeeds when the institution can rely on extracted evidence, not when the model can read more PDFs.

---

0. Disclaimer

本文是学习、作品集、架构训练和内部治理讨论材料, 不构成法律意见、合规结论、记录保留结论、e-discovery 建议、KYC/KYB 充分性判断、贷款或保险承保结论、消费者争议处置意见、欺诈处置指令、模型验证报告或供应商推荐。

正式项目必须由 Legal、Compliance、Privacy、Records Management、Information Governance、Model Risk、Fraud Risk、Financial Crime、Operations、Product、Architecture、Information Security、Data Governance、Vendor Management、Internal Audit 和相关业务 owner 共同判断。记录、证据、法律保留、客户通知、KYC/KYB、信贷、保险、投诉、争议、索赔、跨境数据和 e-discovery 的具体适用性, 取决于 product、record type、jurisdiction、retention schedule、legal hold status、customer segment、channel、policy、contract 和 Legal / Compliance / Records interpretation。

边界原则:

• OCR result 是 extracted text, 不是 verified fact。
• LLM summary 是 productivity aid, 不是原始证据替代品。
• Confidence score 是 routing/control input, 不是业务结论。
• Fraud/tamper model output 是 risk signal, 不是欺诈结论。
• Records retention、legal hold、e-discovery、KYC/KYB、lending、insurance 和 dispute obligations 的具体适用性必须由对应治理职能解释。

---

1. Executive Framing

高管常见目标:

Reduce manual document review.
Accelerate onboarding, claims, disputes and servicing.
Use AI to summarize and extract unstructured documents.

真正的项目目标应改写为:

Create an evidence-grade document intelligence capability
that automates low-risk extraction,
routes ambiguity to skilled review,
prevents unsupported decisions,
preserves records and legal hold controls,
and can replay every material document-driven action.

如果没有 evidence architecture, document AI 会引入隐藏风险:

• bank statement 被 OCR 错读, income calculation 批量错误。
• paystub 伪造模板被高置信度抽取, 进入 underwriting workflow。
• claim photo 被复用, fraud signal 未传给 adjuster。
• dispute evidence summary 省略关键商户证明, 导致不当处置。
• KYB 文件显示 authorized representative 不清楚, 但 AI prefill 直接推进开户。
• legal hold 已触发, 但 OCR JSON、summary、review notes 或 vendor copies 被清理。
• 客户投诉时, 团队找不到“模型看了什么、谁复核了什么、为什么采取该动作”。

Executive one-liner:

Document AI is a controlled evidence supply chain for operations, risk and records.

1.1 Steering Committee Questions

1. 哪些 document classes 和 fields 可以自动抽取, 哪些只能辅助人工?
2. 哪些字段会影响客户权利、资金、身份、保险、信贷、争议或合规?
3. 如何证明字段来自哪份文档、哪一页、哪个区域、哪个模型版本?
4. 置信度阈值如何校准, reviewer overturn 如何反馈?
5. records retention、legal hold、vendor copies 和 derived AI artifacts 如何治理?
6. 出现投诉、审计、监管问询或诉讼保全时, case 能否完整 replay?

---

2. Source Anchors

Source-to-control pattern:

source anchor -> control objective -> product decision
  -> workflow requirement -> evidence artifact -> owner -> metric

---

3. Taxonomy

3.1 Document Use Case Classes

3.2 Evidence Classes

3.3 Criticality Levels

---

4. Target Operating Architecture

channel intake
  -> document provenance and storage
  -> normalization / rendering / safety scan
  -> classification and package splitting
  -> OCR / layout / table extraction
  -> multimodal field and entity extraction
  -> normalization and validation
  -> confidence calibration
  -> fraud and tamper checks
  -> evidence policy gates
  -> human review workbench
  -> workflow orchestration
  -> records retention and legal hold integration
  -> evidence ledger, complaint linkage, monitoring and governance

Architecture capabilities:

---

5. Decision Gates

Gate 0: Use Case Boundary

Gate 1: Evidence Schema

Gate 2: Confidence and Review Policy

Gate 3: Fraud, Security and Access

Gate 4: Records, Legal Hold and Evidence Replay

Gate 5: Production Readiness

---

6. Required Artifacts

6.1 Evidence Field Dictionary Pattern

6.2 Workflow Contract Pattern

Workflow:
Trigger:
Accepted document classes:
Required fields:
Optional fields:
Excluded data:
Confidence thresholds:
Validation rules:
Fraud/tamper routes:
Human review triggers:
Records metadata:
Legal hold behavior:
Case update payload:
Customer communication constraints:
Monitoring metrics:
Owner:

---

7. RACI / Operating Model

Governance cadence:

---

8. Implementation Roadmap

Days 1-30: Baseline and Scope

Days 31-60: Controlled Build and Pilot

Days 61-90: Scale Decision and Assurance

---

9. Evidence Pack

Minimum evidence fields:

Evidence rules:

• Store raw artifacts and derived artifacts with clear lineage, not as one mutable blob。
• Preserve source anchors for every material extracted field。
• Treat missing source anchor for high-impact field as a control defect。
• Separate reviewer-facing summaries from official records and decision evidence unless Records/Legal approves their role。
• Make legal hold state visible to deletion, training-data selection, vendor purge and export jobs。
• Capture final customer/counterparty communication, because disputes often turn on what was said and when。

---

10. Workflow Playbooks

10.1 Bank Statement / Paystub Income Evidence

10.2 Claims Package

10.3 Payment Dispute Evidence

10.4 KYC/KYB Documents

10.5 Complaints and Operational Correspondence

---

11. Checklists

11.1 Release Checklist

11.2 Extraction QA Checklist

11.3 Human Review Checklist

11.4 Records and Legal Hold Checklist

11.5 Workflow Integration Checklist

---

12. Metrics and KRIs

Balanced scorecard:

Productivity: fewer manual minutes per case.
Evidence quality: critical fields accurate, anchored and validated.
Risk: fraud, records, privacy and legal hold controls operate.
Customer outcome: fewer unnecessary document requests and complaint defects.
Governance: every material document-driven action is replayable.

---

13. Anti-Patterns

---

14. Tabletop Scenarios

Scenario 1: High-Confidence Paystub Error

The model extracts gross pay correctly but misreads pay period and YTD amount.
Confidence is high. The income package is about to move forward.

Expected decisions: field criticality threshold, arithmetic validation, reviewer route, downstream case hold, model defect capture。

Scenario 2: Legal Hold Miss on Derived Artifacts

A legal hold is applied to a servicing case. Raw PDFs are preserved,
but OCR JSON, AI summaries and reviewer notes are scheduled for deletion.

Expected decisions: hold propagation graph, disposition stop, vendor copy check, Records/Legal escalation, CAPA。

Scenario 3: Dispute Summary Omits Merchant Evidence

The AI summary says the customer provided strong evidence,
but the merchant delivery proof contradicts the customer statement.

Expected decisions: source-linked summary defect, dispute specialist review, reason-code evidence checklist, eval update。

Scenario 4: Fake Bank Statement Template

A bank statement matches a known visual template but metadata and transaction pattern are inconsistent.
The extraction fields look clean.

Expected decisions: fraud/tamper signal route, no customer-facing rule leakage, manual review, duplicate/template detection improvement。

Scenario 5: KYB Authority Ambiguity

A business registration document and board resolution are uploaded.
The model extracts an officer name but cannot establish authority to open the product.

Expected decisions: authority field specialist review, KYC/KYB interpretation boundary, request-for-info wording, evidence pack。

Scenario 6: Complaint Replay Failure

A customer complains that an AI rejected their claim documents.
The team can find the raw PDF but not the model run, reviewer override or final message.

Expected decisions: evidence pack gap, incident classification, complaint remediation, logging and workflow contract update。

---

15. Portfolio Deliverables

Portfolio storyline:

I designed an AI document intelligence architecture for financial retail operations.
It converts unstructured documents into source-linked evidence,
uses calibrated confidence and validation to route work,
integrates human review, fraud/tamper checks, records retention and legal hold,
and preserves a replayable evidence trail from intake to final workflow action.

---

16. Interview Answers

Q1: 如何向高管解释 document intelligence 的边界?

30 秒:

Document intelligence 不是 OCR 自动化, 而是 evidence supply chain。它把文档采集、provenance、layout、field extraction、confidence、validation、human review、fraud checks、workflow action、records retention 和 legal hold 连成一套可审计系统。能自动化的只是低风险、高质量、可验证字段; 高影响或冲突证据必须复核。

Q2: 什么样的字段可以 straight-through processing?

30 秒:

要满足五个条件: 字段风险低或中等、source anchor 完整、confidence 已校准、业务验证通过、没有 fraud/tamper/legal/authority trigger。高影响字段如 income、claim amount、beneficial ownership、POA、dispute reason 通常不能只靠模型分数推进。

Q3: 如何设计 human-in-the-loop 才不是形式主义?

30 秒:

reviewer 必须看到原文和字段定位、confidence、validation failures、conflicts、history and risk signals, 并用 structured reason code 记录接受、纠正或升级。还要做 QA sampling、overturn monitoring、dual control and feedback governance。否则 human review 很容易变成 rubber-stamp。

Q4: records retention 和 legal hold 为什么会影响 AI 文档项目?

30 秒:

因为系统不只保存原 PDF, 还会生成 OCR text、layout JSON、extracted fields、AI summaries、review notes、exports and final decisions。哪些是 records、保留多久、legal hold 是否覆盖, 取决于 product、record type、jurisdiction、retention schedule、hold status 和 Legal/Compliance/Records interpretation。架构必须让这些 artifact 可分类、可保留、可冻结、可检索。

Q5: 如何衡量 document AI 是否真的可用?

30 秒:

看 field-level accuracy、high-confidence wrong rate、source-anchor completeness、review overturn、QA defects、downstream rework、fraud/tamper yield、records metadata completeness、legal hold propagation、complaint replay success and customer impact。只看 OCR accuracy 或 automation rate 会误导决策。

---

17. Practical Templates

17.1 Use Case Boundary Card

Use case:
Workflow:
Product:
Customer / business segment:
Jurisdiction / policy scope:
Channels:
Document classes:
Unsupported documents:
AI role:
Allowed workflow actions:
Prohibited workflow actions:
High-impact fields:
Fraud risks:
Records owner:
Legal hold considerations:
Human review triggers:
Evidence replay requirement:
Product owner:
Risk owner:

17.2 Document Class Card

Document class:
Subtypes:
Typical source channels:
Expected pages / sections:
Required fields:
Optional fields:
High-impact fields:
Known fraud/tamper patterns:
Quality gates:
Classification ambiguity handling:
Retention metadata:
Legal hold propagation:
Workflow route:
Specialist review triggers:

17.3 Evidence Acceptance Rule

Rule ID:
Workflow:
Document class:
Field:
Criticality:
Required source anchor:
Minimum confidence:
Required validations:
Cross-document checks:
Fraud/tamper exclusions:
Human review required when:
Auto-populate allowed:
Policy acceptance state:
Customer-facing reason:
Evidence fields:
Owner:
Review cadence:

17.4 Reviewer Decision Record

Review task ID:
Case ID:
Document ID:
Field / evidence item:
Model value:
Model confidence:
Source anchor:
Validation result:
Fraud/tamper signal:
Reviewer decision:
Corrected value:
Reason code:
Rationale:
Second approval:
Workflow action:
Customer communication:
Timestamp:
QA sample flag:

17.5 Evidence Replay Script

Case:
Question to answer:
Raw documents:
Derived artifacts:
Extraction model versions:
Field source anchors:
Validation results:
Fraud/tamper results:
Human reviews:
Policy decisions:
Workflow actions:
Customer/counterparty messages:
Records metadata:
Legal hold status:
Complaint / audit / CAPA links:
Replay conclusion:

---

18. Final Operating Principle

这套 playbook 的成熟度可以用一个问题检验:

When a bank statement, paystub, claim package, dispute file, KYB document
or servicing letter changes a customer or business outcome,
can the institution prove exactly what document was used,
what fields were extracted,
where they came from,
how confidence and validation were handled,
who reviewed exceptions,
how fraud and records controls applied,
and why the workflow action was appropriate at that time?

如果答案不清楚, 不是缺一个更强的 OCR vendor。问题是 document intelligence、operations workflow、records governance、fraud controls、model risk and evidence quality 还没有成为同一套 operating model。