AI Agentic Process Audit / Workflow Replay / Assurance Architecture 解读

面向对象: Senior AI PM / AI Architect / Internal Audit Partner / Process Owner / CBAP-level BA / AI Governance Lead / Financial Retail Operations Leader。 核心问题: 当 AI agent 不只是回答问题, 而是理解意图、制定计划、调用工具、请求人工批准、执行流程动作、处理例外并从事故中学习时, 组织如何设计一套可审阅、可重放、可质询、可持续改进的 evidence architecture? 学习目标: 建立 agentic process audit model、replayable trace、event-sourced agent workflow、plan/action/observation/approval/output schema、audit query、sampling strategy、process conformance、incident replay 和 assurance operating model 的高级心智模型。

重要说明: 本文是学习、作品集和内部架构训练材料, 不构成法律意见、监管解释、审计意见、模型验证结论、内控有效性结论、风险接受决定或生产上线批准。本文讨论的 audit、assurance、evidence、control effectiveness 和 process owner review, 均指支持内部审计、风险、合规、业务 Owner 和管理层进行独立审阅与挑战的证据架构, 不代表任何正式审计签署或监管认可。访问日期按 2026-06-30 记录。

---

Source Anchors

以下来源用于组织 AI 风险管理、AI 管理体系、架构描述、需求工程、可观测性、provenance 和金融机构 IT 审查语言。本文只将它们作为产品、架构、BA 和内部 assurance 的设计锚点, 不声称任何 schema、trace 或 replay workbench 自动满足审计、监管或模型风险批准。

一句话:

Agentic process audit is the evidence architecture discipline that turns an AI agent's intent, plan, tool actions, policy decisions, human approvals, exceptions, outputs and learning signals into replayable process evidence for review, challenge and improvement.

---

1. Executive Summary

普通 AI 系统的审阅重点通常是:

• 模型是否表现足够好。
• RAG 是否引用了正确来源。
• 输出是否合规、安全、可解释。
• 上线后是否有监控和事故响应。

Agentic workflow 的复杂度更高, 因为系统会跨越多个业务步骤:

User intent
  -> intent interpretation
  -> plan generation
  -> policy decision
  -> tool selection
  -> tool call
  -> observation
  -> human approval
  -> exception handling
  -> compensating action
  -> final output
  -> feedback
  -> incident learning

如果组织只保存最终答案或普通 API 日志, 内审、业务 Owner、风险团队和架构团队无法回答关键问题:

成熟的 agentic process audit architecture 不是让 AI 团队多填表, 而是在 workflow runtime 中原生生成证据:

Process claim
  -> replayable trace
  -> event-sourced agent workflow
  -> evidence chain
  -> audit query
  -> sampling and testing
  -> process conformance review
  -> incident replay
  -> control improvement

高级 AI PM、AI Architect 和 CBAP-level BA 的任务是把 agentic workflow 从 "看起来自动化" 升级成 "可证明地受控"。这里的 "可证明" 不意味着输出完全 deterministic, 也不意味着自动获得审计结论, 而是指组织可以在权限、隐私和保留边界内重建足够证据, 支持独立挑战、管理审阅和持续改进。

---

2. Target Audience and Role Expectations

成熟组织会把 agentic process audit 当成流程自动化产品的一部分, 不是上线前由内审临时提出的证据补丁。

---

3. Learning Objectives

完成本文后应能:

1. 区分 audit trail、observability、provenance、evidence chain 和 workflow replay。
2. 定义 process claim, 并把它拆成可验证的 workflow、decision、control 和 outcome 证据。
3. 设计 event-sourced agent workflow, 覆盖 intent、plan、action、observation、approval、policy、exception、output、feedback 和 incident learning。
4. 写出 plan/action/observation/approval/output event schema, 明确字段、版本、保留、脱敏和访问边界。
5. 解释 causality vs chronology 的差异, 避免把时间线当作因果证明。
6. 设计 workflow replay architecture, 支持 technical replay、business replay、control replay 和 incident replay。
7. 建立 audit query catalog, 支持内审、流程 Owner、风险、合规、模型风险和事故复盘。
8. 设计 risk-based sampling strategy, 测试控制设计有效性和运行有效性。
9. 区分 process conformance、justified exception、policy override 和 control failure。
10. 在 AML investigation copilot、payment dispute assistant、KYC onboarding agent、collections hardship case agent、regulatory reporting narrative drafter 和 payment operations repair queue agent 中应用。

---

4. Core Concepts

4.1 Process Claim

Process claim 是团队对 agentic workflow 的可验证主张。它不能写成空泛的 "agent improves operations", 而要写成可审阅、可测试、可采样的声明。

Process claim 的组成:

Business outcome
  + workflow scope
  + agent responsibility
  + human responsibility
  + policy boundary
  + evidence requirement
  + control and exception handling
  + outcome measurement

4.2 Replayable Trace

Replayable trace 不是简单地把原始 prompt、答案和日志全部保存。它是一个最小充分证据集合, 能在授权条件下重建关键流程事实:

• 谁发起了请求, 角色和权限是什么。
• 用户意图如何被解释, 是否存在歧义。
• Agent 生成了什么计划, 哪些步骤被执行或跳过。
• 哪些工具被调用, 输入、输出、side effect 和 idempotency 证据是什么。
• 哪些 policy decisions 允许、拦截、升级或附加 obligations。
• 哪些人工审批、覆盖、拒绝或补充信息发生。
• 哪些异常被识别, 采取了什么补偿动作。
• 最终输出如何生成、保存、发送或丢弃。
• 反馈、事故信号和改进动作如何进入 learning loop。

Replayable 不等于 bit-for-bit deterministic。对 LLM 和第三方模型而言, 完全复现输出常常不可承诺。成熟表达应是:

We can reconstruct the approved version set, inputs at the agreed evidence level, policy decisions, tool actions, approvals, outputs and observed outcomes, while documenting reproducibility limits.

4.3 Event-Sourced Agent Workflow

Event-sourced agent workflow 把 agent 行为记录为不可随意改写的业务事件序列。当前 workflow state 可以由事件投影而来, 事故复盘和过程审阅可以从事件重新构造。

---

5. Agentic Process Audit Model

5.1 Audit Layers

Agentic process audit model 可分为八层。

5.2 Process Claim to Evidence Chain

一个 process claim 必须连到 evidence chain:

Process claim
  -> requirement
  -> control objective
  -> workflow event
  -> evidence object
  -> audit query
  -> sampling approach
  -> control result
  -> management action

示例: Payment dispute assistant

---

6. Event Schema

6.1 Common Event Envelope

Agentic process events should share a stable envelope, even if each event type has domain-specific payload.

6.2 Plan / Action / Observation / Approval / Output Schema

6.3 Example Event

{
  "event_id": "evt_20260630_kyc_000184",
  "event_type": "ai.approval.decided",
  "schema_version": "1.0",
  "occurred_at": "2026-06-30T18:42:10Z",
  "recorded_at": "2026-06-30T18:42:11Z",
  "trace_id": "trc_kyc_onboarding_72f1",
  "workflow_id": "wf_kyc_case_review_9482",
  "case_id_hash": "sha256:1d4f...",
  "use_case_id": "kyc_onboarding_agent",
  "risk_tier": "high",
  "producer": "kyc-review-workbench",
  "actor_type": "reviewer",
  "actor_id_hash": "hash:user:9a27",
  "data_class": "restricted",
  "retention_policy_id": "ret_ai_high_business_record_7y",
  "redaction_profile": "pii-minimized-v3",
  "prev_event_ids": ["evt_20260630_kyc_000181", "evt_20260630_kyc_000183"],
  "evidence_refs": [
    "evidence://visible-set/vis_kyc_5521",
    "evidence://policy-decision/pol_kyc_8910",
    "evidence://tool-input/tool_kyc_followup_2322"
  ],
  "integrity_hash": "sha256:ab77...",
  "data": {
    "approval_id": "appr_kyc_7721",
    "request_reason": "customer_followup_message",
    "approver_role": "KYC_Senior_Reviewer",
    "decision": "approved_with_edit",
    "reason_code": "missing_ubo_evidence_clearer_language",
    "visible_evidence_hash": "sha256:392e...",
    "approval_scope": "draft_customer_followup_only",
    "expiry": "2026-07-01T18:42:10Z"
  }
}

---

7. Causality vs Chronology

Workflow replay must not confuse chronological order with causal explanation.

Agentic workflow needs both:

• Chronological timeline for human-readable incident review.
• Causal graph for control testing, evidence lineage and independent challenge.

Example:

Approval A happened before Tool Call B.
That is chronology.

Approval A authorized input hash H1.
Tool Call B executed input hash H2.
H1 != H2.
That is causal control failure.

成熟 replay architecture 应记录 prev_event_ids、causal_refs、policy_decision_id、approval_scope、input_hash 和 side_effect_id, 让审阅者能判断 "按时间发生过" 与 "被正确授权" 的差异。

---

8. Workflow Replay Architecture

8.1 Logical Architecture

Business workflow UI / queue / API
        |
AI agent orchestrator
  intent classifier | planner | policy engine | tool gateway | HITL workflow
        |
OpenTelemetry traces + agentic process events
        |
Evidence collection layer
  schema validation | redaction | hashing | retention tagging | integrity checks
        |
Event store + trace store + evidence lake
        |
Provenance graph
  entities | activities | agents | causal edges | version set
        |
Replay workbench
  timeline view | causal graph | evidence chain | policy decision view | approval view
        |
Audit query and assurance reporting
  process conformance | sampling | incident replay | control testing | process owner review

8.2 Core Components

8.3 Replay Modes

8.4 Reproducibility Limits

For agentic AI, replay architecture must explicitly document reproducibility limits:

---

9. Audit Trail vs Observability

Audit trail and observability overlap but are not the same.

Mature architecture connects both:

Observability shows that something happened and how the system behaved.
Audit trail shows whether the behavior was authorized, controlled, evidenced and reviewable.

For example, an OTel span may show tool.invoke completed in 250 ms. Audit evidence must additionally show:

• Which tool contract and schema version.
• Which user or agent authority was used.
• Which policy decision allowed it.
• Whether approval was required and completed.
• What side effect occurred.
• Whether the action matched approval scope.
• How to reverse or compensate if needed.

---

10. Evidence Chain

10.1 Evidence Chain Model

Agentic process evidence should be organized as a chain:

Business outcome evidence
  <- process conformance evidence
  <- workflow replay evidence
  <- event evidence
  <- source system evidence
  <- version and control evidence

10.2 Business Outcome Evidence

Business outcome evidence prevents audit architecture from becoming purely technical. For financial retail examples:

---

11. Exception and Override Handling

Exceptions are expected in real processes. The assurance question is whether they are explicit, justified, owned, time-bound and learned from.

11.1 Exception Types

11.2 Good Override Record

Weak override:

Supervisor approved exception.

Strong override:

Supervisor approved hardship option deviation for case class H2 because customer submitted verified disaster impact documentation not covered by standard script. AI recommendation was restricted to draft language. Final treatment selected by hardship specialist. Case added to monthly vulnerable-customer QA sample and policy team review.

---

12. Segregation of Duties

Agentic AI can blur roles because the same platform may propose, execute, document and monitor a process. Segregation of duties must be explicit.

SoD event evidence should include:

• requester_role
• approver_role
• relationship_check_result
• same_user_blocked
• delegated_authority_source
• approval_scope
• independent_challenge_required
• break_glass_reason

Independent challenge does not mean every event is manually reviewed. It means the evidence architecture allows authorized second-line, third-line, QA or process-owner reviewers to test whether controls operated as designed.

---

13. Sampling and Testing Approach

13.1 Population Definition

Sampling begins with a population definition. Without it, teams cherry-pick good examples.

13.2 Testing Types

13.3 Sampling Strategy

Risk-based sampling should combine:

• 100% automated queries for missing mandatory events.
• Attribute sampling for required control evidence.
• Judgmental samples for high-risk, high-value, high-complexity or complaint-linked cases.
• Stratified samples by channel, segment, language, geography, model route and reviewer.
• Incident-driven samples from near misses, overrides and QA failures.
• Negative samples where agent refused, escalated or blocked action.

Sampling should record:

---

14. Process Conformance

Process conformance asks whether actual workflow execution matches the approved process model.

Approved model:
intent -> plan -> policy -> tool dry-run -> approval -> tool execution -> output -> feedback

Actual trace:
intent -> plan -> tool execution -> output

Conformance result:
non-conformant because policy and approval events are missing before customer-impacting tool execution.

14.1 Conformance vs Justified Exception

Mature review does not punish every deviation. It classifies deviations and updates process, controls, training or tooling based on evidence.

14.2 Audit Queries for Conformance

---

15. Incident Replay

Incident replay reconstructs what happened, why it happened, who or what allowed it, what impact occurred and what changed afterward.

15.1 Incident Replay Packet

15.2 Example: Payment Operations Repair Queue Agent

Incident signal:

Duplicate repair actions increased after a workflow release.

Replay findings:

Assurance conclusion should be framed carefully:

The replay packet supports process owner, risk and internal audit review of the incident evidence. It does not by itself constitute audit sign-off or regulatory closure.

---

16. Financial Retail Examples

16.1 AML Investigation Copilot

16.2 Payment Dispute Assistant

16.3 KYC Onboarding Agent

16.4 Collections Hardship Case Agent

16.5 Regulatory Reporting Narrative Drafter

16.6 Payment Operations Repair Queue Agent

---

17. Operating Model

17.1 RACI

R = Responsible, A = Accountable, C = Consulted, I = Informed.

17.2 Cadence

---

18. Anti-Patterns

---

19. PM / BA / Architect Implications

19.1 For Senior AI PM

• Define process claims before building automation.
• Treat workflow replay as a product capability, not just internal logging.
• Pair value metrics with control and customer outcome evidence.
• Require release gates for evidence coverage, replay readiness and exception handling.
• Write scale recommendations that show uncertainty, residual risk owner and monitoring triggers.

19.2 For CBAP-level BA

• Model agentic workflow as state, event, decision and evidence objects.
• Translate stakeholder needs into process claims, acceptance criteria and audit queries.
• Make exception paths first-class requirements.
• Distinguish conformance, justified exception, process drift and control failure.
• Ensure every material process claim has evidence, owner and test approach.

19.3 For AI Architect

• Design trace context, event schema and provenance graph early.
• Use event-sourced workflow where replay and conformance matter.
• Bind approvals to exact action inputs and side effects.
• Record version sets for model, prompt, RAG, policy, tool schema and workflow.
• Build redaction, retention and access control into the evidence architecture.

19.4 For Internal Audit Partner

• Challenge whether process claims are testable.
• Review whether evidence is complete, reliable and independently queryable.
• Help shape sampling strategy and audit query catalog without owning management controls.
• Distinguish evidence sufficiency for review from formal audit conclusions.

---

20. Interview Answers

Q1: How do you make an agentic AI workflow auditable without slowing it down?

30 秒版本:

I would design auditability into the workflow runtime. Every agentic workflow should emit structured events for intent, plan, observations, policy decisions, tool calls, human approvals, exceptions, outputs and feedback. Those events connect to traces, source records and a provenance graph, so process owners and audit partners can replay cases, test controls and sample exceptions without asking teams to manually reconstruct evidence.

2 分钟版本:

I start from process claims. For example, a payment dispute assistant may draft evidence packets but cannot submit a chargeback without maker-checker approval. That claim becomes requirements, controls, events and audit queries. The architecture emits a replayable trace: intent classification, plan, RAG evidence, policy decision, tool dry-run, approval visible evidence, tool execution, output and feedback.

To avoid slowing teams, evidence is generated as work happens. The tool gateway enforces approval binding, idempotency and side-effect capture. The HITL workflow records visible evidence and reason codes. OpenTelemetry traces provide operational context, while event store and provenance graph provide process and causal evidence. Internal audit or process owners can then run queries such as "show all customer-impacting tool actions without valid approval" and sample cases by risk. The result is not automatic audit sign-off, but a stronger evidence architecture for review and assurance.

Q2: What is the difference between observability and audit trail for agentic AI?

30 秒版本:

Observability explains system behavior: latency, errors, traces, costs and dependencies. Audit trail proves authority and evidence: who requested, what plan was approved, which policy allowed it, what tool action happened, what output was delivered and whether exceptions were justified. Agentic AI needs both connected by trace ids and evidence references.

Q3: Why is causality more important than chronology in workflow replay?

30 秒版本:

Chronology tells us that one event happened before another. Causality tells us whether the later action was actually authorized, supported and caused by the earlier evidence. An approval before a tool call is not enough; the approval must apply to the same input hash, scope, policy decision and side effect.

Q4: How would you test control effectiveness for a KYC onboarding agent?

30 秒版本:

I would define the population, such as all KYC agent cases with customer follow-up drafts in the month. I would run automated exception queries for missing reviewer approval, unsupported document claims and automated rejection. Then I would sample by risk tier, document type, geography and reviewer to verify source evidence, policy decisions, human review, final output and customer outcome. Exceptions would be classified as justified exception, documentation defect, process drift or control failure.

Q5: What evidence is needed for incident replay?

30 秒版本:

An incident replay packet needs scope, affected cases, version set, event timeline, causal graph, policy decisions, approvals, tool side effects, outputs, business impact, evidence gaps, control analysis, remediation and learning actions. It should also state reproducibility limits and privacy constraints.

Q6: How do you handle reproducibility limits with LLM agents?

30 秒版本:

I do not promise exact deterministic reproduction unless the stack supports it. I preserve the version set, prompt/config hash, model route, RAG index, retrieved chunks, policy version, tool inputs, approvals, output hash and business state. Replay focuses on reconstructing evidence and control behavior, while documenting model nondeterminism, vendor version limits and redaction boundaries.

---

21. Portfolio Exercise

Build an "Agentic Process Audit and Workflow Replay Pack" for one financial retail use case. Recommended use cases:

Required Artifacts

1. Process claim with workflow scope, agent boundary, human boundary, policy boundary and outcome evidence.
2. Event dictionary covering intent, plan, observation, policy, action, approval, exception, output, feedback and incident.
3. Plan/action/observation/approval/output schema with required fields and retention class.
4. Replay architecture diagram showing orchestrator, policy engine, tool gateway, HITL, event store, trace store, evidence lake, provenance graph and replay workbench.
5. Causal graph for one representative case.
6. Audit query catalog with at least 10 queries.
7. Control-to-evidence matrix with at least 12 controls.
8. Sampling strategy with population definition, method, sample size rationale and pass criteria.
9. Exception and override register with owner, expiry and compensating control.
10. Incident replay packet for one failure or near miss.
11. Process conformance report distinguishing conformant cases, justified exceptions, process drift and control failures.
12. Executive assurance narrative for process owner review.

Scoring Rubric

---

22. Final Mental Model

Agentic workflow assurance should make five truths visible:

The final answer is not the process.
The timeline is not the cause.
The approval is not valid unless bound to exact evidence and action.
The exception is not acceptable unless owned, justified, expiring and monitored.
The replay is not audit sign-off, but it is the evidence architecture that makes serious review possible.

The senior-level move is to design AI agents as replayable process participants, not opaque automations.