返回 Papers
AI 扩展计划 / Playbooks

AI Operating Model / RACI Runbook

这些来源提供运行模型语言。正式项目仍需结合机构政策、司法辖区、业务域和技术架构确认。

638AI_OPERATING_MODEL_RACI_RUNBOOK.md

AI Operating Model / RACI / Runbook

核心问题: AI 系统上线后仍在持续变化。模型、提示词、知识库、供应商、权限、数据、流程和用户行为都会改变系统输出。如果没有明确的运行模型, AI 会从受控能力退化成无人真正负责的生产风险。

系统边界: 本手册聚焦金融零售企业的生产 AI 运行机制, 覆盖 RAG、copilot、agent、工作流自动化、模型网关、知识库、评测、事故处理、供应商变更、采用效果和治理节奏。它不替代法律、合规、审计、模型验证或生产批准结论。

运行判断: AI launch is not the finish line. Production AI needs operating ownership, measurable controls, change discipline, incident routes, evidence capture and business-value review.


Source Anchors

这些来源提供运行模型语言。正式项目仍需结合机构政策、司法辖区、业务域和技术架构确认。

AnchorLink本文使用方式
NIST AI RMFhttps://www.nist.gov/itl/ai-risk-management-framework用 Govern / Map / Measure / Manage 组织 AI 风险、监控、处置和持续改进。
ISO/IEC 42001https://www.iso.org/standard/81230.html用 AI management system 语言定义责任、运行控制、绩效评估和改进闭环。
EU AI Acthttps://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng用风险分层、人类监督、文档和上市后监控视角约束高影响场景。
OWASP LLM Top 10https://owasp.org/www-project-top-10-for-large-language-model-applications/用 prompt injection、数据泄露、供应链和工具滥用等风险设计运行手册。
TOGAFhttps://www.opengroup.org/togaf用 architecture governance、review board 和变更治理把运行发现反馈到标准和架构决策。
SRE conceptshttps://sre.google/借鉴 SLO、incident、postmortem、error budget、on-call 和降级思路。

1. 核心问题: 生产 AI 为什么需要运行模型

传统系统上线后主要面对容量、缺陷、权限和发布问题。生产 AI 还会面对行为漂移和上下文漂移:

变化源典型表现运行风险
模型版本供应商调整模型、路由或安全策略旧评测不再代表新行为, 拒答、幻觉和成本变化
提示词团队为改善体验直接改系统提示词行为边界、输出格式、拒答策略和合规措辞漂移
知识库政策、产品、费用、流程、FAQ 更新RAG 引用过期, 客服或员工得到旧规则
工具权限agent 从只读扩展到写入或外发由回答错误升级为行动错误
用户行为员工复制输出、绕开复核、批量使用人工监督失效, 形成事实自动化
供应商SLA、区域、日志、训练配置或模型公告变化数据边界、可用性和审计证据失控
监管和政策新义务、审计发现、投诉趋势上线时合规但运行中失配

运行模型要回答的不是“谁参加会议”, 而是以下问题:

Who owns the AI capability and residual operating risk?
Who owns data, knowledge and allowed sources?
Who approves model, prompt, index, tool and workflow changes?
Who monitors quality, safety, cost, adoption and control effectiveness?
Who handles incidents and customer-impact decisions?
Who can pause, roll back, degrade or retire the capability?
Who proves business value and decides whether to scale?

2. 运行模型总架构

成熟的 AI 运行模型应被设计成一个闭环系统:

Inventory and ownership
-> risk tier and operating controls
-> data / knowledge / tool readiness
-> release and change gates
-> production monitoring
-> incident and issue management
-> adoption and value review
-> architecture and policy feedback
层级核心问题关键产物
Ownership layer谁对业务结果、流程、数据、平台、风险和证据负责owner map、RACI、delegated authority
Control layer哪些控制在运行时真实生效control matrix、policy decisions、tool permission、human review
Evidence layer如何证明系统按批准方式运行trace schema、dashboard、eval report、issue log、attestation
Change layer模型、提示词、索引、工具、供应商和流程如何变更change request、regression eval、rollback plan、materiality rule
Incident layer发生幻觉、越权、泄露、停机或质量退化时如何处置severity matrix、runbooks、postmortem、corrective actions
Value layerAI 是否真正改善流程, 是否值得继续投入adoption metrics、baseline comparison、cost per outcome、scale decision
Governance layer运行发现如何进入政策、架构标准和投资取舍quarterly review、architecture decision record、risk acceptance

运行模型的价值在于减少三类失败: 第一, 控制只存在于文档中; 第二, 问题发生后找不到决策权; 第三, 用例持续扩张但证据、人员和平台能力没有同步扩张。


3. 角色与责任边界

角色不是职称清单, 而是运行责任的边界。一个人可以承担多个角色, 但每项责任必须有明确 owner 和替代人。

RoleOperating responsibilityEvidence owned
AI capability owner对业务目标、范围、价值、路线图、上线建议和停止建议负责use case record、roadmap decision、value review
Business process owner对受影响工作流、人工职责、SOP、容量和操作风险负责process map、SOP、review queue metrics、fallback plan
Process analysis owner对流程状态、需求证据、例外路径、验收条件和 stakeholder alignment 负责workflow model、requirements trace、acceptance evidence
Solution architect对系统边界、集成、非功能属性、控制点、日志、回滚和降级设计负责architecture view、ADR、trace design、rollback record
Data owner对源数据质量、访问、血缘、留存、最小化和数据合约负责data inventory、quality SLO、lineage、access review
Knowledge owner对政策、产品内容、FAQ、SOP、知识库版本和有效日期负责corpus registry、source approval、freshness evidence
Model and platform owner对模型网关、路由、可用性、供应商版本、成本和平台控制负责model route registry、SLA report、gateway logs
EvalOps owner对 golden set、回归评测、红队样本、阈值和质量仪表盘负责eval suite、release gate result、quality dashboard
Risk and compliance owner对风险分类、政策解释、挑战意见、例外、控制标准和证据要求负责risk assessment、challenge memo、exception record
Security and privacy owner对身份权限、数据保护、DLP、供应商安全和隐私事件路径负责security review、privacy assessment、DLP evidence
Operations lead对日常采用、用户支持、缺陷分流、培训和一线反馈闭环负责adoption dashboard、support log、training record
Vendor owner对供应商合同、SLA、模型变更通知、区域、日志和退出方案负责vendor review、notice log、exit plan
Frontline champion对用户反馈、实践障碍、信任缺口和工作流适配反馈负责feedback log、training insight、defect examples

职责设计的基本原则:

  • 业务结果由第一线 owner 承担, 不应由治理团队代持。
  • 数据、知识、模型和工具分别有 owner, 因为它们的变更节奏和失败模式不同。
  • 风险、合规、安全和隐私提供标准、挑战和监控要求, 但不替代业务对残余运行风险的负责。
  • 架构责任包括证据生成能力, 不是只画系统图。
  • 运行责任必须覆盖工作日、非工作日、供应商事故、紧急禁用和回滚。

4. RACI: 用例准入与范围定义

用例准入阶段决定 AI 是不是正确工具、风险是否可控、基线是否存在、上线后是否有人运营。

ActivityCapability ownerProcess analysisProcess ownerArchitectRiskData ownerOperations
Identify business problemARRCCIC
Define baseline metricsARRICCR
Assess AI fit and no-AI optionARCCCCC
Define allowed and prohibited useARRCCCC
Initial risk tierCCCCA/RCI
Discovery decisionARACCCC

准入质量标准:

CheckFailure modePass condition
Business problem为使用 AI 而找场景有人工基线、痛点证据、价值指标和不使用 AI 的比较
Scope用例名称过宽明确用户、渠道、流程步骤、允许动作和禁止动作
Risk tier先试点后补风险判断在 discovery 前做初始风险分类, 高风险场景设更高证据预算
Baseline只看模型 demo有处理时长、错误率、返工、投诉、成本或覆盖率基线
Operating owner项目团队负责到上线为止指定上线后的流程、知识、数据、平台和事故 owner

5. RACI: 数据与知识就绪

RAG 和 copilot 的质量通常由知识库和数据治理决定。运行模型必须把 source-of-truth、权限、有效期和删除路径前置。

ActivityCapability ownerProcess analysisArchitectData ownerKnowledge ownerSecurity / privacyRisk
Source inventoryCRCA/RRCC
Data classificationICCRCA/RC
Access control designICRCCA/RC
Knowledge versioningCCCIA/RCC
Retention and loggingCCRCCA/RA/R
Data readiness sign-offCRCA/RA/RCC

控制设计要点:

  • RAG corpus 必须有 owner、purpose、allowed users、effective date、retention 和 sensitivity。
  • 权限过滤必须在 retrieval time 执行, 不能只依赖生成后过滤。
  • 知识更新要触发回归评测, 尤其是费用、信贷政策、投诉处理、KYC、反洗钱和适当性规则。
  • 日志和评测集不应成为新的未治理数据湖。
  • 数据删除、同意撤回和权限撤销要能传播到索引、缓存、记忆、日志和供应商副本。

6. RACI: 模型、提示词、索引、工具和供应商变更

生产 AI 的关键不是“能不能改”, 而是“什么变化足以改变风险”。高影响场景必须把行为变化纳入变更门禁。

ActivityCapability ownerArchitectEvalOpsKnowledge ownerPlatform ownerRiskOperations
Change requestA/RCCRCCC
Impact assessmentARRRCCC
Regression evalCCA/RCCCI
Risk reviewCCCCIA/RI
Release approvalARRCRCC
Rollback decisionARRCRCR

Material change examples:

ChangeWhy it mattersDefault handling
Model route or provider changes能力、拒答、上下文、安全策略、成本和延迟可能变化regression eval、canary、rollback plan、vendor review
System prompt changes改变角色、边界、输出格式和拒答逻辑prompt diff、targeted eval、approval、post-release sampling
RAG corpus or index refresh影响依据来源、有效日期和检索结果freshness test、citation eval、source owner sign-off
Tool permission expansion从建议进入执行或外发policy check、approval workflow、idempotency、audit trace
Workflow automation change改变人工复核、默认采纳或升级路径process simulation、capacity check、control owner review
Vendor terms or region changes数据处理、留存、训练、审计权和驻留边界变化third-party risk review、privacy/security review、exit option

7. RACI: 事故响应与问题整改

事故响应不是单独的技术流程。AI 事故往往同时涉及客户影响、合规判断、数据保护、模型行为、运营流程和供应商责任。

ActivityCapability ownerOperationsArchitectEvalOpsRiskSecurity / privacyVendor owner
Triage incidentARRRCCC
Severity classificationARCCA/RCC
ContainmentARRCCA/RR
User or customer communicationA/RRIICCI
Root cause analysisCRRRCCC
Corrective actionARRRCCC
PostmortemARRRA/RCC

Severity 参考:

SeverityDefinitionResponse
Critical已造成或极可能造成客户伤害、监管违规、未授权工具动作、重大泄露或资金损失stop route、executive escalation、customer-impact assessment、formal incident process
High高风险流程控制失效, 但可通过人工、回滚或流量控制缓解restrict scope、increase review、open high issue、root cause deadline
Medium局部质量、流程或监控缺口影响效率或中等风险输出fix with due date、targeted regression、monitor recurrence
Low文档、格式、体验或低影响缺陷track in backlog and review trend

8. 治理节奏

运行节奏要匹配风险等级和变化速度。低风险内部工具不需要重治理, 高影响场景不能只靠季度会议。

CadenceForumInputsDecisions
Daily during pilotPilot operating standupincidents、feedback、defects、queue healthquick fixes、support、scope restriction
WeeklyAI quality revieweval results、override、complaints、unsupported answersprompt/index/tool fixes、new eval cases
BiweeklyProduct and operations adoption reviewadoption dashboard、workflow metrics、training feedbackrollout adjustment、training、UX changes
MonthlyAI risk and governance reviewrisk register、incidents、model changes、exceptionsrisk acceptance、control changes、gate decisions
QuarterlyAI capability reviewvalue、cost、quality trend、vendor review、residual riskscale、refresh、merge、retire、platform investment
Event-drivenMaterial change or incident forumchange request、incident record、vendor notice、regulatory signalpause、rollback、release gate、exception

会议质量标准:

  • 每次 forum 都有明确 decision type: approve, hold, reduce scope, escalate, remediate, retire。
  • 指标只作为证据, 不是会议目的。会议要做决策。
  • 过期问题、重复问题和控制漂移必须进入升级路径。
  • 高风险例外必须有到期日、补偿控制、监控指标和退出路径。

9. 必备运行资产

ArtifactPurposeMinimum content
AI use case inventory确认生产 AI 在哪里、谁负责、影响什么use case、risk tier、owner、status、approved use、controls
Owner map防止责任模糊business、process、data、knowledge、platform、risk、security、vendor owners
RACI明确设计、变更、事故和运营决策activities、accountable owner、responsible teams、consulted/informed roles
Data and knowledge registry管理 RAG 和数据依赖sources、owner、sensitivity、freshness、retention、allowed use
Prompt/model/index registry支持版本化和回滚model route、prompt hash、index version、tool schema、release status
Eval dashboard判断质量和风险是否仍在门槛内golden set、red-team、threshold、failures、trend
Incident log追踪事故、近失和根因severity、impact、containment、root cause、corrective action
Risk register管理残余风险和例外risk statement、control、owner、review date、status
Change log管理行为变更request、materiality、approval、eval delta、rollback
Adoption dashboard判断工作方式是否安全改变active users、eligible cases、accept/edit/reject、training gaps
Vendor review log管理外部依赖SLA、model notice、region、data terms、incident、exit readiness
Quarterly capability review决定继续、扩展或退役value、risk、cost、control maturity、roadmap decision

10. Runbook: Hallucination or Unsupported Answer

Trigger

  • 输出包含无依据事实。
  • 引用错误、引用不支持结论或引用过期。
  • 用户报告答案错误。
  • 质量抽样发现虚构理由或政策。

Immediate Containment

  1. Capture request id, prompt hash, source ids, model route, output, user role and channel.
  2. Classify severity by customer impact, regulatory relevance and whether answer reached final workflow.
  3. For high severity, disable affected route, force human review or switch to fallback source.
  4. Notify capability owner, operations, EvalOps, architecture and risk owners.

Diagnosis

QuestionEvidence
Was the correct evidence available?source registry、index version、top-k retrieval trace
Was retrieval wrong or stale?citation log、effective-date metadata、reranker score
Did prompt over-instruct or under-constrain?prompt diff、system instruction review
Did model ignore evidence?answer trace、judge score、manual review
Did evaluator miss this case?eval coverage map、similar failure samples

Corrective Actions

  • Add the failure to golden set or challenge set.
  • Fix retrieval metadata, source freshness or chunking.
  • Update prompt, output schema or citation validator.
  • Add low-confidence human review for similar scenarios.
  • Update monitoring threshold and open issue with closure evidence.

Postmortem Questions

  • Which control was expected to catch this?
  • Why did the release gate or monitoring miss it?
  • Is the root cause local to one use case or a reusable platform control gap?
  • What evidence proves the corrective action works?

11. Runbook: Prompt Injection or Tool Misuse

Trigger

  • Model follows instructions embedded in retrieved content.
  • User asks model to bypass policy, reveal hidden rules or call forbidden tools.
  • Tool call attempts unauthorized lookup, write, notification or account action.
  • Retrieval content includes hostile instructions or data exfiltration pattern.

Immediate Containment

  1. Stop affected tool route if side-effect risk exists.
  2. Preserve prompt, retrieved content, tool request, policy decision and result trace.
  3. Notify security, architecture, risk, capability owner and operations.
  4. Review access policy, action policy and whether any customer-impacting action occurred.

Diagnosis

QuestionEvidence
Was retrieved content treated as data, not instruction?prompt hierarchy、content sanitization record
Did entitlement filter run before retrieval and tool call?access decision log、deny log
Did tool gateway enforce action policy?policy decision、approval token、tool schema
Was red-team coverage sufficient?prompt injection test cases、release results
Was the user interface making unsafe action too easy?UX state、confirmation flow、operator feedback

Corrective Actions

  • Add prompt injection and tool misuse tests.
  • Harden content sanitization and instruction hierarchy.
  • Add explicit action approval for side-effect tools.
  • Reduce tool scope, add rate limits or require exact identifiers.
  • Train operators on unsafe prompts and escalation route.

12. Runbook: Data Leakage

Trigger

  • Output includes unauthorized customer, account, employee or third-party data.
  • Logs contain sensitive fields beyond approved schema.
  • RAG citation exposes restricted title, path or case metadata.
  • Vendor, observability or annotation tool receives unapproved data.

Immediate Containment

  1. Stop affected route or restrict to safe mode.
  2. Preserve evidence in controlled store without expanding exposure.
  3. Identify data classes, affected subjects, systems, logs and external processors.
  4. Notify security/privacy, risk, legal/compliance, capability owner and incident command.
  5. Determine whether customer notification, regulatory notification or contractual notice may be required under institutional process.

Diagnosis

QuestionEvidence
Which control failed: input, retrieval, tool, output, log or vendor?data flow、DLP event、trace
Was data needed for the approved purpose?minimization record、allowed use matrix
Did access control run at source and retrieval time?source entitlement、retrieval filter log
Did logs preserve raw payload by design?trace schema、redaction pipeline
Did vendor terms and configuration prohibit training and retention?vendor record、gateway configuration

Corrective Actions

  • Patch DLP or redaction rules and replay representative traces.
  • Remove or quarantine affected index, cache, memory and logs.
  • Update data copy map and deletion verification.
  • Add regression tests for the leaked data class.
  • Review vendor configuration and contractual evidence.

13. Runbook: Model or Provider Outage

Trigger

  • Model API unavailable or degraded.
  • Latency, timeout or cost spike affects service level.
  • Provider announces behavior, region, logging or route change.
  • Approved model route unavailable and fallback not automatically selected.

Immediate Containment

  1. Switch to approved fallback model, retrieval-only mode or manual process.
  2. Cap traffic or disable non-essential workloads.
  3. Notify platform, operations, vendor owner and affected business owners.
  4. Communicate expected operational impact and manual workaround.

Diagnosis

QuestionEvidence
Is this provider outage, network issue, quota, rate limit or internal gateway failure?gateway metrics、provider status、network logs
Are fallbacks behaviorally approved for this use case?fallback approval、eval comparison
Did degraded mode preserve customer and regulatory controls?SOP、manual queue、control log
Did cost controls prevent runaway spend?cost dashboard、quota policy

Corrective Actions

  • Add provider health check and automatic safe degradation.
  • Validate fallback quality and refusal behavior.
  • Update capacity planning, quota alerts and vendor SLA review.
  • Add outage scenario to quarterly tabletop.

14. Runbook: Eval Regression

Trigger

  • Regression suite falls below release threshold.
  • Golden set trend degrades after model, prompt, index or tool change.
  • New topic cluster lacks coverage.
  • Judge and human expert scores diverge materially.

Immediate Containment

  1. Hold release or block scale request.
  2. Preserve eval run id, dataset version, model route, prompt hash and index version.
  3. Compare with previous approved run.
  4. Classify failures by severity and business workflow.

Diagnosis

QuestionEvidence
Is degradation caused by model, prompt, retrieval, data, judge or sample shift?eval diff、trace sample、component version
Are thresholds still aligned with risk appetite?release gate、risk tier、control matrix
Did new failures expose missing requirements?failure taxonomy、issue log
Is judge scoring calibrated?human review sample、judge drift metric

Corrective Actions

  • Add new samples for uncovered topic or failure mode.
  • Fix component and rerun targeted regression before full run.
  • Recalibrate judge or require human review for high-risk samples.
  • Update release gate and change materiality rules if needed.

15. Runbook: Knowledge Staleness

Trigger

  • Source policy, product, fee, procedure or regulatory content changes.
  • RAG cites old document after effective date.
  • Knowledge owner misses freshness review.
  • User reports answer contradicts official portal or SOP.

Immediate Containment

  1. Identify affected corpus, document ids, effective dates and use cases.
  2. Freeze or roll back index if stale content may affect customer or regulated workflow.
  3. Route affected questions to human or official source.
  4. Notify knowledge owner, capability owner, EvalOps and risk.

Corrective Actions

  • Repair source metadata and effective-date filter.
  • Rebuild index and run stale-citation regression.
  • Add source-change event trigger to change process.
  • Set freshness SLO and overdue escalation.
  • Review whether other use cases share the same corpus.

16. Runbook: User Trust Drop

Trigger

  • Adoption drops despite eligible workload.
  • Accepted suggestion rate falls while edit or reject rate rises.
  • Qualitative feedback says output is too long, too vague, risky or not trustworthy.
  • Users bypass AI or use unofficial tools.

Diagnosis

QuestionEvidence
Is the issue quality, latency, workflow fit or accountability fear?adoption dashboard、feedback tags、session replay
Are users seeing evidence and limitations clearly?UI review、citation visibility、copy
Is AI output too hard to edit or verify?edit distance、time saved、operator notes
Are managers pushing blind acceptance?training record、override trend
Does the use case need narrower scope?failure pattern、eligible case segmentation

Corrective Actions

  • Improve evidence display, citations and short-form output.
  • Narrow scope to tasks with high trust and clear value.
  • Add feedback-to-eval loop and show users fixes made.
  • Train champions with specific workflow examples.
  • Clarify accountability and safe override expectations.

17. 金融零售运行模式示例

AML Copilot

AreaOperating design
Business boundaryAssists analysts with evidence organization and narrative draft; final disposition and suspicious activity conclusion remain with authorized analyst.
OwnersFinancial crime operations owns workflow; compliance owns SAR boundary; knowledge owner owns SOP and typology library; EvalOps owns red-flag eval; architecture owns audit and retrieval/tool controls.
CadenceWeekly case-quality review, monthly typology refresh, quarterly risk review.
EvidenceSource-span linkage, analyst approval event, narrative edit diff, typology coverage, issue/action log.
Runbook focusUnsupported narrative, typology drift, source evidence gap, analyst over-reliance.

Customer Service RAG

AreaOperating design
Business boundarySupports agents with approved-source answers and drafts; fee disputes, complaints and account actions route to controlled workflows.
OwnersKnowledge manager owns articles; contact center operations owns adoption; quality team owns answer sampling; capability owner owns roadmap; EvalOps owns policy regression.
CadenceWeekly source freshness review, weekly quality sample, monthly complaint linkage review.
EvidenceSource approval, citation accuracy, stale-source alerts, agent edit/reject reasons, customer complaint linkage.
Runbook focusOutdated policy, unauthorized promise, escalation miss, privacy leakage.

Payments Exception Agent

AreaOperating design
Business boundaryRecommends repair steps and prepares draft work items; payment rail actions require deterministic tool policy and approval.
OwnersPayment operations owns process; architecture owns tool gateway; operational risk owns action approval criteria; platform owns idempotency and audit trace.
CadenceDaily exception queue health, weekly duplicate-action review, monthly payment rail dependency review.
EvidenceTool decision log, approval token, idempotency key, rollback or compensation record.
Runbook focusDuplicate action, wrong repair recommendation, rail outage, unauthorized tool attempt.

Lending Assistant

AreaOperating design
Business boundaryFinds policy, prepares document checklist and drafts explanation for internal review; does not decide approval, decline, limit or adverse action.
OwnersCredit policy owns content; underwriting owns final decisions; compliance owns fair lending review path; EvalOps owns reason-code and citation eval.
CadenceMonthly policy-source review, monthly override and complaint review, quarterly fairness challenge.
EvidencePolicy citation, reason-code mapping, underwriter approval, override reason, adverse action evidence link.
Runbook focusUnsupported reason code, protected-factor proxy issue, incorrect policy citation, stale policy.

18. Adoption and Change Management

Adoption is not login count. Adoption means users safely change how work is done while controls remain effective.

18.1 Adoption Metrics

MetricWhat it provesFailure signal
Eligible users activatedTraining and access reached intended populationCapability exists but workflow not adopted
Eligible cases touchedAI is used in the target workflow, not only by enthusiastsLow case coverage suggests workflow fit problem
Repeat usageUsers find recurring valueOne-time demo usage
Accept / edit / reject mixWhether output is useful and controlledBlind acceptance or universal rejection
Override reasonsWhy human judgment differs from AIMissing policy, wrong context, trust issue
Time savedValue against baselineEfficiency claim unsupported
Quality defectsRisk and rework impactAI increases downstream QA issues
Escalation rateBoundary and uncertainty handlingToo many escalations or missed escalations
Trust survey themesHuman factors and explainabilityUsers fear blame, audit or replacement

18.2 Trust-Building Controls

  • Start with read-only or draft support in high-risk domains.
  • Show source, version, evidence and limitations at the point of use.
  • Keep the authorized human role able to edit, reject, escalate and pause.
  • Capture feedback inside the workflow, not in disconnected surveys only.
  • Close the feedback loop by adding real failures to eval and communicating fixes.

18.3 Resistance Handling

ResistanceOperating response
AI will replace my judgmentPosition and design it as decision support; final authority stays in named workflow roles.
AI makes mistakesShow eval results, citations, human review and issue closure evidence.
It slows me downShorten outputs, reduce clicks, improve context prefill and remove unnecessary confirmations.
I do not know who is accountableClarify approval boundary, override reason and management expectations.
I do not trust the knowledgeShow source owner, version, effective date and freshness cadence.

19. Failure Modes and Control Design

Failure modeRoot cause patternControl design
Ownerless capabilityProject team disbands after launchProduction owner map, quarterly attestation, support rota
Paper controlControl described but not instrumentedRuntime policy decision log and dashboard evidence
Silent scope creepUsers apply assistant to unapproved tasksApproved-use binding, usage taxonomy, topic drift monitoring
Knowledge driftSource changes without index refreshSource-change event, freshness SLO, stale citation eval
Tool overreachAgent gains write or send actions without stronger controlsTool class tiers, approval token, side-effect audit
Human oversight theaterReviewer lacks time, authority or evidenceReview workflow, workload capacity, override rights, evidence display
Vendor surpriseProvider changes model or terms without route controlVendor notice process, pinned route, regression gate
Evidence gapLogs cannot reconstruct high-risk outputTrace schema, retention rule, evidence store, audit query
Adoption without valueUsage rises but baseline does not improveValue dashboard, stop rule, workflow redesign
Over-governanceLow-risk cases wait for high-risk committeesRisk-tiered gates and reusable platform controls

20. Operating Readiness Check

Before a production release or material scale decision, the team should be able to answer:

CheckPass condition
Capability ownershipBusiness outcome, process, data, knowledge, platform, risk, security and vendor owners are named.
Approved scopeAllowed use, prohibited use, user population, channels and automation level are documented.
Data and knowledge readinessSources are approved, access-filtered, fresh, retained and deletion-aware.
Control operationCritical controls have runtime evidence, owner, frequency, threshold and failure action.
Change disciplineModel, prompt, index, tool, workflow and vendor changes have materiality rules and rollback.
MonitoringQuality, risk, cost, adoption and control metrics are live with alert routing.
Incident readinessRunbooks exist for hallucination, injection/tool misuse, leakage, outage, regression and stale knowledge.
Adoption planTraining, feedback, champions, support and trust metrics are in place.
EvidenceA sampled high-risk output can be reconstructed from input to final human action.
Stop ruleThe accountable owner can pause, route to human, disable tools, roll back or retire the capability.

21. Connections

Existing assetUse
docs/abpa/templates/09-operating-model-raci.mdBaseline RACI structure and responsibility mapping.
docs/AI_ARCHITECTURE_REVIEW_GATE_CHECKLISTS.mdRelease gate evidence and architecture review criteria.
docs/AI_REQUIREMENTS_TO_EVAL_COOKBOOK.mdConnecting requirements, eval suites and release criteria.
docs/AI_VENDOR_BUILD_BUY_ADOPTION_PLAYBOOK.mdVendor dependency, adoption and build-buy governance.
docs/AI_GOVERNANCE_EVALOPS_RISK_90_PLAN.mdDeeper governance and EvalOps practice path.
docs/AI_ARCHITECTURE_DIAGRAM_PLAYBOOK.mdDrawing system boundary, operating model and runbook architecture.

22. Landing Check

An AI system is not production-ready until the operating model can prove:

Quality has an owner.
Data has an owner.
Knowledge has an owner.
Changes have a gate.
Incidents have a route.
Rollback has authority.
Adoption has evidence.
Business value has a baseline.
Residual risk has a named owner and review date.