AI 底层逻辑 / 经典论文
AI Agent Marketplace：工具认证与治理架构
重要说明: 本文是学习、架构训练和作品集材料, 不构成法律意见、合规意见、网络安全认证意见、审计结论、采购建议、供应商认证结论或监管解释。正式项目必须由 AI Governance、Enterprise Architecture、Cyber Security、Privacy、Data Governance、Model Risk、Third-Party Risk、Legal、Compliance、I
553 行ai-foundations/papers/129-ai-agent-marketplace-tool-certification-governance-architecture.md
AI Agent Marketplace / Tool Certification Governance Architecture 解读
面向对象: Advanced AI PM / Senior BA / Product Architect / Enterprise Architect / AI Platform Owner / API Product Manager / Model Risk / Cyber / Data Governance / Third-Party Risk / Internal Audit Partner。 核心问题: 当企业内部开始复用 agents、tools、prompts、MCP servers、APIs、RAG capabilities 和 reusable workflows 时, 如何把“可发现、可复用、可认证、可授权、可审计、可退出”做成一套 marketplace governance architecture, 而不是把高风险 agentic capability 扔进普通 service catalog? 学习目标: 建立 agent marketplace taxonomy、capability card、tool/API certification、risk tiering、signed package provenance、permission scope、owner attestation、runtime policy control、evaluation evidence、deprecation/exit 和 auditability 的产品架构能力。
重要说明: 本文是学习、架构训练和作品集材料, 不构成法律意见、合规意见、网络安全认证意见、审计结论、采购建议、供应商认证结论或监管解释。正式项目必须由 AI Governance、Enterprise Architecture、Cyber Security、Privacy、Data Governance、Model Risk、Third-Party Risk、Legal、Compliance、Internal Audit、Platform Engineering、API/Product Owners 和业务负责人共同判断。适用性取决于行业、监管环境、数据类型、agent autonomy level、tool side effect、供应商依赖、身份体系、运行时控制能力和机构内部政策。
Source Anchors
| Source | Link | 用途 |
|---|---|---|
| OWASP Top 10 for LLM Applications | https://genai.owasp.org/llm-top-10/ | 用 prompt injection、sensitive information disclosure、excessive agency、insecure output handling、supply chain 等风险语言定义 agent/tool marketplace 的 threat-informed certification baseline |
| CISA Secure by Design | https://www.cisa.gov/securebydesign | 用 secure-by-design、默认安全、可证明控制和供应商责任语言设计 signed packages、owner attestation、secure defaults 和 marketplace admission gate |
| NIST AI RMF | https://www.nist.gov/itl/ai-risk-management-framework | 用 Govern / Map / Measure / Manage 组织 marketplace risk tier、evaluation evidence、monitoring、risk acceptance 和 lifecycle review |
| ISO/IEC 42001 overview | https://www.iso.org/standard/42001 | 用 AI management system、policy、roles、operation、performance evaluation、internal audit 和 continual improvement 组织 operating model |
| OpenAPI Specification | https://spec.openapis.org/oas/latest.html | 用 contract-first API specification、schema、security scheme、operation metadata 和 versioning 支持 tool/API certification |
| AsyncAPI Specification | https://www.asyncapi.com/docs/reference/specification/latest | 用 event-driven contract、message schema、channel、binding 和 consumer/producer boundary 支持 agentic event tools 与 asynchronous capabilities 的认证 |
一句话:
Agent marketplace governance is not a catalog problem; it is a permissioned product platform for discovering, certifying, invoking, monitoring and retiring agentic capabilities with evidence.
1. Thesis
企业内部 AI agent ecosystem 会很快从“几个团队试用 Copilot”演化为一组可复用能力:
- 业务 agents: complaint response agent、underwriting assistant、fraud investigation copilot、branch knowledge agent。
- Tools: CRM update、case creation、refund request、KYC lookup、document retrieval、pricing quote、workflow trigger。
- Prompts and policies: system prompt、policy prompt、claim guardrails、tone guide、regulated disclosure template。
- MCP servers and connectors: document systems、ticketing、data warehouse、knowledge graph、code repositories、workflow engines。
- APIs and event capabilities: OpenAPI tools、AsyncAPI event subscriptions、batch jobs、notification channels。
- Reusable evaluation and guardrail assets: prompt injection test set、policy compliance eval、PII leakage eval、tool misuse eval。
普通 service catalog 解决的是:
What services exist, who owns them, how do I request access?
Agent marketplace 要继续解决:
What can an agent do with this capability,
what data can it access,
what side effects can it create,
which risk tier has certified it,
which evidence proves it is safe enough,
which permissions are granted at runtime,
who attested ownership,
which package and specification were signed,
how will invocation be monitored,
and how will consumers exit when it is deprecated?
核心观点: AI agent marketplace 是一个 internal platform product。它的用户不是只有开发者, 还包括业务 owner、risk reviewer、tool provider、agent builder、auditor、data steward、security engineer 和 procurement/vendor owner。它的价值不是“把链接放在一个目录里”, 而是把 product discovery、risk governance、runtime authorization、evidence collection 和 lifecycle management 变成一条可重复的 platform operating model。
2. Why It Matters
Agentic capability 的风险比普通 API 更复杂, 因为调用者不是稳定的人类或固定系统流程, 而可能是会规划、组合、重试、解释、绕路和生成参数的 AI runtime。
| Marketplace failure | 典型表现 | 风险放大点 |
|---|---|---|
| Tool over-permission | 一个知识查询 agent 获得 CRM write、refund trigger 或 customer notification 权限 | excessive agency, unauthorized side effect, weak segregation of duties |
| Prompt/package provenance gap | 团队复制旧 prompt、外部 MCP server 或未签名 tool wrapper | supply chain, stale policy, hidden data flow, weak audit |
| Capability card 不完整 | catalog 只写“customer data lookup”, 不写数据字段、用途、限制、eval、owner | consumer 误用, risk reviewer 无法判断, auditor 无法追溯 |
| Certification 只看代码 | 没有测试 prompt injection、data leakage、unsafe tool chaining、fallback path | agentic risk 未被覆盖 |
| Runtime 与 catalog 脱节 | catalog 显示 Tier 2 read-only, 实际 token 可调用 Tier 1 write tool | governance fiction, evidence mismatch |
| Deprecated tool 继续被调用 | old endpoint、old model route、old prompt bundle 被 agent cache 使用 | stale control, regulatory inconsistency, operational fragility |
| Owner attestation 形式化 | owner 不确认数据用途、SLA、evidence、change notice、exit path | accountability weak, product consumers lose trust |
金融零售场景尤其敏感:
- 工具调用可能影响客户资金、信用、投诉、欺诈冻结、费用减免、KYC/AML、产品适配性和客户通知。
- Prompt injection 可以从 RAG content、ticket text、email attachment、browser page 或 third-party data 中进入 agent planning。
- Agent 可能把多个低风险 read tools 组合成高风险 inference 或 action。
- 同一个 MCP server 可能服务多个 agents, 造成 platform-level systemic dependency。
- 内部 reusable prompt 或 API 一旦被广泛复用, 它的缺陷会跨业务线扩散。
所以 marketplace 的第一性问题不是“如何让团队更快找到工具”, 而是“如何让团队只发现、申请、调用和复用已被正确描述、正确认证、正确授权、正确监控的能力”。
3. Marketplace vs Generic Service Catalog
| 维度 | Generic service catalog | Agent marketplace / certification platform |
|---|---|---|
| 主要对象 | IT services, APIs, applications | agents, tools, prompts, MCP servers, APIs, event capabilities, eval packs, policy bundles |
| 用户意图 | 查找服务、申请访问、了解 owner | 发现可组合能力、判断可用边界、申请 agent runtime permissions、查看 evidence |
| 风险模型 | availability, support, cost, ownership | tool side effect, data sensitivity, prompt injection, model behavior, autonomy, provenance, chain-of-actions |
| 描述粒度 | service name, SLA, owner, docs | capability card, allowed uses, forbidden uses, input/output schema, permission scopes, eval evidence, data classification |
| 认证焦点 | 是否上线、是否支持、是否合规 | 是否可被 agent 安全调用, 是否 contract-first, 是否 signed, 是否 least privilege, 是否 monitored |
| 运行时控制 | access request often outside catalog | catalog policy binds to runtime gateway, token scopes, approval gates and invocation ledger |
| 生命周期 | onboarding/offboarding | certification renewal, breaking-change gate, deprecation, consumer impact, exit readiness, evidence retention |
| 审计问题 | 谁拥有服务, 谁有访问 | 哪个 agent 何时基于哪个 package/spec/prompt/tool scope 对哪个数据执行了什么动作 |
设计判断:
If the catalog cannot constrain runtime invocation, it is documentation.
If certification cannot produce evidence, it is ceremony.
If marketplace discovery ignores risk tier, it accelerates unsafe reuse.
4. Capability Taxonomy
Marketplace 对象必须按 agentic behavior 建模, 而不是按技术组件建模。
| Capability type | 示例 | Certification focus |
|---|---|---|
| Agent product | complaint response agent, onboarding agent, claims triage agent | use case, autonomy, human oversight, eval, runtime policy, business owner |
| Tool / function | create_case, lookup_customer, refund_fee, send_message | input/output schema, permission scope, side effect, idempotency, approval requirement |
| Prompt / policy bundle | system prompt, regulated response prompt, refusal policy | versioning, source of policy truth, test cases, forbidden behavior, owner attestation |
| MCP server / connector | ticketing server, document repository server, data warehouse server | identity delegation, resource boundary, tool manifest, transport security, audit log |
| API product | OpenAPI-described REST API used by agents | contract, auth scheme, rate limit, data classification, error semantics |
| Event capability | AsyncAPI-described stream or command/event channel | message schema, producer/consumer boundary, replay, ordering, retention, side effects |
| RAG / knowledge asset | curated corpus, vector index, knowledge graph endpoint | source lineage, ACL, freshness, retention, citation, data use constraint |
| Evaluation pack | prompt injection suite, regulated content eval, tool misuse eval | scenario coverage, pass/fail thresholds, evidence reproducibility |
| Guardrail / policy service | PII redaction, claim checker, policy-as-code decision | control objective, false positive/negative tradeoff, bypass rule, monitoring |
Marketplace card 不应把这些对象都压扁成“服务”。每类对象有不同的风险、证据、owner 和生命周期。
5. Capability Card
Capability card 是 marketplace 的核心产品资产。它不是营销介绍, 而是 agent builders、business adopters、risk reviewers 和 auditors 共同使用的决策对象。
| Section | Required content |
|---|---|
| Identity | capability id, name, type, owner, steward, provider team, business domain, version |
| Purpose | intended use, user persona, business process, value hypothesis, non-goals |
| Risk tier | data sensitivity, side effect level, autonomy compatibility, regulated workflow flag |
| Contracts | OpenAPI / AsyncAPI / MCP tool manifest / prompt bundle / eval pack references |
| Inputs and outputs | schema, data classifications, allowed fields, prohibited fields, retention expectations |
| Permission scopes | read/write/execute scopes, customer/account/product boundaries, environment boundary |
| Invocation controls | human approval, dual control, rate limit, spend limit, transaction limit, safe-stop |
| Evaluation evidence | eval suite, red-team cases, prompt injection tests, tool misuse tests, policy compliance |
| Provenance | signed package, source repository, build attestation, dependency/SBOM or AI BOM, release notes |
| Runtime telemetry | required traces, audit events, metrics, alerting, log retention, evidence export path |
| Consumer obligations | allowed use cases, forbidden use cases, integration checklist, incident escalation |
| Lifecycle | certification date, renewal date, deprecation date, replacement path, exit support |
高级判断:
- Capability card should be machine-readable enough to drive policy and human-readable enough to support product discovery。
- Risk tier should influence search ranking、request workflow、default permissions、eval requirement and renewal cadence。
- Owner attestation should not be a checkbox; it should bind named owners to data use、side effect、support、evidence and lifecycle obligations。
6. Risk Tiering Model
Risk tiering 的目的不是给所有能力贴标签, 而是驱动差异化 controls。
| Tier | Capability pattern | Example | Required posture |
|---|---|---|---|
| Tier 0: Reference | No runtime access, documentation or static prompt sample | prompt pattern library, architecture guide | peer review, source attribution, no production invocation |
| Tier 1: Read-only low sensitivity | read public/internal non-sensitive data, no customer action | product FAQ lookup, policy summary from approved public docs | owner attestation, schema validation, basic eval, monitoring |
| Tier 2: Sensitive read | read customer, employee, financial, complaint or confidential data | KYC lookup, complaint history retrieval | data steward approval, purpose binding, ABAC, prompt injection testing, evidence logging |
| Tier 3: Controlled write/action | create/update internal records, draft customer content, trigger workflow without direct monetary movement | create case, update disposition, draft letter | human approval, policy checks, rate limits, rollback, stronger eval, periodic review |
| Tier 4: High-impact action | money movement, account restriction, credit/insurance/investment impact, legal/regulatory communication | fee refund, fraud hold, adverse action draft, customer notice send | dual control, segregation of duties, model risk review, runtime command gate, incident drill, board/reporting visibility where appropriate |
Risk tier inputs:
- Data classification and customer sensitivity。
- Tool side effect and reversibility。
- Agent autonomy compatibility: draft-only、recommendation、human-approved action、autonomous action。
- Regulated workflow involvement。
- External/customer visibility。
- Vendor dependency and data residency。
- Evidence completeness。
- Prompt injection exposure surface。
- Consumer blast radius: number of agents/products that can use it。
Important nuance:
A tool can be Tier 2 in isolation and Tier 4 in a specific chain.
Marketplace governance must evaluate both capability tier and composition tier.
7. Certification Architecture
Certification 应覆盖 design-time、release-time 和 runtime 三个层面。
provider submits capability
-> contract and manifest validation
-> risk tier classification
-> security and data review
-> eval and red-team evidence review
-> owner attestation
-> signed package and release approval
-> marketplace publication
-> runtime permission binding
-> monitoring, renewal, deprecation
认证不只是“通过一次评审”。它应该产生可持续使用的证据:
| Certification gate | Key questions | Evidence |
|---|---|---|
| Intake completeness | capability 是否有 owner、purpose、scope、contract、data map、risk tier | capability card, owner record |
| Contract-first gate | API/tool/event 是否有 versioned schema and security metadata | OpenAPI/AsyncAPI spec, MCP tool manifest, schema tests |
| Data permission gate | 是否清楚读取/写入哪些数据, 是否 purpose-bound | data classification, steward approval, field allowlist |
| Agentic threat gate | 是否测试 prompt injection、tool misuse、excessive agency、output handling | red-team results, OWASP-aligned test cases |
| Side-effect gate | tool action 是否可逆、幂等、审批、限额、回滚 | approval rules, transaction limits, rollback design |
| Provenance gate | package、prompt、connector、eval 是否可追溯且签名 | signed artifact, build attestation, release hash |
| Runtime binding gate | marketplace policy 是否能下发到 gateway/token/scope | policy id, token scope, enforcement test |
| Monitoring gate | 是否产生 invocation ledger、metrics、alerts、evidence export | trace schema, log retention, dashboard |
| Renewal gate | 是否有过期、复审、drift、breaking change、consumer impact 管理 | renewal record, change log, consumer inventory |
高级 PM/architect 的判断点:
- Certification should be proportional to risk and reusable across consumers。
- The consumer team should not need to re-litigate every control, but must still attest its own use case and composition risk。
- Certification evidence must remain valid after version changes, model route changes, prompt changes and tool permission changes。
8. Signed Packages and Provenance
Agent marketplace 的 supply chain 风险来自多个对象: prompt、tool wrapper、MCP server、API schema、guardrail config、eval set、RAG corpus manifest、policy bundle、agent workflow graph。
Provenance model:
source repo / package registry
-> build and test pipeline
-> security scan and eval run
-> signed artifact and manifest
-> marketplace release record
-> runtime deployment reference
-> invocation trace references artifact hash
| Artifact | Provenance requirement |
|---|---|
| Prompt bundle | version, owner, policy source, eval result, signed hash |
| Tool wrapper | source repo, dependency scan, input/output schema, signed release |
| MCP server | tool manifest, transport/security config, server identity, package signature |
| OpenAPI tool | spec version, operation id, security scheme, schema conformance, breaking-change record |
| AsyncAPI event tool | channel, message schema, binding, consumer group, retention, replay policy |
| RAG corpus | source list, ACL map, document hashes, refresh cadence, deprecation rules |
| Eval pack | scenario source, coverage statement, thresholds, run id, result hash |
| Agent workflow | graph version, allowed tools, fallback paths, approval gates, release attestation |
Without provenance, runtime audit becomes narrative-driven:
We think the agent used the approved tool.
We think the prompt was current.
We think the RAG source was allowed.
With provenance, audit can ask:
Which signed capability version, under which policy scope,
was invoked by which agent, with which user delegation,
against which data object, producing which side effect?
9. Permission Scopes and Runtime Controls
Marketplace governance only matters if it can constrain runtime behavior。
Reference runtime model:
agent request
-> identity and delegation context
-> marketplace entitlement check
-> policy decision point
-> tool gateway / MCP gateway / API gateway
-> schema validation and parameter policy
-> approval or dual-control gate
-> execution
-> invocation ledger and evidence export
Scope design:
| Scope dimension | Examples |
|---|---|
| Action | read, search, summarize, draft, create, update, approve, send, refund, restrict |
| Data boundary | product, customer segment, jurisdiction, account type, case type, document collection |
| Environment | sandbox, test, pilot, production, emergency break-glass |
| Identity | agent id, human delegator, service account, role, business unit |
| Time | session, case duration, campaign window, standing entitlement, expiring token |
| Volume | max calls, max records, max monetary value, max customer messages |
| Approval | none, human review, supervisor approval, dual control, risk approval |
| Purpose | complaint handling, fraud investigation, marketing review, underwriting support |
Controls that distinguish agent marketplace from API gateway alone:
- Parameter-level policy: e.g. an agent may read complaint summaries but not full attachment text。
- Chain-aware policy: a read tool followed by a write tool may require additional approval。
- Prompt-aware and source-aware checks: tool call allowed only if source evidence meets freshness/citation requirements。
- Runtime permission downgrade: high-risk signals can move agent from action mode to draft-only mode。
- Safe-stop and kill switch: disable capability or consumer entitlement without redeploying every agent。
- Invocation ledger: records intent, context, policy decision, parameters, output, side effect and approval。
10. Operating Model
Marketplace governance requires product management, not just architecture review。
| Role | Accountability |
|---|---|
| Marketplace Product Owner | roadmap, user experience, adoption, policy integration, discovery, metrics |
| Capability Owner | business purpose, data use, support, change notice, lifecycle and attestation |
| Tool/API Product Manager | contract quality, versioning, developer experience, consumer communication |
| AI Platform Engineering | registry, gateway, policy enforcement, telemetry, signing, deployment integration |
| Enterprise Architecture | target architecture, standards, composition patterns, technology rationalization |
| Cyber Security | threat model, secure-by-design controls, identity, vulnerability, runtime protection |
| Data Governance / Privacy | data classification, purpose binding, field-level permissions, retention |
| Model Risk / AI Governance | risk tiering, eval sufficiency, monitoring, use-case approval alignment |
| Third-Party Risk | vendor-owned capabilities, contract obligations, assurance evidence, exit path |
| Internal Audit | independent assurance over certification, runtime evidence and lifecycle controls |
| Consumer Team | use-case fit, integration, local controls, user training, incident escalation |
Governance forums:
- Weekly marketplace intake and certification triage。
- Biweekly platform product review: backlog, adoption friction, policy exceptions, developer experience。
- Monthly AI governance committee: Tier 3/4 capabilities, exceptions, incidents, high-risk changes。
- Quarterly architecture/risk review: systemic dependency, vendor concentration, deprecation, audit findings。
- Annual certification framework refresh: standards, threat model, eval library, control effectiveness。
Operating model test:
Can a new team discover a certified tool, understand its risk and obligations,
request the right scopes, integrate through the runtime gateway,
prove evidence, and exit safely when the tool changes?
11. Product and Architecture Decisions
| Decision | Option A | Option B | Senior judgment |
|---|---|---|---|
| Marketplace scope | catalog only approved capabilities | include experimental sandbox assets with clear labels | Include sandbox only if runtime separation and search filters prevent accidental production reuse |
| Certification model | central board reviews everything | risk-tiered delegated certification | Use risk-tiered delegation; central governance handles Tier 3/4, exceptions and standards |
| Capability cards | free-form documentation | structured card linked to policy and evidence | Structured card, with narrative fields only where needed |
| Runtime gateway | optional best practice | mandatory for certified invocation | Mandatory for production agentic invocation; otherwise certification cannot bind runtime |
| Permission model | broad app access | scoped capability entitlement | Scope by action, data, purpose, identity, environment and approval |
| Package provenance | repo links and release notes | signed artifacts with manifest and hash | Signed packages for production; hash referenced in invocation traces |
| Eval evidence | per team ad hoc evals | shared eval packs plus use-case evals | Shared baseline for common risks, local evals for business-specific harm |
| Deprecation | owner sends email | marketplace-driven consumer inventory and sunset gate | Consumer impact map and blocked new usage before shutdown |
| Exceptions | informal risk acceptance | expiring exception with owner, compensating controls and evidence | Expiring exception; renewal requires updated evidence |
12. Control Matrix
| Control objective | Control activity | Evidence |
|---|---|---|
| Prevent unsafe discovery | Marketplace search filters by risk tier, environment, certification status and consumer eligibility | search policy, eligibility logs, publication status |
| Ensure clear ownership | Every capability has accountable owner, steward, support path and attestation | owner attestation, support SLA, renewal record |
| Constrain agentic action | Runtime gateway enforces scopes, approvals, limits and purpose binding | policy decision log, token scopes, approval packet |
| Protect sensitive data | Field-level allowlists, ABAC, data classification and retention controls | data map, steward approval, access logs |
| Reduce prompt injection and tool misuse | Certification includes adversarial evals and tool-call scenario testing | eval run id, test cases, pass/fail evidence |
| Establish provenance | Production packages, prompts, specs and manifests are signed and traceable | signature, hash, release manifest |
| Manage change safely | Breaking changes require consumer impact analysis and recertification | change record, consumer inventory, recertification evidence |
| Enable auditability | Invocation ledger records agent, delegator, tool, parameters, policy decision and side effect | ledger export, trace id, evidence pack |
| Support deprecation/exit | Deprecated capabilities block new adoption, notify consumers and provide migration path | deprecation notice, replacement mapping, shutdown record |
| Govern exceptions | Exceptions expire, include compensating controls and owner approval | exception register, expiry, review outcome |
13. Metrics and KRIs
Marketplace should be measured as a platform product and a governance control surface。
| Metric | Why it matters |
|---|---|
| Certified capability adoption rate | shows whether teams reuse trusted assets instead of creating shadow tools |
| Time from submission to certification | measures governance throughput and bottlenecks |
| Capability card completeness rate | measures discovery and audit readiness |
| Runtime invocation coverage | percentage of production tool calls passing through governed gateway |
| Scope precision ratio | granted scopes vs actually used scopes, highlighting over-permission |
| Eval evidence freshness | days since last eval or red-team run by risk tier |
| Signed artifact coverage | percentage of production capabilities linked to signed package/hash |
| Deprecated capability invocation count | measures lifecycle control effectiveness |
| Exception age and renewal rate | shows governance debt and risk acceptance discipline |
| Consumer exit readiness | percentage of consumers with tested migration or fallback plan |
KRIs:
| KRI | Example threshold |
|---|---|
| Tier 3/4 tool outside runtime gateway | zero tolerance |
| Sensitive read capability without data steward approval | immediate escalation |
| Customer-impacting action without approval evidence | incident review |
| Unsigned package in production agent path | block new invocation |
| Capability card missing data classification | publication blocker |
| Deprecated tool still used after sunset | executive escalation for high-risk capabilities |
| Prompt injection eval failure without compensating control | certification hold |
| Broad wildcard scope granted to agent | security architecture review |
| Owner attestation expired | capability hidden from new consumers |
| Invocation ledger missing side-effect result | audit evidence defect |
14. Failure Modes
| Failure mode | Why it happens | Consequence | Better pattern |
|---|---|---|---|
| Marketplace becomes a link directory | team optimizes for inventory, not runtime governance | unsafe capabilities look official | publish only with capability card, tier, evidence and enforcement path |
| Certification detached from gateway | review happens in slides, runtime uses direct credentials | policy is unenforceable | bind certification to entitlement and runtime policy |
| Tool providers under-describe side effects | API team thinks in endpoints, not agent behavior | agents use tools in unintended chains | require side-effect taxonomy, reversibility and approval model |
| One-time approval culture | capability approved once and forgotten | stale prompt, model, source or endpoint persists | renewal cadence by risk tier and change triggers |
| Marketplace hides risk to improve adoption | high-risk tools marketed as “easy reuse” | business teams bypass review | make risk visible but navigable with clear gates |
| Broad service accounts | agents share generic credentials | weak attribution and excessive access | agent identity, delegated user context and scoped tokens |
| Eval theater | only happy-path tests or generic quality checks | prompt injection/tool misuse missed | threat-informed eval library and use-case harm tests |
| No consumer inventory | owner cannot know who uses capability | deprecation and incident response fail | entitlement graph and invocation graph |
| No exit path | capability becomes critical but unsupported | vendor or owner failure creates operational lock-in | lifecycle and replacement plan before Tier 3/4 adoption |
| Audit evidence is reconstructed manually | traces do not link card, package, scope and invocation | slow regulator/internal audit response | evidence-by-design ledger |
15. Interview-Ready Takeaways
Q1: AI agent marketplace 和普通 service catalog 最大区别是什么?
普通 service catalog 主要解决服务发现和访问申请。AI agent marketplace 还必须解决 agentic risk: tool side effects、data access、permission scopes、prompt injection、package provenance、runtime invocation evidence、owner attestation、risk-tiered certification 和 deprecation/exit。它不是目录, 而是受控的 platform product。
Q2: 你会如何设计 capability card?
我会把 card 做成人机共同可读的产品资产: identity、purpose、risk tier、allowed/forbidden uses、input/output schema、data classification、permission scopes、evaluation evidence、signed package provenance、runtime telemetry、consumer obligations、support and lifecycle。Card 要能驱动 search、access request、policy enforcement 和 audit evidence。
Q3: Tool certification 应该认证什么?
不只是代码质量。要认证 contract-first schema、auth/security scheme、data permissions、side effects、idempotency、approval requirement、prompt injection/tool misuse tests、signed package、runtime gateway enforcement、monitoring and evidence export。对 Tier 3/4, 还要有 human approval、dual control、limits、rollback 和 recertification。
Q4: 为什么 signed package 对 AI governance 重要?
Agent runtime 复用的是 prompt、tool wrapper、MCP server、API spec、guardrail config、RAG manifest 和 eval pack 的组合。没有签名和 hash, 事故后无法证明到底调用了哪个版本。Signed provenance 让 invocation trace 可以连接到 approved artifact。
Q5: 如何避免 marketplace 变成创新阻力?
把治理做成平台产品: 清晰分类、低风险快速通道、标准化 capability card、自动 schema validation、共享 eval packs、self-service sandbox、可解释 certification status、明确 SLA 和 developer experience metrics。高风险能力需要强控制, 低风险能力应快速复用。
Q6: 你如何处理 MCP servers 的治理?
不把 MCP 当特殊例外。MCP server 要有 tool manifest、server identity、transport/security controls、resource boundary、data classification、permission scopes、audit logs、package provenance、owner attestation and deprecation path。关键是把每个 exposed tool 认证为 capability, 并通过 runtime gateway 或 equivalent control enforcement 约束调用。
16. Practical Templates
16.1 Capability Card Example
# Capability Card: Complaint Case Update Tool
Capability ID: cap-servicing-complaint-case-update-v3
Type: Tool
Owner: Head of Customer Servicing Platforms
Steward: Customer Data Steward, Servicing Domain
Business Domain: Customer Servicing and Complaint Operations
Risk Tier: Tier 3
Certification Status: Restricted
Purpose:
Allow approved complaint response agents to draft and submit internal case disposition updates after human reviewer approval.
Allowed Uses:
- Update complaint case status after the assigned complaint specialist confirms evidence and resolution code.
Forbidden Uses:
- Do not send customer-visible messages, issue refunds, close regulatory complaints or override fraud/account restrictions.
Data Boundary:
- Data classes: customer complaint metadata, case notes, resolution codes.
- Allowed fields: case id, disposition code, reviewer id, resolution summary, supporting evidence ids.
- Prohibited fields: full payment card number, authentication secrets, unrelated account balances, medical or HR records.
- Retention: invocation ledger retained for seven years for complaint governance evidence.
Permission Scopes:
- Action: update complaint case disposition.
- Data: assigned cases in servicing complaint workflow.
- Purpose: complaint resolution and quality review.
- Environment: production with restricted entitlement.
- Approval: human complaint specialist approval before submit.
Contracts and Manifests:
- OpenAPI / AsyncAPI / MCP manifest / prompt bundle: servicing-case-tools-openapi.yaml.
- Version: 3.4.1.
- Signed artifact hash: sha256:9f8a5f2c-servicing-case-tool-v3-4-1.
Evaluation Evidence:
- Eval suite: complaint-tool-misuse-and-prompt-injection-v2.
- Last run: 2026-06-15.
- Pass threshold: 100% block rate for unauthorized closure/refund/send attempts.
- Known limitations: does not validate downstream CRM rendering defects; monitored by case platform QA.
Runtime Controls:
- Gateway: agent-tool-gateway-prod.
- Rate/volume limits: 20 updates per specialist per hour, 500 updates per agent per day.
- Human approval: required approval_packet_id for every write.
- Logging: invocation id, agent id, reviewer id, case id, policy decision, side effect.
- Alerting: deny or escalate when prompt injection signal appears before write action.
Lifecycle:
- Certified on: 2026-06-20.
- Renewal date: 2026-12-20.
- Deprecation policy: block new consumers 60 days before shutdown.
- Replacement path: cap-servicing-complaint-case-update-v4 after CRM migration.
16.2 Certification Decision Record Example
# Certification Decision Record
Capability: cap-servicing-complaint-case-update-v3
Version: 3.4.1
Risk Tier: Tier 3
Decision: Restricted
Decision Date: 2026-06-20
Decision Owners: AI Governance Delegate, Customer Servicing Platform Owner, Cyber Security Reviewer
Evidence Reviewed:
- Capability card: approved record cap-servicing-complaint-case-update-v3.
- Contract/spec: servicing-case-tools-openapi.yaml version 3.4.1.
- Data map: complaint-case-field-allowlist-v5.
- Eval results: complaint-tool-misuse-and-prompt-injection-v2 run 2026-06-15.
- Security review: appsec-review-servicing-tools-2026-06.
- Runtime enforcement test: gateway-policy-test-run-8842.
- Owner attestation: signed by Head of Customer Servicing Platforms on 2026-06-18.
Conditions:
- Production use requires approval_packet_id and assigned-case authorization for every update.
Consumer Obligations:
- Consumer agents must remain draft-only when prompt injection score is elevated or source evidence lacks current complaint policy citation.
Renewal Trigger:
- Recertify before CRM v4 migration, any new closure/refund action, or 2026-12-20 renewal date.
Decision Rationale:
Restricted certification is appropriate because the tool creates internal case side effects, but runtime approval, scoped authorization, prompt-injection downgrade and invocation ledger controls reduce risk to the approved Tier 3 posture.
16.3 Runtime Invocation Evidence Schema
| Field | Meaning |
|---|---|
| invocation_id | unique tool/capability call id |
| agent_id | calling agent or workflow |
| human_delegator_id | user or approver where relevant |
| capability_id | marketplace capability id |
| capability_version | certified version |
| artifact_hash | signed package/prompt/spec hash |
| risk_tier | effective risk tier at invocation |
| scope_granted | approved scopes |
| scope_used | scopes actually exercised |
| purpose_code | declared business purpose |
| policy_decision | allow, deny, step-up, draft-only, dual-control |
| input_schema_result | validation outcome |
| data_objects | data categories or object references |
| tool_parameters_hash | hash or redacted parameter record |
| output_hash | output or result reference |
| side_effect | record updated, message drafted, message sent, refund created |
| approval_packet_id | human or dual-control approval evidence |
| control_exceptions | exception id if used |
| trace_export_status | available, partial, failed |
17. Final Operating Principle
成熟的 AI agent marketplace 可以用一句话检验:
Can the enterprise prove that every reusable agentic capability was discoverable,
accurately described, risk-tiered, certified with evidence, signed, permissioned,
monitored at runtime, owned by accountable people, and safely retired when trust expires?
如果答案不清楚, 企业不是缺一个 catalog 页面, 而是缺一套把 AI product architecture、platform governance、tool/API certification、runtime authorization 和 audit evidence 连接起来的 agent capability control plane。