agent-production-operations

agent-production-operations

A runtime control plane for taking an evaluated agent with tools and authority into controlled production operation. This bundle bridges the gap between passing evaluations and safe production use: it defines how an agent is versioned, rolled out, monitored, constrained, escalated, and disabled — and how production evidence feeds back into evaluation and release decisions.

The bundle does not build agents, design evaluations, run release pipelines, or own infrastructure. It composes those concerns from existing specialist skills and owns only the runtime control decisions between them.

When to load this

Load this bundle when you are operating an agent that:

• Has passed evaluation and is cleared for production consideration.
• Has tools with side effects and delegated authority that must be gated.
• Needs a staged rollout with progressive authority expansion.
• Must be monitored for cost, latency, tool health, and authority usage in production.
• Requires a defined fallback, escalation, and disablement path.
• Should feed production traces and incidents back into evaluation cases and release-gate updates.

When not to use

Do not load this bundle for:

• Building or designing an agent from scratch — route to the appropriate framework skill (LangGraph, CrewAI, AutoGen, etc.).
• Designing agent evaluations or observability — route to agent-evals-and-observability.
• General release engineering (CI/CD pipelines, artifact promotion) — route to release-engineering.
• General site reliability engineering (incident response, SLO definition, error budgets) — route to site-reliability-engineering.
• General security engineering (threat modeling, vulnerability assessment) — route to secure-software-engineering.
• General platform infrastructure (compute, networking, storage) — route to platform-engineering.
• Performing a production-readiness review — route to production-readiness; this bundle consumes readiness outcomes, it does not produce them.
• Conducting an incident postmortem — route to incident-learning; this bundle consumes incident records as escalation and feedback inputs.

Autonomy is not assumed uniform

This bundle explicitly does not assume all agents share the same autonomy, user population, or side-effect profile. Agent profiles are distinguished by capability class and side-effect surface:

Every control-plan decision (authority, escalation, fallback, disablement) in this bundle is parameterized by the agent's autonomy profile. A read-only agent and a side-effect-capable agent operating on customer data receive different thresholds and different escalation paths.

Runtime control plane routing

The bundle composes the following specialist skills. Load them when their domain is the active concern; the bundle owns the cross-domain coordination.

Loading protocol

This SKILL.md is the discoverable umbrella entry point. Nested skills are not used in this bundle. Reference files are loaded on trigger:

Concrete operational parameters

Authority

• Definition: the set of actions an agent is permitted to perform, scoped by target (which resources), operation (read/write/delete), and user-context (whose data).
• Trigger to review: any new tool registration, model update, or prompt change that expands the agent's reachable action surface.
• Threshold: authority is binary per action class. No action may be performed that is not explicitly listed in the production contract. An attempt to perform an unauthorized action is an authority breach and triggers immediate escalation.
• Action on breach: log the attempt, block the action, increment the authority-breach counter, and escalate. If breach count exceeds 3 in a rolling 24-hour window, disable the agent.

Escalation

• Definition: transfer of a decision or action from the agent to a designated human operator.
• Triggers (any one triggers escalation):
  • Authority breach (attempted unauthorized action).
  • Uncertainty above threshold: agent confidence < 0.7 on a side-effect-capable action (configurable per profile).
  • Tool failure rate > 5% in a 5-minute sliding window.
  • Cost budget breach > 10% of allocated budget in a billing period.
  • Latency p95 > 2x baseline for > 5 minutes.
  • Human-handoff keyword or explicit user escalation request.
• Action: suspend the agent's side-effect authority, log the escalation context (trigger, state snapshot, pending actions), notify the designated escalation channel, and await human disposition (resume / reduced-authority / disable).

Fallback

• Definition: a predetermined safe behavior when the agent, model, or a tool cannot operate at its normal capability level.
• Triggers:
  • Model endpoint returns 5xx for > 30 seconds.
  • A critical tool is unavailable (health-check failure for > 2 minutes).
  • Cost budget exhausted (100% consumed).
  • Latency p95 > 5x baseline for > 2 minutes.
• Actions per profile:
  • Read-only: return a static/cached response with a "results may be stale" disclaimer.
  • Side-effect-capable, internal: degrade to read-only, queue pending mutations, notify operator.
  • Side-effect-capable, customer-facing: degrade to read-only, surface a "temporarily unavailable" message to the user, escalate immediately.

Disablement

• Definition: complete revocation of the agent's ability to act, including read-only access. Distinct from fallback (which preserves reduced capability).
• Triggers:
  • 3 or more authority breaches in a 24-hour window.
  • Production-readiness review outcome is "no-go" or "defer" for the current agent version.
  • Incident-learning record attributes a severity-1 or severity-2 incident to agent action, and verified closure is not yet complete.
  • Human operator issues an explicit disable directive.
  • Tool compromise or credential leak detected (route through secure-software-engineering).
• Action: revoke all credentials and tokens; remove from routing/load-balancing; record disablement reason, timestamp, and authorizing evidence; notify escalation channel; prevent automatic restart until a new production-readiness review passes.

Cost

• Definition: the financial cost of operating the agent, attributed to a budget owner.
• Budget: allocated per agent per billing period (e.g., $500/day for a customer-facing support agent). Budget is set at production-contract time and reviewed at each production-readiness cycle.
• Thresholds:
  • 50% consumed: notification to budget owner.
  • 80% consumed: warning; cost-optimization review triggered.
  • 100% consumed: fallback to degraded mode (read-only or cached).
  • 110% consumed: disablement (hard stop, no further cost accrual).
• Measurement: per-request model cost + per-call tool cost (where applicable), attributed to the agent instance. Cost data is appended to the tool-authority-health record.

Latency

• Definition: end-to-end response time from user request to agent response, measured at p50 and p95 over 5-minute windows.
• Baseline: established during staged rollout observation window (phase 1 of rollout). Recorded in the agent production contract.
• Thresholds:
  • p95 > 2x baseline for > 5 minutes: escalation trigger.
  • p95 > 5x baseline for > 2 minutes: fallback trigger.
  • p50 > 3x baseline for > 10 minutes while cost budget is > 80% consumed: disablement consideration.
• Action: latency breaches feed the trace-to-eval feedback loop: the latency-impacted requests are sampled and reviewed for eval-case generation.

Privacy

• Definition: constraints on agent access to, processing of, and retention of user data (PII, usage patterns, conversation content).
• Concrete rules:
  • An agent must not access PII without an explicit user-data-access grant in its authority contract.
  • Agent traces and conversation logs must be scrubbed of PII before entering the trace-to-eval feedback pipeline (route scrubbing design to privacy-engineering).
  • Data retention for agent traces defaults to 30 days unless a shorter period is specified in the production contract.
  • Any PII access by a read-only agent triggers immediate escalation (the read-only profile should never touch PII).
• Privacy breach: any PII access outside the granted scope triggers escalation and a mandatory security review via secure-software-engineering.

Core workflow

1. Contract: define the agent's capability, authority, uncertainty, escalation, and side-effect contracts using references/agent-production-contract.md. Consume production-readiness outcomes and incident-learning records as decision inputs.

2. Version and plan: version the model, prompt, tools, policy, and evaluator; plan the staged rollout and fallback path using references/runtime-control-plan.md.

3. Roll out: execute staged rollout with progressive authority expansion, monitoring tool health and authority usage at each stage using references/tool-authority-health.md.

4. Monitor: track cost, latency, tool health, authority breaches, and escalation events. Apply threshold-based actions (notify, degrade, escalate, disable) per the concrete parameters above.

5. Feed back: connect production traces and incidents to evaluation cases and release-gate updates using references/trace-to-eval-feedback.md.

6. Learn: consume incident-learning verified-closure records and production-readiness review outcomes to update agent authority, escalation thresholds, and disablement conditions.

File map