ops-incident-response

Incident Response

Production incident management: detect, triage, mitigate, resolve, learn. This skill covers the human and process side of incidents, not the infrastructure automation (use ops-devops-platform for that).

Quick Reference

Scope

Severity Classification

ASCII Flow

Alert, user report, or suspected incident
  -> Detect and acknowledge within SLA
  -> Open incident channel and assign roles
  -> Triage impact, blast radius, and severity
  -> Communicate status and next update time
  -> Mitigate user impact before deep root-cause work
  -> Verify recovery with metrics and smoke tests
  -> Resolve, close comms, and schedule postmortem
  -> Feed action items back into runbooks, alerts, and platform work

Workflow

1) DETECT — Alert fires or user report received
   - Acknowledge the alert within SLA
   - Open an incident channel (Slack/Teams)
   - Assign incident commander if SEV1/SEV2

   DECLARE-AND-MOBILIZE (T+0, SEV1/SEV2):
   - Auto-create or manually open a dedicated #inc-YYYYMMDD-[short-name] channel
   - Assign roles at declaration: IC, comms lead, scribe — do not wait for volunteers
   - Post status-page automation or manual update within 5 min of declaration
   - Trigger postmortem document creation at incident close (template auto-populated
     with title, severity, IC, timeline start — not at the writing stage)
   - These are process expectations, not product prescriptions; implement with
     whatever incident tooling your team uses (PagerDuty, Opsgenie, homegrown)

2) TRIAGE — Assess impact and classify severity
   - Confirm affected systems, user segments, blast radius
   - Check recent deployments, config changes, dependency status
   - Update status page with initial assessment
   - If SLO/error-budget tracking is active: post current budget consumed in channel

3) MITIGATE — Stop the bleeding
   - Prioritize mitigation over root cause
   - Rollback, feature-flag off, scale up, redirect traffic
   - Communicate ETA or "investigating" to stakeholders

4) RESOLVE — Confirm recovery
   - Verify metrics return to baseline
   - Run smoke tests on affected paths
   - Update status page to "resolved"
   - Close the incident channel
   - Note total error-budget consumed in the postmortem summary line

5) LEARN — Blameless postmortem
   - Write timeline (what happened, when, who did what)
   - Identify contributing factors (not "root cause")
   - Define action items with owners and deadlines
   - Share postmortem with the team
   - For SEV1/SEV2: use the Howie structured-interview approach before group debrief
     (see references/postmortem-facilitation.md)

Incident Commander Checklist

• Open dedicated incident channel
• Assign roles: IC, comms lead, subject matter experts
• Set a timer for status updates (every 15-30 min for SEV1/2)
• Keep a running timeline in the channel
• Delegate investigation — IC coordinates, does not debug
• Post status page updates at each phase transition
• Call "resolved" only when metrics confirm recovery
• Schedule postmortem within 48 hours

Judgment Calls a Checklist Won't Make for You

A checklist tells you what to do. These are the calls where the IC's judgment is the actual value-add — get these wrong and the checklist executes perfectly around a bad decision.

Declare vs. don't declare, under ambiguity. The cost of over-declaring (a false SEV2 that interrupts a few engineers for 20 minutes) is almost always smaller than the cost of under-declaring (a real SEV1 running uncoordinated for the same 20 minutes). When symptoms are genuinely ambiguous in the first two minutes, escalate — this is not optimism, it is the dominant strategy under minimax regret (see references/decision-theory-applied.md P2). The failure mode worth watching for is the opposite: an IC who keeps a live incident at SEV3 past the point where a reasonable second opinion would have escalated it, because declaring SEV1/2 feels like "making a bigger deal of it." Declaring is cheap. Under-declaring is not reversible after the fact.

Comms-cadence trade-offs. A fixed 15-minute cadence is a floor, not a ceremony. Send an update sooner than the timer when: the situation materially changes (new blast radius, new mitigation, escalation to a higher severity); or a stakeholder audience (support, leadership, a named customer) is about to make a decision based on stale information. Send "no change since last update, next update at HH:MM" rather than silence — silence is read as either "resolved" or "abandoned," never as "still working on it." The trade-off cuts the other way too: paging every update to a broad channel when nothing has changed trains people to stop reading incident channels. Match audience size to update frequency — narrow SME channel gets high-frequency raw detail; the stakeholder-facing channel gets only state-change updates.

What a good IC notices that the checklist doesn't ask about. The checklist says "keep a running timeline" — it doesn't say to notice that the timeline has gone quiet for 10 minutes with no explanation, which is the tell that someone is deep in a rabbit hole instead of reporting status. The checklist says "delegate investigation" — it doesn't say to notice when two SMEs are proposing conflicting mitigations (e.g., one wants to roll back, another wants to scale up) and neither has said so out loud; the IC's job is to surface the conflict, not let it resolve by whoever pushes first. The checklist says "call resolved only when metrics confirm recovery" — it doesn't say to notice when the room is exhausted and eager to declare resolution on a metric that looks better but hasn't held for a full cycle. Watch for premature declarations of "investigating" that are actually stalling on an uncomfortable rollback call (see references/decision-theory-applied.md P4/A2) — that's a decision-avoidance pattern, not a diagnostic one.

Postmortem anti-patterns beyond the obvious.

• Blame laundering: the document uses blameless language ("the engineer who deployed the change...") while every sentence still orbits one person's actions; the tell is a timeline that reads like a deposition. Fix by writing every timeline entry as "the system/process allowed X" rather than "X did Y."
• Action-item graveyard: postmortems generate action items that get logged and never revisited — the postmortem completion rate looks healthy (100%) while the action item close rate (a separate metric in ## Metrics) quietly rots. If a team tracks postmortem completion but not action-item closure, it is optimizing the metric that doesn't matter.
• Root-cause theater: closing on a single named root cause because it is psychologically satisfying, then writing exactly one action item against it, when the timeline itself shows three or four independent contributing factors that each deserve an owner.

Communication Templates

Status Page — Investigating

[Service Name] — Investigating
We are aware of [brief description of impact].
Our team is investigating and will provide updates every [interval].
Started: [timestamp UTC]

Status Page — Resolved

[Service Name] — Resolved
The issue affecting [brief description] has been resolved.
Duration: [start] to [end] ([total minutes])
A postmortem will be published within [N] business days.

Stakeholder Update (SEV1/2)

Subject: [SEV-N] [Service] incident update — [status]

Impact: [who is affected, what they see]
Current status: [mitigating / investigating / resolved]
Next update: [time UTC]
Actions taken: [bulleted list]
Incident lead: [name]

Metrics

Distributional caveat: incident durations follow heavy-tailed distributions. The mean (the "M" in MTTx) is easily skewed by outlier incidents and is "poorly suited for decision making or trend analysis" (Davidovič, Google SRE). Use MTTx labels to name the phase; use p50/p90/p99 percentiles and incident frequency/rate for trend analysis and comparison. See references/incident-metrics-guide.md for the full treatment.

Anti-Patterns

Regulatory Reporting Overlays

Some incidents trigger a legal reporting clock in addition to the operational response above. This skill covers the operational process only — it does not draft regulatory filings or make materiality/severity determinations that carry legal weight; route those to legal/compliance or outside counsel. Verify current thresholds locally before relying on any of this table, since regulators amend timelines and technical standards.

Judgment note. These three clocks start at different triggers (classification vs. awareness vs. materiality determination) — an IC who assumes "detection" starts all three clocks will misreport the timeline to legal. Flag any incident with plausible regulatory exposure (financial-services ICT outage, personal-data exposure, anything a reasonable investor might consider material) to legal/compliance during triage, not after resolution — the reporting clock can start before the incident is even mitigated.

Navigation

References

• runbook-design-guide.md — How to write effective on-call runbooks
• on-call-practices.md — On-call rotation, escalation, and fatigue management
• postmortem-facilitation.md — Running blameless postmortems (includes Howie standard for SEV1/SEV2)
• incident-metrics-guide.md — MTTD/MTTR phases + percentile-based alternatives (MTTx distributional caveat)
• slo-incident-triggering.md — Error-budget burn rate as incident trigger; burn rate → severity mapping; postmortem prioritization by budget remaining
• references/control-theory-applied.md — Control-theory applied recipes for incident response: stable autoscaler retune, cascading-failure containment, recovery throttling.
• references/decision-theory-applied.md — Decision-theory applied recipes for incident response: paging thresholds via EU, rollback as real option, MAB runbook ordering.

Assets

Use these as copy-paste starters; fill placeholders before sharing with stakeholders.

• incident-channel-template.md — Use at T+0 when opening a SEV1/SEV2 incident channel; post as the first pinned message to assign roles and set next-update time.
• postmortem-template.md — Use at incident close to create the postmortem document; populate title, severity, IC, and timeline start immediately (do not wait until the writing session).
• runbook-template.md — Use when authoring or reviewing a service runbook; also reference during triage to confirm runbook coverage gaps as action items.

Related Skills

Fact-Checking

• Verify current severity definitions, escalation targets, status-page policy, and postmortem requirements against the team's actual runbooks.
• Use the local on-call docs and paging configuration as the source of truth before final recommendations.
• If current internal runbooks are unavailable, state the assumption and mark placeholders clearly.
• Tooling landscape moves fast — verify before naming a vendor. Confirmed as of 2026-07-11: Freshworks closed its acquisition of FireHydrant (Jan 1, 2026; FireHydrant now sits inside the Freshservice portfolio); Grafana archived the Grafana OnCall OSS project (Mar 24, 2026) and now points teams to Grafana Cloud IRM (which merges OnCall + Incident); PagerDuty is retiring the standalone Jeli UI (EOL Dec 22, 2026) and folding Howie-based post-incident reviews into PagerDuty's native product, though the Howie methodology itself is unaffected; Atlassian stopped new Opsgenie sales (Jun 4, 2025; support ends Apr 5, 2027). incident.io, Rootly, and PagerDuty remain independent. Do not assume any tool name in this skill or its assets is still the current market leader without a fresh check — this list itself will age.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.