Specialist Incident Ally

An incident management agent that helps you handle production incidents from detection through resolution and post-mortem, coordinating response efforts, managing communication, and driving systematic improvements.

When to Use This Agent

Choose Specialist Incident Ally when:

• Responding to production incidents that need coordinated response
• Setting up incident management processes and playbooks
• Conducting post-incident reviews and generating action items
• Building incident response automation and tooling
• Training teams on incident management best practices

Consider alternatives when:

• Debugging specific technical issues (use a debugging agent)
• Setting up monitoring and alerting (use a monitoring specialist)
• Managing PagerDuty configuration (use Advisor PagerDuty Champion)

Quick Start

Copy
# .claude/agents/specialist-incident-ally.yml
name: Specialist Incident Ally
description: Coordinate incident response and post-mortems
model: claude-sonnet
tools:
  - Read
  - Write
  - Bash
  - Glob
  - Grep

Example invocation:

claude "We have a production outage affecting checkout. Help me coordinate the response: assign roles, set up communication, and guide the investigation"

Core Concepts

Incident Response Framework

Role	Responsibility	During Incident
Incident Commander	Coordination, decisions	Manages timeline, delegates
Technical Lead	Root cause investigation	Diagnoses and implements fix
Communications Lead	Status updates	Updates stakeholders, customers
Scribe	Documentation	Records timeline, decisions

Incident Severity Levels

Level	Criteria	Response Time	Communication
SEV-1	Complete outage, data loss	<5 min	All-hands page, exec notice
SEV-2	Major degradation, >10% users	<15 min	Team page, management notice
SEV-3	Minor degradation, <10% users	<1 hour	Team notification
SEV-4	Cosmetic issue, workaround exists	Next business day	Ticket created

Incident Timeline Template

Copy
## Incident #2026-0315: Checkout Service Outage

### Summary
Checkout service returned 500 errors for all users for 23 minutes.
Root cause: database connection pool exhaustion from a slow query
introduced in deploy v2.4.1.

### Timeline (UTC)
| Time | Event |
|------|-------|
| 14:02 | Alert: checkout error rate > 5% |
| 14:04 | IC assigned: @sarah |
| 14:05 | Slack channel #inc-checkout-0315 created |
| 14:08 | Identified: all errors from DB timeout |
| 14:12 | Found: slow query in new product search |
| 14:15 | Decision: rollback to v2.4.0 |
| 14:18 | Rollback deployed |
| 14:22 | Error rate returning to baseline |
| 14:25 | Incident resolved |

### Impact
- Duration: 23 minutes
- Users affected: ~3,200
- Revenue impact: ~$45,000 delayed orders
- Data loss: None

### Root Cause
Deploy v2.4.1 added a product search query using LIKE '%term%'
without an index. Under load, query time exceeded 30s, exhausting
the 20-connection pool and blocking all database operations.

### Action Items
| Priority | Action | Owner | Due |
|----------|--------|-------|-----|
| P0 | Add index for product search | @dev-team | Mar 16 |
| P1 | Increase connection pool to 50 | @sre-team | Mar 17 |
| P1 | Add slow query alerting (>5s) | @sre-team | Mar 20 |
| P2 | Add load testing for search | @qa-team | Mar 25 |

Configuration

Parameter	Description	Default
incident_tool	Incident management platform (rootly, pagerduty, opsgenie)	Auto-detect
communication	Communication channel (slack, teams, discord)	slack
severity_levels	Number of severity levels	4
auto_channel	Auto-create incident channels	true
postmortem_template	Post-incident review template	blameless
action_tracking	Action item tracking tool (jira, linear, github)	Auto-detect

Best Practices

1. Assign the Incident Commander role immediately, even for small incidents. Without an IC, everyone investigates independently, communication is scattered, and no one tracks the timeline. The IC does not need to be the most senior person — they need to be organized and available. The IC manages communication and coordination while the Technical Lead focuses entirely on diagnosis and resolution.

2. Separate diagnosis from repair during active incidents. During an active outage, the priority is restoring service, not fixing the root cause. If a rollback restores service, roll back first and investigate the root cause afterward. Spending 30 minutes debugging while the service is down costs more than rolling back in 5 minutes and debugging at leisure. Mitigation first, root cause second.

3. Send status updates on a fixed schedule, not just when there is news. Stakeholders need regular updates to manage their own response (customer support, executive communication). Update every 15 minutes during SEV-1 incidents, every 30 minutes for SEV-2. If there is no new information, explicitly say: "Still investigating. No new findings. Next update at 15:00." This prevents stakeholders from interrupting the response team.

4. Run blameless post-mortems focused on systemic improvements. The post-mortem goal is preventing recurrence, not assigning blame. Focus on what processes, tools, or systems failed — not who made a mistake. Humans make errors; the system should catch and recover from them. Ask "Why did the system allow this error to reach production?" rather than "Who introduced this bug?"

5. Track and complete action items with the same urgency as feature work. Post-mortem action items that are never completed guarantee incident recurrence. P0 actions (prevent this exact incident from recurring) should be completed within one sprint. Track completion rates and review open items in weekly engineering meetings. An incomplete action item list is a list of future incidents.

Common Issues

Incidents escalate because initial responders do not have enough context. The on-call engineer gets a vague alert, spends 20 minutes understanding the system, and the outage extends while they orient themselves. Create runbooks for each service that include: architecture overview, dependency map, common failure modes, diagnostic commands, and escalation contacts. An engineer with a runbook can diagnose in 5 minutes what takes 30 without one.

Post-mortem action items are created but not prioritized or tracked. Without integration into the team's project management tool, action items live in a document that no one revisits. Create tickets in Jira/Linear immediately during the post-mortem, assign owners, and set due dates. Add a standing agenda item to sprint planning: "Review open incident action items." Track action item completion rate as a team metric.

Multiple incidents occur simultaneously and overwhelm the response capacity. When two SEV-1 incidents hit at the same time, the team cannot staff two incident responses. Prepare for this by maintaining an escalation tree that reaches beyond the immediate on-call team, pre-assigning backup ICs, and training multiple engineers on incident management. Cross-train team members on each other's services to increase response capacity.