# Verify service is healthy
curl -s https://api.company.com/payments/health | jq
# Verify error rate is back to normal
curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))" | jq '.data.result[0].value[1]'# Verify latency is acceptable
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))" | jq
# Smoke test critical flows
./scripts/smoke-test-payments.sh
🚨 INCIDENT: PaymentServiceDegradationSeverity: SEV2Status: InvestigatingImpact: ~20% of payment requests failing
StartTime: [TIME]
IncidentCommander: [NAME]
CurrentActions:
- Investigating root cause
- Scaling up service
- Monitoring dashboards
Updatesin #payments-incidents
Status Update
📊 UPDATE: PaymentServiceIncidentStatus: MitigatingImpact: Reduced to ~5% failure rate
Duration: 25 minutes
ActionsTaken:
- Rolled back deployment v2.3.4 → v2.3.3
- Scaled service from 5 → 10 replicas
NextSteps:
- Continuing to monitor
- Root cause analysis in progress
ETA to Resolution: ~15 minutes
Resolution Notification
✅ RESOLVED: PaymentServiceIncidentDuration: 45 minutes
Impact: ~5,000 affected transactions
RootCause: Memory leak in v2.3.4Resolution:
- Rolled back to v2.3.3
- Transactions auto-retried successfully
Follow-up:
- Postmortem scheduled for [DATE]
- Bug fix in progress
### Template2: Database Incident Runbook
```markdown
# Database Incident Runbook
## Quick Reference
| Issue | Command |
|-------|---------|
| Check connections | `SELECT count(*) FROM pg_stat_activity;` |
| Kill query | `SELECT pg_terminate_backend(pid);` |
| Checkreplication lag | `SELECT extract(epochfrom (now() - pg_last_xact_replay_timestamp()));` |
| Check locks | `SELECT * FROM pg_locks WHERENOT granted;` |
## Connection Pool Exhaustion
```sql-- Check current connectionsSELECT datname, usename, state, count(*)
FROM pg_stat_activity
GROUPBY datname, usename, state
ORDERBY count(*) DESC;
-- Identify long-running connectionsSELECT pid, usename, datname, state, query_start, query
FROM pg_stat_activity
WHERE state != 'idle'ORDERBY query_start;
-- Terminate idle connectionsSELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'AND query_start < now() - interval'10 minutes';
Replication Lag
-- Check lag on replicaSELECTCASEWHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN0ELSEextract(epoch from now() - pg_last_xact_replay_timestamp())
ENDAS lag_seconds;
-- If lag > 60s, consider:-- 1. Check network between primary/replica-- 2. Check replica disk I/O-- 3. Consider failover if unrecoverable
Disk Space Critical
# Check disk usagedf -h /var/lib/postgresql/data
# Find large tables
psql -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC
LIMIT 10;"# VACUUM to reclaim space
psql -c "VACUUM FULL large_table;"# If emergency, delete old data or expand disk