PagerDuty Integration: Escalation Policies for Incidents

PagerDuty is the industry standard for on-call management and incident response. Integrating it with your uptime monitoring ensures critical alerts reach the right people, even at 3 AM. Here's how to set it up properly.

Why PagerDuty?

• On-call management - Schedules, rotations, overrides
• Smart escalations - If one person doesn't respond, alert another
• Multi-channel delivery - Phone, SMS, push, email
• Incident tracking - Timeline, acknowledgment, resolution
• Analytics - Response times, alert patterns

Integration Setup

Step 1: Create a Service

1. Log into PagerDuty
2. Go to Services → Service Directory
3. Click "+ New Service"
4. Name it (e.g., "Production Uptime Monitoring")
5. Select escalation policy
6. Choose integration type

Step 2: Add Integration

For most monitoring tools:

1. In Service, go to Integrations tab
2. Click "+ Add Integration"
3. Select your monitoring tool or "Events API v2"
4. Copy the Integration Key (routing key)

Step 3: Configure Monitoring Tool

In your monitoring service:

1. Go to notification settings
2. Add PagerDuty integration
3. Paste the Integration Key
4. Map severity levels

Escalation Policies

Basic Escalation

Level 1 (0 min):  Primary On-Call
Level 2 (15 min): Secondary On-Call
Level 3 (30 min): Engineering Manager
Level 4 (45 min): VP Engineering

Creating Escalation Policy

1. Go to People → Escalation Policies
2. Click "+ New Escalation Policy"
3. Add levels with targets and delays
4. Assign to services

Best Practices

• Short initial timeout - 10-15 minutes for critical
• Multiple escalation levels - At least 3
• Clear ownership - Someone is always responsible
• Regular updates - Review quarterly

On-Call Schedules

Setting Up Schedules

1. Go to People → On-Call Schedules
2. Click "+ New Schedule"
3. Configure rotation:
   • Rotation type (daily, weekly)
   • Handoff time
   • Team members

Schedule Types

Weekly Rotation:

Week 1: Alice
Week 2: Bob
Week 3: Carol
Week 4: Alice...

Follow-the-Sun:

Americas (8 AM - 4 PM PT): Team A
EMEA (8 AM - 4 PM GMT): Team B
APAC (8 AM - 4 PM JST): Team C

Overrides

For planned absences:

1. Go to schedule
2. Click on the time slot
3. Create override with replacement

Alert Routing

Severity Mapping

Map monitoring severity to PagerDuty:

Service Dependencies

If Service A depends on Service B, configure:

• Service B alerts trigger first
• Service A alerts suppressed if B is down

Incident Workflows

Acknowledgment

When alerted:

1. Acknowledge within SLA (e.g., 5 min)
2. Acknowledgment stops escalation
3. Snooze if needed (with reason)

Resolution

After fixing:

1. Resolve incident in PagerDuty
2. Monitoring should auto-resolve too
3. Add notes for postmortem

Auto-Resolution

Configure your monitoring to resolve PagerDuty incidents when service recovers:

{
  "routing_key": "...",
  "dedup_key": "monitor-123",
  "event_action": "resolve"
}

Event Payload

Trigger Event

{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "trigger",
  "dedup_key": "production-api-down",
  "payload": {
    "summary": "Production API is DOWN",
    "severity": "critical",
    "source": "WizStatus Monitoring",
    "custom_details": {
      "monitor": "Production API",
      "url": "https://api.example.com",
      "error": "HTTP 500",
      "duration": "5 minutes"
    }
  },
  "links": [{
    "href": "https://dashboard.wizstatus.com/monitors/123",
    "text": "View in Dashboard"
  }]
}

Resolve Event

{
  "routing_key": "YOUR_INTEGRATION_KEY",
  "event_action": "resolve",
  "dedup_key": "production-api-down"
}

Deduplication

Use dedup_key to prevent duplicate incidents:

• Same dedup_key = same incident
• New triggers update existing incident
• Resolve closes the specific incident

Good dedup keys:

• production-api-http-check
• ssl-cert-expiry-example-com
• monitor-12345

Testing Integration

Send Test Alert

Most monitoring tools have "Send Test Alert" button.

Or use API directly:

curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "YOUR_KEY",
    "event_action": "trigger",
    "dedup_key": "test-alert-001",
    "payload": {
      "summary": "Test alert from monitoring",
      "severity": "info",
      "source": "Test"
    }
  }'

Verify Escalation

1. Trigger test incident
2. Don't acknowledge
3. Wait for escalation timeout
4. Verify next level is notified

Common Configurations

Small Team (2-3 people)

• Simple weekly rotation
• 2-level escalation
• SMS + push notifications

Medium Team (5-10 people)

• Weekly rotation with primary/secondary
• 3-level escalation
• Different schedules for business/after-hours

Large Team (10+ people)

• Follow-the-sun schedules
• Tiered escalation by severity
• Multiple services per team
• Dedicated incident commander

Troubleshooting

Alerts Not Arriving

1. Check Integration Key is correct
2. Verify service is not in maintenance mode
3. Check user notification rules
4. Review PagerDuty event logs

Wrong Person Alerted

1. Check schedule is current
2. Verify no overrides
3. Check escalation policy assignment
4. Review user time zone settings

Too Many Alerts

1. Review deduplication settings
2. Check for flapping monitors
3. Adjust thresholds
4. Consider suppression rules

Integration Checklist

• PagerDuty service created
• Integration key added to monitoring
• Escalation policy configured
• On-call schedule set up
• Severity mapping configured
• Auto-resolution enabled
• Test alert sent successfully
• Escalation tested
• Team notified of on-call duties