SaaS Incident Management

Incident severity classification is the first decision in every incident — it determines who is paged, how quickly, and what communication SLAs apply. A practical severity model: SEV-1 (Critical): complete product outage or data loss event. Zero users can access the core product OR confirmed data integrity breach. Response: immediate all-hands bridge call, CEO and board notification within 1 hour, public status page update within 15 minutes, customer notification within 30 minutes. SEV-2 (Major): significant feature unavailability affecting > 10% of the active user base, or a critical feature broken for all enterprise tier accounts. Response: engineering lead and on-call rotation immediately engaged, cross-functional coordination meeting within 30 minutes, customer notification within 1 hour. SEV-3 (Minor): isolated feature degradation affecting < 10% of users or a non-critical feature broken. Response: engineer assigned, standard working-hours escalation, status page updated if publicly visible, no proactive customer contact unless the affected feature is commonly used. SEV-4 (Low): minor bug with known workaround, no widespread customer impact. Response: logged as a tracked bug in the engineering backlog, no escalation.

Incident communication quality dramatically affects customer trust and churn risk, often more than the incident itself. Principles: communicate early and often, even when you have nothing to report — silence is interpreted as incompetence or evasion. Structure incident communication across three channels: (1) Public status page (Statuspage.io is standard): updated within 15 minutes of incident classification. Initial update: "We are aware of an issue affecting [Feature X] and are investigating. We will update every 30 minutes." Subsequent updates every 30 minutes with exactly three elements: what is still impacted, what the team has learned, and the next update time. Resolution update: "This incident has been resolved. All [Feature X] functionality has been restored as of [time]. We will publish a full post-mortem within 48 hours." (2) Proactive email to affected accounts: for SEV-1 and SEV-2, the Support or CS team sends a direct email to all enterprise-tier accounts informing them of the impact before they encounter it. Being first to inform is significantly better than waiting for customers to discover and complain. (3) In-app banner during active incident: a visible banner in the product UI acknowledging the issue so users encountering problems immediately understand it is known, not user-error.

The post-incident review (PIR, also called a post-mortem in engineering culture) is the structured debrief that converts a painful incident into organizational learning. An effective PIR is blameless — focused on systemic process and infrastructure improvements, not on finding and criticizing the individual who made a mistake. The "five whys" technique: ask "why did this happen?" and then ask "why" about each answer, five times. This exercise almost always reveals that what appeared to be a human error (an engineer made the wrong configuration change) was actually a systemic failure (the change process did not include a review step that would have caught the error, the monitoring did not alarm in time, the runbook directed the on-call engineer incorrectly). PIR structure: timeline reconstruction (a precise minute-by-minute account of what happened, validated by the team members who were involved); impact quantification (how many customers affected, for how long, what was the estimated ARR at risk?); root cause analysis (what systemic conditions enabled this incident?); immediate mitigations completed; and — most importantly — action items. PIR action items must be specific, owned, and scheduled: "Mehmet will add the deployment configuration change review step to the runbook by March 15." Product Ops tracks PIR action items monthly and reports completion rates to engineering leadership.