Incident Postmortem · Sample Output
RESOLVED

Checkout outage: expired payment credential

Bluepine Outdoors checkout was unavailable for 29 minutes on a weekday afternoon. No orders were lost and no data was corrupted — but every customer who tried to check out in that window hit an error, and the first signal came from customers, not from an alert.

SEV-2 Detected 14:02 UTC Resolved 14:31 UTC Duration 29 min Status: Resolved

Impact

Every number carries its window and its source. Estimates are labelled.

MeasureValueWindowSource
Failed checkouts1,84214:02–14:31 UTCCheckout-service error log
Affected users (est.)~3,10014:02–14:31 UTCSessions with a failed checkout attempt
Orders at risk (est.)$18,40014:02–14:31 UTCFailed carts × average order value
Payments capturedUnaffected—Payment ledger: no failed captures

What was not affected: orders already captured, customer accounts, and all stored data. No data was lost.

Timeline

One event per line, each with its source. Gaps are shown, not hidden.

First customer reports of checkout errors reach support.

Source: support inbox, 3 reports by 14:09

On-call engineer paged; incident bridge opened.

Source: paging system

No recorded activity — the bridge log has no entries in this window.

Source: bridge log (gap, stated plainly)

Deploy log checked: the only change in the window was a credential rotation in staging, not production.

Source: deploy log

Expired production payment credential identified as the cause; fallback credential applied.

Source: payment-service log

Error rate back to baseline; incident declared resolved.

Source: checkout error-rate dashboard

Root cause

Checkout failed because the production payment credential expired, and nothing alerted on its expiry before customers felt it.

  1. Why did checkouts fail? The payment service rejected requests with an expired credential.
  2. Why was the credential expired? It was rotated in staging, but the production rotation job never ran.
  3. Why did the job never run? Its schedule was removed during a cleanup of unused cron jobs.
  4. Why was it removed? The cleanup had no “last successful run” check and no owner review for jobs touching production credentials.
  5. Why did nothing catch that? Credential expiry had no alert — the first signal was customer-facing errors.

Ruled out: a bad deploy (deploy log shows no production change in the window) and a payment-provider outage (provider status was green throughout).

Went well / went wrong / was luck

Went well

  • Fallback credential applied in 12 minutes once identified
  • Support kept customers updated every 10 minutes
  • Rollback path was documented and worked first time

Went wrong

  • No alert existed for credential expiry
  • Customers reported the outage before any alert fired
  • Bridge log gap of 4 minutes with no recorded decisions

Was luck

  • Low-traffic hour: at peak, ~4× as many checkouts would have failed
  • A valid fallback credential already existed

Action items

One change, one owner, one date, one verification step each.

#ChangeTypeOwnerDue
1Block cron-job cleanup for jobs touching production credentials without owner reviewPreventPlatform leadNov 20
2Credential-expiry alert at 14 days and 3 days before expiry, all production credentialsDetectOn-call rotationNov 13
3Automatic fallback to secondary payment provider on credential errorsMitigatePayments teamDec 4
4Require a bridge-log entry every 5 minutes during Sev-1/Sev-2 incidentsPreventIncident commander rotationNov 27

Verification: each item is checked by a named test — for item 2, a test credential set to expire in 10 days must fire the alert in staging.

What customers were told (draft, for human approval)

“On Thursday afternoon, checkout on our store was unavailable for about half an hour due to an expired payment credential. No orders were lost and no payment information was affected. We have fixed the credential and are adding expiry alerts so this is caught automatically in future. If your checkout failed during that window, please try again — and thank you for your patience.”

This is a sample output of the Incident Postmortem Writer skill, made with fictional data.
The full method — timeline reconstruction, five-whys discipline, blameless action-item rules — is inside the paid skill.