Forty-one alerts fired at 02:14, Ruby had the cause by 02:15.
On-call copilot for teams whose pager is louder than their signal. Ruby correlates alerts, deploys and config changes into one confirmed cause, then drafts the rollback while the incident is still open.
p99 latency 2,412 ms > 800 ms for 120 s
pg connections held 187 / 200
queue depth 3,204 > 1,000
write retries 1,190 in 60 s
origin 5xx ratio 4.1%
Deploy 8f2c1a4 raised POOL_MAX_CONSUMERS in pool-router from 24 to 96, so two replicas held 187 of the 200 connections Postgres allows.
- deploy 8f2c1a4 landed 41 m before the first alert
- pg_stat_activity rose 24 -> 187 across the same 90 s window
- same signature as INC-3902 on 14 Mar 2026, resolved by rollback
- checkout-api12,847
origin
- notify-fanout3,204
queued, not lost
- ledger-writer1,190
retried to completion
- edge-cache0
insulated by stale-while-revalidate
12,847 requests affected · 431 checkouts failed · EUR 18,240 at risk
Ruby triaged 1,204 incidents in the last 30 days, and collapsed 8,913 alerts to get there.
- 1,204
- incidents triaged
- 47.2%
- median MTTR reduction
- 8,913
- alerts collapsed
- 1m 04s
- to first confirmed cause
last 30 days, 87 teams
against each team's own 90-day baseline
into 1,204 causes · 7.4 : 1
median · was 11m 20s
Hour 02 is INC-4127. It broke the scale at 41 alerts in nine minutes and still came back as one cause.
- one tick
- one alert that fired
- one node
- one cause Ruby confirmed
- quiet streak
- 14 days, ended 22 Jul
- Halyard
- Northwind Freight
- Basalt Labs
- Kestrel Health
- Loam
- Vantive Systems
What Ruby does between the page and the fix.
Four things, all of them from INC-4127 as it actually happened at Halyard on 22 July.
It reads the deploy that broke it
Ruby diffs every change that landed inside the correlation window and points at the line, not the pull request. Here it read a pool ceiling and knew what the database allows.
Postgres max_connections is 200. Each replica opens two pools, so 96 consumers across two replicas requests 192. Nothing in review expresses that product, which is why this passed.
It runs the rollback, once you say so
Proposed steps, your approval, then execution with a hold-time check on the end. Nothing runs unapproved and every step is logged with its timestamp.
Ruby proposed this, Amina Bello approved it at 02:15:58. Nothing runs unapproved.
1. Scale POOL_MAX_CONSUMERS back to 24
02:16:022. Drain pool-router replicas 2 / 2
02:16:443. Verify p99 under 400 ms for 120 s
holding 68 s of 120 s
It wires the symptom to the change
Ruby drew this line: a latency breach in Datadog against a merge in GitHub, 41 minutes apart, across 31 connected sources.
It holds back the 36 alerts you did not need
Every hold states its reason and every one is reversible. Amina saw five alerts that night instead of forty-one, and the ledger says exactly which thirty-six were kept and why.
- ×9checkout-api · 5xx ratiosame cause as INC-4127
- ×7notify-fanout · queue depthdownstream of pool-router
- ×6ledger-writer · write retriesdownstream of pool-router
- ×5edge-cache · origin 5xxdownstream, cache absorbed it
- ×5synthetic checkout · timeoutprobe traffic, not users
- ×4pool-router · restart loopexpected during rollback
Two things nobody wants to do at 02:14, already done when you get there.
The write-up starts at 02:15, not on Thursday afternoon.
Ruby fills the sections it can prove from the incident record: summary, timeline, impact, root cause. It writes prose, with the numbers in it, and it cites the deploy.
What it cannot know it leaves to you and says so. Contributing factors and action items came back marked for a human, because judgement is not in the telemetry.
- pre-filled
- 78% of sections, 4 of 6
- drafted at
- 02:15:24, one minute in
- cites
- 8f2c1a4, INC-3902, pg_stat_activity
- Summary
- Timeline
- Impact
- Root cause
- Contributing factorsyou
- Action itemsyou
At 02:11:30 UTC on 22 Jul 2026, deploy 8f2c1a4 raised POOL_MAX_CONSUMERS in pool-router from 24 to 96.
Each replica opens two pools, so the two running replicas requested 192 of the 200 connections Postgres allows; 187 were held at the moment checkout-api began timing out.
p99 checkout latency moved from 218 ms to 2,412 ms over 90 seconds and 431 checkouts failed before the rollback completed at 02:16:44.
The change passed review because the pool ceiling is not expressed anywhere near the database's connection limit, and no test asserts the product of replicas, pools and consumers.
p99 by route for checkout-api since the deploy
- POST /v2/checkout/confirm2,412 ms+1,006%
- POST /v2/checkout/session1,884 ms+742%
- GET /v2/checkout/:id604 ms+178%
- GET /v2/health18 ms+2%
Ask the question you would have asked a colleague, get the query and the rows.
Ruby knows which route labels your histograms carry and which service owns them, so a sentence becomes PromQL you can read, edit and keep.
It shows the query it wrote. If it guessed a label wrong you will see that before you trust the number, which is the only way this is useful at two in the morning.
- answered in
- 340 ms over 4.1 M samples
- query languages
- PromQL, LogQL, SQL, Datadog
- saved by halyard
- 62 asks kept as dashboards
It reads what you already run.
Ruby correlates across 31 sources and writes back to 4 of them. Read access is enough to start; write-back is opt-in per target.
Not on the list? OpenTelemetry ingest and raw webhooks cover anything that can emit an event with a timestamp.
- Datadogreads monitors + metrics
- Sentryreads issues + releases
- PagerDutyreads pages, writes notes
- GitHubreads deploys + diffs
- Grafanareads dashboards
- Kubernetesreads events + rollouts
- Linearwrites action items
- Slackwrites incident threads
Priced per engineer, never per alert. A loud night should not cost more.
Starter
for 3 services
Enough to watch one critical path.
- Up to 3 services
- 7-day incident history
- 6 sources
- 1 write-back target
- Correlation + confirmed cause
Team
most rotationsper engineer / month
What on-call rotations of 4 to 40 actually run.
- Unlimited services
- 18-month incident history
- All 31 sources · 4 write-back targets
- Runbook execution with approval gates
- Postmortem drafts, 78% pre-filled on average
- Blast radius + suppression ledger
Scale
annual, from EUR 24k
For fleets where the collectors cannot leave your VPC.
- Everything in Team
- Self-hosted collectors
- SSO + SCIM, SOC 2 Type II report
- 99.95% SLA, 15-minute response
- Audit export to your warehouse
Billed on engineers in the rotation, not seats in the workspace. Halyard runs 9 engineers on Team and pays EUR 342 a month.
Your next page lands at 02:14. Ruby will already be reading it.
Three services free. Median time from connecting Datadog to a first correlated cause is 4 minutes.