# Thirteen Minutes on a Saturday: Disaster Recovery a Hedge Fund Can Actually Trust > Most disaster recovery plans are documents. Nobody has restored from the backups. Nobody has promoted the replica. The plan is fiction with a table of contents. This client did not want fiction. When we started in 2022, they asked us to review the backup and recovery posture of a 10TB PostgreSQL 12 estate and tell them the truth. The truth had rough edges. The pgBackRest topology had configuration drift across hosts. A weekly full backup eventually stretched to 65 hours on a database that had quietly grown past 9TB, eating into Monday-morning performance. And the firm had suffered out-of-memory outages on the primary that nobody had fully explained. # Thirteen Minutes on a Saturday Disaster Recovery a Hedge Fund Can Actually Trust [ Learn More ]() [ Get Support ]() * **Industry:** Financial services (fixed-income relative-value investing) * **Organization:** A multi-billion-dollar fixed-income hedge fund * **Environment:** Self-managed PostgreSQL 12, roughly 10TB estate, streaming replication to a dedicated DR site, pgBackRest * **Engagement:** PostgreSQL support and DR engineering * **Services:** Disaster recovery engineering · Backup architecture · Weekend incident response · Performance forensics Outcome | Verified result ---|--- Response to a Saturday DR emergency (first incident) | 13 minutes Response to a Saturday failover-test failure (second incident) | 7 minutes First incident: reported, recovered, and closed | Same Saturday Second incident: root cause found and documented | Same Saturday, replica rebuild underway before noon Written incident report with hardening recommendations | Delivered for both incidents Results at a glance ## Get treated like a professional Reach out and join us in a partner for your success! [Learn More]() ## The Client A multi-billion-dollar fixed-income hedge fund. Its trading and analytics workloads live on a PostgreSQL estate of roughly 10 terabytes, and the firm treats that estate the way it treats any other risk: measured, tested, and reviewed. Markets open Monday morning whether the database is ready or not. So the firm does something most companies only claim to do. It rehearses its disaster recovery, on real hardware, on weekends. ## The Challenge Most disaster recovery plans are documents. Nobody has restored from the backups. Nobody has promoted the replica. The plan is fiction with a table of contents. This client did not want fiction. They asked us to review the backup and recovery posture of a 10TB PostgreSQL 12 estate and tell them the truth. The truth had rough edges. The pgBackRest topology had configuration drift across hosts. A weekly full backup eventually stretched to 65 hours on a database that had quietly grown past 9TB, eating into Monday-morning performance. And the firm had suffered out-of-memory outages on the primary that nobody had fully explained. Worse, a DR plan that ambitious creates a new problem: what happens when the rehearsal itself breaks, on a Saturday, with nobody watching? ## The Solution We built the program in layers, then stood behind it. * We reviewed the backup and DR architecture end to end, wrote the disaster recovery plan, and proved it with a point-in-time restore test. Then we stood up a dedicated DR test environment so failover procedures could be developed and validated without touching production. * We ran forensics on the OOM outages with pgbadger. The picture was slow memory creep, not one bad query: 332 concurrent sessions before the crash, 567 temp files representing roughly 11GB of potential memory exposure that sessions never released. That forensic detail turned a mystery outage into a tunable configuration. * When the full backup hit 65 hours, we benchmarked the storage with bonnie++ and isolated the bottleneck to VM and network throttling, not PostgreSQL, so the fix landed in the right team's queue instead of ours guessing. Then the rehearsals broke. Twice. Both times on a Saturday morning. The first time, a DR test left pg_rewind unable to run because the WAL it needed had been removed from the primary. The DR replica was stranded. The client reported it at 8:04 AM. We were responding at 8:17. Thirteen minutes, on a weekend. We pulled the missing WAL from the pgBackRest repository, let the DR host recover through it, and had it streaming from the primary again. The ticket opened and closed on the same Saturday. The second time, four months later, reversing a failover test failed for a subtler reason: after promotion, PostgreSQL recycles WAL almost immediately, and archive_mode was off on the replica, so the files pg_rewind needed were simply gone. Reported at 9:39 AM. Responding at 9:46. Seven minutes. We diagnosed the root cause, started a pg_basebackup rebuild of the replica before noon, and delivered the written root-cause analysis that same day. ## The Results Every number in the table above traces to the ticket record. But the durable result is what happened after each incident: a written incident report, with specific recommendations, that hardened the next rehearsal. Separate the backup repository from the database hosts. Archive WAL on the replica, not just the primary. Change the test procedure so promotion does not eat the evidence. The second incident was harder to cause than the first because the first one taught us both something. The relationship outcome says the rest. The client put us through their annual vendor due-diligence review, the same scrutiny they apply to any counterparty. We passed. ## Why it Matters DR you have not rehearsed is fiction. This client understood that and rehearsed anyway, which took nerve, because rehearsals fail. Rehearsals only count if someone answers when they break. Thirteen minutes, then seven. That is what a tested plan looks like. --- [View this page online](https://www.commandprompt.com/about/success-stories-case-studies/thirteen-minutes-on-a-saturday-disaster-recovery-a-hedge-fund-can-actually-trust/)