# Restored to the Millisecond: An 18 TB Point-in-Time Recovery, Straight Through an Unreleased PostgreSQL Bug > Two complications were waiting. First, the retention window: the one full backup old enough to reach the requested day was a single scheduled prune away from deletion. Second, invisible until mid-restore: the freshly installed PostgreSQL minor release on the restore host carried a regression, at that point unfixed in any released version, that silently deadlocks WAL replay. # Restored to the Millisecond An 18 TB Point-in-Time Recovery, Straight Through an Unreleased Postgres bugfix [ Learn More ]() [ Get Support ]() * **Industry:** K-12 education technology (analytics & family engagement) * **Organization:** A K-12 education analytics and family-engagement platform serving school districts across the U.S. * **Environment:** AWS EC2: self-managed three-node PostgreSQL 15 cluster, ~20 TB ZFS volumes, pgBackRest full backups + continuous WAL archiving to S3 * **Engagement:** Proactive DBA managed services with 24/7 monitoring * **Services:** Backup & disaster recovery · Point-in-time recovery · PostgreSQL internals & root-cause diagnosis · Incident response Metric | Outcome ---|--- Recovery precision | Replay paused 1.378 milliseconds after the requested target timestamp Data restored | 17.9 TB database, rebuilt from a ~3.3 TB compressed pgBackRest repository in S3 Restore throughput | 64 parallel pgBackRest workers, sustained ~120 MB/s to disk Turnaround | Request to delivered, queryable copy in 8 days, including root-causing an unfixed PostgreSQL bug Production impact | Zero: the live cluster served schools untouched throughout Results at a glance ## Get in touch We are looking forward to working with you! [Learn More]() ## The Client The client is a K-12 education analytics and family-engagement platform serving school districts across the U.S. Its production PostgreSQL, a self-managed three-node cluster on EC2 with roughly 20 TB of compressed ZFS storage, holds every teacher message, attendance record, and district dashboard the platform serves. We have provided proactive DBA managed services since 2022, including designing and restore-testing the platform's disaster-recovery runbook. ## The Challenge The client's VP of Engineering opened a ticket with a deceptively simple question: what would it take to restore the database as it stood on a specific day two weeks earlier. At 18 TB, nothing about that is simple. A full copy needs its own 20 TB of provisioned disk, terabytes pulled from S3 and decompressed, and days of wall-clock time, all without touching the production cluster still serving schools. Two complications were waiting. First, the retention window: the one full backup old enough to reach the requested day was a single scheduled prune away from deletion. Second, invisible until mid-restore: the freshly installed PostgreSQL minor release on the restore host carried a regression, at that point unfixed in any released version, that silently deadlocks WAL replay. ## The Solution **A same-day answer, from a tested playbook.** Within 72 minutes of the request, we returned the written disaster-recovery plan (built and fully restore-tested in 2024, so transfer-and-recovery timing was already known rather than guessed), the complete pgBackRest backup inventory bracketing the target date, and the exact infrastructure needed: one EC2 host with production-sized disk and enough CPU for decompression and WAL replay. **We caught the retention cliff.** The required full backup would have been pruned by the weekend. We extended pgBackRest retention immediately, keeping it in S3 until the work was done, then reverted the change once the client finished. **We ran the restore at full pipe.** With the target set to midnight UTC on the requested recovery point, pgBackRest pulled the 17.9 TB database from its ~3.3 TB compressed S3 repository using 64 parallel workers, holding a steady ~120 MB/s of disk write and saturating the S3 transfer. The data directory landed in about two days, on schedule. **We diagnosed what the logs could not say.** Then WAL replay simply stopped. No error. Nothing in the logs even at maximum verbosity. No I/O, no CPU, the startup process frozen mid-recovery. Reading kernel process state (/proc//wchan) showed it parked on a futex: PostgreSQL had deadlocked against itself replaying its own WAL. We traced it to MultiXact handling, a regression introduced in the then-current minor releases of PostgreSQL by a fix for a different race condition, triggered when replaying WAL generated by an older minor version. The upstream community had a fix committed but not yet shipped in any point release. Notably, pgBackRest and the backups themselves were flawless; the fault was in the server's recovery path. **We built the fix before it was released.** We compiled PostgreSQL from source with the upstream patch applied and brought the database back into recovery. Replay marched through the remaining WAL and paused exactly where asked, 1.378 milliseconds past the target timestamp, delivered read-only with connection settings mirroring production, retaining the option to advance further in time or promote for writes. ## The Results Eight days after the initial question, the client's team was querying a byte-faithful copy of their 18 TB database as of the precise moment they had named, straight through an unreleased PostgreSQL bug standing in the road. Production was never touched. The client's team completed their data work against the restored copy. We then reverted the temporary retention changes and returned the backup pipeline to normal, leaving the disaster-recovery runbook re-proven against real fire rather than a drill. _"Everything is good to go and the team is wrapping up their work. Thanks for making this happen." (VP of Engineering, the client)_ ## Why it Matters A backup you have never restored is a hope, not a plan. And even a tested plan can meet a failure mode no runbook anticipates. When recovery stalled with empty logs, the difference between a delivered restore and a dead end was engineers who could read kernel process state, correlate it against the PostgreSQL source and upstream commit history, and build a patched server before the fix shipped. At terabyte scale, point-in-time recovery is not a feature you configure. It is an expertise you retain. --- [View this page online](https://www.commandprompt.com/about/success-stories-case-studies/restored-to-the-millisecond-an-18-tb-point-in-time-recovery-straight-through-an-unreleased-postgresql-bugfix/)