Breaking the 20 TB Ceiling

• Industry: K-12 education technology (analytics & family engagement)
• Organization: A K-12 education analytics and family-engagement platform serving school districts across the U.S.
• Environment: AWS: formerly RDS PostgreSQL; now a self-managed three-node PostgreSQL cluster on EC2, ~20 TB ZFS volumes, PgBouncer, pgBackRest to S3
• Engagement: Proactive DBA managed services with 24/7 monitoring
• Services: Cloud replatforming (RDS exit) · Logical-replication migration · Performance & capacity engineering · 24/7 monitoring & incident response

The client is a K-12 education analytics and family-engagement platform serving school districts across the U.S. Its Ruby on Rails application, covering attendance, communications, and student analytics, runs on a PostgreSQL database grown into the tens of terabytes. Every teacher message and district dashboard depends on it. For a platform whose customers are schools, an outage during the school day is not an option.

The platform had outgrown Amazon RDS on three fronts at once. Scale: the database stood at 18.15 TB, closing in on its 20 TB RDS storage ceiling, and RDS autoscaling had already bitten once, when a cooling-off window froze volume growth during a runaway materialized-view refresh and took a replica down. Cost: the reserved instance backing an r6g.16xlarge was up for renewal, and leadership was scrutinizing a cloud bill the managed service kept inflating. Control: performance was being left on the table. Filesystem compression, kernel and memory tuning, direct log access: RDS exposes none of it, and the client's DBA partner could see only what the managed service allowed.

The client's framing was direct: out of RDS, into something they could scale horizontally and tune at every layer. The catch was the physics. Nearly 18 TB of live production data, including one 17 TB (uncompressed) append-heavy logging table, had to move without losing a row and without taking schools offline.

We executed the exit as a logical-replication migration: old and new platforms live in parallel until the moment of cutover.

• Build the destination first. A three-node cluster on EC2, a primary plus two streaming replicas, on Ubuntu with ZFS volumes for transparent compression, provisioned via the client's Terraform and configured with Ansible. We configured and restore-tested pgBackRest backups to S3 before production data arrived, and a dedicated pgBadger log host gave us the query-analysis visibility RDS had withheld.
• Replicate ~18 TB out of RDS logically. The initial sync surfaced everything a heavily written, long-lived database can hide: index rows exceeding the B-tree size limit, incompletely dropped columns, duplicate-key and tuple-header corruption, long-running transactions blocking snapshot acquisition. We engineered around each one. Publications split by write rate. The 17 TB table synced independently. A 5 TB insert-only archive table moved by a custom parallel, day-at-a-time copy that committed incrementally to avoid lock buildup.
• Cut over on the client's calendar. Go-live ran in a single planned overnight weekend window during the winter school break, the platform's natural low tide, with row counts verified before traffic moved.
• Keep the exit reversible. Reverse logical replication flowed changes from the new cluster back to RDS through the first production week, a live rollback path until the client called it. We decommissioned RDS five days after go-live.

The platform crossed from managed to self-managed with zero data loss, confirmed by the client with exact row-count matches, in one planned window instead of a forced extended outage. The 20 TB ceiling is gone. Storage is now compressed ZFS, grown through deliberate capacity planning rather than at the mercy of autoscaling cooldowns. The double spend of running RDS alongside its replacement ended the week of cutover, along with the r6g.16xlarge reserved-instance renewal.

The control dividend kept paying. We have since deployed HugePages, tuned work_mem live in production, fixed replica query cancellations with hot_standby_feedback, and replaced a runaway materialized view with a trigger-maintained aggregate table that freed ~110 GB. When an accidental mass UPDATE later overwrote production data, we recovered it in place by reading prior row versions from the heap with pageinspect, responding in 13 minutes, no restore required. The architecture also absorbed the client's next move: a second application's database migrated off Heroku into the same cluster, roughly doubling the workload on infrastructure the client now fully owns.

"Week one on the new cluster is looking great: thank you for the wild hackery on a Sunday." (Director of Engineering, the client)