From Zero Backups to a Safety Net

• Industry: Hospice & Palliative Electronic Medical Record (EMR) / Healthcare IT
• Platform Scale: Bedside clinical EMR backing multi-region hospice care teams and patient administration.
• System Architecture: Five production PostgreSQL clusters spanning three major versions; integrated with pgBackRest and Google Cloud Storage (GCS).
• Environment: Self-managed PostgreSQL in a colocation datacenter, migrating to Google Cloud (Cloud SQL, GKE, PgBouncer)
• Core Competencies Delivered: Migration support Disaster Recovery (DR) Architecture • pgBackRest PITR Deployment • WAL Archiving &C ompression • HIPAA Compliance Controls • Zabbix Backup Alerting

The client runs the electronic medical record for hospice and palliative care providers across the United States. Nurses chart against this database at the bedside. Medication records, care plans, and compliance audit trails all live in PostgreSQL, and all of it is PHI under HIPAA. When they engaged us, the production estate still ran in a colocation datacenter, months away from a planned migration to Google Cloud.

When assuming stewardship of a mission-critical medical platform, the first rule of database engineering is simple: a backup you have never tested is a hope, not a plan.

When the client asked us to verify their disaster recovery posture, our colocation assessment uncovered an immediate crisis: five production PostgreSQL clusters spanning three major versions had zero working backups.

To make matters worse, storage growth projections showed the primary clusters sitting just 48 days away from a disk-full crash. A single hardware failure or storage limit would have caused unrecoverable data loss and an extended EMR blackout.

The most recent local copies were stale by months. On the warehouse cluster there was a large WAL archive with no base backup behind it, which is a pile of receipts for a purchase that was never made. On the core EMR server, every backup was failing, and a cleanup script was deleting WAL files after 14 days whether they had been archived or not.

The worst of it sat on one compliance-critical database. WAL archiving was broken both locally and to its remote target. 135 GiB of unarchived WAL had piled up, the archive volume could not hold it, and no base backup existed anywhere we could find. At the observed WAL rate, the data volume was about 48 days from full. Not a hypothetical. A date on a calendar.

We intervened immediately to construct a fail-safe disaster recovery pipeline without taking the live EMR offline:

• pgBackRest Architecture: Replaced ad-hoc scripts with an enterprise pgBackRest deployment featuring Point-in-Time Recovery (PITR) capabilities.
• Secure Cloud Storage: Configured continuous WAL archiving with compressed backup streams written directly to encrypted Google Cloud Storage (GCS) buckets.
• Active Monitoring & Alerting: Embedded backup status checks and WAL stream verification directly into our 24/7 Zabbix monitoring stack, ensuring silent backup failures surface instantly before they can impact production."We built the safety net first and argued about architecture later.
• We inventoried every cluster's backup and archiving state and documented it, so the gap was visible instead of assumed.
• We implemented pgBackRest across the estate, archiving WAL and shipping compressed backups to Google Cloud Storage buckets, out of the colocation facility entirely. We sequenced the rollout by risk: the compliance-critical database first, the core EMR last only because it needed access and disk planning, not because it could wait.
• We drained the unarchived WAL backlog once real backups existed to replace it, and wired backup status into the client's monitoring so a silent failure could never accumulate for months again.
• The migration itself was led by the client's cloud partner. Our job was the net under it. We reviewed their zero-downtime migration documentation with multiple engineers, consulted on memory and parallelism settings in the final hours before cutover, and kept an engineer on call through the November migration weekend. The cutover completed without needing us. That is the goal.
• After cutover we stayed on: we reviewed the PgBouncer deployment fronting the production EMR on Cloud SQL, backed the change from one pod to three, monitored the deploy, and ran proactive reviews of long-running and heavyweight transactions.
• We moved from safety net to fire drill: disaster recovery runbooks for backup identification and restoration, tested against a dedicated DR project.

Every production cluster went from zero restorable backups to pgBackRest backups in cloud storage, verified and monitored. The disk that had 48 days left never filled. The EMR moved to GCP without downtime, and the client kept us on afterward for pooling, transaction hygiene, and disaster recovery planning.