Fifteen Years, Seven Emergencies, Zero Data Loss

• Industry: Public sector: public libraries
• Organization: One of the busiest public library systems in the United States, serving well over a million residents
• Environment: Self-managed PostgreSQL on on-premises VMware, PgBouncer connection pooling, pgBackRest backups, behind the Evergreen open-source integrated library system (ILS)
• Engagement: Proactive managed DBA services
• Services: Major-version upgrades · 24×7 monitoring & incident response · Infrastructure migration · Performance engineering

The client is one of the busiest public library systems in the United States, serving well over a million residents across dozens of branches. Its catalog, circulation, holds, and patron accounts all run on Evergreen, the open-source ILS, and PostgreSQL is Evergreen's system of record. Every catalog search, checkout, and due date touches the database. Any database problem is immediately visible to the public at every branch. As a taxpayer-funded institution, the library chose an open-source stack over proprietary library platforms, provided it could be run with enterprise-grade reliability.

Running a mission-critical open-source stack on premises demands deep PostgreSQL expertise the library could not justify as a full-time internal team. When the engagement began in 2010, the estate ran PostgreSQL 9.0 on bare-metal CentOS hosts fronted by pgpool, and over fifteen years every generation of that stack aged into a risk. The operating systems reached end of life. The pgpool connection pooler proved fragile, at one point taking the whole system down over a stale socket file. Long-running report queries sat idle in transaction for days, stalling replication and blocking cleanup. Disk and WAL-archive exhaustion recurred through the bare-metal era. And through it all, the database had to climb five major PostgreSQL versions without interrupting daily circulation for the public.

We have run the library's PostgreSQL estate continuously since 2010 under a flat-rate proactive support agreement, a rate held unchanged for a full decade, combining 24×7 monitoring with planned modernization:

• A repeatable, low-risk upgrade playbook. Each major-version upgrade pulls a replica out of the load-balancing pool as a standing rollback point before the primary is touched. That playbook carried the database from PostgreSQL through modern versions and runs the client's annual Evergreen application upgrades on the same cadence.
• Fleet modernization. We migrated the entire database fleet off end-of-life bare-metal CentOS onto Ubuntu VMs on VMware with monitoring intact, eliminating a large EOL risk surface.
• Connection and backup stack overhaul. We replaced the fragile pgpool tier with PgBouncer and standardized backups on pgBackRest, then validated restores when it mattered: an apparent "missing days" restore scare resolved as a procedure correction, with production backups confirmed healthy.
• Monitoring we operate ourselves. A fully managed Zabbix stack watches the estate. When the monitoring platform itself aged, we rebuilt it end to end, repairing broken history-table partitioning in the process, and added long-running-query and replication-delay alerting.
• Proactive performance engineering. After transaction-ID wraparound pressure surfaced , we built an automated weekly VACUUM FREEZE job that has headed off the problem ever since, and a proactive index review identified 461 unused indexes for safe removal.
• Incident response that closes cleanly. When a data-center power loss with generator failure took down the VMware cluster in 2025, we verified every host on recovery, caught the one real defect, an unmounted backup volume silently blocking WAL archiving, and fixed it the same day.

Across 981 tickets and fifteen years, the engagement has produced a 99% closure rate and 100% resolution of every one of the 88 Emergency and High-priority issues ever raised. There have been just seven emergencies in fifteen years, most of them routine, resolved in 2.4 days on average, with the most recent closed same-day. The database advanced five major PostgreSQL versions with no data-loss incident. The fleet moved off end-of-life hardware. The wraparound, pooling, and archiving failure modes of the early years were engineered out rather than repeatedly firefought.

The relationship keeps expanding. The client stood up a new Kubernetes-fronted production cluster with us providing backups and monitoring from day one, and the next major-version upgrade is already planned.