PostgreSQL backup tool Databasus moved to PG 17 native physical backups

Hacker News

Share card

PostgreSQL backup tool Databasus moved to PG 17 native physical backups

An earlier version of Databasus shipped a backup agent: a binary that ran on the database host to stream WAL and create physical backups locally. That first implementation turned out to be a mistake, and we removed it. Physical backups now run remotely from the Databasus host, as described above. Why the agent was the wrong approach: • It was a naive implementation that only copied WAL on top of full backups, which led to a long RTO • Users had to configure both Databasus and a separate agent, when doing everything remotely from one place is far simpler • Because the agent lived outside the main system, it was hard to cover every test case • There is really only one problem an agent solves: reaching a database that is not accessible from outside. For 99% of users that is already handled by running Databasus inside the private network or connecting over SSH, so the agent was reinventing the wheel and making a simple problem far more complicated than it needed to be • It could not run on managed databases like RDS and Cloud SQL, which forbid host-level installs but already expose the replication protocol, so a remote path was needed anyway • It also came with a lot of edge cases. Broken connections, managing agent updates and gathering logs from a separate process were all painful, and the fewer moving parts a system has, the more reliable it is in everyday use So now we use PostgreSQL 17 native backups instead. What that gives you: • They reuse PostgreSQL's own backup machinery instead of reinventing it, so you get battle-tested internals with thousands of tests and edge cases behind them • They work with remote databases, including managed services like Amazon RDS and Google Cloud SQL that expose the replication protocol but forbid installing software on the host • They give near-zero data loss, letting you restore to any second between backups with low RTO / RPO We made sure existing backups stay safe. If you upgrade from a version that still has agent backups, Databasus won't do it silently: it warns you about the change and lets you either stay on the supported version 3.42.0 or remove the old agent backups yourself before upgrading. The agent-based implementation remains available up to version 3.42.0 and will keep working for a long time, so nothing breaks. You can read the full reasoning in the architecture decision records: ADR-0008: PG17-native backups with mandatory WAL summary https://github.com/databasus/databasus/blob/main/adr/0008-wh... ADR-0009: remote physical backups instead of agents https://github.com/databasus/databasus/blob/main/adr/0009-wh... By the way, Databasus recently crossed 7,500 GitHub stars and over 1 million Docker Hub pulls. Thank you for such love to the project. It really keeps us going, and a lot of what shipped here came straight from your issues, PRs and feedback. We're just getting started!