Barman Setup

The goal of this page is to give our developers a rapid insight of the structure of the Barman setup we’re using. It’s not a deep analisis of all the many features barman offers.

When alternatives are shown the goal is to make it clearer why I choose it.

WAL - Write Ahead Logs

Backup and restore in PostgreSQL relyes on wals, the logs that hold all transactions that change the data. When archive is activated PostgreSQL creates them of a fixed dimention (16 MB) that for a low traffic database may take several hours to be filled up.

The ability to rebuild a database from a backup to a certain moment in time (or at a certain transaction…) relyes on the availability of all the wals produced by the cluster after the completion of the last backup.

The role of Barman is to help us manage in a clean way

  • the backup (anche the number of backup we want to retain)
  • the storing of all wal files, completed or partial (see below)
  • the restoration of the setup to the point we need

Barman backup dir

Barman creates backup of a whole clusters (i.e.: instance running at a TCP port). We use labels to differentiate one from the other. All backup data are subdivided in main directories as:

thux-dev
├── base
│   ├── 20180314T234437
│   └── 20180411T013002
├── errors
├── incoming
├── streaming
│   └── 0000000100000000000000AD.partial
└── wals
    └── 0000000100000000
        ├── 00000001000000000000006F
        ├── 00000001000000000000006F.00000028.backup
        ├── 000000010000000000000070

The configuration we’re using is depicted in the manual as:

setup

This means

  • we use rsync to make the backup. This is opposed to using postgres. In Debian-like systems configurations files are in /etc/postgresql (RedHat stores them in PGDATA) and I find it nicer to have them restored along main data when using barman restore: pg_createcluster knows how to use them. Conf files are postgresql.conf, pg_hba.conf (access configuration) and pg_ident.conf.

  • we use both archive_command and wal_streaming to backup data from pg_server to barman.

    archive_command is the way postgresql allows us to handle wals as they are completed (i.e.: 16 MB big or archive_timeout is passed from the last one produced). We configure PostgreSQL to rsync them to barman server.

    wal_streaming is a parameters that activates a comunication between the PostgreSQL server and the barman server that uses the streaming replication protocol to receive all logs. On the barman server there is a process that receives these data:

    root@backup-db64:/var/lib/barman# ps axf|grep thux-dev
15127 ?        Ss     0:00 /usr/bin/python /usr/bin/barman -c /etc/barman/barman.conf -q receive-wal thux-dev
15134 ?        S      0:30  \_ /usr/lib/postgresql/10/bin/pg_receivewal --dbname=dbname=replication host=thux-dev replication=true user=streaming_barman application_name=barman_receive_wal --verbose --no-loop --no-password --directory=/var/lib/barman/thux-dev/streaming --slot=barman

    The streamed data are stored in a file with .partial extension as seen above. This file holds data that are not yet finalized in a wal and are vital to re-create the state of the database in any point after the last backup

barman recover

Restoring a backup is a pretty simple operation. The barman recover command copies a complete backup and all the wals needed. Due to a current limitation in Barman the partial wal is not copied and we must copy it by hand.

I find it very handy to copy on a server in a clean directory that we’ll use to create a new cluster with standard command:

pg_createcluster –start <version> <name>

In a time that depends on the amount of logs to be replayed it will get to a working server. Any possible error will be logged in the logs of that cluster. Use the command pg_lsclusters to check the state:

root@template-web:~# pg_lsclusters
Ver Cluster Port Status Owner    Data directory               Log file
9.5 main    5432 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log
9.6 main    5433 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log
10  main    5434 online postgres /var/lib/postgresql/10/main  /var/log/postgresql/postgresql-10-main.log

More info

A clear presentation of all terms used in PostgreSQL can be found in this PostgreSQL/Replication wiki book.