.. _barman-setup: ======================== Barman Setup ======================== The goal of this page is to give our developers a rapid insight of the structure of the Barman setup we're using. It's not a deep analisis of all the many features barman offers. When alternatives are shown the goal is to make it clearer why I choose it. WAL - Write Ahead Logs ====================== Backup and restore in PostgreSQL relyes on wals, the logs that hold all transactions that change the data. When archive is activated PostgreSQL creates them of a fixed dimention (16 MB) that for a low traffic database may take several hours to be filled up. The ability to rebuild a database from a backup to a certain moment in time (or at a certain transaction...) relyes on the availability of all the wals produced by the cluster after the completion of the last backup. The role of Barman is to help us manage in a clean way * the backup (anche the number of backup we want to retain) * the storing of all wal files, completed or partial (see below) * the restoration of the setup to the point we need Barman backup dir ================= Barman creates backup of a whole clusters (i.e.: instance running at a TCP port). We use labels to differentiate one from the other. All backup data are subdivided in main directories as:: thux-dev ├── base │ ├── 20180314T234437 │ └── 20180411T013002 ├── errors ├── incoming ├── streaming │ └── 0000000100000000000000AD.partial └── wals └── 0000000100000000 ├── 00000001000000000000006F ├── 00000001000000000000006F.00000028.backup ├── 000000010000000000000070 The configuration we're using is depicted in the manual as: .. image:: http://docs.pgbarman.org/release/2.3/images/barman-architecture-scenario2b.png :alt: setup This means * we use ``rsync`` to make the backup. This is opposed to using ``postgres``. In Debian-like systems configurations files are in ``/etc/postgresql`` (RedHat stores them in PGDATA) and I find it nicer to have them restored along main data when using ``barman restore``: pg_createcluster knows how to use them. Conf files are ``postgresql.conf``, ``pg_hba.conf`` (access configuration) and ``pg_ident.conf``. * we use both ``archive_command`` and ``wal_streaming`` to backup data from pg_server to barman. ``archive_command`` is the way postgresql allows us to handle wals as they are completed (i.e.: 16 MB big or ``archive_timeout`` is passed from the last one produced). We configure PostgreSQL to rsync them to barman server. ``wal_streaming`` is a parameters that activates a comunication between the PostgreSQL server and the barman server that uses the `streaming replication protocol`_ to receive all logs. On the barman server there is a process that receives these data:: root@backup-db64:/var/lib/barman# ps axf|grep thux-dev 15127 ? Ss 0:00 /usr/bin/python /usr/bin/barman -c /etc/barman/barman.conf -q receive-wal thux-dev 15134 ? S 0:30 \_ /usr/lib/postgresql/10/bin/pg_receivewal --dbname=dbname=replication host=thux-dev replication=true user=streaming_barman application_name=barman_receive_wal --verbose --no-loop --no-password --directory=/var/lib/barman/thux-dev/streaming --slot=barman The streamed data are stored in a file with ``.partial`` extension as seen above. This file holds data that are not yet finalized in a wal and are vital to re-create the state of the database in any point after the last backup `barman recover` ================== Restoring a backup is a pretty simple operation. The ``barman recover`` command copies a complete backup and all the wals needed. Due to a current limitation in Barman the partial wal is not copied and we must copy it by hand. I find it very handy to copy on a server in a clean directory that we'll use to create a new cluster with standard command: pg_createcluster --start In a time that depends on the amount of logs to be replayed it will get to a working server. Any possible error will be logged in the logs of that cluster. Use the command pg_lsclusters to check the state:: root@template-web:~# pg_lsclusters Ver Cluster Port Status Owner Data directory Log file 9.5 main 5432 online postgres /var/lib/postgresql/9.5/main /var/log/postgresql/postgresql-9.5-main.log 9.6 main 5433 online postgres /var/lib/postgresql/9.6/main /var/log/postgresql/postgresql-9.6-main.log 10 main 5434 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log More info ========== A clear presentation of all terms used in PostgreSQL can be found in this `PostgreSQL/Replication wiki book`_. .. _`streaming replication protocol`: https://www.postgresql.org/docs/10/static/protocol-replication.html .. _`PostgreSQL/Replication wiki book`: https://en.wikibooks.org/wiki/PostgreSQL/Replication