Backing up and restoring Pulp (container and Kubernetes)

This guide backs up and restores a Pulp instance without losing anything, with podman containers as well as on Kubernetes, and also shows how to export the content of a single repository to serve it again elsewhere. Intended audience: administrators responsible for the continuity of an artifact repository. Prerequisites: a running Pulp instance. The point to understand before anything else: with Pulp, data is spread across three distinct locations, and backing up one without the others gives a broken restore.

Many people think that "everything is in the database". That is wrong, and the mistake is costly on restore day. Understanding where the data lives is the prerequisite for any reliable backup strategy.

What you will learn

• Identify the three data locations of Pulp.
• Back up a container instance (database, artifacts, keys).
• Restore in the right order to avoid dead links.
• Automate backup and restore on Kubernetes.
• Export the content of a repository, and import it on another instance.

Where the data lives: three locations, not one

Pulp does not store everything in the same place. A complete backup must cover all three:

Location	Content	Without it, at restore time
PostgreSQL	Metadata: repositories, versions, publications, distributions, tasks, RBAC	No structure left: the artifacts exist but nothing knows how to serve them
File storage (/var/lib/pulp/media, or S3 object storage)	The artifacts themselves (binaries, addressed by sha256)	The database points to missing files: dead links
Encryption keys (/etc/pulp/certs)	The key that encrypts sensitive fields (remote credentials) and the secret key	The encrypted credentials are unrecoverable

Swipe to see more

The database therefore holds metadata only. The binaries live separately, on a file system or in object storage. Redis only serves as an ephemeral task queue and does not need to be backed up.

The encryption key, the oversight that kills the restore

The database_fields.symmetric.key file encrypts the remote passwords stored in the database. If you restore the database and the artifacts but not this key, Pulp can no longer decrypt these credentials: your synchronizations are dead. Back up this key with the same care as a private key.

Backing up a container instance

In a podman deployment, the three locations are mounted volumes. The backup consists of freezing the database and archiving the volumes.

1. Dump the PostgreSQL database. The text dump is portable:

   podman exec pulp bash -c 'pg_dump -U pulp pulp' > pulp-db.sql

   The output should produce a file of several thousand lines starting with -- PostgreSQL database dump.

2. Archive the artifacts from the storage volume:

   podman exec pulp tar czf - -C /var/lib/pulp media > pulp-artifacts.tar.gz

3. Copy the encryption and session keys:

   podman exec pulp tar czf - -C /etc/pulp certs > pulp-certs.tar.gz

These three files (pulp-db.sql, pulp-artifacts.tar.gz, pulp-certs.tar.gz) form a complete and consistent backup. Store them together, encrypted, off the Pulp machine.

Database / artifacts consistency

Artifacts are addressed by digest (sha256) and immutable: storage only accumulates files, it never modifies them. A slight gap between the database dump and the artifact archive is therefore harmless, as long as the artifacts are archived after or at the same time as the database.

Restoring a container instance, in the right order

The restore order is not arbitrary. You restore the artifacts and the keys first, and the database last.

1. Restore the keys and the artifacts into the volumes of the new instance:

   tar xzf pulp-certs.tar.gz -C /chemin/volume/etc-pulp
   tar xzf pulp-artifacts.tar.gz -C /chemin/volume/var-lib-pulp

2. Reload the database:

   podman exec -i pulp bash -c 'psql -U pulp pulp' < pulp-db.sql

3. Check that the API responds and that the repositories are there:

   pulp status
   pulp file repository list

The logic is simple: a restored database that points to a missing artifact breaks the service, whereas an artifact present without a database record is simply orphaned, hence harmless and cleaned up later by the orphan cleanup. So you put the immutable data first.

On Kubernetes: PulpBackup and PulpRestore

The operator automates the whole cycle through two custom resources. You describe a backup, and the operator captures the database, artifacts and secrets to a dedicated PVC; a restore resource recreates the complete instance from that PVC.

Matching PostgreSQL version

The operator's backup tooling runs a pg_dump of a fixed version (15 in operator 2.0). A pg_dump cannot dump a server of a higher version: your database must therefore run PostgreSQL 15, not 16. See Deploying on Kubernetes for this setting. A database on 16 makes the backup fail.

Backing up with PulpBackup

1. Create a destination PVC for the backup:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: pulp-backup-claim
     namespace: pulp
   spec:
     accessModes: ["ReadWriteOnce"]
     storageClassName: local-path
     resources:
       requests:
         storage: 5Gi

2. Declare the backup with a PulpBackup resource:

   apiVersion: repo-manager.pulpproject.org/v1
   kind: PulpBackup
   metadata:
     name: sauvegarde-pulp
     namespace: pulp
   spec:
     deployment_name: pulp
     backup_pvc: pulp-backup-claim

   kubectl apply -f pulpbackup.yaml

   The operator launches a job which, once finished, sets the BackupComplete condition to True. The PVC then contains a timestamped folder with the pulp.db dump, the cr_object (the instance definition) and all the secrets, including db_fields_encryption_secret.yaml (the encryption key) and pulp_secret_key.yaml.

Restoring with PulpRestore

The restore is declared just as declaratively. It recreates the complete instance and reloads the database and secrets from the backup PVC.

apiVersion: repo-manager.pulpproject.org/v1
kind: PulpRestore
metadata:
  name: restauration-pulp
  namespace: pulp
spec:
  deployment_name: pulp
  backup_name: sauvegarde-pulp
  backup_pvc: pulp-backup-claim

kubectl apply -f pulprestore.yaml

The backup_name field is mandatory: it designates the PulpBackup resource to restore. The operator then recreates the database, the API, the content, the workers and the reverse proxy, then reloads the dump and the secrets. Next, check that your repositories are back:

kubectl -n pulp port-forward svc/pulp-api-svc 24817:24817 &
curl -s -u admin:MOT_DE_PASSE \
  http://localhost:24817/pulp/api/v3/repositories/file/file/

The backup stays on a PVC

PulpBackup writes to a PVC of the cluster: this is not off-site storage. For real protection, then copy the content of this PVC to off-cluster storage (S3 object storage, remote site), or point the PVC to a storage class that already replicates elsewhere.

Backing up the content of a repository: export and import

The two previous methods back up the machine. Pulp can also back up the content of a repository, independently of the instance that hosts it: this is export, and it is what you need when the goal is to serve again elsewhere rather than to rebuild the same instance.

The two levels do not replace each other. An export contains neither the accounts, nor the remotes, nor the encryption key: it contains artifacts and the metadata that describe them. It is restored on a new instance, or alongside, in another repository.

Allowing the paths, and restarting only once

Pulp only exports to a directory listed in ALLOWED_EXPORT_PATHS, and only imports from a directory listed in ALLOWED_IMPORT_PATHS. Without these two settings, creating the exporter fails with a message that leaves no doubt:

Error: {"path":["Path '/var/lib/pulp/exports' is not an allowed export path"]}

These settings are read at startup only. So set both at once, then restart a single time:

ALLOWED_EXPORT_PATHS = ["/var/lib/pulp/exports"]
ALLOWED_IMPORT_PATHS = ["/var/lib/pulp/imports"]

Do not use `podman restart` on this container

Measured three times in a row on a rootless instance: podman restart returns 125 with pasta failed: Failed to bind port 8080 (Address already in use) and leaves the container stopped. The port has not yet been given back to the system when podman asks for it again. You stop, wait for the port to be free, then start:

podman stop pulp
until ! ss -ltn | grep -q ':8080 '; do sleep 2; done
podman start pulp

Exporting, then getting the file off the machine

The exporter is an object that you create once; the export is an operation that you rerun each time. The two have distinct subcommands, and this is the most frequent trap: pulp exporter for the object, pulp export for the operation, which expects --exporter and not --name.

pulp exporter pulp create --name secours \
  --path /var/lib/pulp/exports \
  --repository python:python:interne

pulp export pulp run --exporter secours

The operation produces two inseparable files: a .tar archive and its -toc.json table of contents. One without the other restores nothing.

And above all, get them off the machine. An export left in /var/lib/pulp/exports disappears with the disk it was meant to protect. This is the most widespread and the most costly mistake: you discover on the day of the outage that the backup was inside the outage.

Importing: the operation the CLI cannot launch

Verified on pulp-cli 0.41.1: pulp importer pulp only exposes create, destroy, list, show and update, and pulp import does not exist. The Pulp documentation only shows the operation through the API, so that is the only way.

Three points to know before you start:

• the destination repository must exist and be empty; the import fills it, it does not create it;
• the --repo-map option takes two arguments (source name, target name), not JSON;
• the path of the -toc.json is the one the container sees (/var/lib/pulp/imports/...), never the host's.

pulp python repository create --name restaure
pulp importer pulp create --name secours --repo-map interne restaure

IMP=$(pulp --format json importer pulp show --name secours \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['pulp_href'])")

curl -u admin:MOT_DE_PASSE -X POST \
  -H 'Content-Type: application/json' \
  -d '{"toc": "/var/lib/pulp/imports/export-....-toc.json"}' \
  "http://localhost:8080${IMP}imports/"

The response returns a task_group and not a task: the import starts several tasks, one per imported resource. Monitoring written for a single task would wait forever for an end it will never see.

Once the import is finished, the restored repository must be published and distributed to serve: a filled repository with no distribution serves nobody.

Hands-on practice

The lab has you export a repository, then import it into an empty repository and serve a client with what was restored. Its central check is neither the export nor the import: it is that the backup sits on another machine. An export left in /var/lib/pulp/exports disappears with the disk it was meant to protect, and this is the most common outage. The lab also makes you run into the four refusals of Pulp before the first successful export, including the path seen from the container rather than from the host.

Guided lab Practice

Learn how to do it

Back up a repository, and prove the restore serves

$ dsoxlab run pulp-08-sauvegarder

Environment: VM·Duration: ~40 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start pulp-08-sauvegarder checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/artefacts-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run pulp-08-sauvegarder
dsoxlab check pulp-08-sauvegarder

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key takeaways

• Pulp data lives in three places: database (metadata), file storage (artifacts), encryption keys.
• The database does not contain the artifacts: the binaries are on a file system or in object storage.
• The database_fields.symmetric.key key is essential: without it, the encrypted credentials of the remotes are lost.
• In a container, you restore artifacts and keys first, database last (artifacts are immutable and addressed by digest).
• On Kubernetes, PulpBackup / PulpRestore automate the full cycle; the database must run PostgreSQL 15 for the backup to work.
• Export backs up the content of a repository, not the machine: it is restored on a new instance, but contains no accounts, no remotes and no keys.
• An export produces two inseparable files, the archive and its -toc.json, and it is only worth anything once off the machine it protects.
• Import can only be launched through the API: the CLI does not expose it, and the response returns a task_group, not a task.

Knowledge check

Eight questions on what a backup must cover: the three places the data lives, the key that decides whether syncs survive, and the order that avoids dead links.

Knowledge check

Check what you have learned with this interactive quiz

8 questions

8 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →