Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 78 ยท Storage

Running Databases and Backups

Decide when to run a database on Kubernetes, deploy it with a StatefulSet and PVC or an operator, and protect it with Velero, snapshots and tested restores.

Kubernetes was built for stateless services that can be killed and recreated at will. Databases are the opposite: they hold the one copy of your data, they care about identity and disk, and a mistake is expensive. Can you run them on Kubernetes anyway? Yes, many teams do, but it is a decision with trade-offs, and it comes with a non-negotiable duty: backups that you have actually restored.

This lesson helps you decide when a database belongs in the cluster, shows a minimal StatefulSet with persistent storage, introduces database operators, and then covers backups with volume snapshots and Velero. It builds on StatefulSets, PersistentVolumes and StorageClasses. The manifests are read-only (the site has no cluster), and two Python stand-ins let you play with retention and recovery-point maths.

Run it yourself, or rent it?

Managed database Backups, patches and failover are someone else's job.
Operator in cluster Automation for failover and backups, yours to run.
Hand-made StatefulSet Maximum control, maximum homework.

Choose the least work that still meets your needs.

1 / 2

Should the database run on Kubernetes?

Honest reasons to say no: a managed service (RDS, Cloud SQL, Azure Database and others) gives you backups, point-in-time recovery, patching, failover and monitoring for a fee, and your team may be small. A database failure at 3 a.m. is also a poor time to learn how your storage driver behaves.

Honest reasons to say yes: you need the same tooling in every environment (laptop, staging, production, on-premises), you want a database per tenant or per pull request, the managed price is too high at your scale, or you run where no managed option exists. Operators (below) have made this far more reliable than it was.

A common compromise: run databases for development and test environments inside the cluster, and use a managed service for production data, until you have a team that knows how to operate the stateful side.

A minimal StatefulSet database

A StatefulSet gives each replica a stable name (db-0) and, through volumeClaimTemplates, its own claim that sticks to it across restarts.

YAML
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None            # headless, for stable DNS names
  selector:
    app: db
  ports:
    - port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgres
          image: postgres:17
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - containerPort: 5432
          resources:
            requests:
              cpu: 500m
              memory: 1Gi
            limits:
              memory: 1Gi
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "postgres"]
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 20Gi

Points worth noticing:

  • PGDATA is a subdirectory of the mount, because a fresh disk has a lost+found folder at its root that makes Postgres refuse to initialize.
  • A generous terminationGracePeriodSeconds lets the database shut down cleanly instead of being killed mid-write (pod lifecycle).
  • Memory request equals the limit, so memory use cannot balloon past what the scheduler reserved (requests and limits).
  • Deleting a StatefulSet does not delete its PVCs. That is deliberate and protective, but it also means leftover claims keep costing money.
  • This sets up one database. Replication, failover and upgrades are on you.

Set the PV reclaim policy to Retain for such storage, so an accidental kubectl delete pvc or namespace removal does not erase the disk.

Operators for databases

Running a replicated database well needs knowledge: electing a new primary, rebuilding a replica, rolling a major-version upgrade, rotating certificates, taking base backups plus continuous log archives. An operator packages that knowledge as a controller you teach with a custom resource (custom resources and operators). For example, with the CloudNativePG operator, a three-node Postgres cluster looks like this:

YAML
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: shop-db
spec:
  instances: 3
  storage:
    size: 20Gi
    storageClass: fast-ssd

The operator creates the pods, claims and services, promotes a replica when the primary dies, and can be configured to ship backups and write-ahead logs to object storage for point-in-time recovery. Other well-known operators cover MySQL (Percona, MySQL Operator), MongoDB, Redis, Kafka (Strimzi), Elasticsearch and more. Check how mature and how actively maintained an operator is before betting production data on it, and read its backup and restore documentation first.

Backups: three layers

Think of backups in layers, each protecting against a different failure:

  1. Replication protects against a node or zone failure. It is not a backup: a replicated DROP TABLE is dropped everywhere.
  2. Volume snapshots (see StorageClasses) are fast, cheap copies of the disk. They are crash-consistent: restoring one is like restarting after a power cut. Databases survive that, but for real consistency you should flush or freeze the database first.
  3. Logical or physical backups plus logs, such as pg_dump, mysqldump or base backups with continuous WAL archiving, copied to storage outside the cluster. These survive losing the whole cluster, and the log archive lets you restore to any point in time.

A scheduled job can take logical dumps:

YAML
apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-dump
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: dump
              image: postgres:17
              command: ["sh", "-c", "pg_dump -h db -U postgres shop | gzip > /backup/shop-$(date +%F).sql.gz"]
              env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: db-credentials
                      key: password
              volumeMounts:
                - name: backup
                  mountPath: /backup
          volumes:
            - name: backup
              persistentVolumeClaim:
                claimName: backup-volume

That writes to another claim, which is better than nothing but still lives in the same cluster. In real setups the dump is uploaded to object storage in another account or region. More on cron schedules in Jobs and CronJobs.

Velero: backing up the cluster

Velero is a widely used open-source tool that backs up Kubernetes objects (Deployments, Services, ConfigMaps, claims...) to object storage and can snapshot or copy the volume data too. It also moves workloads between clusters. You normally install the Velero CLI, point it at a bucket, and use schedules:

YAML
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: nightly
  namespace: velero
spec:
  schedule: "0 3 * * *"
  template:
    includedNamespaces:
      - shop
    snapshotVolumes: true
    ttl: 720h0m0s
Terminal
velero backup create before-upgrade --include-namespaces shop
velero backup get
velero restore create --from-backup before-upgrade

To make the data consistent, annotate the pod so Velero runs hooks around the snapshot:

YAML
metadata:
  annotations:
    pre.hook.backup.velero.io/container: postgres
    pre.hook.backup.velero.io/command: '["/bin/sh","-c","psql -U postgres -c CHECKPOINT"]'

Velero's ttl expires old backups automatically. You also need your own thinking about retention: how many daily, weekly and monthly copies do you keep? The stand-in below applies a classic grandfather-father-son rotation.

Python Backup rotation: keep dailies, weeklies and monthlies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The seven newest nights are kept as dailies, the newest backup from each of the last four weeks and the last three months is kept as well, and everything else is pruned. With these defaults roughly 3 months of history costs about a dozen backups instead of one hundred.

Measuring recovery: RPO and RTO

Two numbers define a backup strategy. The recovery point objective (RPO) is how much data you can afford to lose, measured in time: with nightly backups and a failure just before the next one, you lose almost 24 hours. The recovery time objective (RTO) is how long you can be down while restoring. Choosing a schedule is really choosing an RPO; continuous log archiving can lower it to seconds.

Python Recovery point, data loss and a restore drill

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the newest backup (n-03) failed, you fall back to n-02, 35.5 hours old, which breaks the 24-hour target. The warning at the end is the lesson: a backup system needs alerts on failed jobs and regular restore drills, ideally automated into a scratch namespace, with a note of how long they take.

A checklist

  • Backups are stored outside the cluster, in another account or region, with access controls.
  • Failed backup jobs raise an alert; backup age is monitored.
  • A restore is rehearsed at least quarterly, and the time it takes is written down.
  • PVs holding data use Retain; namespaces and claims are protected from casual deletion.
  • Passwords and encryption keys needed to read a backup are stored separately from the backup itself.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is database replication not a substitute for backups?

    Replicas protect against hardware failure, not human error. Only backups taken at earlier points in time let you go back before the mistake.
  2. 2. Why does the Postgres example set PGDATA to a subdirectory of the mounted volume?

    The initdb step needs an empty data directory. The lost+found folder on a fresh ext4 disk is not empty, so the data goes one level deeper.
  3. 3. A StatefulSet is deleted. What happens to its PersistentVolumeClaims?

    Claims created from volumeClaimTemplates are kept when the StatefulSet is deleted, to avoid accidental data loss. You remove them explicitly.
  4. 4. What does a nightly backup schedule imply for the RPO?

    If the failure happens right before the next run, everything written since the last backup is gone. Lower RPO needs more frequent backups or continuous log archiving.
  5. 5. Which statement about volume snapshots of a running database is most accurate?

    A snapshot captures the disk at one instant, like a power cut. Databases can recover from that, but pre-snapshot hooks or a checkpoint give a cleaner copy.

Summary

  • Running a database on Kubernetes is a trade-off; managed services remove operational work, operators automate much of it, and a hand-made StatefulSet is the most work.
  • A StatefulSet with volumeClaimTemplates gives each replica a stable identity and its own disk; set Retain and generous termination time for data volumes.
  • Operators encode failover, backup and upgrade knowledge for databases such as Postgres, MySQL, MongoDB and Kafka.
  • Protect data in layers: replication for availability, snapshots for speed, dumps or base backups plus logs stored off-cluster for survival and point-in-time recovery.
  • Velero backs up cluster objects and volume data on a schedule, with TTL-based expiry and hooks for consistency.
  • Define RPO and RTO, alert on failed backups, and rehearse restores.

Next: How the Scheduler Works