Backup K3s with Litestream

This blog post will give you a tutorial on how to back up a K3s single node control plane backed by sqlite3 with Litestream.

K3s is a lightweight Kubernetes distribution targeting Edge workloads and small-scale Kubernetes setups. It bundles many core components like the kubelet, all control plane components, a container runtime, and flannel as container network implementation into a single binary. The binary can be used for running both, a control plane node and a worker node. Another core feature of K3s is the support of sqlite3 or other SQL databases instead of etcd as storage for the Kubernetes API state. Using a single master node with a sqlite3 instead of etcd has many advantages since it reduces the overall complexity of the system. The setup has fewer components one has to master. Especially in Edge workloads, you do not need a load balancer to distribute the traffic to multiple Kubernetes API server nodes. What is more, sqlite3 is a very well-tested and well-known data store with a very low resource footprint. The cost of this setup is reduced resilience since a single master system cannot automatically mitigate hardware failures, or failover to another node during planned maintenance. Kubernetes is designed as an eventually consistent system, so during a control plane node downtime, the existing workload runs without interruption, but no new workload can be scheduled or auto-scaling cannot happen. This is often a valid trade-off to make for small-scale Kubernetes setups.

Litestream can replicate a sqlite3 database to an object store, another file, or SFTP. Supported object store protocols are S3, Azure Blob Storage and Google Cloud Storage. Ben Johnson is the author of Litestream, he also wrote the key-value store boltdb, initially used by etcd. Litestream uses the write-ahead log (WAL) feature of sqlite3 to offer point-in-time restores. Therefore, Litestream upon start copies a snapshot of the sqlite3 database to the replication target and after that subsequently copies the WAL files into the replication target as well. The default synchronization interval for WAL files is 1 second. A more detailed introduction to the motivation behind Litestream and the inner mechanism can be found on the Litestream blog.

In the next few paragraphs, I want to show you an example setup of how to use sqlite3 to back up and restore a sqlite3 backed K3s control plane. To restore a K3s control plane of a running cluster, we only need the sqlite3 database and the K3S_TOKEN used for that cluster.

Step by Step Tutorial for a Backup with Litestream

This tutorial was written for Ubuntu 22.04. This should work for other Linux distributions as well, but you may need to use a different Litestream installation process. In this tutorial, we want to sync to a local minio installation. If you want to replicate this with another object store, skip the minio install and setup process. We start minio, via docker:

Install docker if not already installed:

sudo docker ls || sudo snap install docker

Install jq and pwgen if not previously installed, we use these helper tools during the tutorial:

sudo apt-get update
sudo apt-get install jq pwgen

Install a minio instance and create a bucket:

# Export credentials
export MINIO_USER=admin
export MINIO_PASSWORD=$(pwgen 16 1)
export MINIO_BUCKET_NAME=”k3s-backup”

# Start Minio Server
sudo -E sudo docker run -d --label k3s-tutorial=litestream -p 9000:9000 -p 9001:9001  -e "MINIO_ROOT_USER=${MINIO_USER}"  -e "MINIO_ROOT_PASSWORD=${MINIO_PASSWORD}" quay.io/minio/minio server /data --console-address ":9001"

# Get Host IP (first IPv4 which is not lo or docker0)
export MINIO_IP=$(ip -j a  | jq '.[] | select(.master!="docker0" and .ifname!="lo" and .ifname!="docker0") |.addr_info[] | select(.family=="inet") | .local ' -r)

# Save the MINIO URL
export MINIO_HOST=”http://${MINIO_USER}:${MINIO_PASSWORD}@${MINIO_IP}:9000”

# Create the bucket with the client tool
sudo docker run --label k3s-tutorial=litestream -e "MC_HOST_minio=${MINIO_HOST}" quay.io/minio/mc mb minio/${MINIO_BUCKET_NAME}

Install Litestream:

Shell
wget https://github.com/benbjohnson/litestream/releases/download/v0.3.9/litestream-v0.3.9-linux-amd64.deb
sudo dpkg -i litestream-v0.3.9-linux-amd64.deb

After we have installed litestream, we can create a litestream configuration file:

sudo -E sh -c "cat > /etc/litestream.yml" <<EOF
addr: ":9438"
dbs:
    # K3s stores its state database here:
  - path: /var/lib/rancher/k3s/server/db/state.db
    replicas:
      # Store a replica in an s3 compatible storage, here minio. Use the credentials, host and buckets we used while setting up minio
      - type: s3
        bucket: ${MINIO_BUCKET_NAME}
        path: state.db
        endpoint: http://${MINIO_IP}:9000
        access-key-id: ${MINIO_USER}
        secret-access-key: ${MINIO_PASSWORD}
        snapshot-interval: 1h

Install K3s via the official install script:

export K3S_TOKEN=$(pwgen 16 1)
echo $K3S_TOKEN > /tmp/k3s_token
# We set the kubeconfig world readable for easier steps in the tutorial, in a production setup, you shouldn’t do that
curl -sfL https://get.k3s.io |  sh -s - --write-kubeconfig-mode=644

Now, after we have installed K3s, we can start the Litestream replication:

Shell
sudo systemctl start litestream
sudo systemctl enable litestream

With the Litestream tool, we can check the snapshots stored in our minio storage:

litestream snapshots /var/lib/rancher/k3s/server/db/state.db
replica  generation        index  size    created
s3       625ada9e456b9d3b  3      602216  2022-09-20T11:31:35Z

We can see the data structure in minio bucket as well:

sudo docker run -e "MC_HOST_minio=${MINIO_HOST}" quay.io/minio/mc tree minio/${MINIO_BUCKET_NAME}
minio/k3s-backup
└─ state.db
   └─ generations
      └─ 625ada9e456b9d3b
         ├─ snapshots
         └─ wal

After we have verified that our backup stores data at the target location, we can test the restore process. The most crucial step to verify a backup solution is to test the restoring process. To have some example data in our backup, we create a Kubernetes deployment:

kubectl create deployment --image=nginx restore-test --replicas=2
deployment.apps/restore-test created

kubectl get deployments,pods
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/restore-test   0/2     2            0           7s

NAME                                READY   STATUS              RESTARTS   AGE
pod/restore-test-687fddf875-pdn84   0/1     ContainerCreating   0          7s
pod/restore-test-687fddf875-m4dgt   0/1     ContainerCreating   0          7s

Now, let us uninstall K3s:

# Run the uninstall script, K3s creates at installation time:    
/usr/local/bin/k3s-uninstall.sh

# Check if there is still data present:
sudo ls /var/lib/rancher/k3s/server/db/state.db
ls: cannot access '/var/lib/rancher/k3s/server/db/state.db': No such file or directory

Eventually, we have successfully uninstalled K3s and purged all data. The next step is to use Litestream to restore K3s sqlite database:

sudo litestream restore /var/lib/rancher/k3s/server/db/state.db

The next step is to reinstall K3s with the same K3S_TOKEN as before.

export K3S_TOKEN=$(cat /tmp/k3s_token)
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode=644

After the installation and restore, let’s check if our deployment is still there:

kubectl get pods,deployments
NAME                                READY   STATUS    RESTARTS   AGE
pod/restore-test-687fddf875-pdn84   1/1     Running   1          7m11s
pod/restore-test-687fddf875-m4dgt   1/1     Running   1          7m11s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/restore-test   2/2     2            2           7m11s

It is.
If you want to automate the Litestream restore process, the litestream restore command has two helpful flags:

-if-db-not-exists
    Returns exit code of 0 if the database already exists.

-if-replica-exists
    Returns exit code of 0 if no backups are found.

The first flag, “-if-db-not-exists“ will only restore the database if there is no database currently present. If there is already a database, the file is not overwritten. “-if-replica-exists“ will only restore, if there is a replica in the S3 bucket. Otherwise, it will return with code 0. This is helpful on first deployments. This allows restore the database before you install K3s:

litestream restore -if-db-not-exists -if-replica-exists /var/lib/rancher/k3s/server/db/state.db
export K3S_TOKEN=$(cat /tmp/k3s_token) 
curl -sfL https://get.k3s.io |  sh -s - --write-kubeconfig-mode=644

This allows you in, e.g., Edge Setups or setups where you handle the control plane as cattle to not make a distinction between the first install and subsequent starts/reinstallations.

Cleanup

Finally, let’s clean up our the K3s, litestream and minio installation:

# Uninstall K3s
/usr/local/bin/k3s-uninstall.sh
rm -rf /tmp/k3s_token

# Stop litestream process
sudo systemctl stop litestream
sudo systemctl disable litestream

# Uninstall litestream
sudo apt-get remove litestream

# Remove all containers created during this tutorial
sudo docker rm -f $(sudo docker ps -f label="k3s-tutorial=litestream" -q)

That’s it. We have set up a database-level backup for our K3s clusters. This can be handy in lab setups, allowing you to revert your cluster to 3h ago, or in smaller installations where you only have a single control plane machine running. Be aware that if you restore a database-level backup of a Kubernetes cluster used by multiple tenants, there is a chance that you do a rollback of their applications if the backup is too old. Therefore, you should prefer a GitOps style workflow for your disaster recovery workflows, or ensure you continuously backup your cluster’s sqlite database.