I wanted to dip my toes into Kubernetes for my homelab, but I knew I would need some flexibility to experiment. So instead of deploying k3s directly on my server, I

1. Installed a base Debian on my server, encrypting the disk with LUKS and using LVM to partition it.
2. Installed the Proxmox hypervisor on that base Debian
3. Spun up a Debian VM, and installed k3s on it.

Proxmox supports several storage plugins . It allows me to create LVM Local Volumes for the VM disks for example.

This setup allows me to spin up fresh VMs for my experiments, all while leaving my production k3s intact. This is great, but it came up with two problems:

1. When I provision the VM for k3s I need to allocate it a massive amount of disk space. This is because k3s uses a local path provisioner to provision new Persistent Volumes directly on the VM.
2. I can't take snapshots of the Persistent Volumes when doing backups. There's a risk that the data will change while I perform the backup.

The situation looks like the following.

On the LVM disk of the host, I create a VM for k3s. This VM has a virtual disk that doesn't rely on LVM, so it can't create LVM Logical Volumes. The local provisioner can only create volumes on the virtual disk, because it can't escape the VM to create volumes on the Proxmox host.

Because the volumes are created on the virtual disk that doesn't rely on LVM, I can't use LVM snapshots to take snapshots of my volumes.

[!question] Why not LVM Thin?

One solution to address the massive disk requirement could be to use LVM Thin : it would allow me to allocate a lot of space in theory, but in practice in only fills up as the VM storage gets used.

I don't want to use LVM Thin because it puts me at risk of overprovisioning. I could allocate more storage than I actually have, and it would be difficult to realize that my disks are filling up before it's too late.

My colleague Quentin mentioned the Proxmox CSI Plugin . It is a plugin that replaces k3s' local path provisioner. Instead of creating the kubernetes Persistent Volumes inside the VM, it calls the Proxmox host, asks it to create a LVM Logical Volume and binds it to a Persistent Volume in kubernetes.

Using the Proxmox CSI volume, the situation would look like this.

It solves the two problems for me:

1. I can now only provision a small disk for the k3s VM, since the Persistent Volumes will be created outside of the VM.
2. Since Proxmox will create LVM Logical Volumes to provision the Persistent Volumes, I can either do a LVM Snapshot from Proxmox or use Kubernete's Volume Snapshot feature, with some caveats .

Setting up the Proxmox-CSI-Plugin for k3s can be a bit involved, but I'm writing a longer blog post about it.