Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

Kubernetes storage: PV, PVC, StorageClass and permissions

By updated

Improve this page

Logo Kubernetes

When a Pod disappears, everything it wrote disappears with it, unless the file is in a persistent volume. In this lesson, you declare that volume by hand, you request it with a claim, you mount it in a Pod, then you check where the data lives and what happens to it when the claim is deleted. Next you observe a volume created automatically, you fix a write refusal with fsGroup, and you learn to read a claim stuck in Pending.

What you will learn

  • Link Pod, PVC, PV and StorageClass, and say which one creates which
  • Write a static PersistentVolume, a claim that binds to it and a Pod that mounts it
  • Check that data written by the Pod is found on the node's disk
  • Choose an access mode, knowing that ReadWriteOnce limits access to one node, not one Pod
  • Decide on a reclaim policy, fix it on an existing volume and say what remains of the data
  • Tell static and dynamic provisioning apart, understand WaitForFirstConsumer and diagnose a claim in Pending
  • Set the permissions of a volume for a Pod that does not run as root

Prerequisites

You know how to create a Pod and read its events, as in Pods and Kubernetes resources. You have Docker, kubectl and kind 0.33 or later, installed as in First cluster. This lesson uses a four-node cluster, because storage depends on the node where the Pod runs. The commands were run with kind 0.33.0, kubectl 1.37.0 and nodes in v1.37.0.

Why is storage special in Kubernetes?

A Pod can be deleted, recreated and rescheduled on another node at any time. A file written inside the container does not follow that movement. Three questions follow: how to keep the data when the Pod disappears, how to make it available when the Pod changes node, and who decides what becomes of it when it is no longer needed.

Kubernetes answers with two families of volumes. Ephemeral volumes, such as emptyDir, live and die with the Pod. Persistent volumes, described by a PersistentVolume, exist independently of it.

The mental model: Pod → PVC → PV → StorageClass

Four objects each play a role. The Pod mounts a claim. The claim (PersistentVolumeClaim, PVC) expresses a need: a size and an access mode. The volume (PersistentVolume, PV) is the actual piece of storage. The StorageClass is the recipe that creates a PV on demand. The diagram shows the two paths: a PV prepared by hand by an administrator, or a PV created by a StorageClass.

Storage model: the user creates a 10Gi PVC in ReadWriteOnce, bound either to a static PV created by the admin or to a PV dynamically provisioned by a StorageClass; the PV represents a real volume on the storage backend, through CSI.User / developercreates a PVC (10Gi, ReadWriteOnce)PersistentVolumeClaim (PVC)“I want 10Gi in ReadWriteOnce”Staticexisting PV, created by the adminDynamica StorageClass provisions a PVPersistentVolume (PV)actual volume on the backendStorage backend, through CSI or in-treeAWS EBS, Ceph, NFS…
Source · SVG
Esc to close, + and − to zoom, scroll to move around. 100 %

This lesson prepares the first path by hand, then observes the second. The lesson on StorageClass details the recipe itself.

Prepare the cluster and the namespace

The demo-multi cluster has one control plane node and three worker nodes. The kind-multi.yaml file is the one from the lesson on kind: a control plane and three workers, labelled by zone.

Terminal window
mkdir -p ~/atelier-k8s/stockage && cd ~/atelier-k8s/stockage
cat > kind-multi.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
labels:
topology.kubernetes.io/zone: zone-a
- role: worker
labels:
topology.kubernetes.io/zone: zone-b
- role: worker
labels:
topology.kubernetes.io/zone: zone-c
EOF
kind create cluster --name demo-multi --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5 --config kind-multi.yaml

Creation took 29 seconds. The command prints progress lines, which we cut. The last useful line confirms the context:

Set kubectl context to "kind-demo-multi"

Wait until the four nodes are ready, then look at them:

Terminal window
kubectl wait --for=condition=Ready nodes --all --timeout=120s
kubectl get nodes -o wide
node/demo-multi-control-plane condition met
node/demo-multi-worker condition met
node/demo-multi-worker2 condition met
node/demo-multi-worker3 condition met
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
demo-multi-control-plane Ready control-plane 30s v1.37.0 172.18.0.3 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4
demo-multi-worker Ready <none> 14s v1.37.0 172.18.0.2 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4
demo-multi-worker2 Ready <none> 14s v1.37.0 172.18.0.4 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4
demo-multi-worker3 Ready <none> 14s v1.37.0 172.18.0.5 <none> Debian GNU/Linux 13 (trixie) 6.12.115+deb13.1-amd64 (amd64) containerd://2.3.4

Look at the STATUS column: the four nodes are Ready. With kind, each node is a Docker container. You can therefore read a node's disk with docker exec, which you will do to prove where the data is. Create the working namespace:

Terminal window
kubectl create namespace stockage-modele
namespace/stockage-modele created

The names and messages of this lesson are in French on purpose, so that the outputs stay those of the tested cluster: stockage-modele means "storage model", ephemere "ephemeral", temoin.txt "marker.txt", ecrivain "writer", lecteur "reader", ailleurs "elsewhere", donnees "data", statique "static", dynamique "dynamic", demo-manuel "demo-manual", donnee-du-pod "pod-data", demo-droits "demo-permissions", classe-inconnue "unknown-class", n-existe-pas "does-not-exist", sans-classe "no-class" and absent "missing". In the custom-columns output, the headers NOM and CLASSE mean "name" and "class". The echo message "bonjour" means "hello", and the retrait du PV : … s apres la commande de suppression de la PVC line means "PV removal: … s after the PVC deletion command".

emptyDir and hostPath: two volumes without a PersistentVolume

Before making data persistent, let us see what happens without a persistent volume. Application volumes, including emptyDir, are detailed in Application volumes. Here, we use them to see what persistence fixes.

emptyDir: temporary storage inside the Pod

An emptyDir is an empty directory, created when the Pod starts and deleted with it. The ephemere Pod writes a file into it, then we delete the Pod.

Terminal window
cat > pod-emptydir.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: ephemere
namespace: stockage-modele
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: temp
mountPath: /data
volumes:
- name: temp
emptyDir: {}
EOF
kubectl apply -f pod-emptydir.yaml
kubectl -n stockage-modele wait --for=condition=Ready pod/ephemere --timeout=60s
kubectl -n stockage-modele exec ephemere -- sh -c 'echo bonjour > /data/temoin.txt && cat /data/temoin.txt'
pod/ephemere created
pod/ephemere condition met
bonjour

The file exists. Delete the Pod, recreate it under the same name and list /data:

Terminal window
kubectl -n stockage-modele delete pod ephemere
kubectl apply -f pod-emptydir.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/ephemere --timeout=60s && kubectl -n stockage-modele exec ephemere -- ls -la /data
pod "ephemere" deleted from stockage-modele namespace
pod/ephemere created
pod/ephemere condition met
total 8
drwxrwxrwx 2 root root 4096 Nov 11 22:36 .
drwxr-xr-x 1 root root 4096 Nov 11 22:36 ..

Look for temoin.txt in the list: it is not there. The recreated Pod starts again with an empty directory. Deletion takes about thirty seconds: sleep ignores the stop signal, and Kubernetes waits for the grace period to end. Remove the Pod before continuing:

Terminal window
kubectl -n stockage-modele delete pod ephemere --wait=true
pod "ephemere" deleted from stockage-modele namespace

hostPath: access to the node's filesystem

The Application volumes lesson already introduced hostPath and its danger: the Pod reads the disk of the node that hosts it. On the persistence side, keep the other limit in mind. The data survives the Pod, but it stays tied to one node: a Pod rescheduled elsewhere does not find it. For persistent data, use a PersistentVolume, as in the sections that follow.

Static provisioning

With static provisioning, an administrator creates the PV by hand. You do it here with a local volume, which designates a directory on a specific node. This directory must exist beforehand: Kubernetes does not create it (see the troubleshooting section below). Create it on demo-multi-worker:

Terminal window
docker exec demo-multi-worker sh -c 'mkdir -p /mnt/demo-pv-statique && ls -ld /mnt/demo-pv-statique'
drwxr-xr-x 2 root root 4096 Nov 11 22:36 /mnt/demo-pv-statique

The PersistentVolume carries four useful pieces of information: its size, its access mode, its reclaim policy (see below) and the node where it lives, in nodeAffinity. Its storageClassName is only a label used to match it with a claim. No demo-manuel StorageClass object exists.

Terminal window
cat > pv-statique.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: demo-pv-statique
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: demo-manuel
local:
path: /mnt/demo-pv-statique
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- demo-multi-worker
EOF
kubectl apply -f pv-statique.yaml
kubectl get pv demo-pv-statique
persistentvolume/demo-pv-statique created
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
demo-pv-statique 1Gi RWO Retain Available demo-manuel <unset> 0s

Look at STATUS and CLAIM: the volume is Available, nobody has claimed it yet. Now create the claim, which asks for 1Gi in ReadWriteOnce with the same class label:

Terminal window
cat > pvc-statique.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: donnees-statique
namespace: stockage-modele
spec:
accessModes:
- ReadWriteOnce
storageClassName: demo-manuel
resources:
requests:
storage: 1Gi
EOF
kubectl apply -f pvc-statique.yaml
kubectl -n stockage-modele get pvc donnees-statique
kubectl get pv demo-pv-statique
persistentvolumeclaim/donnees-statique created
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-statique Bound demo-pv-statique 1Gi RWO demo-manuel <unset> 0s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
demo-pv-statique 1Gi RWO Retain Bound stockage-modele/donnees-statique demo-manuel <unset> 1s

The claim is Bound as soon as it is created, and the PV shows its owner in CLAIM. Binding is immediate because a compatible PV already existed: same class, enough size, compatible access mode. Mount the claim in a Pod named ecrivain:

Terminal window
cat > pod-statique.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: ecrivain
namespace: stockage-modele
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: donnees
mountPath: /data
volumes:
- name: donnees
persistentVolumeClaim:
claimName: donnees-statique
EOF
kubectl apply -f pod-statique.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/ecrivain --timeout=60s
kubectl -n stockage-modele get pod ecrivain -o wide
pod/ecrivain created
pod/ecrivain condition met
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ecrivain 1/1 Running 0 0s 10.244.1.4 demo-multi-worker <none> <none>

Look at the NODE column: the Pod runs on demo-multi-worker, although the manifest contains no nodeSelector. It is the PV's affinity that guided the scheduler. Write a file, then read it on the node's disk:

Terminal window
kubectl -n stockage-modele exec ecrivain -- sh -c 'echo donnee-du-pod > /data/temoin.txt'
docker exec demo-multi-worker cat /mnt/demo-pv-statique/temoin.txt
donnee-du-pod

The file written in the Pod is indeed in the node's directory. This is where persistence is proven: the data lives on the disk, not in the Pod. Delete the Pod, recreate it and read again:

Terminal window
kubectl -n stockage-modele delete pod ecrivain && kubectl apply -f pod-statique.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/ecrivain --timeout=60s && kubectl -n stockage-modele exec ecrivain -- cat /data/temoin.txt
pod "ecrivain" deleted from stockage-modele namespace
pod/ecrivain created
pod/ecrivain condition met
donnee-du-pod

The recreated Pod reads donnee-du-pod back. Compare with the emptyDir: same gesture, opposite result.

Access modes: who can access the volume?

A claim's access mode says how the volume can be mounted. There are four, and a PV only accepts some of them depending on its storage type:

ModeAbbreviationWhat it allows
ReadWriteOnceRWOread and write from a single node
ReadOnlyManyROXread-only from several nodes
ReadWriteManyRWXread and write from several nodes
ReadWriteOncePodRWOPread and write from a single Pod

The right-hand column matters: RWO is about the node, RWOP about the Pod. The RWX mode requires shared storage, such as a network file system. The local storage of kind does not offer it.

ReadWriteOnce means one node, not one Pod

An RWO volume is mounted on a single node at a time, but all the Pods of that node can use it. Start a second Pod that mounts the same claim, while ecrivain is running:

Terminal window
cat > pod-lecteur.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: lecteur
namespace: stockage-modele
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: donnees
mountPath: /data
volumes:
- name: donnees
persistentVolumeClaim:
claimName: donnees-statique
EOF
kubectl apply -f pod-lecteur.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/lecteur --timeout=60s && kubectl -n stockage-modele exec lecteur -- cat /data/temoin.txt
kubectl -n stockage-modele get pod -o wide
pod/lecteur created
pod/lecteur condition met
donnee-du-pod
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ecrivain 1/1 Running 0 2s 10.244.1.5 demo-multi-worker <none> <none>
lecteur 1/1 Running 0 1s 10.244.1.6 demo-multi-worker <none> <none>

The second Pod starts and reads the first one's data. Both are on demo-multi-worker: RWO did not stop them. Now force a third Pod onto another node with nodeSelector:

Terminal window
cat > pod-ailleurs.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: ailleurs
namespace: stockage-modele
spec:
nodeSelector:
kubernetes.io/hostname: demo-multi-worker2
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: donnees
mountPath: /data
volumes:
- name: donnees
persistentVolumeClaim:
claimName: donnees-statique
EOF
kubectl apply -f pod-ailleurs.yaml
sleep 10; kubectl -n stockage-modele get pod ailleurs
pod/ailleurs created
NAME READY STATUS RESTARTS AGE
ailleurs 0/1 Pending 0 10s

The Pod stays Pending. Read the cause in the Pod's events (only the end of kubectl -n stockage-modele describe pod ailleurs is shown):

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 10s default-scheduler 0/4 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

The event comes from the scheduler, not from the volume. It counts the refused nodes: one does not match the PV's affinity, another carries a taint (the control plane node), and two do not match the nodeSelector. The volume is tied to one node, and the Pod cannot go elsewhere. Delete the three Pods; the claim stays bound:

Terminal window
kubectl -n stockage-modele delete pod ailleurs lecteur ecrivain
kubectl -n stockage-modele get pvc donnees-statique
pod "ailleurs" deleted from stockage-modele namespace
pod "lecteur" deleted from stockage-modele namespace
pod "ecrivain" deleted from stockage-modele namespace
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-statique Bound demo-pv-statique 1Gi RWO demo-manuel <unset> 76s

To limit usage to a single Pod, even on the same node, the mode is ReadWriteOncePod. The Kubernetes documentation presents it as reserved for CSI volumes, those of storage drivers that follow the standard Container Storage Interface (see CNI, CSI and CRI). The local provisioner of kind accepts it anyway. A second Pod that mounts the same claim then stays Pending, with the event 3 node(s) unavailable due to PersistentVolumeClaim with ReadWriteOncePod access mode already in-use by another pod. On the security side, of the four modes, it is the only one that prevents a second consumer from reading or writing the volume.

Reclaim policies: what happens when the PVC is deleted?

A PV's reclaim policy says what becomes of the volume when its claim is deleted. Delete removes the PV and its content. Retain keeps the PV and the data, so that the administrator decides. The demo-pv-statique PV is in Retain. Delete its claim:

Terminal window
kubectl -n stockage-modele delete pvc donnees-statique
kubectl get pv demo-pv-statique
docker exec demo-multi-worker cat /mnt/demo-pv-statique/temoin.txt
persistentvolumeclaim "donnees-statique" deleted from stockage-modele namespace
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
demo-pv-statique 1Gi RWO Retain Released stockage-modele/donnees-statique demo-manuel <unset> 77s
donnee-du-pod

The PV is Released and CLAIM still names the old claim. The data is still there. Recreate the claim:

Terminal window
kubectl apply -f pvc-statique.yaml && sleep 5 && kubectl -n stockage-modele get pvc donnees-statique
persistentvolumeclaim/donnees-statique created
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-statique Pending demo-manuel <unset> 5s

The new claim stays Pending: a Released PV is not reassigned on its own. It keeps the name and the identifier (uid) of the old claim in its spec.claimRef field, and it reserves them. The choice is deliberate: an administrator must confirm that a new consumer may read the previous one's data. To confirm, remove the claimRef:

Terminal window
kubectl patch pv demo-pv-statique --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'
kubectl -n stockage-modele get pvc donnees-statique && kubectl get pv demo-pv-statique && docker exec demo-multi-worker cat /mnt/demo-pv-statique/temoin.txt
persistentvolume/demo-pv-statique patched
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-statique Pending demo-manuel <unset> 5s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
demo-pv-statique 1Gi RWO Retain Available demo-manuel <unset> 82s
donnee-du-pod

Right after the patch, the PV is Available, but the claim is still Pending. Binding is not instantaneous. Read again after 20 seconds:

Terminal window
sleep 20; kubectl -n stockage-modele get pvc donnees-statique && kubectl get pv demo-pv-statique
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-statique Bound demo-pv-statique 1Gi RWO demo-manuel <unset> 25s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
demo-pv-statique 1Gi RWO Retain Bound stockage-modele/donnees-statique demo-manuel <unset> 102s

The claim is Bound and the new consumer can read donnee-du-pod. With Retain, a reassigned volume gives the old data to the new consumer. Empty the directory before reusing a volume that has held sensitive data.

volumeBindingMode: when should the PVC bind to the PV?

Dynamic provisioning creates the PV on demand, thanks to a StorageClass. The class's volumeBindingMode field decides the timing: Immediate binds the claim as soon as it is created, WaitForFirstConsumer waits until a Pod mounts it, so that the volume is created on the node chosen by the scheduler. The kind cluster already has a default class:

Terminal window
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 25s

Look at VOLUMEBINDINGMODE and RECLAIMPOLICY: standard waits for a first consumer and deletes its volumes along with their claim. Create a claim without a class: it takes the default class.

Terminal window
cat > pvc-dynamique.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: donnees-dynamique
namespace: stockage-modele
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
kubectl apply -f pvc-dynamique.yaml && sleep 5 && kubectl -n stockage-modele get pvc donnees-dynamique
kubectl -n stockage-modele describe pvc donnees-dynamique
persistentvolumeclaim/donnees-dynamique created
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-dynamique Pending standard <unset> 5s
Name: donnees-dynamique
Namespace: stockage-modele
StorageClass: standard
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 0s (x2 over 5s) persistentvolume-controller waiting for first consumer to be created before binding

Pending is not a failure here. The event is of type Normal and it says what we are waiting for: waiting for first consumer to be created before binding. Create a Pod that mounts the claim:

Terminal window
cat > pod-dynamique.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: dynamique
namespace: stockage-modele
spec:
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: donnees
mountPath: /data
volumes:
- name: donnees
persistentVolumeClaim:
claimName: donnees-dynamique
EOF
kubectl apply -f pod-dynamique.yaml && date +%T && kubectl -n stockage-modele wait --for=condition=Ready pod/dynamique --timeout=90s && date +%T
kubectl -n stockage-modele get pvc donnees-dynamique
pod/dynamique created
22:38:23
pod/dynamique condition met
22:38:31
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-dynamique Bound pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 1Gi RWO standard <unset> 13s

The volume appeared with the Pod, in 8 seconds. Nobody wrote a PV. Look at what the provisioner created: the reclaim policy, the class and the volume's node.

Terminal window
kubectl get pv -o custom-columns=NOM:.metadata.name,RECLAIM:.spec.persistentVolumeReclaimPolicy,CLASSE:.spec.storageClassName,CLAIM:.spec.claimRef.name
kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"\n"}{end}'
NOM RECLAIM CLASSE CLAIM
demo-pv-statique Retain demo-manuel donnees-statique
pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 Delete standard donnees-dynamique
demo-pv-statique demo-multi-worker
pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 demo-multi-worker2

The dynamic PV is in Delete, a policy inherited from the standard class. It is tied to a node as well, demo-multi-worker2, the one of the first Pod. The kind provisioner created a directory on that node:

Terminal window
PV=$(kubectl -n stockage-modele get pvc donnees-dynamique -o jsonpath='{.spec.volumeName}'); kubectl get pv "$PV" -o jsonpath='hostPath={.spec.hostPath.path}{"\n"}local={.spec.local.path}{"\n"}'
hostPath=/var/local-path-provisioner/pvc-7e1017da-0381-4fc5-926b-2e0a402606d5_stockage-modele_donnees-dynamique
local=

The directory is under /var/local-path-provisioner, named after the PV, the namespace and the claim.

Change the reclaim policy of an existing PV

A dynamic volume is created in Delete: deleting its claim deletes the data. To keep it, change the PV's policy before deleting the claim. The reclaim policy of an existing PV is changed with kubectl patch, without stopping the Pod. First write a file, then switch the volume to Retain:

Terminal window
kubectl -n stockage-modele exec dynamique -- sh -c 'echo dyn > /data/temoin.txt'
PV=$(kubectl -n stockage-modele get pvc donnees-dynamique -o jsonpath='{.spec.volumeName}'); kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' && kubectl get pv "$PV"
persistentvolume/pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 patched
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 1Gi RWO Retain Bound stockage-modele/donnees-dynamique standard <unset> 4s

The RECLAIM POLICY column changes to Retain, and the Pod keeps running. Now delete the Pod, then the claim:

Terminal window
kubectl -n stockage-modele delete pod dynamique && kubectl -n stockage-modele delete pvc donnees-dynamique && sleep 5 && kubectl get pv
pod "dynamique" deleted from stockage-modele namespace
persistentvolumeclaim "donnees-dynamique" deleted from stockage-modele namespace
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
demo-pv-statique 1Gi RWO Retain Bound stockage-modele/donnees-statique demo-manuel <unset> 2m33s
pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 1Gi RWO Retain Released stockage-modele/donnees-dynamique standard <unset> 41s

The dynamic PV is Released, not deleted. Look on each node to see where the data is:

Terminal window
for n in demo-multi-worker demo-multi-worker2 demo-multi-worker3; do echo "## $n"; docker exec "$n" sh -c 'ls -l /var/local-path-provisioner/ 2>&1; cat /var/local-path-provisioner/*/temoin.txt 2>&1'; done
## demo-multi-worker
ls: cannot access '/var/local-path-provisioner/': No such file or directory
cat: '/var/local-path-provisioner/*/temoin.txt': No such file or directory
## demo-multi-worker2
total 4
drwxrwxrwx 2 root root 4096 Nov 11 22:38 pvc-7e1017da-0381-4fc5-926b-2e0a402606d5_stockage-modele_donnees-dynamique
dyn
## demo-multi-worker3
ls: cannot access '/var/local-path-provisioner/': No such file or directory
cat: '/var/local-path-provisioner/*/temoin.txt': No such file or directory

The directory and its dyn file are on demo-multi-worker2. The other two nodes have no directory: the error messages are normal. Now see the contrast with Delete: the same sequence, without the patch, with a stopwatch on the PV's disappearance.

Terminal window
kubectl apply -f pvc-dynamique.yaml && kubectl apply -f pod-dynamique.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/dynamique --timeout=90s >/dev/null && PV=$(kubectl -n stockage-modele get pvc donnees-dynamique -o jsonpath='{.spec.volumeName}') && echo "$PV" && kubectl -n stockage-modele delete pod dynamique && T0=$(date +%s.%N) && kubectl -n stockage-modele delete pvc donnees-dynamique && kubectl wait --for=delete pv/"$PV" --timeout=60s && T1=$(date +%s.%N) && echo "retrait du PV : $(awk -v a="$T0" -v b="$T1" 'BEGIN{printf "%.1f", b-a}') s apres la commande de suppression de la PVC"
persistentvolumeclaim/donnees-dynamique created
pod/dynamique created
pvc-6075098c-e86f-48be-b8f8-c0d2165f0d66
pod "dynamique" deleted from stockage-modele namespace
persistentvolumeclaim "donnees-dynamique" deleted from stockage-modele namespace
persistentvolume/pvc-6075098c-e86f-48be-b8f8-c0d2165f0d66 condition met
retrait du PV : 2.1 s apres la commande de suppression de la PVC

The PV disappeared 2.1 seconds after the claim. Its directory disappeared too:

Terminal window
for n in demo-multi-worker demo-multi-worker2 demo-multi-worker3; do echo "## $n"; docker exec "$n" sh -c 'ls -l /var/local-path-provisioner/ 2>&1'; done
## demo-multi-worker
total 0
## demo-multi-worker2
total 4
drwxrwxrwx 2 root root 4096 Nov 11 22:38 pvc-7e1017da-0381-4fc5-926b-2e0a402606d5_stockage-modele_donnees-dynamique
## demo-multi-worker3
ls: cannot access '/var/local-path-provisioner/': No such file or directory

On demo-multi-worker, which held the volume in Delete, /var/local-path-provisioner/ is empty (total 0). Only the directory of the volume kept in Retain remains, on demo-multi-worker2. Choose the reclaim policy before deleting the claim: with Delete, there is no way back.

Volume permissions: the security angle

A Pod that does not run as root can have a volume it cannot write to. Prepare a second static volume, whose directory belongs to root, with its claim:

Terminal window
cat > pv-droits.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolume
metadata:
name: demo-pv-droits
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: demo-droits
local:
path: /mnt/demo-pv-droits
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- demo-multi-worker
EOF
cat > pvc-droits.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: donnees-droits
namespace: stockage-modele
spec:
accessModes:
- ReadWriteOnce
storageClassName: demo-droits
resources:
requests:
storage: 1Gi
EOF
docker exec demo-multi-worker sh -c 'mkdir -p /mnt/demo-pv-droits && ls -ld /mnt/demo-pv-droits' && kubectl apply -f pv-droits.yaml -f pvc-droits.yaml && sleep 3 && kubectl -n stockage-modele get pvc donnees-droits
drwxr-xr-x 2 root root 4096 Nov 11 22:40 /mnt/demo-pv-droits
persistentvolume/demo-pv-droits created
persistentvolumeclaim/donnees-droits created
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
donnees-droits Bound demo-pv-droits 1Gi RWO demo-droits <unset> 3s

The directory belongs to root and only root can modify it (drwxr-xr-x). Start a Pod that runs with user ID 1000 and try to write:

Terminal window
cat > pod-non-root.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: non-root
namespace: stockage-modele
spec:
securityContext:
runAsUser: 1000
runAsNonRoot: true
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: donnees
mountPath: /data
volumes:
- name: donnees
persistentVolumeClaim:
claimName: donnees-droits
EOF
kubectl apply -f pod-non-root.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/non-root --timeout=60s && kubectl -n stockage-modele exec non-root -- sh -c 'echo test > /data/test.txt'
pod/non-root created
pod/non-root condition met
sh: line 0: can't create /data/test.txt: Permission denied
command terminated with exit code 1

The Pod starts, but the write is refused: Permission denied. The process is user 1000, in group 0, and the directory belongs to root with no write access for others. You already fixed a read refusal with fsGroup in Application volumes, on a ConfigMap file. The field plays the same role here on a persistent volume: it adds a group to the Pod and changes the directory's group. Add fsGroup: 1000 and recreate the Pod:

Terminal window
cat > pod-non-root-fsgroup.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
name: non-root
namespace: stockage-modele
spec:
securityContext:
runAsUser: 1000
runAsNonRoot: true
fsGroup: 1000
containers:
- name: app
image: busybox:1.38.0@sha256:a46257e06c82ad85242d0d38077d0a52d9896389ca60a1a59d3463db586e5fa4
command: ["sleep", "3600"]
volumeMounts:
- name: donnees
mountPath: /data
volumes:
- name: donnees
persistentVolumeClaim:
claimName: donnees-droits
EOF
kubectl -n stockage-modele delete pod non-root && kubectl apply -f pod-non-root-fsgroup.yaml && kubectl -n stockage-modele wait --for=condition=Ready pod/non-root --timeout=60s && kubectl -n stockage-modele exec non-root -- sh -c 'echo test > /data/test.txt && ls -ld /data'
pod "non-root" deleted from stockage-modele namespace
pod/non-root created
pod/non-root condition met
drwxrwsr-x 2 root 1000 4096 Nov 11 22:41 /data

The write succeeds. Look at the directory's group and permissions: /data now belongs to group 1000, with write access for that group (rws). See the result from the node:

Terminal window
kubectl -n stockage-modele exec non-root -- sh -c 'id; ls -ln /data'; docker exec demo-multi-worker ls -lnd /mnt/demo-pv-droits /mnt/demo-pv-droits/test.txt
uid=1000 gid=0(root) groups=0(root),1000
total 4
-rw-r--r-- 1 1000 1000 5 Nov 11 22:41 test.txt
drwxrwsr-x 2 0 1000 4096 Nov 11 22:41 /mnt/demo-pv-droits
-rw-r--r-- 1 1000 1000 5 Nov 11 22:41 /mnt/demo-pv-droits/test.txt

The change is visible on the node's disk. fsGroup modifies the directory's permissions on the node, and the change remains after the Pod is deleted. This is convenient for a non-root Pod, but a Pod of the same group can then read and modify the content. This setting only applies to volume types that support permission changes. The other security context fields are covered in Security Context.

Troubleshooting: why does my PVC stay Pending?

A PVC in Pending is not always a failure: with WaitForFirstConsumer, it is the normal state before the first Pod. When it lasts while a Pod mounts it, kubectl describe pvc gives the cause in its events.

Diagnostic checklist

What you seeLikely causeWhat to do
waiting for first consumer to be created before bindingclass in WaitForFirstConsumer, no Pod mounts the claimcreate the Pod
storageclass.storage.k8s.io "<name>" not foundthe claim names a class that does not existfix the class name or create the class
Pod in ContainerCreating, FailedMount event and path ... does not existlocal volume whose directory does not exist on the nodecreate the directory on the node
Pod in Pending, didn't match PersistentVolume's node affinitythe Pod cannot go to the PV's noderemove the nodeSelector or lift the constraint

For a local volume without a directory, the measured event is MountVolume.NewMounter initialization failed for volume "demo-pv-absent" : path "/mnt/demo-pv-absent" does not exist. The Pod is scheduled on the right node, then stays in ContainerCreating. A local volume does not create its directory: it is up to you to prepare it on the node.

Diagnostic commands

The second row of the table can be reproduced with a claim that names an unknown class:

Terminal window
cat > pvc-classe-inconnue.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: classe-inconnue
namespace: stockage-modele
spec:
accessModes:
- ReadWriteOnce
storageClassName: n-existe-pas
resources:
requests:
storage: 1Gi
EOF
kubectl apply -f pvc-classe-inconnue.yaml && sleep 5 && kubectl -n stockage-modele get pvc classe-inconnue && kubectl -n stockage-modele describe pvc classe-inconnue
persistentvolumeclaim/classe-inconnue created
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
classe-inconnue Pending n-existe-pas <unset> 5s
Name: classe-inconnue
Namespace: stockage-modele
StorageClass: n-existe-pas
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 1s (x2 over 5s) persistentvolume-controller storageclass.storage.k8s.io "n-existe-pas" not found

Read the last line: the event names the missing class. Fix the class, or create it. A claim without storageClassName does not have this problem: the server writes the default class into the claim.

Terminal window
cat > pvc-sans-classe.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: sans-classe
namespace: stockage-modele
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
EOF
kubectl apply -f pvc-sans-classe.yaml && kubectl -n stockage-modele get pvc sans-classe -o jsonpath='{.spec.storageClassName}{"\n"}'
persistentvolumeclaim/sans-classe created
standard

The claim received standard. It therefore no longer looks for a static PV of another class: it waits for a Pod to create a dynamic volume.

Clean up the cluster

Delete the namespace, then the PVs, which do not belong to a namespace.

Terminal window
kubectl delete namespace stockage-modele --wait=true --timeout=120s
kubectl delete pv demo-pv-statique demo-pv-droits --ignore-not-found
kubectl get pv
namespace "stockage-modele" deleted
persistentvolume "demo-pv-statique" deleted
persistentvolume "demo-pv-droits" deleted
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-7e1017da-0381-4fc5-926b-2e0a402606d5 1Gi RWO Retain Released stockage-modele/donnees-dynamique standard <unset> 4m3s

The dynamic PV kept in Retain remains: deleting the namespace does not delete it. Delete it, then the node directories, which Retain leaves in place:

Terminal window
kubectl get pv -o name | xargs -r kubectl delete
docker exec demo-multi-worker sh -c 'rm -rf /mnt/demo-pv-statique /mnt/demo-pv-droits'
docker exec demo-multi-worker2 sh -c 'rm -r /var/local-path-provisioner/pvc-*'

Keep the demo-multi cluster: the lesson on StorageClass uses it.

Put it into practice

The lab A persistent volume: PersistentVolume, PersistentVolumeClaim and a Pod that writes covers static provisioning. On a cluster without dynamic provisioning, you create a 1Gi lab-pv volume in ReadWriteOnce, backed by the /mnt/lab-data directory of a node, with the manual StorageClass. You claim it with lab-pvc for 500Mi, then mount it on /data in the data-pod Pod, which writes hello into test.txt. The validation checks the volume's capacity, access mode and class, the Bound binding, the mount and the file in the Pod. It also reads the same file back on the node's disk, because that is where persistence is proven. This lab runs on a kubeadm cluster made of virtual machines, not on kind: the node's disk is read there over ssh, and not with docker exec as in this lesson's demonstration.

Key takeaways

  • The Pod mounts a claim (PVC), which is bound to a volume (PV) created by hand or by a StorageClass.
  • Persistence is proven on the node's disk: the file stays, the Pod can disappear.
  • ReadWriteOnce limits access to one node, not one Pod; ReadWriteOncePod limits it to a single Pod.
  • A local volume is tied to a node, and its directory must exist before the Pod.
  • Delete removes the volume and its data along with the claim; Retain keeps them, and the PV becomes Released until its claimRef is removed.
  • The reclaim policy of an existing PV is fixed with kubectl patch pv, before deleting the claim.
  • A PVC in Pending is read with kubectl describe pvc: WaitForFirstConsumer is normal, storageclass ... not found is a class mistake.
  • fsGroup sets write access for a non-root Pod and changes the directory's group on the node.

Test your knowledge

The questions cover the scope of ReadWriteOnce, what becomes of data depending on the reclaim policy, and how to read a PVC in Pending.

Knowledge check

Check what you have learned with this interactive quiz

7 questions
5 min
80% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer