NFS subdir external provisioner (henceforth NFS SEP) is an automatic provisioner, that use your existing and already configured NFS server to support dynamic provisioning of Kubernetes PV via PVC.
Prerequisites
As just mentioned, it is necessary to install on the VMs, both servers and clients that are part of the NFS, the packages necessary for the correct functioning of the NFS (usually these packages are already installed by default in common operating systems). Then insert the client list in the /etc/exports file and update the service
# Install the below package for NFS server on RedHat distro
$ sudo dnf install -y nfs-utils
# Make sure the service is active on all VMs
$ systemctl status nfs-server
# Edit /etc/exports by inserting the exported directory, the IPs of the clients and how to access the exported directory
/<path>/<exported_dir> <IP_client1>(ro,sync,no_root_squash,no_all_squash)
/<path>/<exported_dir> <IP_client2>(ro,sync,no_root_squash,no_all_squash)
# Update the service
$ sudo exportfs -rav
How to deploy NFS SEP
Install the tool using Helm
$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
$ helm repo update
$ helm show values nfs-subdir-external-provisioner/nfs-subdir-external-provisioner > values.yaml
$ helm install <chart_name> nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --values values.yaml --namespace <namespace> --create-namespace
$ helm uninstall <chart_name> -n <namespace>
# It's possible to proceed with the installation in a classic way, through manifest, by downloading the git repository
$ git clone https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git
Before installation, let's see how to configure our chart via the values.yaml file. The basic parameters are nfs.server and nfs.path.
replicaCount: 1
strategyType: Recreate
image:
repository: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner
tag: v4.0.2
pullPolicy: IfNotPresent
imagePullSecrets: []
nfs:
server: <server_IP>
path: <exported_path>
mountOptions:
volumeName: nfs-external-provisioner
# Reclaim policy for the main nfs volume
reclaimPolicy: Delete
# For creating the StorageClass automatically:
storageClass:
create: true
# Set a provisioner name. If unset, a name will be generated.
# provisionerName:
# Set StorageClass as the default StorageClass. Ignored if storageClass.create is false
defaultClass: false
# Set a StorageClass name. Ignored if storageClass.create is false
name: nfs-sc
# Allow volume to be expanded dynamically
allowVolumeExpansion: true
# Method used to reclaim an obsoleted volume
reclaimPolicy: Delete
# When set to false your PVs will not be archived by the provisioner upon deletion of the PVC.
archiveOnDelete: false
# If it exists and has 'delete' value, delete the directory. If it exists and has 'retain' value, save the directory.
# Overrides archiveOnDelete. Ignored if value not set.
onDelete:
# Specifies a template for creating a directory path via PVC metadata's such as labels, annotations, name or namespace. Ignored if value not set.
pathPattern: ${.PVC.namespace}-${.PVC.name}
# Set access mode - ReadWriteOnce, ReadOnlyMany or ReadWriteMany
accessModes: ReadWriteMany
# Set volume bindinng mode - Immediate or WaitForFirstConsumer
volumeBindingMode: Immediate
# Storage class annotations
annotations: {}
leaderElection:
# When set to false leader election will be disabled
enabled: true
## For RBAC support:
rbac:
# Specifies whether RBAC resources should be created
create: true
# If true, create & use Pod Security Policy resources
# https://kubernetes.io/docs/concepts/policy/pod-security-policy/
podSecurityPolicy:
enabled: false
# Deployment pod annotations
podAnnotations: {}
## Set pod priorityClassName
# priorityClassName: ""
podSecurityContext: {}
securityContext: {}
serviceAccount:
# Specifies whether a ServiceAccount should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the ServiceAccount to use. If not set and create is true, a name is generated using the fullname template
name:
resources: {}
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
nodeSelector: {}
tolerations: []
affinity: {}
# Additional labels for any resource created
labels: {}
podDisruptionBudget:
enabled: false
maxUnavailable: 1
How to use PVC into NFS SEP
Let's quickly see what happens when a PVC is created, both in read-only and read-write mode.
Read-Only mode
The steps to follow in this case are:
- create a PVC (the created PVC will be in pending state, because the provisioner does not have write permissions);
- create a directory, inside the
<exported_path>, according to the pathPattern parameter (see values.yaml above); - once the PVC is in bound state, it can be used.
Read-Write mode
The steps to follow in this case are:
- create a PVC: this will create a directory inside the
<exported_path>, according to the pathPattern parameter, and the corresponding PV; - use the newly created PVC.
Example
Let's try the following example, working in the nfs-test namespace. Once the provisioner is implemented, we create a PVC
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim
spec:
storageClassName: nfs-sc # Use the name of the SC created
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 5Mi
The PVC will remain in the pending state until the folder is created within the <exported_path>, according to the pathPattern parameter. In our case, the folder name must be equal to <namespace>-<PVC_name>, that is nfs-test-test-claim. In read-write mode, the folder is created automatically.
$ k apply -f test-claim.yaml -n nfs-test
# The PVC is in pending state
$ k get pvc -n nfs-test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-claim Pending nfs-sc 60s
# Create the folder and a test file inside it
$ mkdir <exported_path>/nfs-test-test-claim
$ echo "test" > <exported_path>/nfs-test-test-claim/file_test.txt
# PVC is in bound state now
$ k get pvc -n nfs-test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/test-claim Bound pvc-10663683-eece-46ca-b83d-53f88a1cabc9 5Mi ROX nfs-sc 4m3s
Use PVC in a trial application
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mystate
spec:
serviceName: mysvc
selector:
matchLabels:
app: myapp
replicas: 3
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: mycontainer
image: nginx
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 80
volumeMounts:
- name: mydata
mountPath: /usr/share/nginx/html
# Create another PVC to mount a different folder inside the Pod
# - name: mydata2
# mountPath: /data
volumes:
- name: mydata
persistentVolumeClaim:
claimName: test-claim
# - name: mydata2
# persistentVolumeClaim:
# claimName: test-claim2
Create the Pods and verify that the volume is properly mounted within them
$ k apply -f trial_app.yaml -n nfs-test
# Enter one of the created replicas
$ k exec -it -n nfs-test mystate-0 -- bash
# Check the contents and permissions of the folder on which the volume is mounted
root@mystate-0:/# cat /usr/share/nginx/html/file_test.txt
test
root@mystate-0:/# echo "test2" > /usr/share/nginx/html/file_test.txt
bash: /usr/share/nginx/html/file_test.txt: Read-only file system
Limitations and pitfalls
The software is subject to some limitations, listed below:
- The storage space provided is not guaranteed: you can allocate more than the total shared size via NFS, because there is no check for it.
- The provisioned storage limit is not enforced. The application can expand to use all the available storage regardless of the provisioned size.
- Storage resize/expansion operations are not presently supported in any form. You will end up in an error state:
Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. - The read-write permissions are not governed by the
access-mode parameter, but by the settings used in the NFS configuration.