Task:
Provide dynamic Persistent Volume Claims (PVCs) support to a NFS 4 server/host.
NFS Server: (NFS 4.1)
Server: mwnfs01.mindwatering.net (10.0.99.101)
Export base path: /local/storage
Steps:
Notes:
- The nfs-provisioner name mapping: nfs-provisioner
- The server and export path used below, are from NFS server information above
1. Create privileged namespace, nfs-system:
- Create namespace:
$ oc new-project nfs-system --description="NFS System"
<view success and switch to new project>
- Update nfs-system pod.security to allow privileged usage:
$ oc label ns/nfs-system pod-security.kubernetes.io/enforce=privileged --overwrite=true
<confirm success>
$ oc label ns/nfs-system pod-security.kubernetes.io/warn=privileged --overwrite=true
<confirm success>
$ oc label ns/nfs-system pod-security.kubernetes.io/audit=privileged --overwrite=true
<confirm success>
- Update the Security Context Constraint SCC (system:serviceaccount) for NSF usage:
$ oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:nfs-system:nfs-provisioner
<confirm success: clusterrole.rbac.authorization.k8s.io/system.openshift:scc:hostmount-anyuid added: "nfs-provisioner">
2. Create NFS Provisioner:
- Modify the NFS Provisioner for our NFS server:
$ cd ~/tmp/
$ vi mwfs01-provisioner.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
namespace: nfs-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nfs-provisioner-runner
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: run-nfs-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
namespace: nfs-system
roleRef:
kind: ClusterRole
name: nfs-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: leader-locking-nfs-provisioner
namespace: nfs-system
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: leader-locking-nfs-provisioner
namespace: nfs-system
subjects:
- kind: ServiceAccount
name: nfs-provisioner
namespace: nfs-system
roleRef:
kind: Role
name: leader-locking-nfs-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-provisioner
namespace: nfs-system
spec:
replicas: 1
selector:
matchLabels:
app: nfs-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccountName: nfs-provisioner
containers:
- name: nfs-provisioner
image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-provisioner
- name: NFS_SERVER
value: mwnfs01.mindwatering.net
- name: NFS_PATH
value: /local/storage
volumes:
- name: nfs-client-root
nfs:
server: mwnfs01.mindwatering.net
path: /local/storage
- Create the provisioner entry:
$ oc apply -f mwfs01-provisioner.yaml
<confirm success>
3. Create the Storage Class:
Notes:
- Set pathPattern with the naming convention of <project (namespace)>/<PVC name>
- Set archiveOnDelete to true to keep after PVCs are deleted/unbound, set to false to auto-clean-up
- The name of our storage class is nfs-dynamic; we'll use this for our PVCs.
- This storage class deletes when PVCs are unbound/deleted, and does NOT allow volume to be expanded/disk increased (plan accordingly)
- Create the storage class setting it to use the nfs-provisioner just added and the naming convention
$ vi mwnfs01-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-dynamic
provisioner: nfs-provisioner
parameters:
archiveOnDelete: "false"
pathPattern: "${.PVC.namespace}/${.PVC.name}"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: false
mountOptions:
- hard
- nfsvers=4.1
- noatime
- nodiratime
- Create the storage class:
$ oc apply -f mwnfs01-sc.yaml
<confirm success>
- Verify:
$ oc get storageclass/nfs-dynamic
<review>
4. Create a PersistentVolumeClaim for a Domino Server:
Notes:
- Ensure to use the storageClassName: nfs-dynamic
- Update the name, project/namespace, and accessMode (ReadWriteMany vs ReadWriteOnce) for your app
- Use the app's namespace for the PVC, not nfs-system; example below assumes the mwdom namespace already exists.
$ vi mwdom01-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mwdom01-local-path-pvc
namespace: mwdom
spec:
accessModes:
- ReadWriteOnce
storageClassName: nfs-dynamic
resources:
requests:
storage: 20Gi
4b. Review Config:
- Check OKD/OCP 4:
$ oc get pvc
<review and confirm new pvc bound, e.g. mwdom01-local-path-pvc Bound ... >
- Verify on NFS Server:
$ sudo su -
<enter pwd>
# ls -l /local/storage/
<confirm one folder nfs-system - the name of the namespace>
# ls -l /local/storage/nfs-system/
<confirm one folder mwdom01-local-path-pvc>
# ls -l /local/storage/nfs-system/mwdom01-local-path-pvc/
<folder will be empty until the mwdom01 server/app is deployed>
4c. Deploy Domino's custom ConfigMap and autoconfig deployment YAML (see other MW Support Reference support doc with these)
$ oc apply -f mwdom01-configmap.yml
<confirm successful, or in our case, it already exists, e.g. configmap/mwdom01-cfg unchanged>
$ oc apply -f mwdom-autoconfig.yml
<confirm success>
---
Troubleshooting:
- View logs:
$ oc logs -n nfs-system deployment/nfs-provisioner
<view logs>
- View app's volume usage in human readable:
$ oc exec -it <pod-name> -- df -h /mount/path
- View app's view of NFS mount options:
$ oc exec -it <pod-name> --mount | grep nfs
<view config output>
- Verify mounted NFS export seen:
$ oc exec -n nfs-system deployment/nfs-provisioner test -d /persistentvolumes
<verify>
- View PVC details and recent events:
$ oc describe pvc/<pvc-name>
<view details>
- Verify NFS server by deploying a temporary busybox to confirm connectivity:
$ oc run --rm debug --image=busybox --restart=Never -- nc -zv mwnfs01.mindwatering.net 2049
> ...
- Scaling the Storage Class mountOptions:
Timeout:
timeo=300 (10ths of seconds, so 30 seconds would be 300)
Client file attributes caching time:
actimeo=60 (seconds, 60 = 60 seconds)
Read and write sizing:
rsize=32768 (bytes, 32k is default, increasing to 64k (rsize=65536) or 1024k (rsize=1048576), up to network limit)
wsize=32768 (bytes, 32k is default, increasing to 64k (wsize=65536) or 1024k (wsize=1048576), up to network limit)
Retransmitting (lost packet support):
retrans=2 (number of times a timeout and retry occurs before a "major timeout" occurs; default is retrans=3)
previous page
|