Deploying an NFS Server in K8S and exposing it externally

Question

I am in the process of setting up a NFS server on my K8S cluster. I want it to act as a NFS server for external entities i.e. client will be from outside the K8S cluster such as VMs.

The port requirements for the Docker image are :

==================================================================
      SERVER STARTUP COMPLETE
==================================================================
----> list of enabled NFS protocol versions: 4.2, 4.1, 4
----> list of container exports:
---->   /exports *(rw,no_subtree_check)
----> list of container ports that should be exposed:
---->   111 (TCP and UDP)
---->   2049 (TCP and UDP)
---->   32765 (TCP and UDP)
---->   32767 (TCP and UDP)

So I have created a Debian Stretch docker image. When I run it using docker run, I can successfully expose /exports and mount it from other systems.

docker run -v /data:/exports -v /tmp/exports.txt:/etc/exports:ro \
--cap-add SYS_ADMIN -p 2049:2049 -p 111:111 -p 32765:32765 \
-p 32767:32767 8113b6abeac

The above command spins up my docker container and when I do

mount.nfs4 <DOKCER_HOST_IP>:/exports /mount/

from another VM, I can successfully mount the volume.

So everything up until here is A OK!

Now the task is to deploy this in K8S.

My stateful-set definition is:

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: nfs-provisioner
spec:
  selector:
    matchLabels:
      app: nfs-provisioner
  serviceName: "nfs-provisioner"
  replicas: 1
  template:
    metadata:
      labels:
        app: nfs-provisioner
    spec:
      serviceAccount: nfs-provisioner
      terminationGracePeriodSeconds: 10
      imagePullSecrets:
      - name: artifactory
      containers:
        - name: nfs-provisioner
          image: repository.hybris.com:5005/test/nfs/nfs-server:1.2
          ports:
            - name: nfs
              containerPort: 2049
            - name: mountd
              containerPort: 20048
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
            - name: filenet
              containerPort: 32767
            - name: filenet-udp
              containerPort: 32767
              protocol: UDP
            - name: unknown
              containerPort: 32765
            - name: unknown-udp
              containerPort: 32765
              protocol: UDP
          securityContext:
            privileged: true
          env:
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: NFS_EXPORT_0
              value: '/exports *(rw,no_subtree_check)'
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: export-volume
              mountPath: /exports
      volumes:
        - name: export-volume
          hostPath:
            path: /var/tmp

As you can see, I have specified all the ports (both TCP and UDP)

And now to expose this to the outside world and not just inside the cluster, my service.yaml file deceleration is :

kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  type: NodePort
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
    - name: rpcbind-udp
      port: 111
      protocol: UDP
    - name: filenet
      port: 32767
    - name: filenet-udp
      port: 32767
      protocol: UDP
    - name: unknown
      port: 32765
    - name: unknown-udp
      port: 32765
      protocol: UDP
  selector:
    app: nfs-provisioner

This results in

kubectl get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                      AGE
nfs-provisioner   NodePort    10.233.43.135   <none>        2049:30382/TCP,20048:31316/TCP,111:32720/TCP,111:32720/UDP,32767:30173/TCP,32767:30173/UDP,32765:31215/TCP,32765:31215/UDP   32m

Now I try to mount /exports from another node/VM that is external to the K8S cluster.

I've tried

mount.nfs4 <K8S_Node_IP>:/exports /mount/

and I've tried

mount.nfs4 -o port=<NodePort> <K8S_Node_IP>:/exports /mount/

Ive tried each NodePort one at a time. But none of them work. I get the error :

mount.nfs4 -o port=31316 <K8S_Node_IP>:/exports /mount/
mount.nfs4: mount to NFS server '<K8S_Node_IP>:/exports' failed: RPC Error: Unable to receive

I'm unsure as to what might be the issue here. Is it that I need to specify all the nodePorts? If so, how can I do that?

Answer 1

The issue here is that all the NodePorts are different as seen externally as from:

---->   111 (TCP and UDP)
---->   2049 (TCP and UDP)
---->   32765 (TCP and UDP)
---->   32767 (TCP and UDP)

You can try an L4 load balancer that exposes exactly those ports on a given IP address (internal or external) and forwards them to the nodePorts (which is what type=LoadBalancer does too).

Another option is to hard code the NodePorts in your services to match exactly the ones of the containers:

kind: Service
apiVersion: v1
metadata:
  name: nfs-provisioner
  labels:
    app: nfs-provisioner
spec:
  type: NodePort
  ports:
    - name: nfs
      port: 2049
      nodePort: 2049
    - name: mountd
      port: 20048
      nodePort: 20048
    - name: rpcbind
      port: 111
      nodePort: 111
    - name: rpcbind-udp
      port: 111
      nodePort: 111
      protocol: UDP
    - name: filenet
      port: 32767
      nodePort: 32767
    - name: filenet-udp
      port: 32767
      nodePort: 32767
      protocol: UDP
    - name: unknown
      port: 32765
      nodePort: 32765
    - name: unknown-udp
      port: 32765
      nodePort: 32765
      protocol: UDP
  selector:
    app: nfs-provisioner

You will have to change the nodePort range (--service-node-port-range) on the kubelet though. This is so that you can use 2049 and 111.

You can also change the ports that you NFS server listens on for 2049 (nfs) and 111 (portmapper) for example, that way you don't have to change --service-node-port-range