Tuesday, October 22, 2019

Kubernetes NFS encrypted communication: Kubernetes pod applications (as NFS client) and Linux based machine (as NFS server) – secure traffic using Tunnel Over SSH

As we all know, to encrypt NFS share traffic b/w NFS client and NFS server the couple of options are used in general are Kerberos Authentication with privacy (krb5p) Or Tunnel over SSH known as port forwarding.

This article I am going to discuss about the option of Tunnel over SSH with Kubernetes pods application which mount the shard path from the NFS server. In general, Tunnel over SSH implementation is common and easy to implement for the scenarios of port forwarding between two machines NFS server and NFS server. This machines can be either windows or Linux or combination of both.

The challenging part comes into picture for the scenarios with Kubernetes cluster in place and when your NFS clients wants to mount the NFS server shared path into a Kubernetes application. The reason why it’s challenging is because Kubernetes pods does not mount the shared path directly instead it depends on cluster “Persisted Volume Claims” and this would raise a request resource to the “Persistent volume” of the cluster. 

1. RHEL – Linux master as NFS server
2. RHEL – Linux node as NFS client and also maintaining running pods and providing the Kubernetes runtime environment.

A share with name “ NFS_Senstive_Data_Share” will be created in NFS server and which will be accessed from an Kubernetes pod application as an mounted path.

Before we start into implementation, would like to give quick explanation of how tunnel over SSH works with a sample in short.

ssh -fNv -c aes192-ctr -L 2049: SERVICEUSER@NFSServerIP sleep 365d

The above command runs in NFS client takes any traffic directed at NFS client's local port 2049 just forwards it, first through SSHD on the remote server (NFS server), and then on to the remote server's(NFS Server) port 2049. This port forwarding can run as background process which can be running in defined long periods. The user session b/w NFS client and NFS Server will be created by the SSH Session Key pair (RSA public & private keys) and login will happen through the key files instead of typing passwords.

Hoping it would have given a basic understanding of how Tunnel over SSH port forwarding work.

Lets move into the implementation:

Configuring NFS Server and NFS client

Now the Tunnel over SSH successfully enabled, all incoming traffic to NFS client ports will be forwarded to NFS server ports through SSHD.

Few points to notice in above commands
Aes256 – forward forwarding uses AES 256 cryptography algorithm
-f - which makes the port forwarding to run in background ssh persists until you explicitly kill it with the Unix kill command.

Now let's configure the Kubernetes

Configuring Kubernetes persistent volume and claims

That’s all, now just deploy this pod and K8s PV volume files. Once deployment done, a persistent volume within K8s with Tunnel over SSH enabled mount will be created in NFS client (linux node)

Let’s verify things :

First, lets verify the PV volume mount is created in the NFS client (linux node)

[root@NFSClient ~]# mount | grep nfs

You would get an output like

localhost:/NFS_Senstive_Data_Share on /var/lib/kubelet/pods/794ea09e-0354-436d-9498-6038f352e64c/volumes/kubernetes.io~nfs/nfs-pvclaim-sensitivedata type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=

and also verify SSH Tunnel is active using below command

sudo lsof -i -n | egrep '\<ssh\>'

Second, let’s try to access the volume mount path inside Kubernetes pods.

[root@NFSServer ~]# kubectl exec -it nfs-in-a-pod -n myproductNamespace -- sh
[root@NFSServer ~]# cd /mnt
[root@NFSServer ~]# ls  ------ here you can see the files inside the NFS shared folder.

That’s all, now the volume mount is created inside Kubernetes POD and the traffic between NFS Server (Linux Mode) and NFS Client (Linux node or K8s pods) are ENCRYPTED !!!