Enable SRIOV - Kernel Driver

SR-IOV WorkerNode Prerequisites

In order for the cni-sriov-plugin to start reading Virtual Functions resources as well HostNetworkTemplate objects can configure virtual functions you need to perform the following steps.

1. Enable VT-D at BIOS level.
2. Enable SRIOV feature globally and/or per nic according to the nic you are working with, for example Intel x710 should be enabled per nic at BIOS level.
3. Upgrade Kernel boot line to enable intel_iommu=on and iommu=pt

$ grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on"
$ reboot

Huge page Support

Most network applications use huge pages, so you may want to enable that. Please edit /etc/default/grub and add huge-pages

GRUB_CMDLINE_LINUX="nofb nomodeset vga=normal iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16"
#Rebuild grub.cfg
grub2-mkconfig -o /boot/grub2/grub.cfg && reboot

Create SR-IOV ConfigMap

SR-IOV ConfigMap

Please copy paste the following definition update the values of your Physical Functions and ranges or Virtual Functions as well as Drivers and apply it to our cluster, in case that you have multiple nics, that each one uses different kernels drivers please omit the drivers section and just mentioned pfName within the selectors section.

SR-IOV - Kernel Driver ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: default
data:
  config.json: |
    {
    	"resourceList": [{
    			"resourceName": "intel_sriov_kernel0",
    			"selectors": {
    				"pfNames": ["em1#0-6"],
    				"drivers": ["igbvf"]
    			}
    		}
    	]
    }

Create Network Attach Definition SR-IOV type

Please copy paste the following definition update the values of your subnet and apply it to our cluster:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-kernelnet0
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_kernel0
spec:
  config: '{
  "type": "sriov",
  "cniVersion": "0.3.1",
  "name": "sriov-kernelnet0",
  "spoofchk": "off",
  "type": "sriov",
  "vlan": 80,
  "ipam": {
    "type": "whereabouts",
    "range": "192.168.80.0/24",
    "range_start": "192.168.80.20",
    "range_end": "192.168.80.50",
    "gateway": "192.168.80.1"
  }
}'

The master key value is the reference to the *_second nic *_in our worker nodes.

#  kubectl apply -f networkattachdefinition-sriov.yaml

Network Attach Definition Validation SR-IOV type

Let’s validate our work by listing and describing our new Network Attach Definition

$ kubectl get net-attach-def
NAME          AGE
sriov-net-a   5d21h

$ kubectl describe net-attach-def sriov-kernelnet0
Name:         sriov-kernelnet0
Namespace:    default
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_kernel0
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"k8s.v1.cni.cncf.io/resourceName":"inte...
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2020-09-23T18:53:31Z
  Generation:          1
  Resource Version:    174547
  Self Link:           /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sriov-kernelnet0
  UID:                 c6fd410a-97d8-42d4-86d5-56ef447e1dfa
Spec:
  Config:  { "type": "sriov", "name": "sriov-kernelnet0", "spoofchk": "off", "type": "sriov", "vlan": 80, "ipam": { "type": "whereabouts", "range": "192.168.80.0/24", "range_start": "192.168.80.20", "range_end": "192.168.80.50", "gateway": "192.168.80.1" } }
Events:    <none>

Create Pods with SR-IOV interfaces (Kernel-Driver)

$ cat pod0-case3.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod0-case-03-sriov-kernel
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-kernelnet0
spec:
  containers:
  - name: pod0-case-03
    image: docker.io/centos/tools:latest
    command:
    - /sbin/init
    resources:
      requests:
        intel.com/intel_sriov_kernel0: '1'
      limits:
        intel.com/intel_sriov_kernel0: '1'

Deploy the new pods

$ kubectl apply -f pod0-case3.yaml

Validate Pods Creation with SR-IOV interfaces (Kernel Driver)

Let’s validate your work by confirming that the pods got created with an additional interface by doing the following commands:

$ kubectl get pods -o wide
NAME                                      READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES          
pod0-case-02                              1/1     Running   0          66m     10.135.1.147    192.168.50.14   <none>           <none>
pod1-case-02                              1/1     Running   0          62m     10.135.1.148    192.168.50.14   <none>           <none>

$ kubectl exec -it pod0-case-02 -- ip -d address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default
    link/ether de:ff:c2:57:c6:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth numtxqueues 1 numrxqueues 1
    inet 10.135.1.147/24 brd 10.135.1.255 scope global eth0
       valid_lft forever preferred_lft forever
4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:10:31:25 brd ff:ff:ff:ff:ff:ff promiscuity 0
    ipvlan  mode l2 numtxqueues 1 numrxqueues 1
    inet 192.168.80.20/24 brd 192.168.70.255 scope global net1
       valid_lft forever preferred_lft forever

$ kubectl exec -it pod1-case-02 -- ip -d address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 numtxqueues 1 numrxqueues 1
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default
    link/ether aa:2d:2e:e8:5e:19 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth numtxqueues 1 numrxqueues 1
    inet 10.135.1.148/24 brd 10.135.1.255 scope global eth0
       valid_lft forever preferred_lft forever
4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default
    link/ether fa:16:3e:10:31:25 brd ff:ff:ff:ff:ff:ff promiscuity 0
    ipvlan  mode l2 numtxqueues 1 numrxqueues 1
    inet 192.168.80.21/24 brd 192.168.70.255 scope global net1
       valid_lft forever preferred_lft forever

SR-IOV VF Testing

SR-IOV Kernel Driver Validation

$ kubectl exec pod0-case-03 -- ethtool -i net2
driver: igbvf
version: 2.4.0-k
firmware-version:
expansion-rom-version:
bus-info: 0000:01:10.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

$ kubectl exec pod0-case-03 -- ethtool -i net1
driver: igbvf
version: 2.4.0-k
firmware-version:
expansion-rom-version:
bus-info: 0000:01:10.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no