How To Luigi Sriov Dpdk Driver

SRI-IOV WorkerNode Prerequisites

In order for the cni-sriov-plugin to start reading Virtual Functions resources as well HostNetworkTemplate objects can configure virtual functions you need to perform the following steps.

1. Enable** VT-D** at BIOS level.

2. Enable SRIOV feature globally and/or per nic according to the nic you are working with, for example Intel x710 should be enabled per nic at BIOS level.

3. Upgrade Kernel boot line to enable intel_iommu=on and iommu=pt

None

$ grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on"
$ reboot

Huge page Support

Most network applications use huge pages, so you may want to enable that. Please edit /etc/default/grub and add huge-ages

Bash

GRUB_CMDLINE_LINUX="nofb nomodeset vga=normal iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16"
#Rebuild grub.cfg
grub2-mkconfig -o /boot/grub2/grub.cfg && reboot

Create SR-IOV ConfigMap

SR-IOV ConfigMap

Please copy paste the following definition update the values of your Physical Functions and ranges or Virtual Functions as well as Drivers and apply it to our cluster, in case that you have multiple nics, that each one uses different kernels drivers please omit the drivers section and just mentioned pfName within the selectors section.

SR-IOV Kernel Driver plus DPDK ConfigMap

YAML

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "intel_sriov_dpdk0",
                "selectors": {
                    "drivers": ["vfio-pci"],
                    "pfNameis": ["enp3s0f0#0-7"]
                }
            },
            {
                "resourceName": "intel_sriov_kernel0",
                "selectors": {
                    "drivers": ["ixgbevf"],
                    "pfNames": ["enp3s0f0#8-15"]
                }
            },
            {
                "resourceName": "intel_sriov_dpdk1",
                "selectors": {
                    "drivers": ["vfio-pci"],
                    "pfNames": ["enp3s0f1#0-7"]
                }
            },
            {
                "resourceName": "intel_sriov_kernel1",
                "selectors": {
                    "drivers": ["ixgbevf"],
                    "pfNames": ["enp3s0f1#8-15"]
                }
            }
        ]
    }

Create NetworkAttachDefinition of DPDK type

YAML

kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net-a
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk_a
spec:
  config: '{
  "type": "sriov",
  "cniVersion": "0.3.1",
  "name": "sriov-network-a",
  "vlan": 100,
  "vlanQoS": 1,
  "max_tx_rate": 100,
  "spoofchk": "off",
  "trust": "on"
}'

Network Attach Definition Validation SR-IOV-DPDK type

Let’s validate our work by listing and describing our new Network Attach Definition

Bash

$ kubectl  get net-attach-def
NAME                      AGE
sriov-dpdk0               15s

Bash

$  kubectl describe net-attach-def sriov-dpdk0
Name:         sriov-dpdk0
Namespace:    default
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk0
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"k8s.v1.cni.cncf.io/resourceName":"inte...
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2020-10-05T20:25:57Z
  Generation:          1
  Resource Version:    2310115
  Self Link:           /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sriov-dpdk0
  UID:                 5307140a-3771-41b4-845e-1aa8460acffc
Spec:
  Config:  { "cniVersion": "0.3.1", "name": "sriov-dpdk0", "type": "sriov", "vlan": 90 }
Events:    <none>

Create Pods with SR-IOV interfaces (DPDK Driver - vfio-pci)

YAML

# cat sriov-pod-2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: sriov-pod-2
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-net-a
spec:
  containers:
  - name: sriov-example
    image: rparikh/dpdk-app-centos
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /etc/podnetinfo
      name: podnetinfo
      readOnly: false
    - mountPath: /dev/hugepages
      name: hugepage
    resources:
      requests:
        memory: 1Gi
        #cpu: "4"
        intel.com/intel_sriov_dpdk_a: '1'
      limits:
        hugepages-1Gi: 2Gi
        #cpu: "4"
        intel.com/intel_sriov_dpdk_a: '1'
    # Uncomment to control which DPDK App is running in container.
    # If not provided, l3fwd is default.
    #   Options: l2fwd l3fwd testpmd
    env:
    - name: DPDK_SAMPLE_APP
      value: "testpmd"
    #
    # Uncomment to debug DPDK App or to run manually to change
    # DPDK command line options.
    command: ["sleep", "infinity"]
  volumes:
  - name: podnetinfo
    downwardAPI:
      items:
        - path: "labels"
          fieldRef:
            fieldPath: metadata.labels
        - path: "annotations"
          fieldRef:
            fieldPath: metadata.annotations
  - name: hugepage
    emptyDir:
      medium: HugePages

Deploy the new pods

Bash

$ kubectl apply -f sriov-pod-2.yaml

Validate Pods Creation with SR-IOV interfaces (DPDK - Driver vfio-pci)

Let’s validate your work by confirming that the created pods got successfully created by doing the following commands:

Bash

$ kubectl get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE     IP              NODE             NOMINATED NODE   READINESS GATES
sriov-pod-2   1/1     Running   0          5d21h   10.167.181.11   10.128.237.204   <none>           <none>

SR-IOV DPDK Driver - vfio-pci Validation

Bash

$ kubectl exec -it sriov-pod-2 -- sh
sh-4.4# dpdk-app
ENTER dpdk-app:
 argc=1
 dpdk-app
  cpuRsp.CPUSet = 0-31
  Interface[0]:
    IfName=""  Name=""  Type=SR-IOV
    MAC=""  IP="10.143.181.4"
    PCIAddress=0000:03:10.7
 myArgc=14
 dpdk-app -n 4 -l 1 --master-lcore 1 -w 0000:03:10.7 -- -p 0x1 -P --config="(0,0,1)" --parse-ptype