# Enable SRIOV with DPDK

### SRIOV and DPDK <a href="#sriov-and-dpdk" id="sriov-and-dpdk"></a>

#### SRI-IOV WorkerNode Prerequisites <a href="#sri-iov-workernode-prerequisites" id="sri-iov-workernode-prerequisites"></a>

In order for the cni-sriov-plugin to start reading Virtual Functions resources as well HostNetworkTemplate objects can configure virtual functions you need to perform the following steps.

1. Enable\*\* VT-D\*\* at BIOS level.
2. Enable **SRIOV** feature globally and/or per nic according to the nic you are working with, for example Intel x710 should be enabled per nic at BIOS level.
3. Upgrade Kernel boot line to enable **intel\_iommu=on and iommu=pt**

<i class="fa-caret-square-up">:caret-square-up:</i>None<i class="fa-sort-down">:sort-down:</i>+<i class="fa-arrows-alt">:arrows-alt:</i><i class="fa-clone">:clone:</i>    &#x20;

```
$ grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on"
$ reboot
```

#### Huge page Support <a href="#huge-page-support" id="huge-page-support"></a>

Most network applications use huge pages, so you may want to enable that. Please edit /etc/default/grub and add huge-ages

```
GRUB_CMDLINE_LINUX="nofb nomodeset vga=normal iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=16"
#Rebuild grub.cfg
grub2-mkconfig -o /boot/grub2/grub.cfg && reboot
```

#### Create SR-IOV ConfigMap <a href="#create-sr-iov-configmap" id="create-sr-iov-configmap"></a>

#### SR-IOV ConfigMap <a href="#sr-iov-configmap" id="sr-iov-configmap"></a>

Please copy paste the following definition update the values of your Physical Functions and ranges or Virtual Functions as well as Drivers and apply it to our cluster, in case that you have multiple nics, that each one uses different kernels drivers please omit the drivers section and just mentioned pfName within the selectors section.

#### SR-IOV Kernel Driver plus DPDK ConfigMap <a href="#sr-iov-kernel-driver-plus-dpdk-configmap" id="sr-iov-kernel-driver-plus-dpdk-configmap"></a>

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "intel_sriov_dpdk0",
                "selectors": {
                    "drivers": ["vfio-pci"],
                    "pfNameis": ["enp3s0f0#0-7"]
                }
            },
            {
                "resourceName": "intel_sriov_kernel0",
                "selectors": {
                    "drivers": ["ixgbevf"],
                    "pfNames": ["enp3s0f0#8-15"]
                }
            },
            {
                "resourceName": "intel_sriov_dpdk1",
                "selectors": {
                    "drivers": ["vfio-pci"],
                    "pfNames": ["enp3s0f1#0-7"]
                }
            },
            {
                "resourceName": "intel_sriov_kernel1",
                "selectors": {
                    "drivers": ["ixgbevf"],
                    "pfNames": ["enp3s0f1#8-15"]
                }
            }
        ]
    }
```

### Create NetworkAttachDefinition of DPDK type

```
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net-a
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk_a
spec:
  config: '{
  "type": "sriov",
  "cniVersion": "0.3.1",
  "name": "sriov-network-a",
  "vlan": 100,
  "vlanQoS": 1,
  "max_tx_rate": 100,
  "spoofchk": "off",
  "trust": "on"
}'
```

#### Network Attach Definition Validation SR-IOV-DPDK type <a href="#network-attach-definition-validation-sr-iov-dpdk-type" id="network-attach-definition-validation-sr-iov-dpdk-type"></a>

Let’s validate our work by listing and describing our new Network Attach Definition

```
$ kubectl  get net-attach-def
NAME                      AGE
sriov-dpdk0               15s
```

```
$  kubectl describe net-attach-def sriov-dpdk0
Name:         sriov-dpdk0
Namespace:    default
Labels:       <none>
Annotations:  k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk0
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"k8s.cni.cncf.io/v1","kind":"NetworkAttachmentDefinition","metadata":{"annotations":{"k8s.v1.cni.cncf.io/resourceName":"inte...
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2020-10-05T20:25:57Z
  Generation:          1
  Resource Version:    2310115
  Self Link:           /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sriov-dpdk0
  UID:                 5307140a-3771-41b4-845e-1aa8460acffc
Spec:
  Config:  { "cniVersion": "0.3.1", "name": "sriov-dpdk0", "type": "sriov", "vlan": 90 }
Events:    <none>
```

### Create Pods with SR-IOV interfaces (DPDK Driver - vfio-pci)

```
# cat sriov-pod-2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: sriov-pod-2
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-net-a
spec:
  containers:
  - name: sriov-example
    image: rparikh/dpdk-app-centos
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /etc/podnetinfo
      name: podnetinfo
      readOnly: false
    - mountPath: /dev/hugepages
      name: hugepage
    resources:
      requests:
        memory: 1Gi
        #cpu: "4"
        intel.com/intel_sriov_dpdk_a: '1'
      limits:
        hugepages-1Gi: 2Gi
        #cpu: "4"
        intel.com/intel_sriov_dpdk_a: '1'
    # Uncomment to control which DPDK App is running in container.
    # If not provided, l3fwd is default.
    #   Options: l2fwd l3fwd testpmd
    env:
    - name: DPDK_SAMPLE_APP
      value: "testpmd"
    #
    # Uncomment to debug DPDK App or to run manually to change
    # DPDK command line options.
    command: ["sleep", "infinity"]
  volumes:
  - name: podnetinfo
    downwardAPI:
      items:
        - path: "labels"
          fieldRef:
            fieldPath: metadata.labels
        - path: "annotations"
          fieldRef:
            fieldPath: metadata.annotations
  - name: hugepage
    emptyDir:
      medium: HugePages
```

Deploy the new pods

```
$ kubectl apply -f sriov-pod-2.yaml
```

#### Validate Pods Creation with SR-IOV interfaces (DPDK - Driver vfio-pci) <a href="#validate-pods-creation-with-sr-iov-interfaces-dpdk---driver-vfio-pci" id="validate-pods-creation-with-sr-iov-interfaces-dpdk---driver-vfio-pci"></a>

Let’s validate your work by confirming that the created pods got successfully created by doing the following commands:

```
$ kubectl get pods -o wide
NAME          READY   STATUS    RESTARTS   AGE     IP              NODE             NOMINATED NODE   READINESS GATES
sriov-pod-2   1/1     Running   0          5d21h   10.167.181.11   10.128.237.204   <none>           <none>
```

### SR-IOV DPDK Driver - vfio-pci Validation

```
$ kubectl exec -it sriov-pod-2 -- sh
sh-4.4# dpdk-app
ENTER dpdk-app:
 argc=1
 dpdk-app
  cpuRsp.CPUSet = 0-31
  Interface[0]:
    IfName=""  Name=""  Type=SR-IOV
    MAC=""  IP="10.143.181.4"
    PCIAddress=0000:03:10.7
 myArgc=14
 dpdk-app -n 4 -l 1 --master-lcore 1 -w 0000:03:10.7 -- -p 0x1 -P --config="(0,0,1)" --parse-ptype
```