Copyright (c) 2018-2019, Linaro Limited Copyright (c) 2019, Nokia All rights reserved. SPDX-License-Identifier: BSD-3-Clause 1. Rationale ================================================= This is an effort to port ODP on top of DPDK to accelerate packet processing on systems equipped with NIC's that supports DPDK. Prerequisites and considerations: -------------------------------- - 8GB of RAM recommended, 4 might be enough too - it's recommended to obtain a DPDK supported network card (many come in dual port configuration, mostly dedicated to server usage) - it's also possible to use odp-dpdk for evaluation purposes without a DPDK compatible NIC, using the pcap poll mode driver - DPDK code must be downloaded, configured and compiled, details below - ODP-DPDK has been compiled and tested on an x86 host with Ubuntu 16.04 LTS (4.4.0 kernel). - DPDK only works on a selected range of network cards. The list of known and supported devices can be found in the DPDK documentation: http://doc.dpdk.org/guides-18.11/nics/index.html 2. Preparing DPDK ================================================= Fetching the DPDK code: ---------------------- Right now odp-dpdk supports DPDK v18.11 (recommended version) and v17.11: git clone http://dpdk.org/git/dpdk-stable --branch 18.11 --depth 1 ./ or git clone http://dpdk.org/git/dpdk-stable --branch 17.11 --depth 1 ./ Compile DPDK: ------------ Please refer to http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html for more details on how to build DPDK. Best effort is done to provide some help on DPDK cmds below for Ubuntu, where it has been compiled and tested. On Ubuntu install pcap development library: sudo apt-get install libpcap-dev This has to be done only once: cd make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc Enable pcap pmd to use ODP-DPDK without DPDK supported NIC's: cd /x86_64-native-linuxapp-gcc sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_PCAP=).*,\1y,' .config Enable openssl crypto pmd to use openssl with odp-dpdk: sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_OPENSSL=).*,\1y,' .config Optionally, add more queues to eventdev: sed -ri 's,(CONFIG_RTE_EVENT_MAX_QUEUES_PER_DEV=).*,\1255,' .config Now return to parent directory and build DPDK: cd .. The last step depends on whether ODP shared libraries will be built with this deployment of DPDK: SHARED libraries: make install T=x86_64-native-linuxapp-gcc DESTDIR=./install EXTRA_CFLAGS="-fPIC" STATIC libraries: make install T=x86_64-native-linuxapp-gcc DESTDIR=./install This only ensures building DPDK, but traffic is not tested with this build yet. 3. Compile ODP-DPDK ================================================= Build dependencies are listed in DEPENDENCIES. Use absolute DPDK directory path with the --with-dpdk-path option. cd ./bootstrap The following step depends on whether ODP shared libraries are to be built. SHARED libraries (requires building DPDK with -fPIC, see above): ./configure --with-dpdk-path=/install STATIC libraries (better performance): ./configure --with-dpdk-path=/install --disable-shared make 4. Prepare DPDK for running ODP-DPDK examples ================================================= Reserve hugepages: ----------------- To reserve huge pages, which is needed for DPDK, execute following commands (these are usually needed only once after the system has started): sudo sh -c 'echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages' If you are running on a multi-node machine then hugepages have to be reserved on each node: ls /sys/devices/system/node sudo sh -c 'echo 1024 > /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages' Mount hugetlbfs: --------------- sudo mkdir /mnt/huge sudo mount -t hugetlbfs nodev /mnt/huge Insert DPDK kernel module: ------------------------- DPDK uses userspace poll mode drivers, so it's necessary to insert a couple of modules to allow DPDK to map the NIC's registers to userspace: sudo /sbin/modprobe uio ulimit -Sn 2048 cd sudo insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko Bind NIC's to DPDK: ------------------ The DPDK code contains a tool used to bind drivers to the network cards. cd ./usertools/dpdk-devbind.py --status This command produces output that is similar to the one given below: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Network devices using DPDK-compatible driver ============================================ 0000:05:00.0 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused= 0000:05:00.1 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:01:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth0 drv=bnx2 unused= *Active* 0000:01:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth1 drv=bnx2 unused= 0000:07:00.0 'T320 10GbE Dual Port Adapter' if=eth2,eth3 drv=cxgb3 unused= Other network devices ===================== ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Bind using interface name: ------------------------- The easiest way is to let the tool automatically switch the regular drivers. For that the interface must not be active i.e. no IP addresses assigned: sudo ifconfig eth0 0 sudo ifconfig eth1 0 sudo ./usertools/dpdk-devbind.py --bind=igb_uio eth0 sudo ./usertools/dpdk-devbind.py --bind=igb_uio eth1 Bind using PCI ids: ------------------ Another way is to remove the regular drivers and use PCI ids: sudo rmmod ixgbe sudo ./usertools/dpdk-devbind.py --bind=igb_uio 05:00.0 sudo ./usertools/dpdk-devbind.py --bind=igb_uio 05:00.1 Unbind network cards from DPDK: ------------------------------ To restore the NIC's back to kernel use something like this: sudo ./usertools/dpdk-devbind.py --bind=ixgbe 05:00.0 sudo ./usertools/dpdk-devbind.py --bind=ixgbe 05:00.1 5. Running ODP apps ================================================= ODP-DPDK applications need to be run as root. You may also need to supply DPDK command line parameters either as a null-terminated array of char's to odp_global_init()'s platform_params parameter: odp_global_init([params], "--no-huge"); Or, if it's NULL the platform tries to read the ODP_PLATFORM_PARAMS environment variable. The coremask (-c) is calculated by ODP-DPDK based on the process affinity at startup. You can influence that with 'taskset'. DPDK init changes the affinity of the calling thread, so after it returns the original affinity is restored. Only the first active core is passed to rte_eal_init(), as the rest would be used for DPDK's special lcore threads, which are only available through rte_eal_[mp_]remote_launch(), but not through ODP API's. Nevertheless, odp_local_init() makes sure for the rest of the DPDK libraries ODP threads look like proper DPDK threads. Exaple how to run an ODP-DPDK L2 forwarding application: sudo ./odp_l2fwd -i 0,1 -c 2 -i 0,1 - interface numbers -c 2 - number of worker cpus 6. Howto debug DPDK apps on the host ================================================= For example you need to debug some l2fwd application. Then network configuration might be: <-----> (iface0 DPDK L2FWD APP iface1) <-----> Where: vethX-Y - virtual devices for host. Packet sent to veth1-2 goes to chain and appears on veth3-2 Steps: Recompile with: CONFIG_RTE_LIBRTE_PMD_PCAP=y ip link add veth1-2 type veth peer name veth2-1 ip link add veth2-3 type veth peer name veth3-2 ifconfig veth1-2 up -arp ifconfig veth2-1 up -arp ifconfig veth2-3 up -arp ifconfig veth3-2 up -arp mount -t hugetlbfs none /mnt/huge Finally give l2fwd fake devices: ./l2fwd -c '0xf' --vdev "eth_pcap0,iface=veth2-1" --vdev="eth_pcap1,iface=veth2-3" -- -p 3 7. Upgrading ODP-DPDK to newer ODP API level ================================================= This repository is based on odp.git, it also retains the history of that. There are some modifications in configure.ac plus a few other files, but most of the changes are in platform/linux-dpdk. That directory's Makefile.am builds our code and the required parts from platform/linux-generic. This allows us to easily pull the necessary changes from odp.git with git: git remote add odp_base https://github.com/OpenDataPlane/odp.git git pull odp_base master This will result in a merge commit, and possibly some conflict resolving if one of the files we touch outside platform/linux-dpdk is modified. After resolving this conflict you need to implement the changes in the API. Also, some of the code in our files are exact copy of the linux-generic counterpart, it's important to port the fixes to the original. The git-transplant script can do this: scripts/git-transplant.py platform/linux-generic/ platform/linux-dpdk/ \ [the previous last commit merged from odp.git]..HEAD It prints the list of prospective patches to be ported. See its comments about what it does. 8. Building odp-dpdk with dpdk crypto PMD's ====================================================== Refer to dpdk crypto documentation for detailed building of crypto PMD's: http://dpdk.org/doc/guides/cryptodevs/index.html. To build odp-dpdk with dpdk virtual crypto devices, we need to build supporting intel multi-buffer library prior to dpdk build. Get the Intel multi-buffer crypto library from, https://github.com/intel/intel-ipsec-mb and follow the README from the repo on how to build the library. building dpdk: Follow the instructions from "Compile DPDK" section. Make sure to enable necessary crypto PMD's, # Compile PMD for AESNI backed device sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_AESNI_MB=).*,\1y,' .config # Compile PMD for AESNI GCM device sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_AESNI_GCM=).*,\1y,' .config # Compile PMD for SNOW 3G device sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_SNOW3G=).*,\1y,' .config AESNI_MULTI_BUFFER_LIB_PATH=/path-to/Intel-multi-buffer-crypto/ \ make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS="-fPIC" when building odp-dpdk application, add the multi-buffer crypto library path to make file. Before running the application, export ODP_PLATFORM_PARAMS with corresponding crypto vdev's. ex: ODP_PLATFORM_PARAMS="--vdev cryptodev_aesni_mb_pmd,max_nb_sessions=32 \ --vdev cryptodev_null_pmd,max_nb_sessions=32" 9. Using eventdev scheduling and queues (experimental) ====================================================== ODP-DPDK includes experimental implementations of ODP scheduler, queues, and scheduled pktio using DPDK event device library. The initial implementation has been validated using only the standard software eventdev poll mode driver with DPDK v18.11. Due to some pending eventdev bugs the implementation is not yet tested in the ODP CI. To use eventdev one must set ODP_SCHEDULER environment variable to "eventdev" and provide the necessary platform parameters to DPDK. Refer to DPDK event device driver documentation for platform defails: https://doc.dpdk.org/guides/eventdevs/index.html In case of the standard software eventdev implementation one must enable a DPDK service core, which will perform scheduling and receive packets for the scheduled pktio input queues. The DPDK service cores and the ODP application cores should not overlap. Exaple how to run odp_scheduling test application using eventdev: sudo ODP_SCHEDULER="eventdev" ODP_PLATFORM_PARAMS="--vdev event_sw0 -s 0x4" \ ./odp_scheduling -c 1