1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
|
Copyright (c) 2014, Linaro Limited
All rights reserved.
SPDX-License-Identifier: BSD-3-Clause
1. Rationale
=================================================
This is an effort to port ODP on top of DPDK to accelerate packet processing on
Intel systems equiped with NIC's that supports DPDK.
DPDK and ODP have been compiled and tested on Ubuntu 14.04 LTS
3.13.0-29-generic kernel.
2. Preparing DPDK
=================================================
Fetching the DPDK code:
----------------------
git clone http://92.243.14.124/git/dpdk ./<dpdk-dir>
Right now odp-dpdk only supports DPDK v1.7.1. During upgrade, make sure that PMD
constuctors are correctly and fully referred in refer_constructors():
git tag -l -- will list all the tags available
git checkout -b 1.7.1 tags/v1.7.1
Compile DPDK:
------------
Please refer to http://dpdk.org/doc for more details on how to build DPDK.
Getting started guide for Linux might be of help.
Best effort is done to provide some help on DPDK cmds below for Ubuntu, where it
has been compiled and tested.
This has to be done only once:
cd <dpdk-dir>
make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc
Set CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=y in
./x86_64-native-linuxapp-gcc/.config file.
Note: if non-intel SFP's are used in IXGBE, then set
CONFIG_RTE_LIBRTE_IXGBE_ALLOW_UNSUPPORTED_SFP=y in .config file.
make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS="-fPIC"
If "conflicting types for skb_set_hash" error happens during DPDK
build, then please knock-off skb_set_hash function from kcompat.h as
shown below. This was seen in Ubuntu 3.13.0-30-generic.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
index 19df483..78a794a 100644
--- a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
+++ b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
@@ -3845,11 +3845,6 @@ static inline struct sk_buff *__kc__vlan_hwaccel_put_tag(struct sk_buff *skb,
#if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) )
#ifdef NETIF_F_RXHASH
#define PKT_HASH_TYPE_L3 0
-static inline void
-skb_set_hash(struct sk_buff *skb, __u32 hash, __always_unused int type)
-{
- skb->rxhash = hash;
-}
#endif /* NETIF_F_RXHASH */
#endif /* < 3.14.0 */
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This only ensures building DPDK, but traffic is not tested with this build yet.
3. Compile odp-dpdk
=================================================
cd <odp-dir>
./bootstrap
./configure --with-platform=linux-dpdk --with-sdk-install-path=<dpdk-dir>/x86_64-native-linuxapp-gcc
make
4. Prepare DPDK for running odp-dpdk examples
=================================================
Reserve hugepages:
-----------------
To reserve huge pages, which is needed for DPDK, execute following commands
(these are usually needed only once after the system has started):
sudo sh -c 'echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'
If you are running on a multi-node machine then hugepages have to be reserved on
each node:
ls /sys/devices/system/node
sudo sh -c 'echo 1024 > /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages'
Mount hugetlbfs:
---------------
sudo mkdir /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
Insert DPDK kernel module:
-------------------------
DPDK uses userspace poll mode drivers, so it's necessary to insert a couple of
modules to allow DPDK to map the NIC's registers to userspace:
sudo /sbin/modprobe uio
ulimit -Sn 2048
cd <dpdk-dir>
sudo insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
Bind NIC's to DPDK:
------------------
The DPDK code contains a tool used to bind drivers to the network cards.
cd <dpdk-dir>
./tools/dpdk_nic_bind.py --status
This command produces output that is similar to the one given below:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Network devices using IGB_UIO driver
====================================
0000:05:00.0 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused=
0000:05:00.1 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused=
Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth0 drv=bnx2 unused=<none> *Active*
0000:01:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth1 drv=bnx2 unused=<none>
0000:07:00.0 'T320 10GbE Dual Port Adapter' if=eth2,eth3 drv=cxgb3 unused=<none>
Other network devices
=====================
<none>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bind using interface name:
-------------------------
The easiest way is to let the tool automatically switch the regular drivers. For
that the interface must not be active i.e. no IP addresses assigned:
ifconfig eth0 0
ifconfig eth1 0
sudo ./tools/dpdk_nic_bind --bind=igb_uio eth0
sudo ./tools/dpdk_nic_bind --bind=igb_uio eth1
Bind using PCI ids:
------------------
Another way is to remove the regular drivers and use PCI ids:
sudo rmmod ixgbe
If the SFP's used are non-intel, then
sudo modprobe ixgbe allow_unsupported_sfp=1
sudo ./tools/igb_uio_bind.py --bind=igb_uio 05:00.0
sudo ./tools/igb_uio_bind.py --bind=igb_uio 05:00.1
Unbind network cards from DPDK:
------------------------------
To restore the NIC's back to kernel use something like this:
sudo ./tools/igb_uio_bind.py --bind=ixgbe 05:00.0
sudo ./tools/igb_uio_bind.py --bind=ixgbe 05:00.1
5. Running ODP apps
=================================================
Some useful ODP examples and how to run them:
l2fwding app - sudo ./example/l2fwd/odp_l2fwd -i 0,1 -m 0 -c 2
loopback app - sudo ./example/packet/odp_pktio -i 0,1 -m 0 -c 2
-i 0,1 - interface number
-m 0 - burst mode
-c 2 - number of cpus
6. Howto debug DPDK apps on the host
=================================================
For example you need to debug some l2fwd application. Then network configuration
might be:
<veth1-2> <-----> <veth2-1> (iface0 DPDK L2FWD APP iface1) <veth2-3> <-----> <veth3-2>
Where:
vethX-Y - virtual devices for host.
Packet sent to veth1-2 goes to chain and appears on veth3-2
Steps:
Recompile with:
CONFIG_RTE_LIBRTE_PMD_PCAP=y
ip link add veth1-2 type veth peer name veth2-1
ip link add veth2-3 type veth peer name veth3-2
ifconfig veth1-2 up -arp
ifconfig veth2-1 up -arp
ifconfig veth2-3 up -arp
ifconfig veth3-2 up -arp
mount -t hugetlbfs none /mnt/huge
Finally give l2fwd fake devices:
./l2fwd -c '0xf' -n 4 --vdev "eth_pcap0,iface=veth2-1" --vdev="eth_pcap1,iface=veth2-3" -- -p 3
|
