1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
|
.. _iperf3:
Introduction
============
This tutorial shows how to use VPP use iperf3 and TRex to get some basic performance
numbers from a few basic configurations. Four examples are shown. In the first two
examples, the **iperf3** tool is used to generate traffic, and in the last two examples
the Cisco's `TRex Realistic Traffic Generator <http://trex-tgn.cisco.com/>`_ is used. For
comparison purposes, the first example shows packet forwarding using ordinary kernel
IP forwarding, and the second example shows packet forwarding using VPP.
Three Intel Xeon processor platform systems are used to connect to the VPP host to pass traffic
using **iperf3** and Cisco’s `TRex <http://trex-tgn.cisco.com/>`_.
Intel 40 Gigabit Ethernet (GbE) network interface cards (NICs) are used to connect the hosts.
Using Kernel Packet Forwarding with Iperf3
===========================================
In this test, 40 GbE Intel Ethernet Network Adapters are used to connect the three
systems. Figure 1 illustrates this configuration.
.. figure:: /_images/iperf3fig1.png
Figure 1: VPP runs on a host that connects to two other systems via 40 GbE NICs.
For comparison purposes, in the first example, we configure kernel forwarding in
*csp2s22c03* and use the **iperf3** tool to measure network bandwidth between
*csp2s22c03* and *net2s22c05*.
In the second example, we start the VPP engine in *csp2s22c03* instead of using
kernel forwarding. On *csp2s22c03*, we configure the system to have the addresses
10.10.1.1/24 and 10.10.2.1/24 on the two 40-GbE NICs. To find all network interfaces
available on the system, use the lshw Linux command to list all network interfaces
and the corresponding slots *[0000:xx:yy.z]*.
In this example, the 40-GbE interfaces are *ens802f0* and *ens802f1*.
.. code-block:: console
csp2s22c03$ sudo lshw -class network -businfo
Bus info Device Class Description
========================================================
pci@0000:03:00.0 enp3s0f0 network Ethernet Controller 10-Gig
pci@0000:03:00.1 enp3s0f1 network Ethernet Controller 10-Gig
pci@0000:82:00.0 ens802f0 network Ethernet Controller XL710
pci@0000:82:00.1 ens802f1 network Ethernet Controller XL710
pci@0000:82:00.0 ens802f0d1 network Ethernet interface
pci@0000:82:00.1 ens802f1d1 network Ethernet interface
Configure the system *csp2s22c03* to have 10.10.1.1 and 10.10.2.1 on the two 40-GbE NICs
*ens802f0* and *ens802f1*, respectively.
.. code-block:: console
csp2s22c03$ sudo ip addr add 10.10.1.1/24 dev ens802f0
csp2s22c03$ sudo ip link set dev ens802f0 up
csp2s22c03$ sudo ip addr add 10.10.2.1/24 dev ens802f1
csp2s22c03$ sudo ip link set dev ens802f1 up
List the route table:
.. code-block:: console
csp2s22c03$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default jf111-ldr1a-530 0.0.0.0 UG 0 0 0 enp3s0f1
default 192.168.0.50 0.0.0.0 UG 100 0 0 enp3s0f0
10.10.1.0 * 255.255.255.0 U 0 0 0 ens802f0
10.10.2.0 * 255.255.255.0 U 0 0 0 ens802f1
10.23.3.0 * 255.255.255.0 U 0 0 0 enp3s0f1
link-local * 255.255.0.0 U 1000 0 0 enp3s0f1
192.168.0.0 * 255.255.255.0 U 100 0 0 enp3s0f0
.. code-block:: console
csp2s22c03$ ip route
default via 10.23.3.1 dev enp3s0f1
default via 192.168.0.50 dev enp3s0f0 proto static metric 100
10.10.1.0/24 dev ens802f0 proto kernel scope link src 10.10.1.1
10.10.2.0/24 dev ens802f1 proto kernel scope link src 10.10.2.1
10.23.3.0/24 dev enp3s0f1 proto kernel scope link src 10.23.3.67
169.254.0.0/16 dev enp3s0f1 scope link metric 1000
192.168.0.0/24 dev enp3s0f0 proto kernel scope link src 192.168.0.142 metric 100
On *csp2s22c04*, we configure the system to have the address 10.10.1.2 and use
the interface *ens802* to route IP packets 10.10.2.0/24. Use the lshw Linux
command to list all network interfaces and the corresponding slots *[0000:xx:yy.z]*.
For example, the interface *ens802d1* *(ens802)* is connected to slot *[82:00.0]*:
.. code-block:: console
csp2s22c04$ sudo lshw -class network -businfo
Bus info Device Class Description
=====================================================
pci@0000:03:00.0 enp3s0f0 network Ethernet Controller 10-Gigabit X540-AT2
pci@0000:03:00.1 enp3s0f1 network Ethernet Controller 10-Gigabit X540-AT2
pci@0000:82:00.0 ens802d1 network Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:82:00.0 ens802 network Ethernet interface
For kernel forwarding, set 10.10.1.2 to the interface *ens802*, and add a static
route for IP packet 10.10.2.0/24:
.. code-block:: console
csp2s22c04$ sudo ip addr add 10.10.1.2/24 dev ens802
csp2s22c04$ sudo ip link set dev ens802 up
csp2s22c04$ sudo ip route add 10.10.2.0/24 via 10.10.1.1
.. code-block:: console
csp2s22c04$ ifconfig
enp3s0f0 Link encap:Ethernet HWaddr a4:bf:01:00:92:73
inet addr:10.23.3.62 Bcast:10.23.3.255 Mask:255.255.255.0
inet6 addr: fe80::a6bf:1ff:fe00:9273/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3411 errors:0 dropped:0 overruns:0 frame:0
TX packets:1179 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:262230 (262.2 KB) TX bytes:139975 (139.9 KB)
ens802 Link encap:Ethernet HWaddr 68:05:ca:2e:76:e0
inet addr:10.10.1.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::6a05:caff:fe2e:76e0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:5480 (5.4 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:31320 errors:0 dropped:0 overruns:0 frame:0
TX packets:31320 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:40301788 (40.3 MB) TX bytes:40301788 (40.3 MB)
After setting the route, we can ping from *csp2s22c03* to *csp2s22c04*, and vice versa:
.. code-block:: console
csp2s22c03$ ping 10.10.1.2 -c 3
PING 10.10.1.2 (10.10.1.2) 56(84) bytes of data.
64 bytes from 10.10.1.2: icmp_seq=1 ttl=64 time=0.122 ms
64 bytes from 10.10.1.2: icmp_seq=2 ttl=64 time=0.109 ms
64 bytes from 10.10.1.2: icmp_seq=3 ttl=64 time=0.120 ms
.. code-block:: console
csp2s22c04$ ping 10.10.1.1 -c 3
PING 10.10.1.1 (10.10.1.1) 56(84) bytes of data.
64 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=0.158 ms
64 bytes from 10.10.1.1: icmp_seq=2 ttl=64 time=0.096 ms
64 bytes from 10.10.1.1: icmp_seq=3 ttl=64 time=0.102 ms
Similarly, on *net2s22c05*, we configure the system to have the address *10.10.2.2*
and use the interface *ens803f0* to route IP packets *10.10.1.0/24*. Use the lshw
Linux command to list all network interfaces and the corresponding slots
*[0000:xx:yy.z]*. For example, the interface *ens803f0* is connected to slot *[87:00.0]*:
.. code-block:: console
NET2S22C05$ sudo lshw -class network -businfo
Bus info Device Class Description
========================================================
pci@0000:03:00.0 enp3s0f0 network Ethernet Controller 10-Gigabit X540-AT2
pci@0000:03:00.1 enp3s0f1 network Ethernet Controller 10-Gigabit X540-AT2
pci@0000:81:00.0 ens787f0 network 82599 10 Gigabit TN Network Connection
pci@0000:81:00.1 ens787f1 network 82599 10 Gigabit TN Network Connection
pci@0000:87:00.0 ens803f0 network Ethernet Controller XL710 for 40GbE QSFP+
pci@0000:87:00.1 ens803f1 network Ethernet Controller XL710 for 40GbE QSFP+
For kernel forwarding, set 10.10.2.2 to the interface ens803f0, and add a static
route for IP packet 10.10.1.0/24:
.. code-block:: console
NET2S22C05$ sudo ip addr add 10.10.2.2/24 dev ens803f0
NET2S22C05$ sudo ip link set dev ens803f0 up
NET2S22C05$ sudo ip route add 10.10.1.0/24 via 10.10.2.1
After setting the route, you can ping from *csp2s22c03* to *net2s22c05*, and vice
versa. However, in order to ping between *net2s22c05* and *csp2s22c04*, kernel IP
forwarding in *csp2s22c03* has to be enabled:
.. code-block:: console
csp2s22c03$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
csp2s22c03$ echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
csp2s22c03$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
If successful, verify that now you can ping between *net2s22c05* and *csp2s22c04*:
.. code-block:: console
NET2S22C05$ ping 10.10.1.2 -c 3
PING 10.10.1.2 (10.10.1.2) 56(84) bytes of data.
64 bytes from 10.10.1.2: icmp_seq=1 ttl=63 time=0.239 ms
64 bytes from 10.10.1.2: icmp_seq=2 ttl=63 time=0.224 ms
64 bytes from 10.10.1.2: icmp_seq=3 ttl=63 time=0.230 ms
We use the **iperf3** utility to measure network bandwidth between hosts. In this
test, we download the **iperf3** utility tool on both *net2s22c05* and *csp2s22c04*.
On *csp2s22c04*, we start the **iperf3** server with “iperf3 –s”, and then on *net2s22c05*,
we start the **iperf3** client to connect to the server:
.. code-block:: console
NET2S22C05$ iperf3 -c 10.10.1.2
Connecting to host 10.10.1.2, port 5201
[ 4] local 10.10.2.2 port 54074 connected to 10.10.1.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 936 MBytes 7.85 Gbits/sec 2120 447 KBytes
[ 4] 1.00-2.00 sec 952 MBytes 7.99 Gbits/sec 1491 611 KBytes
[ 4] 2.00-3.00 sec 949 MBytes 7.96 Gbits/sec 2309 604 KBytes
[ 4] 3.00-4.00 sec 965 MBytes 8.10 Gbits/sec 1786 571 KBytes
[ 4] 4.00-5.00 sec 945 MBytes 7.93 Gbits/sec 1984 424 KBytes
[ 4] 5.00-6.00 sec 946 MBytes 7.94 Gbits/sec 1764 611 KBytes
[ 4] 6.00-7.00 sec 979 MBytes 8.21 Gbits/sec 1499 655 KBytes
[ 4] 7.00-8.00 sec 980 MBytes 8.22 Gbits/sec 1182 867 KBytes
[ 4] 8.00-9.00 sec 1008 MBytes 8.45 Gbits/sec 945 625 KBytes
[ 4] 9.00-10.00 sec 1015 MBytes 8.51 Gbits/sec 1394 611 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 9.45 GBytes 8.12 Gbits/sec 16474 sender
[ 4] 0.00-10.00 sec 9.44 GBytes 8.11 Gbits/sec receiver
iperf Done.
|
