1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
XL710 Statless benchmark
========================
:email: trex.tgen@gmail.com
:quotes.++:
:numbered:
:web_server_url: https://trex-tgn.cisco.com/trex
:local_web_server_url: csi-wiki-01:8181/trex
:toclevels: 6
:tabledef-default.subs: normal,callouts
include::trex_ga.asciidoc[]
// PDF version - image width variable
ifdef::backend-docbook[]
:p_width: 450
endif::backend-docbook[]
// HTML version - image width variable
ifdef::backend-xhtml11[]
:p_width: 800
endif::backend-xhtml11[]
== TRex stateless L2 benchmarks using XL710 40G NICs
=== Setup details
[cols="1,5"]
|=================
| Server: | UCSC-C240-M4SX
| CPU: | 2 x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
| RAM: | 65536 @ 2133 MHz
| NICs: | 2 x Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
| QSFP: | Cisco QSFP-H40G-AOC1M
| OS: | Fedora 18
| Switch: | Cisco Nexus 3172 Chassis, System version: 6.0(2)U5(2).
| TRex: | v2.09 using 7 cores per dual interface.
|=================
=== Topology
TRex port 1 ↔ Switch port Eth1/50 (vlan 1005) ↔ Switch port Eth1/52 (vlan 1005) ↔ TRex port 2
=== Results
.Cached VM
[cols="2,2^,2^,2^,2^,2^,2^,2^,3", options="header"]
|=================
| Packet size | Line Utilization (%) | Total L1 (Gb/s) | Total L2 (Gb/s) | CPU Util (%) | Total MPPS | BW per core (Gb/s) <1> | MPPS per core <2> | Multiplier
| Imix | 100.04 | 80.03 | 76.03 | 2.7 | 25.03 | 89.74 | 28.07 | 100%
| 1514 | 100.12 | 80.1 | 79.05 | 1.33 | 6.53 | 430.18 | 35.07 | 100%
| 590 | 99.36 | 79.49 | 76.89 | 3.2 | 16.29 | 177.43 | 36.36 | 99.5%
| 128 | 99.56 | 79.65 | 68.89 | 15.4 | 67.27 | 36.94 | 31.2 | 99.5%
| 64 | 52.8 | 42.3 | 32.23 | 14.1 | 62.95 | 21.43 | 31.89 | 31.5mpps
|=================
.VM with 1 variable
[cols="2,2^,2^,2^,2^,2^,2^,2^,3", options="header"]
|=================
| Packet size | Line Utilization (%) | Total L1 (Gb/s) | Total L2 (Gb/s) | CPU Util (%) | Total MPPS | BW per core (Gb/s) <1> | MPPS per core <2> | Multiplier
| Imix | 100.04 | 80.03 | 76.03 | 12.6 | 25.03 | 45.37 | 14.19 | 100%
| 1514 | 100.12 | 80.1 | 79.05 | 2.6 | 6.53 | 220.05 | 17.94 | 100%
| 590 | 99.36 | 79.49 | 76.89 | 5.6 | 16.29 | 101.39 | 20.78 | 99.5%
| 128 | 99.56 | 79.65 | 68.89 | 33.1 | 67.27 | 17.19 | 14.52 | 99.5%
| 64 | 52.8 | 42.3 | 32.23 | 31.3 | 63.06 | 9.65 | 14.37 | 31.5mpps
|=================
<1> Extrapolated L1 bandwidth per 1 core @ 100% CPU utilization.
<2> Extrapolated amount of MPPS per 1 core @ 100% CPU utilization.
INFO: XL710 NIC does not support linerate for 64B by design. only 128B is supported.
== Appendix
=== Preparing setup and running the tests.
==== Hardware preparations
Order the UCS with HW described above.
* There are several NICs with this chipset. +
Bare Intel NICs don't work with Cisco QSFP+ optics, for such case you will need Silicom NICs.
* Use NICs with 2x40G ports in each.
* Put the NICs at different NUMAs (first on the left side, second on the right side).
==== Software preparations
* Install the OS (Bare metal Linux, *not* VM!)
* Obtain the latest TRex package: wget https://trex-tgn.cisco.com/trex/release/latest
* Untar the package: tar -xzf latest
* Change dir to unzipped TRex
* Create config file using command: sudo python dpdk_setup_ports.py -i *TBD more info here*
** In case of Ubuntu 16 need python3
** The script by default produces config for loopback or L2 Switch as DUT (ports connection 1↔2, 3↔4 etc.). +
If you have router or L3 switch, change the destination MACs of TRex interfaces to match the DUT one's.
==== The tests
* Run the TRex server: sudo ./t-rex-64 -i -c 7
* In another shell run TRex console: trex-console
** The console can be run from another computer with -s argument, --help for more info.
** Other options for TRex client are automation or GUI
* In the console, run "tui" command, and then send the traffic with commands like:
** start -f stl/bench.py -m 50% --port 0 3 -t size=590,vm=var1
** stop
** clear
** start -f stl/bench.py -m 30mpps --port 0 -t size=64,vm=cached
** start -f stl/bench.py -m 100% -t size=1514,vm=random --force
=== Some of screenshots of console with commands
==== 64 bytes
Utilization:
image:images/64_util.png[title="64 bytes util",align="left",width={p_width}, link="images/64_util.png"]
No drops:
image:images/64_nodrop.png[title="64 bytes no drops",align="left",width={p_width}, link="images/64_nodrop.png"]
==== 128 bytes
Utilization:
image:images/128_util.png[title="128 bytes util",align="left",width={p_width}, link="images/128_util.png"]
No drops:
image:images/128_nodrop.png[title="128 bytes no drops",align="left",width={p_width}, link="images/128_nodrop.png"]
==== 590 bytes
Utilization:
image:images/590_util.png[title="128 bytes util",align="left",width={p_width}, link="images/590_util.png"]
No drops:
image:images/590_nodrop.png[title="590 bytes no drops",align="left",width={p_width}, link="images/590_nodrop.png"]
==== 1514 bytes
Utilization:
image:images/1514_util.png[title="128 bytes util",align="left",width={p_width}, link="images/1514_util.png"]
No drops:
image:images/1514_nodrop.png[title="1514 bytes no drops",align="left",width={p_width}, link="images/1514_nodrop.png"]
|
