1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
|
TRex Performance Trend
======================
:email: trex.tgen@gmail.com
:quotes.++:
:numbered:
:web_server_url: https://trex-tgn.cisco.com/trex
:local_web_server_url: csi-wiki-01:8181/trex
:toclevels: 6
:tabledef-default.subs: normal,callouts
include::trex_ga.asciidoc[]
// PDF version - image width variable
ifdef::backend-docbook[]
:p_width: 450
endif::backend-docbook[]
// HTML version - image width variable
ifdef::backend-xhtml11[]
:p_width: 800
endif::backend-xhtml11[]
= TRex performance trend
This document tracks the performance of TRex Stateless use-cases of various hardware and NICS.
The intention is to track the TRex software performance and not the NIC driver performance, however from this document you can learn the difference between different driver implementation (cycles/packets)
Performance is measured in a metric called MPPS/Core (Norm), which stands for Mega-Packet per second (Norm).
+
How the calculation is made? +
+
We measure the MPPS of the DUT in its best working point. +
The measurement is then scaled to fit 100% CPU utilization, and finally divided by the number of cores the device has,
to get a normalized MPPS/Core which is denoted as MPPS/Core (Norm) +
+
How to run these tests? +
[source,python]
----
trex-core/scripts> ./run_regression --stl -t <name of test> --cfg setups/<name of setup>
----
The Error (%) metric: +
+
You will see a column titled 'Error (%)' next to other metrics (avg,std etc) +
Error(%) is calculated for each batch of test results +
+
Taking the Max value (max) and the Min value (min), calculating the difference max-min +
We take the difference and then normalize it to the Average value to get (max-min)/Avg +
Then, we present it as percentage +
+
Error (%) = ((max-min)/Avg)*100 +
+
== Setup: XL710 (4x40GbE) trex-08
.Setup Details
[options='header',cols="1,5"]
|=================
| Server | Cisco UCS 240M3
| CPU: | 2 sockets x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 8 cores
| NICs: | 2 NICS x ( 2x 40GbE ) Intel XL710, Total ports 4
| NUMA | NIC0-NUMA 0 , NIC1-NUMA1
| Memory | 2x16GB
| PCIe | 2x PCIe16 + 2xPCIe4
| OS: | Fedora 18 - baremetal
|=================
image:images/trex08_latest_test_runs.png[title="trex08 test runs",align="left",width={p_width}, link="images/trex08_latest_test_runs.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex08_latest_test_runs_stats.csv[]
|===
=== Trend: Performance Over Time
image:images/trex08_trend_graph.png[title="trex08_trend_graph",align="left",width={p_width}, link="images/trex08_trend_graph.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex08_trend_stats.csv[]
|===
== Setup: X710 (8x10GbE) trex-09
.Setup Details
[options='header',cols="1,5"]
|=================
| Server | Cisco UCS 240M3
| CPU: | 2 sockets x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 8 cores
| NICs: | 2 NICS x ( 2x 40GbE ) Intel X710, Total 8 ports
| NUMA | NIC0-NUMA0 , NIC1-NUMA1
| Memory | 2x16GB
| PCIe | 2x PCIe16 + 2xPCIe4
| OS: | Fedora 18 - baremetal
|=================
image:images/trex09_latest_test_runs.png[title="trex09 test runs",align="left",width={p_width}, link="images/trex09_latest_test_runs.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex09_latest_test_runs_stats.csv[]
|===
=== Trend: Performance Over Time
image:images/trex09_trend_graph.png[title="trex09_trend_graph",align="left",width={p_width}, link="images/trex09_trend_graph.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex09_trend_stats.csv[]
|===
== Setup: 82599EB (4x10GbE) kiwi02
.Setup Details
[options='header',cols="1,5"]
|=================
| Server | Cisco UCS 220M3
| CPU: | 2 x Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz (low end)
| NICs: | 2(NICS)x (2x10GbE) (4 ports) Intel 82599EB Total 4 ports
| NUMA | NIC0-NUMA0 , NIC1-NUMA1
| Memory | 2x16GB
| PCIe | 1x PCIe16 + 1xPCIe8
| OS: | Fedora 18 - baremetal
|=================
image:images/kiwi02_latest_test_runs.png[title="trex09 test runs",align="left",width={p_width}, link="images/kiwi02_latest_test_runs.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/kiwi02_latest_test_runs_stats.csv[]
|===
=== Trend: Trend: Performance Over Time
image:images/kiwi02_trend_graph.png[title="kiwi02_trend_graph",align="left",width={p_width}, link="images/kiwi02_trend_graph.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/kiwi02_trend_stats.csv[]
|===
== Setup: Cisco 1300 VIC (2x40GbE) trex-11
[IMPORTANT]
=====================================
* this setup is not baremetal, performance are low due to that
=====================================
.Setup Details
[options='header',cols="1,5"]
|=================
| Server | Cisco UCS 240M3
| CPU: | 2 sockets x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 8 cores
| NICs: | 1 NIC x ( 2x 40GbE ) Cisco 1300 VIC
| NUMA | Unknown due to ESXi
| Memory | 2x16GB
| PCIe | 2x PCIe16 + 2xPCIe4
| OS: | Ubuntu 14 - ESXi only 4 cores are utilized
|=================
image:images/trex11_latest_test_runs.png[title="trex11 test runs",align="left",width={p_width}, link="images/trex11_latest_test_runs.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex11_latest_test_runs_stats.csv[]
|===
=== Trend: Performance Over Time
image:images/trex11_trend_graph.png[title="trex11_trend_graph",align="left",width={p_width}, link="images/trex11_trend_graph.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex11_trend_stats.csv[]
|===
== Setup: Mellanox ConnectX-4 (2x100GbE) trex-07
.Setup Details
[options='header',cols="1,5"]
|=================
| Server | Cisco UCS 240
| CPU: | 2 sockets x Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 8 cores
| NICs: | 1 NIC x 100G (2 ports) Mellanox ConnectX-4 NUMA#1
| NUMA | One NIC on NUMA1
| Memory | 2x16GB
| PCIe | 2x PCIe16 + 2xPCIe4
| OS: | CentOS 7.2 - baremetal
|=================
image:images/trex07_latest_test_runs.png[title="trex07 test runs",align="left",width={p_width}, link="images/trex07_latest_test_runs.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex07_latest_test_runs_stats.csv[]
|===
=== Trend: Performance Over Time
image:images/trex07_trend_graph.png[title="trex07_trend_graph",align="left",width={p_width}, link="images/trex07_trend_graph.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/trex07_trend_stats.csv[]
|===
== Comparison between Mellanox ConnectX-4 (2x100GbE) (trex-07) and XL710 (4x40GbE) (trex-08)
image:images/_comparison.png[title="comparison",align="left",width={p_width}, link="images/_comparison.png"]
[format="csv", options="header",halign='center']
|===
include::build/images/_comparison_stats_table.csv[]
|===
== Raw information
raw information can be found here link:images/_detailed_table.csv[raw_info.csv]
|
