aboutsummaryrefslogtreecommitdiffstats
path: root/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst
blob: aba4e919441a3fbe08274139be78356dacac00b0 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
.. raw:: latex

    \clearpage

.. _VPP_Packet_Throughput:

Packet Throughput
=================

Throughput graphs are generated based on the results data obtained from
the |csit-release| test jobs. In order to verify benchmark results
repeatibility selected, CSIT performance tests are executed multiple
times (target: 10 times) on each physical testbed type. Box-and-Whisker
plots are used to display variations in measured throughput values.

Lists of tests selected for multiple execution and graphing are captured
per testbed type in `test_select_list_{testbed_type}.md
<https://git.fd.io/csit/tree/docs/job_specs>`_ files.

Graphs are split into sections as follows:

#. **Header 1**: VPP packet path and lookup types

   - **L2 Ethernet Switching**: L2 bridge-doman, L2 cross-connect and L2 patch
   - **IPv4 Routing**: IPv4 routing with /32 prefixes
   - **IPv6 Routing**: IPv6 routing with /128 prefixes
   - **SRv6 Routing**: SRv6 with IPv6 routing
   - **IPv4 Tunnels**: IPv4 overlay tunnels
   - **KVM VMs vhost-user**: KVM VMs connected over virtio and vhost-user interfaces
   - **LXC/DRC Container Memif**: Linux containers and Docker containers connected over Memif interfaces
   - **IPsec IPv4 Routing**: IPsec encryption/decryption with IPv4 routing
   - **Virtual Topology System**: VXLAN configurations with L2 bridge-domains

#. **Header 2**: testbeds and NIC models

   - section name format:

     - {**testbed_type**}-{**nic_model**}

   - **testbed_type**:

     - 2n-skx: 2-node Xeon Skylake
     - 3n-skx: 3-node Xeon Skylake
     - 2n-clx: 2-node Xeon Cascade Lake
     - 3n-hsw: 3-node Xeon Haswell
     - 3n-tsh: 3-node Arm TaiShan
     - 2n-dnv: 2-node Atom Denverton
     - 3n-dnv: 3-node Atom Denverton

   - **nic_model**:

     - xxv710: xxv710 2p25GE Intel (Fortville)
     - x710: x710 4p10GE Intel (Fortville)
     - xl710: xl710 2p40GE Intel (Fortville)
     - x520: x520 2p10GE Intel (Niantic)
     - x553: x553 2p10GE Intel (Niantic)

#. **Header 3**: test group names

   - section name format:

     - {**frame_size**}-{**worker_thread_core_cfg**}-{**vpp_functionality**}-{**vpp_lookup_type**}-{**baseline_scale**}-{**nic_driver**}

   - **frame_size**:

     - 64b: 64 byte frames, smallest frame size for untagged IPv4 packets
     - 78b: 78 byte frames, smallest frame size for untagged IPv6 packets
     - 114b: VXLAN encapsulated L2 frames
     - imix: a sequence of (7x64B, 4x570, 1x1518) byte frames

   - **worker_thread_core_cfg**:

     - 1t1c: 1 worker thread on 1 core, hyper-threading not used
     - 2t1c: 2 worker threads on 1 core, hyper-threading used

   - **vpp_functionality** (optional):

     - features: including input-acl, output-acl, macip-iacl, nat44
     - srv6: srv6 encap/decap, proxy
     - link-bonding: L2 link aggregation with 1 or 2 bonded links
     - ipsec: IPsec encryption/decryption with different ciphers
     - vts: Virtual Topology System specific tests

   - **vpp_lookup_type**:

     - l2switching, ip4routing, ip6routing, ip4tunnel, vhost, memif

   - **baseline_scale**:

     - base: baseline tests with less than 10 forwarding entries
     - scale: scale tests with up to 2 million forwarding entries
     - base-scale: both baseline and scale tests grouped together

   - **nic_driver**:

     - avf: VPP native avf driver for Intel Fortville NICs
     - i40e: dpdk poll mode driver for Intel Fortville NICs
     - ixgbe: dpdk poll mode driver for Intel Niantic NICs

For each test case, Box-and-Whisker plots show the quartiles (Min, 1st
quartile / 25th percentile, 2nd quartile / 50th percentile / mean, 3rd
quartile / 75th percentile, Max) across collected data set. Outliers are
plotted as individual points.

Additional information about graph data:

#. **Graph Title**: describes tested packet path, testbed topology,
   processor model, NIC model, packet size, number of cores and
   threads used by data plane workers and indication of VPP DUT
   configuration.

#. **X-axis Labels**: indices of individual test suites as listed in
   Graph Legend.

#. **Y-axis Labels**: measured Packets Per Second [pps] throughput
   values.

#. **Graph Legend**: lists X-axis indices with associated CSIT test
   suites executed to generate graphed test results.

#. **Hover Information**: lists minimum, first quartile, median,
   third quartile, and maximum. If either type of outlier is present the
   whisker on the appropriate side is taken to 1.5×IQR from the quartile
   (the "inner fence") rather than the max or min, and individual outlying
   data points are displayed as unfilled circles (for suspected outliers)
   or filled circles (for outliers). (The "outer fence" is 3×IQR from the
   quartile.)

.. note::

    Test results have been generated by
    `FD.io test executor vpp performance job 2n-skx`_,
    `FD.io test executor vpp performance job 3n-skx`_,
    `FD.io test executor vpp performance job 2n-clx`_,
    `FD.io test executor vpp performance job 2n-zn2`_,
    `FD.io test executor vpp performance job 3n-hsw`_,
    `FD.io test executor vpp performance job 3n-tsh`_,
    `FD.io test executor vpp performance job 2n-dnv`_ and
    `FD.io test executor vpp performance job 3n-dnv`_ with RF
    result files csit-vpp-perf-|srelease|-\*.zip
    `archived here <../../_static/archive/>`_.
    Required per test case data set size is **10**, but for VPP tests
    the actual size varies per test case and is <=10.

.. toctree::
    :maxdepth: 3

    l2
    ip4
    ip6
    srv6
    ip4_tunnels
    nat44
    vm_vhost
    container_memif
    ipsec
    vts