diff options
Diffstat (limited to 'docs/report/vpp_performance_tests')
| -rw-r--r-- | docs/report/vpp_performance_tests/csit_release_notes.rst | 42 | ||||
| -rw-r--r-- | docs/report/vpp_performance_tests/overview.rst | 133 | ||||
| -rw-r--r-- | docs/report/vpp_performance_tests/test_environment.rst | 12 |
3 files changed, 91 insertions, 96 deletions
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 0ee3e82f59..c9e62e8415 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -6,18 +6,18 @@ Changes in CSIT |release| #. Test environment changes in VPP data plane performance tests:
- - Further characterization and optimizations of VPP vhost-user and VM
- test methodology and test environment;
+ - Further characterization and optimizations of VPP vhost-user and VM test
+ methodology and test environment;
- Tests with varying Qemu virtio queue (a.k.a. vring) sizes:
[vr256] default 256 descriptors, [vr1024] 1024 descriptors to
optimize for packet throughput;
- - Tests with varying Linux CFS (Completely Fair Scheduler)
- settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1)
- policy applied to all data plane threads handling test packet
- path including all VPP worker threads and all Qemu testpmd
- poll-mode threads;
+ - Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)`
+ settings: [cfs] default settings, [cfsrr1] :abbr:`CFS (Completely Fair
+ Scheduler)` RoundRobin(1) policy applied to all data plane threads
+ handling test packet path including all VPP worker threads and all Qemu
+ testpmd poll-mode threads;
- Resulting test cases are all combinations with [vr256,vr1024] and
[cfs,cfsrr1] settings;
@@ -28,8 +28,8 @@ Changes in CSIT |release| #. Code updates and optimizations in CSIT performance framework:
- Complete CSIT framework code revision and optimizations as descried
- on CSIT wiki page
- `Design_Optimizations <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.
+ on CSIT wiki page `Design_Optimizations
+ <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.
- For more detail see the :ref:`CSIT Framework Design <csit-design>` section
in this report;
@@ -85,14 +85,14 @@ Changes in CSIT |release| Performance Improvements
------------------------
-Substantial improvements in measured packet throughput have been
-observed in a number of CSIT |release| tests listed below, with relative
-increase of double-digit percentage points. Relative improvements for
-this release are calculated against the test results listed in CSIT
-|release-1| report. The comparison is calculated between the mean values
-based on collected and archived test results' samples for involved VPP
-releases. Standard deviation has been also listed for CSIT |release|.
-VPP-16.09 and VPP-17.01 numbers are provided for reference.
+Substantial improvements in measured packet throughput have been observed in a
+number of CSIT |release| tests listed below, with relative increase of
+double-digit percentage points. Relative improvements for this release are
+calculated against the test results listed in CSIT |release-1| report. The
+comparison is calculated between the mean values based on collected and
+archived test results' samples for involved VPP releases. Standard deviation
+has been also listed for CSIT |release|. VPP-16.09 and VPP-17.01 numbers are
+provided for reference.
NDR Throughput
~~~~~~~~~~~~~~
@@ -121,10 +121,10 @@ Measured improvements are in line with VPP code optimizations listed in Other Performance Changes
-------------------------
-Other changes in measured packet throughput, with either minor relative
-increase or decrease, have been observed in a number of CSIT |release|
-tests listed below. Relative changes are calculated against the test
-results listed in CSIT |release-1| report.
+Other changes in measured packet throughput, with either minor relative increase
+or decrease, have been observed in a number of CSIT |release| tests listed
+below. Relative changes are calculated against the test results listed in CSIT
+|release-1| report.
NDR Throughput
~~~~~~~~~~~~~~
diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst index 5cf7b1d355..89f5958075 100644 --- a/docs/report/vpp_performance_tests/overview.rst +++ b/docs/report/vpp_performance_tests/overview.rst @@ -5,9 +5,8 @@ Tested Physical Topologies -------------------------- CSIT VPP performance tests are executed on physical baremetal servers hosted by -LF FD.io project. Testbed physical topology is shown in the figure below. - -:: +:abbr:`LF (Linux Foundation)` FD.io project. Testbed physical topology is shown +in the figure below.:: +------------------------+ +------------------------+ | | | | @@ -60,9 +59,7 @@ N>1 DUT a logical service chain forwarding topology is created on DUT by applying L2 or IPv4/IPv6 configuration depending on the test suite. DUT test topology with N VM/LXC instances is shown in the figure below including applicable packet flow thru the DUTs and VMs/LXCs (marked in -the figure with ``***``). - -:: +the figure with ``***``).:: +-------------------------+ +-------------------------+ | +---------+ +---------+ | | +---------+ +---------+ | @@ -92,22 +89,22 @@ Hence the external throughput rates measured by TG and listed in this report must be multiplied by (N+1) to represent the actual DUT aggregate packet forwarding rate. -Note that reported DUT (VPP) performance results are specific to the -SUTs tested. Current LF FD.io SUTs are based on Intel XEON E5-2699v3 -2.3GHz CPUs. SUTs with other CPUs are likely to yield different results. -A good rule of thumb, that can be applied to estimate VPP packet +Note that reported DUT (VPP) performance results are specific to the SUTs +tested. Current :abbr:`LF (Linux Foundation)` FD.io SUTs are based on Intel +XEON E5-2699v3 2.3GHz CPUs. SUTs with other CPUs are likely to yield different +results. A good rule of thumb, that can be applied to estimate VPP packet thoughput for Phy-to-Phy (NIC-to-NIC, PCI-to-PCI) topology, is to expect the forwarding performance to be proportional to CPU core frequency, assuming CPU is the only limiting factor and all other SUT parameters equivalent to FD.io CSIT environment. The same rule of thumb can be also -applied for Phy-to-VM/LXC-to-Phy (NIC-to-VM/LXC-to-NIC) topology, but -due to much higher dependency on intensive memory operations and -sensitivity to Linux kernel scheduler settings and behaviour, this -estimation may not always yield good enough accuracy. +applied for Phy-to-VM/LXC-to-Phy (NIC-to-VM/LXC-to-NIC) topology, but due to +much higher dependency on intensive memory operations and sensitivity to Linux +kernel scheduler settings and behaviour, this estimation may not always yield +good enough accuracy. -For detailed LF FD.io test bed specification and physical topology -please refer to -`LF FD.io CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_. +For detailed :abbr:`LF (Linux Foundation)` FD.io test bed specification and +physical topology please refer to `LF FD.io CSIT testbed wiki page +<https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_. Performance Tests Coverage -------------------------- @@ -115,7 +112,7 @@ Performance Tests Coverage Performance tests are split into two main categories: - Throughput discovery - discovery of packet forwarding rate using binary search - in accordance to RFC2544. + in accordance to :rfc:`2544`. - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss; followed by one-way packet latency measurements at 10%, 50% and 100% of @@ -185,60 +182,59 @@ CSIT |release| includes following performance test suites, listed per NIC type: - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames with MAC learning. -Execution of performance tests takes time, especially the throughput -discovery tests. Due to limited HW testbed resources available within -FD.io labs hosted by Linux Foundation, the number of tests for NICs -other than X520 (a.k.a. Niantic) has been limited to few baseline tests. -CSIT team expect the HW testbed resources to grow over time, so that -complete set of performance tests can be regularly and(or) continuously -executed against all models of hardware present in FD.io labs. +Execution of performance tests takes time, especially the throughput discovery +tests. Due to limited HW testbed resources available within FD.io labs hosted +by :abbr:`LF (Linux Foundation)`, the number of tests for NICs other than X520 +(a.k.a. Niantic) has been limited to few baseline tests. CSIT team expect the +HW testbed resources to grow over time, so that complete set of performance +tests can be regularly and(or) continuously executed against all models of +hardware present in FD.io labs. Performance Tests Naming ------------------------ -CSIT |release| follows a common structured naming convention for all -performance and system functional tests, introduced in CSIT |release-1|. +CSIT |release| follows a common structured naming convention for all performance +and system functional tests, introduced in CSIT |release-1|. -The naming should be intuitive for majority of the tests. Complete -description of CSIT test naming convention is provided on `CSIT test naming wiki +The naming should be intuitive for majority of the tests. Complete description +of CSIT test naming convention is provided on `CSIT test naming wiki <https://wiki.fd.io/view/CSIT/csit-test-naming>`_. Methodology: Multi-Core and Multi-Threading ------------------------------------------- -**Intel Hyper-Threading** - CSIT |release| performance tests are -executed with SUT servers' Intel XEON processors configured in Intel -Hyper-Threading Disabled mode (BIOS setting). This is the simplest -configuration used to establish baseline single-thread single-core -application packet processing and forwarding performance. Subsequent -releases of CSIT will add performance tests with Intel Hyper-Threading -Enabled (requires BIOS settings change and hard reboot of server). +**Intel Hyper-Threading** - CSIT |release| performance tests are executed with +SUT servers' Intel XEON processors configured in Intel Hyper-Threading Disabled +mode (BIOS setting). This is the simplest configuration used to establish +baseline single-thread single-core application packet processing and forwarding +performance. Subsequent releases of CSIT will add performance tests with Intel +Hyper-Threading Enabled (requires BIOS settings change and hard reboot of +server). -**Multi-core Tests** - CSIT |release| multi-core tests are executed in -the following VPP thread and core configurations: +**Multi-core Tests** - CSIT |release| multi-core tests are executed in the +following VPP thread and core configurations: #. 1t1c - 1 VPP worker thread on 1 CPU physical core. #. 2t2c - 2 VPP worker threads on 2 CPU physical cores. -VPP worker threads are the data plane threads. VPP control thread is -running on a separate non-isolated core together with other Linux -processes. Note that in quite a few test cases running VPP workers on 2 -physical cores hits the tested NIC I/O bandwidth or packets-per-second -limit. +VPP worker threads are the data plane threads. VPP control thread is running on +a separate non-isolated core together with other Linux processes. Note that in +quite a few test cases running VPP workers on 2 physical cores hits the tested +NIC I/O bandwidth or packets-per-second limit. Methodology: Packet Throughput ------------------------------ Following values are measured and reported for packet throughput tests: -- NDR binary search per RFC2544: +- NDR binary search per :rfc:`2544`: - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x <per direction packets-per-second>)" - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per second> Gbps (untagged)" -- PDR binary search per RFC2544: +- PDR binary search per :rfc:`2544`: - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x <per direction packets-per-second>)" @@ -282,27 +278,27 @@ latency values are measured using following methodology: Methodology: KVM VM vhost ------------------------- -CSIT |release| introduced test environment configuration changes to KVM Qemu vhost- -user tests in order to more representatively measure |vpp-release| performance -in configurations with vhost-user interfaces and different Qemu settings. +CSIT |release| introduced test environment configuration changes to KVM Qemu +vhost-user tests in order to more representatively measure |vpp-release| +performance in configurations with vhost-user interfaces and different Qemu +settings. -FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following environment settings +FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following +environment settings: -- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: - [vr256] default 256 descriptors, [vr1024] 1024 descriptors to - optimize for packet throughput; +- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr256] default 256 + descriptors, [vr1024] 1024 descriptors to optimize for packet throughput; -- Tests with varying Linux CFS (Completely Fair Scheduler) - settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1) - policy applied to all data plane threads handling test packet - path including all VPP worker threads and all Qemu testpmd - poll-mode threads; +- Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)` settings: + [cfs] default settings, [cfsrr1] CFS RoundRobin(1) policy applied to all data + plane threads handling test packet path including all VPP worker threads and + all Qemu testpmd poll-mode threads; - Resulting test cases are all combinations with [vr256,vr1024] and [cfs,cfsrr1] settings; -- Adjusted Linux kernel CFS scheduler policy for data plane threads used - in CSIT is documented in +- Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)` scheduler policy + for data plane threads used in CSIT is documented in `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_. The purpose is to verify performance impact (NDR, PDR throughput) and same test measurements repeatability, by making VPP and VM data plane @@ -312,15 +308,14 @@ FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following env Methodology: LXC Container memif -------------------------------- -CSIT |release| introduced new tests - VPP Memif virtual interface -(shared memory interface) tests interconnecting VPP instances over -memif. VPP vswitch instance runs in bare-metal user-mode handling Intel -x520 NIC 10GbE interfaces and connecting over memif (Master side) -virtual interfaces to another instance of VPP running in bare-metal -Linux Container (LXC) with memif virtual interfaces (Slave side). LXC -runs in a priviliged mode with VPP data plane worker threads pinned to -dedicated physical CPU cores per usual CSIT practice. Both VPP run the -same version of software. This test topology is equivalent to existing +CSIT |release| introduced new tests - VPP Memif virtual interface (shared memory +interface) tests interconnecting VPP instances over memif. VPP vswitch instance +runs in bare-metal user-mode handling Intel x520 NIC 10GbE interfaces and +connecting over memif (Master side) virtual interfaces to another instance of +VPP running in bare-metal :abbr:`LXC (Linux Container)` with memif virtual +interfaces (Slave side). LXC runs in a priviliged mode with VPP data plane worker +threads pinned to dedicated physical CPU cores per usual CSIT practice. Both VPP +run the same version of software. This test topology is equivalent to existing tests with vhost-user and VMs. Methodology: IPSec with Intel QAT HW cards @@ -363,7 +358,7 @@ TRex is installed and run on the TG compute node. The typical procedure is: - There are traffic streams dynamically prepared for each test, based on traffic profiles. The traffic is sent and the statistics obtained using - trex_stl_lib.api.STLClient. + :command:`trex_stl_lib.api.STLClient`. **Measuring packet loss** diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst index ffc47fc558..494799ffaf 100644 --- a/docs/report/vpp_performance_tests/test_environment.rst +++ b/docs/report/vpp_performance_tests/test_environment.rst @@ -1,17 +1,17 @@ Test Environment
================
-CSIT performance tests are executed on the three identical physical
-testbeds hosted by Linux Foundation for FD.io project. Each testbed
-consists of two servers acting as Systems Under Test (SUT) and one
-server acting as Traffic Generator (TG).
+CSIT performance tests are executed on the three identical physical testbeds
+hosted by :abbr:`LF (Linux Foundation)` for FD.io project. Each testbed
+consists of two servers acting as Systems Under Test (SUT) and one server
+acting as Traffic Generator (TG).
Server Specification and Configuration
--------------------------------------
Complete specification and configuration of compute servers used in CSIT
-physical testbeds is maintained on wiki page
-`CSIT LF Testbeds <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
+physical testbeds is maintained on wiki page `CSIT LF Testbeds
+<https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
SUT Configuration
-----------------
|