diff options
| -rw-r--r-- | docs/report/introduction/introduction.rst | 3 | ||||
| -rw-r--r-- | docs/report/introduction/methodology.rst | 81 | ||||
| -rw-r--r-- | docs/report/vpp_performance_tests/test_environment.rst | 2 |
3 files changed, 60 insertions, 26 deletions
diff --git a/docs/report/introduction/introduction.rst b/docs/report/introduction/introduction.rst index 24c5970725..9990d47d5d 100644 --- a/docs/report/introduction/introduction.rst +++ b/docs/report/introduction/introduction.rst @@ -13,7 +13,8 @@ available for download. - **Test Scenarios Overview**: A brief overview of test scenarios covered in this report. - **Physical Testbeds**: Description of physical testbeds. - - **Performance Test Methodology**: Benchmarking test methodologies. + - **Test Methodology**: Performance benchmarking and functional test + methodologies. #. VPP PERFORMANCE: VPP performance tests executed in physical FD.io testbeds. diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index 89db393973..f6217d89a6 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -1,16 +1,17 @@ -.. _performance_test_methodology: +.. _test_methodology: -Performance Test Methodology -============================ +Test Methodology +================ -Throughput ----------- +Data Plane Throughput +--------------------- -Packet and bandwidth throughput are measured in accordance with -:rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search (MLRsearch), an -optimized binary search algorithm, that measures SUT/DUT throughput at -different Packet Loss Ratio (PLR) values. +Network data plane packet and bandwidth throughput are measured in +accordance with :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search +(MLRsearch), an optimized throughput search algorithm, that measures +SUT/DUT packet throughput rates at different Packet Loss Ratio (PLR) +values. Following MLRsearch values are measured across a range of L2 frame sizes and reported: @@ -39,13 +40,15 @@ All rates are reported from external Traffic Generator perspective. .. _mlrsearch_algorithm: -MLRsearch Algorithm -------------------- +MLRsearch Tests +--------------- -Multiple Loss Rate search (MLRsearch) is a new search algorithm +Multiple Loss Rate search (MLRsearch) tests use new search algorithm implemented in FD.io CSIT project. MLRsearch discovers multiple packet throughput rates in a single search, with each rate associated with a -distinct Packet Loss Ratio (PLR) criteria. +distinct Packet Loss Ratio (PLR) criteria. MLRsearch is being +standardized in IETF with `draft-vpolak-mkonstan-mlrsearch-XX +<https://tools.ietf.org/html/draft-vpolak-mkonstan-mlrsearch-00>`_. Two throughput measurements used in FD.io CSIT are Non-Drop Rate (NDR, with zero packet loss, PLR=0) and Partial Drop Rate (PDR, with packet @@ -318,15 +321,22 @@ but without detailing their mutual interaction. than pure binary search, the implemented tests fail themselves when the search takes too long (given by argument *timeout*). -Maximum Receive Rate MRR ------------------------- +(B)MRR Throughput +----------------- -MRR tests measure the packet forwarding rate under the maximum -load offered by traffic generator over a set trial duration, +Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests, +as they provide a maximum "raw" throughput benchmark for development and +testing community. MRR tests measure the packet forwarding rate under +the maximum load offered by traffic generator over a set trial duration, regardless of packet loss. Maximum load for specified Ethernet frame size is set to the bi-directional link rate. -Current parameters for MRR tests: +In |csit-release| MRR test code has been updated with a configurable +burst MRR parameters: trial duration and number of trials in a single +burst. This enabled a new Burst MRR (BMRR) methodology for more precise +performance trending. + +Current parameters for BMRR tests: - Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all quoted sizes include frame CRC, but exclude per frame transmission @@ -346,17 +356,25 @@ Current parameters for MRR tests: XL710. Packet rate for other tested frame sizes is limited by PCIe Gen3 x8 bandwidth limitation of ~50Gbps. -- Trial duration: 10sec. +- Trial duration: 1 sec. + +- Number of trials per burst: 10. Similarly to NDR/PDR throughput tests, MRR test should be reporting bi- directional link rate (or NIC rate, if lower) if tested VPP configuration can handle the packet rate higher than bi-directional link rate, e.g. large packet tests and/or multi-core tests. -MRR tests are used for continuous performance trending and for -comparison between releases. Daily trending job tests subset of frame -sizes, focusing on 64B (78B for IPv6) for all tests and IMIX for -selected tests (vhost, memif). +MRR tests are currently used for FD.io CSIT continuous performance +trending and for comparison between releases. Daily trending job tests +subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and +IMIX for selected tests (vhost, memif). + +MRR-like measurements are being used to establish starting conditions +for experimental Probabilistic Loss Ratio Search (PLRsearch) used for +soak testing, aimed at verifying continuous system performance over an +extended period of time, hours, days, weeks, months. PLRsearch code is +currently in experimental phase in FD.io CSIT project. Packet Latency -------------- @@ -510,7 +528,8 @@ following environment settings: [cfs,cfsrr1] settings. - Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)` scheduler policy for data plane threads used in CSIT is documented in - `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_. + `CSIT Performance Environment Tuning wiki + <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_. - The purpose is to verify performance impact (MRR and NDR/PDR throughput) and same test measurements repeatability, by making VPP and VM data plane threads less susceptible to other Linux OS system @@ -562,6 +581,20 @@ VMs as described earlier in :ref:`tested_physical_topologies`. Further documentation is available in :ref:`container_orchestration_in_csit`. +VPP_Device Functional +--------------------- + +|csit-release| added new VPP_Device test environment for functional VPP +device tests integrated into LFN CI/CD infrastructure. VPP_Device tests +run on 1-Node testbeds (1n-skx, 1n-arm) and rely on Linux SRIOV Virtual +Function (VF), dot1q VLAN tagging and external loopback cables to +facilitate packet passing over exernal physical links. Initial focus is +on few baseline tests. Existing CSIT VIRL tests can be moved to +VPP_Device framework by changing L1 and L2 KW(s). RF test definition +code stays unchanged with the exception of requiring adjustments from +3-Node to 2-Node logical topologies. CSIT VIRL to VPP_Device migration +is expected in the next CSIT release. + IPSec on Intel QAT ------------------ diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst index 6dd6ca35d8..8f92c6ef20 100644 --- a/docs/report/vpp_performance_tests/test_environment.rst +++ b/docs/report/vpp_performance_tests/test_environment.rst @@ -103,6 +103,6 @@ below: } Description of VPP startup settings used in CSIT is provided in -:ref:`performance_test_methodology`. +:ref:`test_methodology`. .. include:: ../introduction/test_environment_tg.rst |