diff options
| -rw-r--r-- | docs/report/vpp_performance_tests/csit_release_notes.rst | 108 |
1 files changed, 60 insertions, 48 deletions
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 8f37fed284..d4b0fc275f 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -6,54 +6,71 @@ Changes in |csit-release| #. VPP PERFORMANCE TESTS - - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now - configurable trial duration and number of consecutive executions. - Coverage of MRR tests has been extended across more test - scenarios. MRR tests are used for continuous performance trending - and for comparison between VPP releases. - - - **MLRsearch Throughput**: MLRsearch algorithm has been introduced - for all NDR and PDR throughput tests. All tests that previously - used binary search got converted to MLRsearch. Coverage of NDR/PDR - tests has been extended across more test scenarios. - - - **L2patch Tests**: Tests measure performance of VPP L2patch, the - fastest L2 forwarding path implemented in VPP, that cross-links - RX and TX of two physical interfaces. - - - **2-Node Tests**: A new baseline set of 2-node tests covering base - ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake - testbeds. - - - **Generated tests**: Simplified and unified test structure, semi- - autogenerated by generator script. Test generator is currently - able to create test combinations with various frame size and - cores combinations. All existing test cases were converted to new - format. - - - **Simultaneous Multi-Threading**: SMT-aware detection of server - processor operation mode (HyperThreading enabled/disabled) with - associated compute resource configuration including thread - affinity, number of Rx queues and DPDK I/O mbufs. Tests are - automatically tagged during execution to indicate executed thread - configuration. - - - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node - physical testbed topologies based on the new SuperMirco servers - each with two Intel Xeon Skylake Platinum processors. Full - Ansible playbooks refactor for quick server (re)installation and - reference pointers of configuration. + - **BMRR Throughput**: MRR (Maximum Receive Rate) test code has been + updated with a configurable burst MRR parameters: trial duration + and number of trials in a single burst. Enables a new Burst MRR + (BMRR for short) methodology for more precise performance + trending. See updated :ref:`performance_test_methodology` section + for more details. -#. PRESENTATION AND ANALYTICS LAYER + - **2n-skx tests**: Added performamce tests for 2n-skx (2-Node Xeon + Skylake) testbeds: focus on baseline and scale tests, including + VM vhost and Container memif tests. - - **Performance trending**: Further improvements of continuous - performance trending, anomaly detection and analysis. + - **3n-skx tests**: Added performamce tests for 3n-skx (3-Node Xeon + Skylake) testbeds: VM vhost and Container memif tests. + + - **VXLAN Scale Tests**: Added performamce tests for VXLAN scale with + dot1q and VPP L2BD. + + - **AVF Driver Tests**: Added performamce tests for i40e AVF driver + on VPP, no DPDK required. + + - **QAT**: Fixed reoccuring issues with QAT crypto accelerator cards. + + - **K8s/Ligato in Trending**: Added K8s/Ligato Container memif tests + to daily trending. + +#. TEST FRAMEWORK + + - **Experimental Soak Tests**: Added performamce soak tests framework + code for extended time duration tests and throughput discovery + at given PLR and at give total test time e.g. minutes, hours, + days, weeks, months, years. See updated + :ref:`performance_test_methodology` section for more details. + + - **Trending Tests BMRR**: Used new Burst MRR (BMRR) tests for daily + trending. + + - **Per VPP Patch Performance Checks**: Per VPP gerrit patch vs. + parent performance tests, anomaly detection and no verify voting + (-1/0/+1) yet. Manual trigger only. Not "marketed" to FD.io + community yet to avoid excessive LFN FD.io physical performance + testbed blocking. -#. TEST FRAMEWORK OPTIMIZATIONS + - **Patch-on-Patch Infra**: Added capability to run performance tests + using CSIT gerrit patch code testing VPP gerrit patch code, i.e. + before any VPP and/or CSIT code is merged into git branch. + + - **CSIT PAPI Support**: Initial implementation of PAPI L1 KWs in + CSIT using VPP Python bindings. Required for migraing away from + VAT. Very few L1 KWs implemented ("show version", "show + interfaces"). - **General Code Housekeeping**: Ongoing RF keywords optimizations, removal of redundant RF keywords. +#. PRESENTATION AND ANALYTICS LAYER + + - **Graphs Layout Improvements**: Improved performance graphs layout + for better readibility and maintenance: test grouping, axis + labels, descriptions, other informative decoration. Master report + generated. 744 graphs(!) + + + - **Performance Trending**: Further improvements of continuous + performance trending, anomaly detection and analysis. + .. raw:: latex \clearpage @@ -80,12 +97,7 @@ List of known issues in |csit-release| for VPP performance tests: | 4 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | | | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | <https://jira.fd.io/browse/CSIT-1245>`_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. | -+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 6 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | +| 5 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | | | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either. | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 7 | `CSIT-1253 | VPP lbdpdk link bonding tests failing due to interfaces not coming up. | -| | <https://jira.fd.io/browse/CSIT-1253>`_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail. | -+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ + |