Report: VPP perf, updated changes in rls and list of known issues.

Change-Id: Iaeb4eac41df6bd688ce5b2bcf2baf8db956f77bb
Signed-off-by: Tibor Frank <tifrank@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

1 files changed, 60 insertions, 48 deletions

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 8f37fed284..d4b0fc275f 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -6,54 +6,71 @@ Changes in |csit-release|
 
 #. VPP PERFORMANCE TESTS
 
-   - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now
-     configurable trial duration and number of consecutive executions.
-     Coverage of MRR tests has been extended across more test
-     scenarios. MRR tests are used for continuous performance trending
-     and for comparison between VPP releases.
-
-   - **MLRsearch Throughput**: MLRsearch algorithm has been introduced
-     for all NDR and PDR throughput tests. All tests that previously
-     used binary search got converted to MLRsearch. Coverage of NDR/PDR
-     tests has been extended across more test scenarios.
-
-   - **L2patch Tests**: Tests measure performance of VPP L2patch, the
-     fastest L2 forwarding path implemented in VPP, that cross-links
-     RX and TX of two physical interfaces.
-
-   - **2-Node Tests**: A new baseline set of 2-node tests covering base
-     ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake
-     testbeds.
-
-   - **Generated tests**: Simplified and unified test structure, semi-
-     autogenerated by generator script. Test generator is currently
-     able to create test combinations with various frame size and
-     cores combinations. All existing test cases were converted to new
-     format.
-
-   - **Simultaneous Multi-Threading**: SMT-aware detection of server
-     processor operation mode (HyperThreading enabled/disabled) with
-     associated compute resource configuration including thread
-     affinity, number of Rx queues and DPDK I/O mbufs. Tests are
-     automatically tagged during execution to indicate executed thread
-     configuration.
-
-   - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node
-     physical testbed topologies based on the new SuperMirco servers
-     each with two Intel Xeon Skylake Platinum processors. Full
-     Ansible playbooks refactor for quick server (re)installation and
-     reference pointers of configuration.
+   - **BMRR Throughput**: MRR (Maximum Receive Rate) test code has been
+     updated with a configurable burst MRR parameters: trial duration
+     and number of trials in a single burst. Enables a new Burst MRR
+     (BMRR for short) methodology for more precise performance
+     trending. See updated :ref:`performance_test_methodology` section
+     for more details.
 
-#. PRESENTATION AND ANALYTICS LAYER
+   - **2n-skx tests**: Added performamce tests for 2n-skx (2-Node Xeon
+     Skylake) testbeds: focus on baseline and scale tests, including
+     VM vhost and Container memif tests.
 
-   - **Performance trending**: Further improvements of continuous
-     performance trending, anomaly detection and analysis.
+   - **3n-skx tests**: Added performamce tests for 3n-skx (3-Node Xeon
+     Skylake) testbeds: VM vhost and Container memif tests.
+
+   - **VXLAN Scale Tests**: Added performamce tests for VXLAN scale with
+     dot1q and VPP L2BD.
+
+   - **AVF Driver Tests**: Added performamce tests for i40e AVF driver
+     on VPP, no DPDK required.
+
+   - **QAT**: Fixed reoccuring issues with QAT crypto accelerator cards.
+
+   - **K8s/Ligato in Trending**: Added K8s/Ligato Container memif tests
+     to daily trending.
+
+#. TEST FRAMEWORK
+
+   - **Experimental Soak Tests**: Added performamce soak tests framework
+     code for extended time duration tests and  throughput discovery
+     at given PLR and at give total test time e.g. minutes, hours,
+     days, weeks, months, years. See updated
+     :ref:`performance_test_methodology` section for more details.
+
+   - **Trending Tests BMRR**: Used new Burst MRR (BMRR) tests for daily
+     trending.
+
+   - **Per VPP Patch Performance Checks**: Per VPP gerrit patch vs.
+     parent performance tests, anomaly detection and no verify voting
+     (-1/0/+1) yet. Manual trigger only. Not "marketed" to FD.io
+     community yet to avoid excessive LFN FD.io physical performance
+     testbed blocking.
 
-#. TEST FRAMEWORK OPTIMIZATIONS
+   - **Patch-on-Patch Infra**: Added capability to run performance tests
+     using CSIT gerrit patch code testing VPP gerrit patch code, i.e.
+     before any VPP and/or CSIT code is merged into git branch.
+
+   - **CSIT PAPI Support**: Initial implementation of PAPI L1 KWs in
+     CSIT using VPP Python bindings. Required for migraing away from
+     VAT. Very few L1 KWs implemented ("show version", "show
+     interfaces").
 
    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
      removal of redundant RF keywords.
 
+#. PRESENTATION AND ANALYTICS LAYER
+
+   - **Graphs Layout Improvements**: Improved performance graphs layout
+     for better readibility and maintenance: test grouping, axis
+     labels, descriptions, other informative decoration. Master report
+     generated. 744 graphs(!)
+
+
+   - **Performance Trending**: Further improvements of continuous
+     performance trending, anomaly detection and analysis.
+
 .. raw:: latex
 
     \clearpage
@@ -80,12 +97,7 @@ List of known issues in |csit-release| for VPP performance tests:
 | 4  | `CSIT-1243                              | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds.                      |
 |    | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%.                                                          |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 5  | `CSIT-1245                              | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
-|    | <https://jira.fd.io/browse/CSIT-1245>`_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring.                                |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 6  | `CSIT-1246                              | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07.        |
+| 5  | `CSIT-1246                              | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07.        |
 |    | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either.                                                                              |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 7  | `CSIT-1253                              | VPP lbdpdk link bonding tests failing due to interfaces not coming up.                                                          |
-|    | <https://jira.fd.io/browse/CSIT-1253>`_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail.                                                   |
-+----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+