rls1707 report: edits of vpp_performance_tests/rls_notes

Change-Id: Ia7fbe0fbfa83c7f61f59850ea98e00405fc82711
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit 912acb80b6607a6b904242a0ec695c874580bd02)

1 files changed, 82 insertions, 26 deletions

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 0a5d25e6b6..782fa42840 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -4,45 +4,95 @@ CSIT Release Notes
 Changes in CSIT |release|

 -------------------------

 

-#. VPP performance test environment changes

+#. Test environment changes in VPP data plane performance tests:

 

-   - Further optimizations of VM and vhost-user test environment - various

-     Qemu virtio queue size testing with value of 256 and 1024. Applied

-     Linux CFS optimization to run VPP worker threads and Qemu worker threads

-     with highest priority.

+   - Further characterization and optimizations of VPP vhost-user and VM

+     test methodology and test environment;

 

-#. VPP performance test framework changes

+     - Tests with varying Qemu virtio queue (a.k.a. vring) sizes:

+       [vr256] default 256 descriptors, [vr1024] 1024 descriptors to

+       optimize for packet throughput;

 

-   - Full code review, optimization and refactor.

+     - Tests with varying Linux CFS (Completely Fair Scheduler)

+       settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1)

+       policy applied to all data plane threads handling test packet

+       path including all VPP worker threads and all Qemu testpmd

+       poll-mode threads;

 

-#. T-rex changes

+     - Resulting test cases are all combinations with [vr256,vr1024] and

+       [cfs,cfsrr1] settings;

 

-   - Full refactor of T-rex driver and introduce of traffic profiles that

-     improves readability, manageability of traffic profiles for various

-     test scenarios.

+     - For more detail see performance results observations section in

+       this report;

+

+#. Code updates and optimizations in CSIT performance framework:

+

+   - Complete CSIT framework code revision and optimizations as descried

+     on CSIT wiki page

+     `Design_Optimizations <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.

+

+   - For more detail see the CSIT Framework Design section in this

+     report;

+

+#. Changes to CSIT driver for TRex Traffic Generator:

+

+   - Complete refactor of TRex CSIT driver;

+

+   - Introduction of packet traffic profiles to improve usability and

+     manageability of traffic profiles for a growing number of test

+     scenarios.

+

+   - Support for packet traffic profiles to test IPv4/IPv6 stateful and

+     stateless DUT data plane features;

 

 #. Added VPP performance tests

 

-   - **LXC memif**

+   - **Linux Container VPP memif virtual interface tests**

 

-     - Memif interface tests interconnecting two VPP instances on single SUT.

-       Master VPP instance running on native OS with Intel x520 NIC and guest

-       VPP instance running in Linux Container (LXC) doing the L2 cross

-       connect loop. LXC running in privileged mode is pinned to dedicated

-       cores. All VPP instances are same version.

+     - VPP Memif virtual interface (shared memory interface) tests

+       interconnecting VPP instances over memif. VPP vswitch

+       instance runs in bare-metal user-mode handling Intel x520 NIC

+       10GbE interfaces and connecting over memif (Master side) virtual

+       interfaces to another instance of VPP running in bare-metal Linux

+       Container (LXC) with memif virtual interfaces (Slave side). LXC

+       runs in a priviliged mode with VPP data plane worker threads

+       pinned to dedicated physical CPU cores per usual CSIT practice.

+       Both VPP run the same version of software. This test topology is

+       equivalent to existing tests with vhost-user and VMs.

 

    - **Stateful Security Groups**

 

-   - **VM vhost use cases**

+     - New tests of VPP stateful security-groups a.k.a. acl-plugin

+       functionally compatible with networking-vpp OpenStack;

+

+     - New tested security-groups access-control-lists (acl)

+       configuration variants include: [iaclNsl] input acl stateless,

+       [oaclNsl] output acl stateless, [iaclNsf] input acl stateful

+       a.k.a. reflect, [oaclNsf] output acl stateful a.k.a. reflect,

+       where N is number of access-control-entries (ace) in the acl.

+

+     - Testing packet flows transmitted by TG: 100, 10k, 100k, always

+       hitting the last permit entry in acl.

+

+   - **VPP vhost and VM tests**

+

+     - New VPP vhost-user and VM test cases to benchmark performance of

+       VPP and VM topologies with Qemu and CFS policy combinations of

+       [vr256,vr1024] x [cfs,cfsrr1];

+

+     - Statistical analysis of repeatibility of results;

 

 Performance Improvements

 ------------------------

 

 Substantial improvements in measured packet throughput have been

 observed in a number of CSIT |release| tests listed below, with relative

-increase  of double-digit percentage points. Relative improvements are

-calculated against the test results listed in CSIT |release-1| report.

-VPP-16.09 numbers are provided for reference.

+increase  of double-digit percentage points. Relative improvements for

+this release are calculated against the test results listed in CSIT

+|release-1| report. The comparison is calculated between the mean values

+based on collected and archived test results' samples for involved VPP

+releases. Standard deviation has been also listed for CSIT |release|.

+VPP-16.09 and VPP-17.01 numbers are provided for reference.

 

 NDR Throughput

 ~~~~~~~~~~~~~~

@@ -105,21 +155,27 @@ Here is the list of known issues in CSIT |release| for VPP performance tests:
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 | # | Issue                                           | Jira ID    | Description                                                     |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 1 | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |

+| 1 | Security-groups acl-plugin scale tests failure  | CSIT-xxx   | VPP with 2 worker threads crashes during security-groups        |

+|   | with stateful acls if VPP with 2 worker threads | VPP-912    | iaclNsf and oaclNsf tests with 100k flows.                      |

++---+-------------------------------------------------+------------+-----------------------------------------------------------------+

+| 2 | VPP fails memif tests in 4 worker 2 core setup  | CSIT-xxx   | VPP with 4 worker threads running on 2 physical cores crashes   |

+|   |                                                 | VPP-xxx    | during memif tests. Initial debugging points to DPDK code       |

++---+-------------------------------------------------+------------+-----------------------------------------------------------------+

+| X | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |

 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |

 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 2 | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |

+| X | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |

 |   |                                                 |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 3 | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |

+| X | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |

 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 4 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |

+| X | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |

 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |

 |   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 5 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |

+| X | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |

 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+