rls1804 report: updates to perf rls notes, methodology.

Change-Id: I54ef2e63ce03aee509fba7dcbfd1d7faabe6ef91
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

4 files changed, 321 insertions, 292 deletions

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 8fd8a3b634..74517c3064 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -4,43 +4,38 @@ CSIT Release Notes
 Changes in CSIT |release|

 -------------------------

 

-#. Added VPP performance tests

+#. **Added VPP performance tests**

 

-   - **Container Service Chain Topologies Orchestrated by K8s with VPP Memif**

+   - **MRR tests** : New MRR tests measure the packet forwarding rate

+       under the maximum load offered by traffic generator over a set

+       trial duration, regardless of packet loss. Maximum load for

+       specified Ethernet frame size is set to the bi-directional link

+       rate. MRR tests are used for continuous performance trending and

+       for comparison between releases.

 

-     - Added tests with VPP vswitch in container connecting a number of VPP-

-       in-container service chain topologies with L2 Cross-Connect and L2

-       Bridge-Domain configurations, orchestrated by Kubernetes. Added

-       following forwarding topologies: i) "Parallel" with packets flowing from

-       NIC via VPP to container and back to VPP and NIC; ii) "Chained" (a.k.a.

-       "Snake") with packets flowing via VPP to container, back to VPP, to next

-       container, back to VPP and so on until the last container in a chain,

-       then back to VPP and NIC; iii) "Horizontal" with packets flowing via VPP

-       to container, then via "horizontal" memif to next container, and so on

-       until the last container, then back to VPP and NIC;

+   - **SRv6** : Initial SRv6 (Segment Routing IPv6) tests verifying

+       performance of IPv6 and SRH (Segment Routing Header)

+       encapsulation, decapsulation, lookups and rewrites based on

+       configured End and End.DX6 SRv6 egress functions.

 

-   - **MRR tests**

+#. **Presentation and Analytics Layer (PAL)**

 

-     - <placeholder>;

+     - Added continuous performance measuring, trending and anomaly

+       detection. Includes new PAL code and Jenkins jobs for Performance

+       Trending (PT) and Performance Analysis (PA) producing performance

+       trending dashboard and trendline graphs with summary and drill-

+       down views across all specified tests that can be reviewed and

+       inspected regularly by FD.io developers and users community.

 

-   - **SRv6**

+#. **Test Framework Optimizations**

 

-     - Initial SRv6 (Segment Routing IPv6) tests verifying performance of

-       IPv6 and SRH (Segment Routing Header) encapsulation, decapsulation,

-       lookups and rewrites based on configured End and End.DX6 SRv6 egress

-       functions;

+     - **Performance tests efficiency** : Qemu build/install

+     optimizations, warmup phase handling, vpp restart handling.

+     Resulted in improved stability and reduced total execution time by

+     30% for single pkt size e.g. 64B/78B.

 

-#. Presentation and Analytics Layer

-

-     - Added throughput speedup analysis for multi-core and multi-thread

-       VPP tests into Presentation and Analytics Layer (PAL) for automated

-       CSIT test results analysis;

-

-#. Other changes

-

-     - **Framework optimizations**

-

-       - Performance test duration improvements and stability;

+     - **General code housekeeping** : ongoing RF keywords

+     optimizations, removal of redundant RF keywords.

 

 Performance Changes

 -------------------

@@ -98,6 +93,8 @@ pretty ASCII formats:
 Known Issues

 ------------

 

+<to be updated before rls1804 release>

+

 Here is the list of known issues in CSIT |release| for VPP performance tests:

 

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

diff --git a/docs/report/vpp_performance_tests/index.rst b/docs/report/vpp_performance_tests/index.rst
index 1292051a26..45efc3e10b 100644
--- a/docs/report/vpp_performance_tests/index.rst
+++ b/docs/report/vpp_performance_tests/index.rst
@@ -5,6 +5,7 @@ VPP Performance Tests
 
     overview
     csit_release_notes
+    methodology
     packet_throughput_graphs/index
     throughput_speedup_multi_core/index
     packet_latency_graphs/index
diff --git a/docs/report/vpp_performance_tests/methodology.rst b/docs/report/vpp_performance_tests/methodology.rst
new file mode 100644
index 0000000000..1de3fa4217
--- /dev/null
+++ b/docs/report/vpp_performance_tests/methodology.rst
@@ -0,0 +1,293 @@
+Test Methodology
+================
+
+Multi-Core and Multi-Threading
+------------------------------
+
+**Intel Hyper-Threading** - CSIT |release| performance tests are executed with
+SUT servers' Intel XEON processors configured in Intel Hyper-Threading Disabled
+mode (BIOS setting). This is the simplest configuration used to establish
+baseline single-thread single-core application packet processing and forwarding
+performance. Subsequent releases of CSIT will add performance tests with Intel
+Hyper-Threading Enabled (requires BIOS settings change and hard reboot of
+server).
+
+**Multi-core Tests** - CSIT |release| multi-core tests are executed in the
+following VPP thread and core configurations:
+
+#. 1t1c - 1 VPP worker thread on 1 CPU physical core.
+#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.
+#. 4t4c - 4 VPP worker threads on 4 CPU physical cores.
+
+VPP worker threads are the data plane threads. VPP control thread is
+running on a separate non-isolated core together with other Linux
+processes. Note that in quite a few test cases running VPP workers on 2
+or 4 physical cores hits the I/O bandwidth or packets-per-second limit
+of tested NIC.
+
+Section :ref:`throughput_speedup_multi_core` includes a set of graphs
+illustrating packet throughout speedup when running VPP on multiple
+cores.
+
+Packet Throughput
+-----------------
+
+Following values are measured and reported for packet throughput tests:
+
+- NDR binary search per :rfc:`2544`:
+
+  - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
+    (2x <per direction packets-per-second>)";
+  - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+    second> Gbps (untagged)";
+
+- PDR binary search per :rfc:`2544`:
+
+  - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
+    <per direction packets-per-second>)";
+  - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+    second> Gbps (untagged)";
+  - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
+    lost at PDR rate>";
+
+- NDR and PDR are measured for the following L2 frame sizes (untagged
+  Ethernet):
+
+  - IPv4 payload: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B;
+  - IPv6 payload: 78B, 1518B, 9000B;
+
+- NDR and PDR binary search resolution is determined by the final value of the
+  rate change, referred to as the final step:
+
+  - The final step is set to 50kpps for all NIC to NIC tests and all L2
+    frame sizes except 9000B (changed from 100kpps used in previous
+    releases).
+
+  - The final step is set to 10kpps for all remaining tests, including 9000B
+    and all vhost VM and memif Container tests.
+
+All rates are reported from external Traffic Generator perspective.
+
+Maximum Receive Rate (MRR)
+--------------------------
+
+MRR tests measure the packet forwarding rate under the maximum
+load offered by traffic generator over a set trial duration,
+regardless of packet loss. Maximum load for specified Ethernet frame
+size is set to the bi-directional link rate.
+
+Current parameters for MRR tests:
+
+- Ethernet frame sizes: 64B (78B for IPv6 tests) for all tests, IMIX for
+  selected tests (vhost, memif); all quoted sizes include frame CRC, but
+  exclude per frame transmission overhead of 20B (preamble, inter frame
+  gap).
+
+- Maximum load offered: 10GE and 40GE link (sub-)rates depending on NIC
+  tested, with the actual packet rate depending on frame size,
+  transmission overhead and traffic generator NIC forwarding capacity.
+
+  - For 10GE NICs the maximum packet rate load is 2* 14.88 Mpps for 64B,
+    a 10GE bi-directional link rate.
+  - For 40GE NICs the maximum packet rate load is 2* 18.75 Mpps for 64B,
+    a 40GE bi-directional link sub-rate limited by TG 40GE NIC used,
+    XL710.
+
+- Trial duration: 10sec.
+
+Similarly to NDR/PDR throughput tests, MRR test should be reporting bi-
+directional link rate (or NIC rate, if lower) if tested VPP
+configuration can handle the packet rate higher than bi-directional link
+rate, e.g. large packet tests and/or multi-core tests.
+
+MRR tests are used for continuous performance trending and for
+comparison between releases.
+
+Packet Latency
+--------------
+
+TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported
+latency values are measured using following methodology:
+
+- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
+  for each NDR throughput test and packet size (except IMIX).
+- TG sends dedicated latency streams, one per direction, each at the rate of
+  10kpps at the prescribed packet size; these are sent in addition to the main
+  load streams.
+- TG reports min/avg/max latency values per stream direction, hence two sets
+  of latency values are reported per test case; future release of TRex is
+  expected to report latency percentiles.
+- Reported latency values are aggregate across two SUTs due to three node
+  topology used for all performance tests; for per SUT latency, reported value
+  should be divided by two.
+- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
+  FD.io labs used by CSIT project.
+- TRex setup introduces an always-on error of about 2*2usec per latency flow -
+  additonal Tx/Rx interface latency induced by TRex SW writing and reading
+  packet timestamps on CPU cores without HW acceleration on NICs closer to the
+  interface line.
+
+vhostuser with KVM VMs
+----------------------
+
+FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following
+environment settings:
+
+- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr256] default 256
+  descriptors, [vr1024] 1024 descriptors to optimize for packet throughput.
+
+- Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)` settings:
+  [cfs] default settings, [cfsrr1] CFS RoundRobin(1) policy applied to all data
+  plane threads handling test packet path including all VPP worker threads and
+  all Qemu testpmd poll-mode threads.
+
+- Resulting test cases are all combinations with [vr256,vr1024] and
+  [cfs,cfsrr1] settings.
+
+- Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)` scheduler policy
+  for data plane threads used in CSIT is documented in
+  `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
+  The purpose is to verify performance impact (MRR and NDR/PDR
+  throughput) and same test measurements repeatability, by making VPP
+  and VM data plane threads less susceptible to other Linux OS system
+  tasks hijacking CPU cores running those data plane threads.
+
+Memif with LXC and Docker Containers
+------------------------------------
+
+CSIT |release| includes tests taking advantage of VPP memif virtual
+interface (shared memory interface) to interconnect VPP running in
+Containers. VPP vswitch instance runs in bare-metal user-mode handling
+NIC interfaces and connecting over memif (Slave side) to VPPs running in
+:abbr:`Linux Container (LXC)` or in Docker Container (DRC) configured
+with memif (Master side). LXCs and DRCs run in a priviliged mode with
+VPP data plane worker threads pinned to dedicated physical CPU cores per
+usual CSIT practice. All VPP instances run the same version of software.
+This test topology is equivalent to existing tests with vhost-user and
+VMs as described earlier in :ref:`tested_physical_topologies`.
+
+More information about CSIT LXC and DRC setup and control is available
+in :ref:`container_orchestration_in_csit`.
+
+Memif with K8s Pods/Containers
+------------------------------
+
+CSIT |release| includes tests of VPP topologies running in K8s
+orchestrated Pods/Containers and connected over memif virtual
+interfaces. In order to provide simple topology coding flexibility and
+extensibility container orchestration is done with `Kubernetes
+<https://github.com/kubernetes>`_ using `Docker
+<https://github.com/docker>`_ images for all container applications
+including VPP. `Ligato <https://github.com/ligato>`_ is used for the
+Pod/Container networking orchestration that is integrated with K8s,
+including memif support.
+
+In these tests VPP vswitch runs in a K8s Pod with Docker Container (DRC)
+handling NIC interfaces and connecting over memif to more instances of
+VPP running in Pods/DRCs. All DRCs run in a priviliged mode with VPP
+data plane worker threads pinned to dedicated physical CPU cores per
+usual CSIT practice. All VPP instances run the same version of software.
+This test topology is equivalent to existing tests with vhost-user and
+VMs as described earlier in :ref:`tested_physical_topologies`.
+
+Further documentation is available in
+:ref:`container_orchestration_in_csit`.
+
+IPSec with Intel QAT HW cards
+-----------------------------
+
+VPP IPSec performance tests are using DPDK cryptodev device driver in
+combination with HW cryptodev devices - Intel QAT 8950 50G - present in
+LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
+data plane functions supported by VPP.
+
+Currently CSIT |release| implements following IPSec test cases:
+
+- AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
+  with Intel xl710 NIC.
+- CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
+  IPv4-over-IPv4 with Intel xl710 NIC.
+
+TRex Traffic Generator Usage
+----------------------------
+
+`TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
+CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
+throughputs using binary search (NDR and PDR discovery tests) and for quick
+checks of DUT performance against the reference NDRs (NDR check tests) for
+specific configuration.
+
+TRex is installed and run on the TG compute node. The typical procedure is:
+
+- If the TRex is not already installed on TG, it is installed in the
+  suite setup phase - see `TRex intallation`_.
+- TRex configuration is set in its configuration file
+  ::
+
+  /etc/trex_cfg.yaml
+
+- TRex is started in the background mode
+  ::
+
+  $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /tmp/trex.log 2>&1 &' > /dev/null
+
+- There are traffic streams dynamically prepared for each test, based on traffic
+  profiles. The traffic is sent and the statistics obtained using
+  :command:`trex_stl_lib.api.STLClient`.
+
+**Measuring packet loss**
+
+- Create an instance of STLClient
+- Connect to the client
+- Add all streams
+- Clear statistics
+- Send the traffic for defined time
+- Get the statistics
+
+If there is a warm-up phase required, the traffic is sent also before test and
+the statistics are ignored.
+
+**Measuring latency**
+
+If measurement of latency is requested, two more packet streams are created (one
+for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In
+that case, returned statistics will also include min/avg/max latency values.
+
+TCP/IP tests with WRK tool
+--------------------------
+
+`WRK HTTP benchmarking tool <https://github.com/wg/wrk>`_ is used for
+experimental TCP/IP and HTTP tests of VPP TCP/IP stack and built-in
+static HTTP server. WRK has been chosen as it is capable of generating
+significant TCP/IP and HTTP loads by scaling number of threads across
+multi-core processors.
+
+This in turn enables quite high scale benchmarking of the main TCP/IP
+and HTTP service including HTTP TCP/IP Connections-Per-Second (CPS),
+HTTP Requests-Per-Second and HTTP Bandwidth Throughput.
+
+The initial tests are designed as follows:
+
+- HTTP and TCP/IP Connections-Per-Second (CPS)
+
+  - WRK configured to use 8 threads across 8 cores, 1 thread per core.
+  - Maximum of 50 concurrent connections across all WRK threads.
+  - Timeout for server responses set to 5 seconds.
+  - Test duration is 30 seconds.
+  - Expected HTTP test sequence:
+
+    - Single HTTP GET Request sent per open connection.
+    - Connection close after valid HTTP reply.
+    - Resulting flow sequence - 8 packets: >S,<S-A,>A,>Req,<Rep,>F,<F,> A.
+
+- HTTP Requests-Per-Second
+
+  - WRK configured to use 8 threads across 8 cores, 1 thread per core.
+  - Maximum of 50 concurrent connections across all WRK threads.
+  - Timeout for server responses set to 5 seconds.
+  - Test duration is 30 seconds.
+  - Expected HTTP test sequence:
+
+    - Multiple HTTP GET Requests sent in sequence per open connection.
+    - Connection close after set test duration time.
+    - Resulting flow sequence: >S,<S-A,>A,>Req[1],<Rep[1],..,>Req[n],<Rep[n],>F,<F,>A.
diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst
index 5f85b77b51..4f8fd19388 100644
--- a/docs/report/vpp_performance_tests/overview.rst
+++ b/docs/report/vpp_performance_tests/overview.rst
@@ -237,265 +237,3 @@ and system functional tests, introduced in CSIT |release-1|.
 The naming should be intuitive for majority of the tests. Complete description
 of CSIT test naming convention is provided on `CSIT test naming wiki
 <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
-
-Methodology: Multi-Core and Multi-Threading
--------------------------------------------
-
-**Intel Hyper-Threading** - CSIT |release| performance tests are executed with
-SUT servers' Intel XEON processors configured in Intel Hyper-Threading Disabled
-mode (BIOS setting). This is the simplest configuration used to establish
-baseline single-thread single-core application packet processing and forwarding
-performance. Subsequent releases of CSIT will add performance tests with Intel
-Hyper-Threading Enabled (requires BIOS settings change and hard reboot of
-server).
-
-**Multi-core Tests** - CSIT |release| multi-core tests are executed in the
-following VPP thread and core configurations:
-
-#. 1t1c - 1 VPP worker thread on 1 CPU physical core.
-#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.
-#. 4t4c - 4 VPP worker threads on 4 CPU physical cores.
-
-VPP worker threads are the data plane threads. VPP control thread is
-running on a separate non-isolated core together with other Linux
-processes. Note that in quite a few test cases running VPP workers on 2
-or 4 physical cores hits the I/O bandwidth or packets-per-second limit
-of tested NIC.
-
-Section :ref:`throughput_speedup_multi_core` includes a set of graphs
-illustrating packet throughout speedup when running VPP on multiple
-cores.
-
-Methodology: Packet Throughput
-------------------------------
-
-Following values are measured and reported for packet throughput tests:
-
-- NDR binary search per :rfc:`2544`:
-
-  - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
-    (2x <per direction packets-per-second>)";
-  - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
-    second> Gbps (untagged)";
-
-- PDR binary search per :rfc:`2544`:
-
-  - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
-    <per direction packets-per-second>)";
-  - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
-    second> Gbps (untagged)";
-  - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
-    lost at PDR rate>";
-
-- NDR and PDR are measured for the following L2 frame sizes:
-
-  - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B;
-  - IPv6: 78B, 1518B, 9000B;
-
-- NDR and PDR binary search resolution is determined by the final value of the
-  rate change, referred to as the final step:
-
-  - The final step is set to 50kpps for all NIC to NIC tests and all L2
-    frame sizes except 9000B (changed from 100kpps used in previous
-    releases).
-
-  - The final step is set to 10kpps for all remaining tests, including 9000B
-    and all vhost VM and memif Container tests.
-
-All rates are reported from external Traffic Generator perspective.
-
-Methodology: Packet Latency
----------------------------
-
-TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported
-latency values are measured using following methodology:
-
-- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
-  for each NDR throughput test and packet size (except IMIX).
-- TG sends dedicated latency streams, one per direction, each at the rate of
-  10kpps at the prescribed packet size; these are sent in addition to the main
-  load streams.
-- TG reports min/avg/max latency values per stream direction, hence two sets
-  of latency values are reported per test case; future release of TRex is
-  expected to report latency percentiles.
-- Reported latency values are aggregate across two SUTs due to three node
-  topology used for all performance tests; for per SUT latency, reported value
-  should be divided by two.
-- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
-  FD.io labs used by CSIT project.
-- TRex setup introduces an always-on error of about 2*2usec per latency flow -
-  additonal Tx/Rx interface latency induced by TRex SW writing and reading
-  packet timestamps on CPU cores without HW acceleration on NICs closer to the
-  interface line.
-
-
-Methodology: KVM VM vhost
--------------------------
-
-CSIT |release| introduced test environment configuration changes to KVM Qemu
-vhost-user tests in order to more representatively measure |vpp-release|
-performance in configurations with vhost-user interfaces and different Qemu
-settings.
-
-FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following
-environment settings:
-
-- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr256] default 256
-  descriptors, [vr1024] 1024 descriptors to optimize for packet throughput;
-
-- Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)` settings:
-  [cfs] default settings, [cfsrr1] CFS RoundRobin(1) policy applied to all data
-  plane threads handling test packet path including all VPP worker threads and
-  all Qemu testpmd poll-mode threads;
-
-- Resulting test cases are all combinations with [vr256,vr1024] and
-  [cfs,cfsrr1] settings;
-
-- Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)` scheduler policy
-  for data plane threads used in CSIT is documented in
-  `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
-  The purpose is to verify performance impact (NDR, PDR throughput) and
-  same test measurements repeatability, by making VPP and VM data plane
-  threads less susceptible to other Linux OS system tasks hijacking CPU
-  cores running those data plane threads.
-
-Methodology: LXC and Docker Containers memif
---------------------------------------------
-
-CSIT |release| introduced additional tests taking advantage of VPP memif virtual
-interface (shared memory interface) tests to interconnect VPP instances. VPP
-vswitch instance runs in bare-metal user-mode handling Intel x520 NIC 10GbE,
-Intel x710 NIC 10GbE, Intel xl710 NIC 40GbE interfaces and connecting over memif
-(Slave side) virtual interfaces to more instances of VPP running in
-:abbr:`LXC (Linux Container)` or in Docker Containers, both with memif virtual
-interfaces (Master side). LXCs and Docker Containers run in a priviliged mode
-with VPP data plane worker threads pinned to dedicated physical CPU cores per
-usual CSIT practice. All VPP instances run the same version of software.
-This test topology is equivalent to existing tests with vhost-user and
-VMs as described earlier in :ref:`tested_physical_topologies`.
-
-More information about CSIT LXC and Docker Container setup and control
-is available in :ref:`container_orchestration_in_csit`.
-
-Methodology: Container Topologies Orchestrated by K8s
------------------------------------------------------
-
-CSIT |release| introduced new tests of Container topologies connected
-over the memif virtual interface (shared memory interface). In order to
-provide simple topology coding flexibility and extensibility container
-orchestration is done with `Kubernetes <https://github.com/kubernetes>`_
-using `Docker <https://github.com/docker>`_ images for all container
-applications including VPP. `Ligato <https://github.com/ligato>`_ is
-used to address the container networking orchestration that is
-integrated with K8s, including memif support.
-
-For these tests VPP vswitch instance runs in a Docker Container handling
-Intel x520 NIC 10GbE, Intel x710 NIC 10GbE interfaces and connecting over memif
-virtual interfaces to more instances of VPP running in Docker Containers
-with memif virtual interfaces. All Docker Containers run in a priviliged mode
-with VPP data plane worker threads pinned to dedicated physical CPU cores per
-usual CSIT practice. All VPP instances run the same version of software. This
-test topology is equivalent to existing tests with vhost-user and VMs as
-described earlier in :ref:`tested_physical_topologies`.
-
-More information about CSIT Container Topologies Orchestrated by K8s is
-available in :ref:`container_orchestration_in_csit`.
-
-Methodology: IPSec with Intel QAT HW cards
-------------------------------------------
-
-VPP IPSec performance tests are using DPDK cryptodev device driver in
-combination with HW cryptodev devices - Intel QAT 8950 50G - present in
-LF FD.io physical testbeds. DPDK cryptodev can be used for all IPSec
-data plane functions supported by VPP.
-
-Currently CSIT |release| implements following IPSec test cases:
-
-- AES-GCM, CBC-SHA1 ciphers, in combination with IPv4 routed-forwarding
-  with Intel xl710 NIC.
-- CBC-SHA1 ciphers, in combination with LISP-GPE overlay tunneling for
-  IPv4-over-IPv4 with Intel xl710 NIC.
-
-Methodology: TRex Traffic Generator Usage
------------------------------------------
-
-`TRex traffic generator <https://wiki.fd.io/view/TRex>`_ is used for all
-CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
-throughputs using binary search (NDR and PDR discovery tests) and for quick
-checks of DUT performance against the reference NDRs (NDR check tests) for
-specific configuration.
-
-TRex is installed and run on the TG compute node. The typical procedure is:
-
-- If the TRex is not already installed on TG, it is installed in the
-  suite setup phase - see `TRex intallation`_.
-- TRex configuration is set in its configuration file
-  ::
-
-  /etc/trex_cfg.yaml
-
-- TRex is started in the background mode
-  ::
-
-  $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /tmp/trex.log 2>&1 &' > /dev/null
-
-- There are traffic streams dynamically prepared for each test, based on traffic
-  profiles. The traffic is sent and the statistics obtained using
-  :command:`trex_stl_lib.api.STLClient`.
-
-**Measuring packet loss**
-
-- Create an instance of STLClient
-- Connect to the client
-- Add all streams
-- Clear statistics
-- Send the traffic for defined time
-- Get the statistics
-
-If there is a warm-up phase required, the traffic is sent also before test and
-the statistics are ignored.
-
-**Measuring latency**
-
-If measurement of latency is requested, two more packet streams are created (one
-for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In
-that case, returned statistics will also include min/avg/max latency values.
-
-Methodology: TCP/IP tests with WRK tool
----------------------------------------
-
-`WRK HTTP benchmarking tool <https://github.com/wg/wrk>`_ is used for
-experimental TCP/IP and HTTP tests of VPP TCP/IP stack and built-in
-static HTTP server. WRK has been chosen as it is capable of generating
-significant TCP/IP and HTTP loads by scaling number of threads across
-multi-core processors.
-
-This in turn enables quite high scale benchmarking of the main TCP/IP
-and HTTP service including HTTP TCP/IP Connections-Per-Second (CPS),
-HTTP Requests-Per-Second and HTTP Bandwidth Throughput.
-
-The initial tests are designed as follows:
-
-- HTTP and TCP/IP Connections-Per-Second (CPS)
-
-  - WRK configured to use 8 threads across 8 cores, 1 thread per core.
-  - Maximum of 50 concurrent connections across all WRK threads.
-  - Timeout for server responses set to 5 seconds.
-  - Test duration is 30 seconds.
-  - Expected HTTP test sequence:
-
-    - Single HTTP GET Request sent per open connection.
-    - Connection close after valid HTTP reply.
-    - Resulting flow sequence - 8 packets: >S,<S-A,>A,>Req,<Rep,>F,<F,> A.
-
-- HTTP Requests-Per-Second
-
-  - WRK configured to use 8 threads across 8 cores, 1 thread per core.
-  - Maximum of 50 concurrent connections across all WRK threads.
-  - Timeout for server responses set to 5 seconds.
-  - Test duration is 30 seconds.
-  - Expected HTTP test sequence:
-
-    - Multiple HTTP GET Requests sent in sequence per open connection.
-    - Connection close after set test duration time.
-    - Resulting flow sequence: >S,<S-A,>A,>Req[1],<Rep[1],..,>Req[n],<Rep[n],>F,<F,>A.