CSIT-1430 Update release notes

Change-Id: Id67de7b23d48509b3118d24b6d054b8d50a3d0ec
Signed-off-by: Peter Mikus <pmikus@cisco.com>

6 files changed, 53 insertions, 74 deletions

diff --git a/docs/report/introduction/introduction.rst b/docs/report/introduction/introduction.rst
index c15df8d9bc..80f926e83a 100644
--- a/docs/report/introduction/introduction.rst
+++ b/docs/report/introduction/introduction.rst
@@ -26,10 +26,15 @@ available for download.
    - **Packet Throughput**: NDR, PDR throughput graphs based on results
      from repeated same test job executions to verify repeatibility of
      measurements.
-   - **Packet Latency**: Latency graphs based on results from test job
-     executions.
    - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup
      graphs based on results from test job executions.
+   - **Packet Latency**: Latency graphs based on results from test job
+     executions.
+   - **Soak Tests**: Long duration soak tests are executed using PLRsearch
+     algorithm.
+   - **NFV Service Density**: Network Function Virtualization (NFV) service
+     density tests focus on measuring total per server throughput at varied NFV
+     service "packing" densities with vswitch providing host dataplane.
    - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance
      graphs.
    - **Comparisons**: Performance comparisons between VPP releases and
@@ -59,6 +64,17 @@ available for download.
    - **Documentation**: Pointers to CSIT source code documentation for
      DPDK performance tests.
 
+#. VPP DEVICE: VPP functional tests executed in physical FD.io
+   testbeds using containers.
+
+   - **Overview**: Tested virtual topologies, test coverage and naming
+     specifics;
+   - **Release Notes**: Changes in |csit-release|, added tests,
+     environment or methodology changes, known issues.
+   - **Integration Tests**: Functional test environment configuration.
+   - **Documentation**: Pointers to CSIT source code documentation for
+     VPP functional tests.
+
 #. VPP FUNCTIONAL: VPP functional tests executed in virtual FD.io
    testbeds.
 
@@ -70,7 +86,7 @@ available for download.
    - **Documentation**: Pointers to CSIT source code documentation for
      VPP functional tests.
 
-#. HONEYCOMBE FUNCTIONAL: Honeycomb functional tests executed in
+#. HONEYCOMB FUNCTIONAL: Honeycomb functional tests executed in
    virtual FD.io testbeds.
 
    - **Overview**: Tested virtual topologies, test coverage and naming
diff --git a/docs/report/introduction/methodology_k8s_container_memif.rst b/docs/report/introduction/methodology_k8s_container_memif.rst
index 2e5ce0e017..ec7a3c9ccc 100644
--- a/docs/report/introduction/methodology_k8s_container_memif.rst
+++ b/docs/report/introduction/methodology_k8s_container_memif.rst
@@ -1,7 +1,7 @@
 K8s Container Memif
 -------------------
 
-|csit-release| includes tests of VPP topologies running in K8s
+CSIT includes tests of VPP topologies running in K8s
 orchestrated Pods/Containers and connected over memif virtual
 interfaces. In order to provide simple topology coding flexibility and
 extensibility container orchestration is done with `Kubernetes
diff --git a/docs/report/introduction/methodology_kvm_vms_vhost_user.rst b/docs/report/introduction/methodology_kvm_vms_vhost_user.rst
index 34e3bc0447..0a0ea1be1e 100644
--- a/docs/report/introduction/methodology_kvm_vms_vhost_user.rst
+++ b/docs/report/introduction/methodology_kvm_vms_vhost_user.rst
@@ -4,14 +4,13 @@ KVM VMs vhost-user
 FD.io CSIT performance lab is testing VPP vhost with KVM VMs using
 following environment settings:
 
-- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr256]
-  default 256 descriptors, [vr1024] 1024 descriptors to optimize for
-  packet throughput.
+- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr1024] 1024
+  descriptors to optimize for packet throughput.
 - Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)`
   settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1) policy
   applied to all data plane threads handling test packet path including
   all VPP worker threads and all Qemu testpmd poll-mode threads.
-- Resulting test cases are all combinations with [vr256,vr1024] and
+- Resulting test cases are all combinations with [vr1024] and
   [cfs,cfsrr1] settings.
 - Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)`
   scheduler policy for data plane threads used in CSIT is documented in
diff --git a/docs/report/introduction/methodology_lxc_drc_container_memif.rst b/docs/report/introduction/methodology_lxc_drc_container_memif.rst
index 56f052550c..125a71ccc7 100644
--- a/docs/report/introduction/methodology_lxc_drc_container_memif.rst
+++ b/docs/report/introduction/methodology_lxc_drc_container_memif.rst
@@ -1,7 +1,7 @@
 LXC/DRC Container Memif
 -----------------------
 
-|csit-release| includes tests taking advantage of VPP memif virtual
+CSIT includes tests taking advantage of VPP memif virtual
 interface (shared memory interface) to interconnect VPP running in
 Containers. VPP vswitch instance runs in bare-metal user-mode handling
 NIC interfaces and connecting over memif (Slave side) to VPPs running in
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 5d87b85c74..50dce62687 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -6,36 +6,13 @@ Changes in |csit-release|
 
 #. VPP PERFORMANCE TESTS
 
-   - **BMRR Throughput**: MRR (Maximum Receive Rate) test code has been
-     updated with a configurable burst MRR parameters: trial duration
-     and number of trials in a single burst. Enables a new Burst MRR
-     (BMRR for short) methodology for more precise performance
-     trending. See updated :ref:`test_methodology` section
-     for more details.
-
-   - **2n-skx tests**: Added performamce tests for 2n-skx (2-Node Xeon
-     Skylake) testbeds: focus on baseline and scale tests, including
-     VM vhost and Container memif tests.
-
-   - **3n-skx tests**: Added performamce tests for 3n-skx (3-Node Xeon
-     Skylake) testbeds: VM vhost and Container memif tests.
-
-   - **VXLAN Scale Tests**: Added performamce tests for VXLAN scale with
-     dot1q and VPP L2BD.
-
-   - **AVF Driver Tests**: Added performamce tests for i40e AVF driver
-     on VPP, no DPDK required.
-
-   - **QAT**: Fixed reoccuring issues with QAT crypto accelerator cards.
-
-   - **VM Vhost Virtio Params Combinations**: Added performance tests
-     for VM vhost with different virtio parameters combinations:
-     indirect buffers, mergeable buffers.
-
-   - **K8s/Ligato in Trending**: Added K8s/Ligato Container memif tests
-     to daily trending.
-
-#. TEST FRAMEWORK
+   - **Service density 2n-skx tests**: Network Function Virtualization (NFV)
+     service density tests focus on measuring total per server throughput at
+     varied NFV service *packing* densities with vswitch providing host
+     dataplane. The goal is to compare and contrast performance of a shared
+     vswitch for different network topologies and virtualization technologies,
+     and their impact on vswitch performance and efficiency in a range of NFV
+     service configurations.
 
    - **Experimental Soak Tests**: Added performamce soak tests framework
      code for extended time duration tests and throughput discovery
@@ -43,23 +20,14 @@ Changes in |csit-release|
      days, weeks, months, years. See updated
      :ref:`test_methodology` section for more details.
 
-   - **Trending Tests BMRR**: Used new Burst MRR (BMRR) tests for daily
-     trending.
-
-   - **Per VPP Patch Performance Checks**: Per VPP gerrit patch vs.
-     parent performance tests, anomaly detection and no verify voting
-     (-1/0/+1) yet. Manual trigger only. Not "marketed" to FD.io
-     community yet to avoid excessive LFN FD.io physical performance
-     testbed blocking.
+#. TEST FRAMEWORK
 
-   - **Patch-on-Patch Infra**: Added capability to run performance tests
-     using CSIT gerrit patch code testing VPP gerrit patch code, i.e.
-     before any VPP and/or CSIT code is merged into git branch.
+   - **Container code optimizations**: Optimized container library allows to
+     run containre_memif tests faster.
 
-   - **CSIT PAPI Support**: Initial implementation of PAPI L1 KWs in
-     CSIT using VPP Python bindings. Required for migraing away from
-     VAT. Very few L1 KWs implemented ("show version", "show
-     interfaces").
+   - **CSIT PAPI Support**: Continue converting existing VAT L1 keywords to
+     PAPI L1 KWs in CSIT using VPP Python bindings. Required for migrating away
+     from VAT.
 
    - **General Code Housekeeping**: Ongoing RF keywords optimizations,
      removal of redundant RF keywords.
@@ -68,11 +36,7 @@ Changes in |csit-release|
 
    - **Graphs Layout Improvements**: Improved performance graphs layout
      for better readibility and maintenance: test grouping, axis
-     labels, descriptions, other informative decoration. Master report
-     generated. 744 graphs(!)
-
-   - **Performance Trending**: Further improvements of continuous
-     performance trending, anomaly detection and analysis.
+     labels, descriptions, other informative decoration.
 
 #. MISCELLANEOUS
 
@@ -100,16 +64,18 @@ List of known issues in |csit-release| for VPP performance tests:
 | 1  | `CSIT-570                               | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue.                         |
 |    | <https://jira.fd.io/browse/CSIT-570>`_  | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds.                           |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 2  | `CSIT-1234                              | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04.        |
-|    | <https://jira.fd.io/browse/CSIT-1234>`_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%.                          |
+| 2  | `VPP-1562                               | Link bonding (mode LACP, transmit policy l34) test are failing due to VPP crashing producing core dump.                         |
+|    | <https://jira.fd.io/browse/VPP-1562>`_  |                                                                                                                                 |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 3  | `CSIT-1242                              | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04.                  |
-|    | <https://jira.fd.io/browse/CSIT-1242>`_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev.                                                     |
+| 3  | `CSIT-1427                              | Scale HW IPsec are failing due to issue with selecting ipsec backend in VPP.                                                    |
+|    | <https://jira.fd.io/browse/CSIT-1427>`_ |                                                                                                                                 |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 4  | `CSIT-1243                              | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds.                      |
-|    | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%.                                                          |
+| 4  | `VPP-1563                               | AVF L2patch tests are failing for all packet size and core combination. Reason: null-node blackholed packets in show error.     |
+|    | <https://jira.fd.io/browse/VPP-1563>`_  |                                                                                                                                 |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 5  | `CSIT-1246                              | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07.        |
-|    | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either.                                                                              |
+| 5  | `CSIT-1234                              | VPP IPSecHW/IPSecsW scale interface mode, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-19.01 vs. CSIT-18.10.      |
+|    | <https://jira.fd.io/browse/CSIT-1234>`_ |                                                                                                                                 |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 6  | `CSIT-1428                              | VM vhost tests with RXQ more than 2 are failing on 3n-skx. Suspecting RXQ configuration issues.                                 |
+|    | <https://jira.fd.io/browse/CSIT-1428>`_ |                                                                                                                                 |
 +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-
diff --git a/docs/vpp-device.rst b/docs/vpp-device.rst
index 3dae59a8d1..4289887c6b 100644
--- a/docs/vpp-device.rst
+++ b/docs/vpp-device.rst
@@ -255,6 +255,10 @@ including those required by DUT/TG.
     # Mount vfio to be able to bind to see binded interfaces. We cannot use
     # --device=/dev/vfio as this does not see newly binded interfaces.
     --volume /dev/vfio:/dev/vfio
+    # Mount nested_vm image to be able to run VM tests.
+    --volume /var/lib/vm/vhost-nested.img:/var/lib/vm/vhost-nested.img
+    # Mount docker.sock to be able to use docker deamon of the host.
+    --volume /var/run/docker.sock:/var/run/docker.sock
     # Image of csit-sut-dcr
     snergster/csit-vpp-device-test:latest
 
@@ -506,12 +510,6 @@ Stability
 
 .. note::
 
-    Improve NIC selection pair-wise: As of now script is taking first two
-    interfaces from discovered list regardless of sibling pairing. Implement
-    more advance method of selection of interfaces based on VF 802.1Q siblings.
-
-.. note::
-
     Implement queueing mechanism: Currently there is no mechanics that
     would place starving jobs in queue in case of no resources available.