CSIT-618 Release report update VIII

Change-Id: Idec390861aa5ac8a1a485d5996fbfef444ac8a29
Signed-off-by: Peter Mikus <pmikus@cisco.com>

13 files changed, 82 insertions, 88 deletions

diff --git a/docs/report/dpdk_performance_tests/test_environment.rst b/docs/report/dpdk_performance_tests/test_environment.rst
index e8ed76d761..a8c0a485d2 100644
--- a/docs/report/dpdk_performance_tests/test_environment.rst
+++ b/docs/report/dpdk_performance_tests/test_environment.rst
@@ -369,7 +369,8 @@ Below a subset of the running configuration:
 

 - **isolcpus=<cpu number>-<cpu number>** used for all cpu cores apart from

   first core of each socket used for running VPP worker threads and Qemu/LXC

-  processes https://www.kernel.org/doc/Documentation/kernel-parameters.txt

+  processes

+  https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt

 - **intel_pstate=disable** - [X86] Do not enable intel_pstate as the default

   scaling driver for the supported processors. Intel P-State driver decide what

   P-state (CPU core power state) to use based on requesting policy from the

@@ -387,7 +388,7 @@ Below a subset of the running configuration:
 - **rcu_nocbs** - [KNL] In kernels built with CONFIG_RCU_NOCB_CPU=y, set the

   specified list of CPUs to be no-callback CPUs, that never queue RCU callbacks

   (read-copy update).

-  https://www.kernel.org/doc/Documentation/kernel-parameters.txt

+  https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt

 

 **Applied command line boot parameters:**

 

diff --git a/docs/report/honeycomb_functional_tests/test_environment.rst b/docs/report/honeycomb_functional_tests/test_environment.rst
index 92431c6c32..b55bfad602 100644
--- a/docs/report/honeycomb_functional_tests/test_environment.rst
+++ b/docs/report/honeycomb_functional_tests/test_environment.rst
@@ -13,7 +13,7 @@ versions:
 Current VPP tests have been executed on a single VM operating system and

 version only, as described in the following paragraphs.

 

-In CSIT terminology, the VM operating system for both SUTs and TG that

+In CSIT terminology, the VM operating system for both DUTs and TG that

 |vpp-release| has been tested with, is the following:

 

   |virl-image-ubuntu|

@@ -23,10 +23,8 @@ package versions are those that would have been installed by a "apt-get update",
 "apt-get upgrade" on that day), produced by CSIT disk image build scripts.

 

 The exact list of installed packages and their versions (including the Linux

-kernel package version) are included in CSIT source repository:

+kernel package version) are included in `VIRL images lists`_.

 

-  resources/tools/disk-image-builder/ubuntu/lists/|virl-image-ubuntu|

-

-A replica of this VM image can be built by running the "build.sh" script in CSIT

-repository resources/tools/disk-image-builder/ubuntu.

+A replica of this VM image can be built by running the "build.sh" script in

+`VIRL nested`_.

 

diff --git a/docs/report/honeycomb_performance_tests/csit_release_notes.rst b/docs/report/honeycomb_performance_tests/csit_release_notes.rst
index 51b62a7a6a..75d5e9eb3b 100644
--- a/docs/report/honeycomb_performance_tests/csit_release_notes.rst
+++ b/docs/report/honeycomb_performance_tests/csit_release_notes.rst
@@ -4,7 +4,7 @@ CSIT Release Notes
 Changes in CSIT |release|

 -------------------------

 

-#. First release with honeycomb performance testing

+#. First release with Honeycomb performance testing.

 

 Known Issues

 ------------

diff --git a/docs/report/honeycomb_performance_tests/test_environment.rst b/docs/report/honeycomb_performance_tests/test_environment.rst
index 1cafe26aa4..db187b2c1b 100644
--- a/docs/report/honeycomb_performance_tests/test_environment.rst
+++ b/docs/report/honeycomb_performance_tests/test_environment.rst
@@ -2,7 +2,7 @@ Test Environment
 ================
 
 To execute performance tests, there are three identical testbeds, each testbed
-consists of two SUTs and one TG.
+consists of two DUTs and one TG.
 
 Server HW Configuration
 -----------------------
diff --git a/docs/report/introduction/csit_design.rst b/docs/report/introduction/csit_design.rst
index baba58f904..7586e1dc8f 100644
--- a/docs/report/introduction/csit_design.rst
+++ b/docs/report/introduction/csit_design.rst
@@ -1,3 +1,5 @@
+.. _csit-design:

+

 CSIT Design

 ===========

 

diff --git a/docs/report/nsh_sfc_functional_tests/overview.rst b/docs/report/nsh_sfc_functional_tests/overview.rst
index c9d6ce7502..79403a2272 100644
--- a/docs/report/nsh_sfc_functional_tests/overview.rst
+++ b/docs/report/nsh_sfc_functional_tests/overview.rst
@@ -100,19 +100,19 @@ NSH_SFC Functional Tests Coverage
 Following NSH_SFC functional test areas are covered in the CSIT |release| with

 results listed in this report:

 

-- **NSH SFC Classifier** - TG send some TCP packets to test NSH SFC

+- **NSH SFC Classifier** - TG sends some TCP packets to test NSH SFC

   Classifier functional. DUT1 will receive these packets from one NIC and loopback

   the VXLAN-GPE-NSH encapsulated packets to the TG from other NIC.

   - Test case count: 7

-- **NSH SFC Proxy Inbound** - TG send some VXLAN-GPE-NSH encapsulated packets

+- **NSH SFC Proxy Inbound** - TG sends some VXLAN-GPE-NSH encapsulated packets

   to test NSH SFC Proxy Inbound functional. DUT1 will receive these packets from one

   NIC and loopback the VXLAN encapsulated packets to the TG from other NIC.

   - Test case count: 6

-- **NSH SFC Proxy Outbound** - TG send some VXLAN encapsulated packets to test

+- **NSH SFC Proxy Outbound** - TG sends some VXLAN encapsulated packets to test

   NSH SFC Proxy Outbound functional. DUT1 will receive these packets from one NIC

   and loopback the VXLAN-GPE-NSH encapsulated packets to the TG from other NIC.

   - Test case count: 6

-- **NSH SFC Service Function Forward** - TG send some VXLAN-GPE-NSH encapsulated

+- **NSH SFC Service Function Forward** - TG sends some VXLAN-GPE-NSH encapsulated

   packets to test NSH SFC Service Function Forward functional. DUT1 will receive these

   packets from one NIC and swap the VXLAN-GPE-NSH header, after that DUT1 loopback the

   VXLAN-GPE-NSH encapsulated packtes to the TG from other NIC.

diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst
index 08fe87053d..c547258091 100644
--- a/docs/report/vpp_functional_tests/csit_release_notes.rst
+++ b/docs/report/vpp_functional_tests/csit_release_notes.rst
@@ -23,8 +23,8 @@ Changes in CSIT |release|
      on CSIT wiki page
      `Design_Optimizations <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_;
 
-   - For more detail see the CSIT Framework Design section in this
-     report;
+   - For more detail see the :ref:`CSIT Framework Design <csit-design>` section
+     in this report;
 
 Known Issues
 ------------
@@ -68,6 +68,6 @@ Here is the list of known issues in CSIT |release| for VPP functional tests in V
 |   |                                                 |          | environment or CSIT issue, but can not exclude VPP,  |
 |   |                                                 |          | further troubleshooting in progress.                 |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
-| 7 | IPSEC: Tests cover old crypto code path         |    --    | There are used default conf settings so IPSEC tests  |
+| 7 | IPSEC: Tests cover old crypto code path         | CSIT-733 | There are used default conf settings so IPSEC tests  |
 |   |                                                 |          | use old security code not the new Crypto SW code.    |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
diff --git a/docs/report/vpp_functional_tests/test_environment.rst b/docs/report/vpp_functional_tests/test_environment.rst
index 93e3955f6f..15c27ec708 100644
--- a/docs/report/vpp_functional_tests/test_environment.rst
+++ b/docs/report/vpp_functional_tests/test_environment.rst
@@ -42,9 +42,7 @@ VIRL simulations.
 SUT Configuration - VIRL Guest VM

 ---------------------------------

 

-Configuration of the SUT VMs is defined in file

-

-   /csit/resources/tools/virl/topologies/double-ring-nested.xenial.virl

+Configurations of the SUT VMs is defined in `VIRL topologies directory`_

 

 - List of SUT VM interfaces:::

 

@@ -65,48 +63,43 @@ SUT Configuration - VIRL Guest OS Linux
 In CSIT terminology, the VM operating system for both SUTs that |vpp-release| has

 been tested with, is the following:

 

-**#. Ubuntu VIRL image**

-

-This image implies Ubuntu 16.04.1 LTS, current as of yyyy-mm-dd (that is,

-package versions are those that would have been installed by a "apt-get update",

-"apt-get upgrade" on that day), produced by CSIT disk image build scripts.

-

-The exact list of installed packages and their versions (including the Linux

-kernel package version) are included in CSIT source repository:

-

-  resources/tools/disk-image-builder/ubuntu/lists/|virl-image-ubuntu|

-

-A replica of this VM image can be built by running the "build.sh" script in CSIT

-repository resources/tools/disk-image-builder/ubuntu.

-

-**#. CentOS VIRL image**

-

-The Centos7.3 image is ready to be used but no tests running on it now.

-Corresponding Jenkins jobs are under preparation.

-

-The exact list of installed packages and their versions (including the Linux

-kernel package version) are included in CSIT source repository:

-

-  resources/tools/disk-image-builder/ubuntu/lists/|virl-image-centos|

-

-A replica of this VM image can be built by running the "build.sh" script in CSIT

-repository resources/tools/disk-image-builder/centos.

-

-**#. Nested VM image**

-

-In addition to the "main" VM image, tests which require VPP to communicate to a

-VM over a vhost-user interface, utilize a "nested" VM image.

-

-This "nested" VM is dynamically created and destroyed as part of a test case,

-and therefore the "nested" VM image is optimized to be small, lightweight and

-have a short boot time. The "nested" VM image is not built around any

-established Linux distribution, but is based on `BuildRoot

-<https://buildroot.org/>`_, a tool for building embedded Linux systems. Just as

-for the "main" image, scripts to produce an identical replica of the "nested"

-image are included in CSIT GIT repository, and the image can be rebuilt using

-the "build.sh" script at:

-

-   resources/tools/disk-image-builder/ubuntu/lists/nested

+#. **Ubuntu VIRL image**

+

+   This image implies Ubuntu 16.04.1 LTS, current as of yyyy-mm-dd (that is,

+   package versions are those that would have been installed by a "apt-get

+   update", "apt-get upgrade" on that day), produced by CSIT disk image build

+   scripts.

+   

+   The exact list of installed packages and their versions (including the Linux

+   kernel package version) are included in `VIRL images lists`_.

+   

+   A replica of this VM image can be built by running the "build.sh" script in

+   CSIT repository.

+

+#. **CentOS VIRL image**

+

+   The Centos7.3 image is ready to be used but no tests running on it now.

+   Corresponding Jenkins jobs are under preparation.

+   

+   The exact list of installed packages and their versions (including the Linux

+   kernel package version) are included in `VIRL images lists`_.

+   

+   A replica of this VM image can be built by running the "build.sh" script in

+   CSIT repository.

+

+#. **Nested VM image**

+

+   In addition to the "main" VM image, tests which require VPP to communicate to

+   a VM over a vhost-user interface, utilize a "nested" VM image.

+

+   This "nested" VM is dynamically created and destroyed as part of a test case,

+   and therefore the "nested" VM image is optimized to be small, lightweight and

+   have a short boot time. The "nested" VM image is not built around any

+   established Linux distribution, but is based on `BuildRoot

+   <https://buildroot.org/>`_, a tool for building embedded Linux systems. Just

+   as for the "main" image, scripts to produce an identical replica of the

+   "nested" image are included in CSIT GIT repository, and the image can be

+   rebuilt using the "build.sh" script at `VIRL nested`_.

 

 DUT Configuration - VPP

 -----------------------

@@ -117,9 +110,7 @@ Under Test (DUT) node.
 **DUT port configuration**

 

 Port configuration of DUTs is defined in topology file that is generated per

-VIRL simulation based on the definition stored in file

-

-   /csit/resources/tools/virl/topologies/double-ring-nested.xenial.yaml

+VIRL simulation based on the definition stored in `VIRL topologies directory`_.

 

 Example of DUT nodes configuration:::

 

@@ -373,7 +364,7 @@ Traffic scripts of test cases are executed on this VM.
 

 **TG VM configuration**

 

-Configuration of the TG VMs is defined in file

+Configuration of the TG VMs is defined in `VIRL topologies directory`_.

 

    /csit/resources/tools/virl/topologies/double-ring-nested.xenial.virl

 

@@ -389,9 +380,7 @@ Configuration of the TG VMs is defined in file
 **TG node port configuration**

 

 Port configuration of TG is defined in topology file that is generated per VIRL

-simulation based on the definition stored in file

-

-   /csit/resources/tools/virl/topologies/double-ring-nested.xenial.yaml

+simulation based on the definition stored in `VIRL topologies directory`_.

 

 Example of TG node configuration:::

 

diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 3daff961b4..0ee3e82f59 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -31,8 +31,8 @@ Changes in CSIT |release|
      on CSIT wiki page

      `Design_Optimizations <https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.

 

-   - For more detail see the CSIT Framework Design section in this

-     report;

+   - For more detail see the :ref:`CSIT Framework Design <csit-design>` section

+     in this report;

 

 #. Changes to CSIT driver for TRex Traffic Generator:

 

@@ -155,27 +155,27 @@ Here is the list of known issues in CSIT |release| for VPP performance tests:
 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 | # | Issue                                           | Jira ID    | Description                                                     |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 1 | Security-groups acl-plugin scale tests failure  | CSIT-xxx   | VPP with 2 worker threads crashes during security-groups        |

+| 1 | Security-groups acl-plugin scale tests failure  | CSIT-731   | VPP with 2 worker threads crashes during security-groups        |

 |   | with stateful acls if VPP with 2 worker threads | VPP-912    | iaclNsf and oaclNsf tests with 100k flows.                      |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| 2 | VPP fails memif tests in 4 worker 2 core setup  | CSIT-xxx   | VPP with 4 worker threads running on 2 physical cores crashes   |

-|   |                                                 | VPP-xxx    | during memif tests. Initial debugging points to DPDK code       |

+| 2 | VPP fails memif tests in 4 worker 2 core setup  | CSIT-732   | VPP with 4 worker threads running on 2 physical cores crashes   |

+|   |                                                 | VPP-920    | during memif tests. Initial debugging points to DPDK code       |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| X | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |

+| 3 | NDR discovery test failures 1518B frame size    | VPP-663    | VPP reporting errors: dpdk-input Rx ip checksum errors.         |

 |   | for ip4scale200k, ip4scale2m scale IPv4 routed- |            | Observed frequency: all test runs.                              |

 |   | forwarding tests. ip4scale20k tests are fine.   |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| X | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |

+| 4 | Vic1385 and Vic1227 low performance.            | VPP-664    | Low NDR performance.                                            |

 |   |                                                 |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| X | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |

+| 5 | Sporadic NDR discovery test failures on x520.   | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |

 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| X | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |

+| 6 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |

 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |

 |   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

-| X | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |

+| 7 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |

 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 

diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst
index 98a4d06681..5cf7b1d355 100644
--- a/docs/report/vpp_performance_tests/overview.rst
+++ b/docs/report/vpp_performance_tests/overview.rst
@@ -361,8 +361,9 @@ TRex is installed and run on the TG compute node. The typical procedure is:
 
   $ sh -c 'cd /opt/trex-core-2.25/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /dev/null 2>&1 &' > /dev/null
 
-- There are traffic streams dynamically prepared for each test. The traffic
-  is sent and the statistics obtained using trex_stl_lib.api.STLClient.
+- There are traffic streams dynamically prepared for each test, based on traffic
+  profiles. The traffic is sent and the statistics obtained using
+  trex_stl_lib.api.STLClient.
 
 **Measuring packet loss**
 
diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst
index 9392259a97..20ea77f31a 100644
--- a/docs/report/vpp_performance_tests/test_environment.rst
+++ b/docs/report/vpp_performance_tests/test_environment.rst
@@ -251,7 +251,8 @@ Below a subset of the running configuration:
 

 - **isolcpus=<cpu number>-<cpu number>** used for all cpu cores apart from

   first core of each socket used for running VPP worker threads and Qemu/LXC

-  processes https://www.kernel.org/doc/Documentation/kernel-parameters.txt

+  processes

+  https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt

 - **intel_pstate=disable** - [X86] Do not enable intel_pstate as the default

   scaling driver for the supported processors. Intel P-State driver decide what

   P-state (CPU core power state) to use based on requesting policy from the

@@ -269,7 +270,7 @@ Below a subset of the running configuration:
 - **rcu_nocbs** - [KNL] In kernels built with CONFIG_RCU_NOCB_CPU=y, set the

   specified list of CPUs to be no-callback CPUs, that never queue RCU callbacks

   (read-copy update).

-  https://www.kernel.org/doc/Documentation/kernel-parameters.txt

+  https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt

 

 **Applied command line boot parameters:**

 

diff --git a/resources/tools/report_gen/conf.py b/resources/tools/report_gen/conf.py
index b16cc4e42f..89dd45824f 100644
--- a/resources/tools/report_gen/conf.py
+++ b/resources/tools/report_gen/conf.py
@@ -69,8 +69,11 @@ rst_epilog = """
 .. _tag documentation rst file: https://git.fd.io/csit/tree/docs/tag_documentation.rst?h=rls1707
 .. _TRex intallation: https://git.fd.io/csit/tree/resources/tools/trex/trex_installer.sh?h=rls1707
 .. _TRex driver: https://git.fd.io/csit/tree/resources/tools/trex/trex_stateless_profile.py?h=rls1707
-.. _CSIT Honeycomb Functional Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.func.html
-.. _CSIT Honeycomb Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.perf.html
+.. _VIRL topologies directory: https://git.fd.io/csit/tree/resources/tools/virl/topologies/?h=rls1707
+.. _VIRL images lists: https://git.fd.io/csit/tree/resources/tools/disk-image-builder/ubuntu/lists/?h=rls1707
+.. _VIRL nested: https://git.fd.io/csit/tree/resources/tools/disk-image-builder/nested/?h=rls1707
+.. _CSIT Honeycomb Functional Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.func.honeycomb.html
+.. _CSIT Honeycomb Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.perf.honeycomb.html
 .. _CSIT DPDK Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.dpdk.perf.html
 .. _CSIT VPP Functional Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.func.html
 .. _CSIT VPP Performance Tests Documentation: https://docs.fd.io/csit/rls1707/doc/tests.vpp.perf.html
@@ -83,7 +86,6 @@ rst_epilog = """
 .. _FD.io test executor honeycomb performance jobs: https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-perf-master-ubuntu1604/lastSuccessfulBuild
 .. _FD.io test executor NSH_SFC functional jobs: https://jenkins.fd.io/view/csit/job/csit-nsh_sfc-verify-func-1707-ubuntu1604-virl/lastSuccessfulBuild
 .. _FD.io VPP compile job: https://jenkins.fd.io/view/vpp/job/vpp-merge-1707-ubuntu1604/
-.. _FD.io VPP compile job: https://jenkins.fd.io/view/vpp/job/vpp-merge-1707-ubuntu1604/
 .. _CSIT Testbed Setup: https://git.fd.io/csit/tree/resources/tools/testbed-setup/README.md?h=rls1707
 """
 
diff --git a/resources/tools/report_gen/run_report.cfg b/resources/tools/report_gen/run_report.cfg
index 57b2be0f94..5cd6b37c58 100644
--- a/resources/tools/report_gen/run_report.cfg
+++ b/resources/tools/report_gen/run_report.cfg
@@ -54,8 +54,8 @@ JOB[PERF,HC]=hc2vpp-csit-perf-master-ubuntu1604
 JOB[PERF,HC,BLD]="8 9"
 JOB[FUNC,HC]=hc2vpp-csit-integration-1707-ubuntu1604
 JOB[FUNC,HC,BLD]=lastSuccessfulBuild
-JOB[FUNC,NSH]=hc2vpp-csit-integration-1707-ubuntu1604
-JOB[FUNC,NSH,BLD]=lastSuccessfulBuild
+JOB[FUNC,NSH]=csit-nsh_sfc-verify-func-1707-ubuntu1604-virl
+JOB[FUNC,NSH,BLD]=2
 
 JOB[1704,PERF,VPP]=csit-vpp-perf-1704-all
 JOB[1704,VPP,BLD]="6 7 8 9 10 12 14 15 16 17"