CSIT rls1704 report - update to handcraftec sections:

- introduction /overview /general_notes /csit_test_naming
- vpp_performance_tests /overview /csit_release_notes /test_environment /documentation /xx_graphs
- vpp_functional_tests /overview /csit_release_notes /documentation
- testpmd_performance_tests /overview /csit_release_notes /test_environment /documentation /xx_graphs

Change-Id: I10593d4155edc663ca07ae5cd29eb679c5f5d85f
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

6 files changed, 48 insertions, 110 deletions

diff --git a/docs/report/testpmd_performance_tests/csit_release_notes.rst b/docs/report/testpmd_performance_tests/csit_release_notes.rst
index 09d7f727da..6d415a1add 100644
--- a/docs/report/testpmd_performance_tests/csit_release_notes.rst
+++ b/docs/report/testpmd_performance_tests/csit_release_notes.rst
@@ -6,11 +6,5 @@ Changes in CSIT |release|
 
 #. Added Testpmd tests
 
-    - new NICs - Intel x520
+    - new NICs - Intel xl710
 
-Tests to be Added - Additional NICs
------------------------------------
-
-Adding test cases with more NIC types. Once the results become available, they
-will be published as an addendum to the current version of CSIT |release|
-report.
diff --git a/docs/report/testpmd_performance_tests/documentation.rst b/docs/report/testpmd_performance_tests/documentation.rst
index 737e46505b..4a0fe5dce5 100644
--- a/docs/report/testpmd_performance_tests/documentation.rst
+++ b/docs/report/testpmd_performance_tests/documentation.rst
@@ -2,5 +2,5 @@ Documentation
 =============

 

 `CSIT Testpmd Performance Tests Documentation

-<https://docs.fd.io/csit/master/doc/tests.perf.html>`_ contains  detailed

+<https://docs.fd.io/csit/rls1704/doc/tests.perf.html>`_ contains  detailed

 functional description and input parameters for each test case.

diff --git a/docs/report/testpmd_performance_tests/overview.rst b/docs/report/testpmd_performance_tests/overview.rst
index db1ac7849d..2182a00cd8 100644
--- a/docs/report/testpmd_performance_tests/overview.rst
+++ b/docs/report/testpmd_performance_tests/overview.rst
@@ -42,8 +42,7 @@ tested NIC models include:
 #. 2port40GE VIC1385 Cisco.
 #. 2port40GE XL710 Intel.
 
-Detailed LF FD.io test bed specification and topology is described in
-`wiki CSIT LF testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
+For detailed LF FD.io test bed specification and physical topology please refer to `LF FDio CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
 
 Performance Tests Coverage
 --------------------------
@@ -71,6 +70,11 @@ CSIT |release| includes following performance test suites, listed per NIC type:
   - **L2IntLoop** - L2 Interface Loop forwarding any Ethernet frames between
     two Interfaces.
 
+- 2port40GE XL710 Intel
+
+  - **L2IntLoop** - L2 Interface Loop forwarding any Ethernet frames between
+    two Interfaces.
+
 Execution of performance tests takes time, especially the throughput discovery
 tests. Due to limited HW testbed resources available within FD.io labs hosted
 by Linux Foundation, the number of tests for NICs other than X520 (a.k.a.
@@ -94,7 +98,6 @@ following Testpmd thread and core configurations:
 
 #. 1t1c - 1 Testpmd pmd thread on 1 CPU physical core.
 #. 2t2c - 2 Testpmd pmd threads on 2 CPU physical cores.
-#. 4t4c - 4 Testpmd pmd threads on 4 CPU physical cores.
 
 Note that in many tests running Testpmd reaches tested NIC I/O bandwidth
 or packets-per-second limit.
diff --git a/docs/report/testpmd_performance_tests/packet_latency_graphs/l2.rst b/docs/report/testpmd_performance_tests/packet_latency_graphs/l2.rst
index fc2e833967..4ffc9b00ec 100644
--- a/docs/report/testpmd_performance_tests/packet_latency_graphs/l2.rst
+++ b/docs/report/testpmd_performance_tests/packet_latency_graphs/l2.rst
@@ -8,7 +8,7 @@ configurations of Testpmd worker thread(s), a.k.a. Testpmd data plane thread
 (s), and their physical CPU core(s) placement.

 

 *Title of each graph* is a regex (regular expression) matching all plotted

-latency test cases, *X-axis labels* are indeces of csit-dpdk-perf-1701 jobs

+latency test cases, *X-axis labels* are indeces of csit-dpdk-perf-1704 jobs

 that created result output files used as data sources for the graph,

 *Y-axis labels* are measured packet Latency [uSec] values, and the *graph

 legend* identifes the plotted test suites.

@@ -16,11 +16,9 @@ legend* identifes the plotted test suites.
 .. note::

 

     Data sources for reported test results: i) FD.io test executor jobs

-    `csit-dpdk-perf-1701-all <https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-1701-all/>`_

-    and

-    `csit-dpdk-perf-1701-long <https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-1701-long/>`_

-    , ii) archived FD.io jobs test result output files

-    `csit-dpdk-perf-1701-all#job-number <https://nexus.fd.io/link>`_.

+    `csit-dpdk-perf-1704-all <https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-1704-all/>`_ ,

+    ii) archived FD.io jobs test result output files

+    `csit-dpdk-perf-1704-all#job-number <https://nexus.fd.io/link>`_.

 

 Testpmd packet latency - running in configuration of **one worker thread (1t) on one

 physical core (1c)** - is presented in the figure below.

@@ -60,22 +58,3 @@ git repository by filtering with specified regex as follows:
 

     10ge2p1x520-eth-l2xcbase-ndrdisc.robot:| tc07-64B-2t2c-eth-l2xcbase-ndrdisc

 

-Testpmd packet latency - running in configuration of **four worker threads (4t)

-on four physical cores (4c)** - is presented in the figure below.

-

-.. raw:: html

-

-    <iframe width="700" height="700" frameborder="0" scrolling="no" src="../../_static/testpmd/64B-4t4c-l2-ndrdisc-lat50.html"></iframe>

-

-*Figure 3. Testpmd 4thread 4core - packet latency for Phy-to-Phy L2 Ethernet Looping.*

-

-CSIT test cases used to generate results presented above can be found in CSIT

-git repository by filtering with specified regex as follows:

-

-.. code-block:: bash

-

-    $ csit/tests/perf

-    $ grep -E "64B-4t4c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" *

-

-    10ge2p1x520-eth-l2xcbase-ndrdisc.robot:| tc13-64B-4t4c-eth-l2xcbase-ndrdisc

-

diff --git a/docs/report/testpmd_performance_tests/packet_throughput_graphs/l2.rst b/docs/report/testpmd_performance_tests/packet_throughput_graphs/l2.rst
index ee79a4c6d0..90dd456473 100644
--- a/docs/report/testpmd_performance_tests/packet_throughput_graphs/l2.rst
+++ b/docs/report/testpmd_performance_tests/packet_throughput_graphs/l2.rst
@@ -8,18 +8,16 @@ Testpmd worker thread (a.k.a. VPP data plane thread) and physical CPU core
 configurations.

 

 *Title of each graph* is a regex (regular expression) matching all plotted

-throughput test cases, *X-axis labels* are indeces of csit-dpdk-perf-1701 jobs

+throughput test cases, *X-axis labels* are indeces of csit-dpdk-perf-1704 jobs

 that created result output files used as data sources for the graph,

 *Y-axis labels* are measured Packets Per Second [pps] values, and the *graph

 legend* identifes the plotted test suites.

 

 .. note::

 

-    Sources of test results data: i) FD.io executor jobs `csit-dpdk-perf-1701-all

-    <https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-1701-all/>`_ and

-    `csit-dpdk-perf-1701-long

-    <https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-1701-long/>`_, ii)

-    FD.io jobs archieved result files.

+    Sources of test results data: i) FD.io executor jobs `csit-dpdk-perf-1704-all

+    <https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-1704-all/>`_ ,

+    ii) FD.io jobs archieved result files.

 

 NDR Throughput

 ~~~~~~~~~~~~~~

@@ -64,26 +62,6 @@ git repository by filtering with specified regex as follows:
 

     10ge2p1x520-eth-l2xcbase-ndrdisc.robot:| tc07-64B-2t2c-eth-l2xcbase-ndrdisc

 

-NDR throughput of Testpmd - running in configuration of **four worker threads

-(4t) on four physical cores (4c)** - is presented in the figure below.

-

-.. raw:: html

-

-    <iframe width="700" height="700" frameborder="0" scrolling="no" src="../../_static/testpmd/64B-4t4c-l2-ndrdisc.html"></iframe>

-

-*Figure 3. Testpmd 4threads 4cores - NDR Throughput for Phy-to-Phy L2 Ethernet

-Looping.*

-

-CSIT test cases used to generate results presented above can be found in CSIT

-git repository by filtering with specified regex as follows:

-

-.. code-block:: bash

-

-    $ csit/dpdk-tests/perf

-    $ grep -E "64B-4t4c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-ndrdisc" *

-

-    10ge2p1x520-eth-l2xcbase-ndrdisc.robot:| tc13-64B-4t4c-eth-l2xcbase-ndrdisc

-

 PDR Throughput

 ~~~~~~~~~~~~~~

 

@@ -95,7 +73,7 @@ below 0.5% packet loss ratio.
 

     <iframe width="700" height="700" frameborder="0" scrolling="no" src="../../_static/testpmd/64B-1t1c-l2-pdrdisc.html"></iframe>

 

-*Figure 4. Testpmd 1thread 1core - PDR Throughput for Phy-to-Phy L2 Ethernet

+*Figure 3. Testpmd 1thread 1core - PDR Throughput for Phy-to-Phy L2 Ethernet

 Looping.*

 

 CSIT test cases used to generate results presented above can be found in CSIT

@@ -115,7 +93,7 @@ PDR throughput of Testpmd - running in configuration of **two worker threads
 

     <iframe width="700" height="700" frameborder="0" scrolling="no" src="../../_static/testpmd/64B-2t2c-l2-pdrdisc.html"></iframe>

 

-*Figure 5. Testpmd 2thread 2core - PDR Throughput for Phy-to-Phy L2 Ethernet

+*Figure 4. Testpmd 2thread 2core - PDR Throughput for Phy-to-Phy L2 Ethernet

 Looping.*

 

 CSIT test cases used to generate results presented above can be found in CSIT

@@ -128,22 +106,3 @@ git repository by filtering with specified regex as follows:
 

     10ge2p1x520-eth-l2xcbase-ndrdisc.robot:| tc08-64B-2t2c-eth-l2xcbase-pdrdisc

 

-PDR throughput of Testpmd - running in configuration of **four worker threads

-(4t) on four physical cores (4c)** - is presented in the figure below.

-

-.. raw:: html

-

-    <iframe width="700" height="700" frameborder="0" scrolling="no" src="../../_static/testpmd/64B-4t4c-l2-pdrdisc.html"></iframe>

-

-*Figure 6. Testpmd 4thread 4core - PDR Throughput for Phy-to-Phy L2 Ethernet

-Looping.*

-

-CSIT test cases used to generate results presented above can be found in CSIT

-git repository by filtering with specified regex as follows:

-

-.. code-block:: bash

-

-    $ csit/tests/perf

-    $ grep -E "64B-4t4c-(eth|dot1q|dot1ad)-(l2xcbase|l2bdbasemaclrn)-pdrdisc" *

-

-    10ge2p1x520-eth-l2xcbase-ndrdisc.robot:| tc14-64B-4t4c-eth-l2xcbase-pdrdisc

diff --git a/docs/report/testpmd_performance_tests/test_environment.rst b/docs/report/testpmd_performance_tests/test_environment.rst
index e4f301daa0..dd51137b2f 100644
--- a/docs/report/testpmd_performance_tests/test_environment.rst
+++ b/docs/report/testpmd_performance_tests/test_environment.rst
@@ -6,9 +6,11 @@ consists of two SUTs and one TG.
 

 SUT Configuration - Host HW

 ---------------------------

-Hardware details (CPU, memory, NIC layout) are described in

-`CSIT/CSIT_LF_testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_ in

-summary:

+Host hardware details (CPU, memory, NIC layout) and physical topology are

+described in detail in

+`LF FDio CSIT testbed wiki page <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.

+

+**Host configuration**

 

 - All hosts are Cisco UCS C240-M4 (2x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz,

   18c, 512GB RAM),

@@ -35,7 +37,7 @@ summary:
   - This allows for a total of five ring topologies, each using ports on

     specific NIC model, enabling per NIC model benchmarking.

 

-**NIC types**

+**NIC models and placement**

 

 - 0a:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+

   Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server

@@ -67,9 +69,10 @@ summary:
 SUT Configuration - Host OS Linux

 ---------------------------------

 

-Software details (OS, configuration) is described in

-`CSIT/CSIT_LF_testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_ in

-summary:

+Software details (OS, configuration) are described in

+`CSIT/CSIT_LF_testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.

+

+Below a subset of the configuration:

 

 .. code-block:: xml

 

@@ -2107,10 +2110,10 @@ summary:
         <exec_command><![CDATA[lsblk -l]]></exec_command>

         <exec_return_code>0</exec_return_code>

         <exec_output><![CDATA[NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

-  sda    8:0    0  1.8T  0 disk 

+  sda    8:0    0  1.8T  0 disk

   sda1   8:1    0  243M  0 part /boot

   sda2   8:2    0  1.8T  0 part /

-  sda3   8:3    0    1K  0 part 

+  sda3   8:3    0    1K  0 part

   sda5   8:5    0  976M  0 part [SWAP]

   ]]></exec_output>

       </function>

@@ -2192,29 +2195,29 @@ summary:
   dev.cdrom.check_media = 0

   dev.cdrom.debug = 0

   dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17

-  dev.cdrom.info = 

-  dev.cdrom.info = drive name:	

-  dev.cdrom.info = drive speed:	

+  dev.cdrom.info =

+  dev.cdrom.info = drive name:

+  dev.cdrom.info = drive speed:

   dev.cdrom.info = drive # of slots:

-  dev.cdrom.info = Can close tray:	

-  dev.cdrom.info = Can open tray:	

-  dev.cdrom.info = Can lock tray:	

+  dev.cdrom.info = Can close tray:

+  dev.cdrom.info = Can open tray:

+  dev.cdrom.info = Can lock tray:

   dev.cdrom.info = Can change speed:

   dev.cdrom.info = Can select disk:

   dev.cdrom.info = Can read multisession:

-  dev.cdrom.info = Can read MCN:	

+  dev.cdrom.info = Can read MCN:

   dev.cdrom.info = Reports media changed:

-  dev.cdrom.info = Can play audio:	

-  dev.cdrom.info = Can write CD-R:	

+  dev.cdrom.info = Can play audio:

+  dev.cdrom.info = Can write CD-R:

   dev.cdrom.info = Can write CD-RW:

-  dev.cdrom.info = Can read DVD:	

+  dev.cdrom.info = Can read DVD:

   dev.cdrom.info = Can write DVD-R:

   dev.cdrom.info = Can write DVD-RAM:

-  dev.cdrom.info = Can read MRW:	

-  dev.cdrom.info = Can write MRW:	

-  dev.cdrom.info = Can write RAM:	

-  dev.cdrom.info = 

-  dev.cdrom.info = 

+  dev.cdrom.info = Can read MRW:

+  dev.cdrom.info = Can write MRW:

+  dev.cdrom.info = Can write RAM:

+  dev.cdrom.info =

+  dev.cdrom.info =

   dev.cdrom.lock = 0

   dev.hpet.max-user-freq = 64

   dev.mac_hid.mouse_button2_keycode = 97

@@ -2276,7 +2279,7 @@ summary:
   kernel.hardlockup_all_cpu_backtrace = 0

   kernel.hardlockup_panic = 0

   kernel.hostname = t1-sut1

-  kernel.hotplug = 

+  kernel.hotplug =

   kernel.hung_task_check_count = 4194304

   kernel.hung_task_panic = 0

   kernel.hung_task_timeout_secs = 120

@@ -2577,7 +2580,7 @@ summary:
   net.ipv4.ip_forward = 0

   net.ipv4.ip_forward_use_pmtu = 0

   net.ipv4.ip_local_port_range = 32768	60999

-  net.ipv4.ip_local_reserved_ports = 

+  net.ipv4.ip_local_reserved_ports =

   net.ipv4.ip_no_pmtu_disc = 0

   net.ipv4.ip_nonlocal_bind = 0

   net.ipv4.ipfrag_high_thresh = 4194304

@@ -3495,7 +3498,7 @@ DUT Configuration - DPDK
 

 **DPDK Version**

 

-16.09

+17.02

 

 **DPDK Compile Parameters**