CSIT rls1704 report - small updates

Change-Id: I80b5aaf2bab77a6131fca8cf4e903ef6aecd1195
Signed-off-by: Jan Gelety <jgelety@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

5 files changed, 31 insertions, 22 deletions

diff --git a/docs/report/testpmd_performance_tests/csit_release_notes.rst b/docs/report/testpmd_performance_tests/csit_release_notes.rst
index 7f87d247c9..72bef87ce4 100644
--- a/docs/report/testpmd_performance_tests/csit_release_notes.rst
+++ b/docs/report/testpmd_performance_tests/csit_release_notes.rst
@@ -20,6 +20,6 @@ Here is the list of known issues in CSIT |release| for Testpmd performance tests
 |   | of discovered NDR throughput values across        |            | throughput. Applies to XL710 and X710 NICs, no issues observed  |
 |   | multiple test runs with xl710 and x710 NICs.      |            | on x520 NICs.                                                   |
 +---+---------------------------------------------------+------------+-----------------------------------------------------------------+
-| 2 | Lower than expected NDR throughput with xl710     | CSIT-?     | Suspected NIC firmware or DPDK driver issue affecting NDR       |
+| 2 | Lower than expected NDR throughput with xl710     | CSIT-571   | Suspected NIC firmware or DPDK driver issue affecting NDR       |
 |   | and x710 NICs, compared to x520 NICs.             |            | throughput. Applies to XL710 and X710 NICs.                     |
 +---+---------------------------------------------------+------------+-----------------------------------------------------------------+
\ No newline at end of file
diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst
index 3b2d02ebb1..25ca27c03f 100644
--- a/docs/report/vpp_functional_tests/csit_release_notes.rst
+++ b/docs/report/vpp_functional_tests/csit_release_notes.rst
@@ -4,16 +4,21 @@ CSIT Release Notes
 Changes in CSIT |release|
 -------------------------
 
-#. Introduction of Centos7 tests
+#. VPP functional test environment changes
 
-#. Implemented VAT command history collection for every test case as part of teardown
+    - Implemented VAT command history collection for every test case as part of teardown.
+    - Introduction of Centos7 tests in VIRL environment.
+
+#. VPP functional test framework changes
+
+    - Added VAT command history collection for every test case as part of teardown.
 
 #. Added VPP functional tests
 
-    - IPv4 routed-forwarding with dot1q VLAN sub-interfaces
-    - L2BD switched-forwarding with dot1q VLAN sub-interfaces and vhost-user to VM
-    - IPv4 routed-forwarding with vhost-user interfaces to VM
-    - Vhost-User interface re-connect tests
+    - IPv4 routed-forwarding with dot1q VLAN sub-interfaces.
+    - L2BD switched-forwarding with dot1q VLAN sub-interfaces and vhost-user to VM.
+    - IPv4 routed-forwarding with vhost-user interfaces to VM.
+    - Vhost-user interface re-connect tests.
 
 Known Issues
 ------------
@@ -48,17 +53,21 @@ Here is the list of known issues in CSIT |release| for VPP functional tests in V
 |   |                                                 |          | destination address is reported.                     |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
 | 6 | IPv4 + VLAN: Processing of tagged frame doesn't | CSIT-564 | Dot1q tagged packets are thrown away in case of IPv4 |
-|   | work with virtio driver                         |          | routing on interface binded to virtio driver.        |
+|   | work with virtio driver                         |          | routing on interface binded to virtio driver. Issue  |
+|   |                                                 |          | with VIRL test simulation environment                |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
-| 7 | VHOST-user: QEMU re-connect doesn't work        | CSIT-565 | Used QEMU 2.5.0 doesn't support re-connection. Usage |
-|   |                                                 |          | of QEMU 2.7.0 is not recommended by vpp-dev team at  |
-|   |                                                 |          | the moment.                                          |
+| 7 | Vhost-user: QEMU reconnect doesn't work         | CSIT-565 | Using QEMU 2.5.0 that does not support vhost-user    |
+|   |                                                 |          | reconnect. It requires moving CSIT VIRL environment  |
+|   |                                                 |          | to QEMU 2.7.0.                                       |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
-| 8 | Centos7: LISP and VXLAN test failures           | CSIT-566 | We can observe crash of SUT1 VM during LISP tests    |
-|   |                                                 |          | execution, that leads to failure of all other tests  |
-|   |                                                 |          | (sometimes observed  during VXLAN tests execution    |
-|   |                                                 |          | too). NOTE: Possible root cause identified after     |
-|   |                                                 |          | upgrade of one of VIRL servers. When fix is          |
-|   |                                                 |          | confirmed in a short time it will be merged to CSIT  |
-|   |                                                 |          | rls1704 branch.                                      |
+| 8 | Centos7: LISP and VXLAN sporadic test failures  | CSIT-566 | Observing sporadic crashes of DUT1 VM during LISP    |
+|   |                                                 |          | and VXLAN test executions, what leads to failure of  |
+|   |                                                 |          | all other tests in the suite. NOTE: After upgrading  |
+|   |                                                 |          | one of the VIRL servers host from Ubuntu14.04 to     |
+|   |                                                 |          | 16.04, DUT1 VM stops crashing, but tests keep        |
+|   |                                                 |          | failing sporadically 1-in-100. Possible root cause   |
+|   |                                                 |          | identified - unexpected IPv6 RA packets upsetting    |
+|   |                                                 |          | some test component. Currently suspecting            |
+|   |                                                 |          | environment or CSIT issue, but can not exclude VPP,  |
+|   |                                                 |          | further troubleshooting in progress.                 |
 +---+-------------------------------------------------+----------+------------------------------------------------------+
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 7c17e0f448..b4fc9de3c9 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -215,14 +215,14 @@ Here is the list of known issues in CSIT |release| for VPP performance tests:
 |   | routes - 2M in this case. ip6scale2kk works.    |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 | 3 | Vic1385 and Vic1227 low performance             | VPP-664    | Low NDR performance.                                            |

-|   |                                                 |            |                                       .                         |

+|   |                                                 |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 | 4 | Sporadic NDR discovery test failures on x520    | CSIT-750   | Suspected issue with HW settings (BIOS, FW) in LF               |

 |   |                                                 |            | infrastructure. Issue can't be replicated outside LF.           |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 | 5 | VPP in 2t2c setups - large variation            | CSIT-568   | Suspected NIC firmware or DPDK driver issue affecting NDR       |

 |   | of discovered NDR throughput values across      |            | throughput. Applies to XL710 and X710 NICs, x520 NICs are fine. |

-|   | multiple test runs with xl710 and x710 NICs.    |            |                                       .                         |

+|   | multiple test runs with xl710 and x710 NICs.    |            |                                                                 |

 +---+-------------------------------------------------+------------+-----------------------------------------------------------------+

 | 6 | Lower than expected NDR and PDR throughput with | CSIT-569   | Suspected NIC firmware or DPDK driver issue affecting NDR and   |

 |   | xl710 and x710 NICs, compared to x520 NICs.     |            | PDR throughput. Applies to XL710 and X710 NICs.                 |

diff --git a/docs/report/vpp_unit_tests/overview.rst b/docs/report/vpp_unit_tests/overview.rst
index 2848db7ddc..4ef50f53c0 100644
--- a/docs/report/vpp_unit_tests/overview.rst
+++ b/docs/report/vpp_unit_tests/overview.rst
@@ -33,7 +33,7 @@ Unit Tests Coverage
 -------------------

 

 Following VPP functional test areas are covered in VPP unit test code included

-in VPP rls1701 with results listed in this report:

+in VPP rls1704 with results listed in this report:

 

 - CRUD Loopback - create, read, update, delete Loopback interfaces.

 - Flow-per-packet plugin - collect and report L2 and IP4 flow statistics.

diff --git a/resources/tools/report_gen/run_report.sh b/resources/tools/report_gen/run_report.sh
index c48b78e651..afff43f6f2 100755
--- a/resources/tools/report_gen/run_report.sh
+++ b/resources/tools/report_gen/run_report.sh
@@ -238,7 +238,7 @@ sed -i -e "s@###LINK###@${JEN_URL}\/${JEN_JOB}\/${JEN_BUILD[-1]}@g" \
 ### FUNCTIONAL SOURCE DATA
 
 JEN_JOB='csit-vpp-functional-1704-ubuntu1604-virl'
-JEN_BUILD=54
+JEN_BUILD=63
 
 if [[ ${DEBUG} -eq 1 ]] ;
 then