aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorTibor Frank <tifrank@cisco.com>2018-09-05 13:21:19 +0200
committerTibor Frank <tifrank@cisco.com>2018-09-05 13:21:19 +0200
commit79ecf968f27fca0a75d6db5ab40920ba04873448 (patch)
tree9dfe302f2ea6a207912a369ec51ec6ecff8349f0
parentb4e5c717f5e2c39ded81f0c6f7b0f9f61945befd (diff)
Add known issues to report 18.07
Change-Id: Ide101c80e2ef13021f25096da6afd9e4364ee263 Signed-off-by: Tibor Frank <tifrank@cisco.com>
-rw-r--r--docs/report/introduction/index.rst1
-rw-r--r--docs/report/introduction/report_history.rst15
-rw-r--r--docs/report/vpp_performance_tests/csit_release_notes.rst75
-rwxr-xr-xresources/tools/presentation/run_report.sh2
4 files changed, 62 insertions, 31 deletions
diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst
index ba6599cfc8..81a7dc3c49 100644
--- a/docs/report/introduction/index.rst
+++ b/docs/report/introduction/index.rst
@@ -3,6 +3,7 @@ Introduction
.. toctree::
+ report_history
introduction
test_scenarios_overview
physical_testbeds
diff --git a/docs/report/introduction/report_history.rst b/docs/report/introduction/report_history.rst
new file mode 100644
index 0000000000..d54f526f55
--- /dev/null
+++ b/docs/report/introduction/report_history.rst
@@ -0,0 +1,15 @@
+Report History
+==============
+
++---------+--------------------------------------------------------------------+
+| Version | Change |
++=========+====================================================================+
+| 1.1 | The known issues |
+| | `CSIT-1246 <https://jira.fd.io/browse/CSIT-1246>`_, |
+| | `CSIT-1253 <https://jira.fd.io/browse/CSIT-1253>`_, |
+| | `CSIT-1266 <https://jira.fd.io/browse/CSIT-1266>`_, |
+| | `CSIT-1267 <https://jira.fd.io/browse/CSIT-1267>`_ |
+| | added to :ref:`vpp_known_issues`. |
++---------+--------------------------------------------------------------------+
+| 1.0 | Initial version |
++---------+--------------------------------------------------------------------+
diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst
index 32bbbbcfb6..379072c119 100644
--- a/docs/report/vpp_performance_tests/csit_release_notes.rst
+++ b/docs/report/vpp_performance_tests/csit_release_notes.rst
@@ -54,38 +54,53 @@ Changes in |csit-release|
- **General Code Housekeeping**: Ongoing RF keywords optimizations,
removal of redundant RF keywords.
+.. _vpp_known_issues:
+
Known Issues
------------
List of known issues in |csit-release| for VPP performance tests:
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| # | JiraID | Issue Description |
-+===+=========================================+=================================================================================================================================+
-| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. |
-| | <https://jira.fd.io/browse/CSIT-570>`_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 2 | `VPP-1361 | High failure rate of api call sw_interface_set_flags [admin-up|link-up]. |
-| | <https://jira.fd.io/browse/VPP-1361>`_ | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 3 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
-| | <https://jira.fd.io/browse/CSIT-1234>`_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 4 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
-| | <https://jira.fd.io/browse/CSIT-1242>`_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 5 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. |
-| | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 6 | `CSIT-1244 | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. |
-| | <https://jira.fd.io/browse/CSIT-1244>`_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 7 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
-| | <https://jira.fd.io/browse/CSIT-1245>`_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 8 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. |
-| | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
-| 9 | `CSIT-1253 | VPP lbdpdk link bonding tests failing due to interfaces not coming up. |
-| | <https://jira.fd.io/browse/CSIT-1253>`_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail. |
-+---+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| # | JiraID | Issue Description |
++====+=========================================+=================================================================================================================================+
+| 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. |
+| | <https://jira.fd.io/browse/CSIT-570>`_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 2 | `VPP-1361 | High failure rate of api call sw_interface_set_flags [admin-up|link-up]. |
+| | <https://jira.fd.io/browse/VPP-1361>`_ | Failure rate: 30-40% of tests failing due to interfaces not in link-up state after API call sw_interface_set_flags. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 3 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
+| | <https://jira.fd.io/browse/CSIT-1234>`_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 4 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
+| | <https://jira.fd.io/browse/CSIT-1242>`_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 5 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. |
+| | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 6 | `CSIT-1244 | VPP lispip4 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. |
+| | <https://jira.fd.io/browse/CSIT-1244>`_ | ip4lispip4-ip4base 2core 3n-skx/3n-hsw relative change: NDR -11%, PDR -18%. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 7 | `CSIT-1245 | VPP srv6proxy-stat and srv6proxy-masq, much higher NDR and PDR 78B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. |
+| | <https://jira.fd.io/browse/CSIT-1245>`_ | Due to wrong test suite configuration in dynamic-proxy mode. Artefact of suite code refactoring. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 8 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. |
+| | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 9 | `CSIT-1253 | VPP lbdpdk link bonding tests failing due to interfaces not coming up. |
+| | <https://jira.fd.io/browse/CSIT-1253>`_ | VPP lbdpdk link bonding tests relying on DPDK functionality for bonding fail. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 10 | `CSIT-1266 | Because of wrong Multi Seq configuration in the suite "csit/tests/vpp/perf/ip4/2n1l-25ge2p1xxv710-ethip4-ip4base-mrr.robot", |
+| | <https://jira.fd.io/browse/CSIT-1266>`_ | the data generated by this suite is incorrect. The bug was fixed by the `patch 14440 <https://gerrit.fd.io/r/#/c/14440>`_. |
+| | | The incorrect data will be replaced by new generated correct data. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
+| 11 | `CSIT-1267 | Because of the wrong NIC configuration in suites |
+| | <https://jira.fd.io/browse/CSIT-1267>`_ | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-mrr.robot, |
+| | | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2bdbasemaclrn-eth-2vhostvr1024-1vm-ndrpdr.robot, |
+| | | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm-mrr.robot, |
+| | | tests/vpp/perf/vm_vhost/10ge2p1x710-1lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm-ndrpdr.robot |
+| | | the data generated by this suites is incorrect. |
+| | | The bug was fixed by the `patch 14395 <https://gerrit.fd.io/r/#/c/14395>`_. |
+| | | The incorrect data will be replaced by new generated correct data. |
++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+
diff --git a/resources/tools/presentation/run_report.sh b/resources/tools/presentation/run_report.sh
index 9435ce01a6..2792190e3e 100755
--- a/resources/tools/presentation/run_report.sh
+++ b/resources/tools/presentation/run_report.sh
@@ -38,7 +38,7 @@ export PYTHONPATH=`pwd`
python pal.py \
--specification specification.yaml \
--release ${RELEASE} \
- --version "1.0" \
+ --version "1.1" \
--logging INFO \
--force