diff options
| author |   Maciek Konstantynowicz <mkonstan@cisco.com> | 2017-02-05 16:33:28 +0000 |
|---|---|---|
| committer |   Peter Mikus <pmikus@cisco.com> | 2017-02-06 16:30:04 +0000 |
| commit | 854e43d1792c153df4801891b3bc65781945ab8e (patch) | |
| tree | c18ded7427f2e8da2618a3b674ab5ddd837e89d1 | |
| parent | 91f196598d989e9c324f919201667c471b9716d9 (diff) | |
csit rls1701 report edits:
- simplified section structure for clarity and readability,
- updated overview sections,
- moved not rls related content from rls_notes to overview sections,
- removed section title suffixes: HW, VIRL.
- completed vpp_unit_tests and vpp_unittest_results sections.
- updated all documentation sections.
- updated rls_notes sections for vpp performance and vpp functional.
Change-Id: Id2c2abbf9d3531ec9f63ecd353f385a0b55ae1ba
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Signed-off-by: pmikus <pmikus@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Signed-off-by: pmikus <pmikus@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
(cherry picked from commit da2aa127ee071e37bc4d6060dd39476179e65365)
76 files changed, 1351 insertions, 915 deletions
diff --git a/docs/report/honeycomb_functional_results_virl/honeycomb_functional_results_virl.rst b/docs/report/detailed_test_results/honeycomb_functional_results/honeycomb_functional_results.rst index b298dca43c..a4bf19f61b 100644 --- a/docs/report/honeycomb_functional_results_virl/honeycomb_functional_results_virl.rst +++ b/docs/report/detailed_test_results/honeycomb_functional_results/honeycomb_functional_results.rst @@ -6,8 +6,8 @@ .. note:: Data sources for reported test results: i) FD.io test executor jobs - `hc2vpp-csit-integration-1701-ubuntu1404 - <https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-integration-1701-ubuntu1404/>`_ + `hc2vpp-csit-integration-1701-ubuntu1404/1 + <https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-integration-1701-ubuntu1404/1>`_ , ii) archived FD.io jobs test result `output files <../../_static/archive/>`_. diff --git a/docs/report/detailed_test_results/honeycomb_functional_results/index.rst b/docs/report/detailed_test_results/honeycomb_functional_results/index.rst new file mode 100644 index 0000000000..e7061694aa --- /dev/null +++ b/docs/report/detailed_test_results/honeycomb_functional_results/index.rst @@ -0,0 +1,7 @@ +HoneyComb Functional Results
+============================
+
+.. toctree::
+
+ honeycomb_functional_results
+
diff --git a/docs/report/detailed_test_results/index.rst b/docs/report/detailed_test_results/index.rst new file mode 100644 index 0000000000..28e8b8d4b1 --- /dev/null +++ b/docs/report/detailed_test_results/index.rst @@ -0,0 +1,10 @@ +Detailed Test Results +===================== + +.. toctree:: + + vpp_performance_results/index + testpmd_performance_results/index + vpp_functional_results/index + honeycomb_functional_results/index + vpp_unittest_results/index diff --git a/docs/report/detailed_test_results/testpmd_performance_results/index.rst b/docs/report/detailed_test_results/testpmd_performance_results/index.rst new file mode 100644 index 0000000000..7a5cf77938 --- /dev/null +++ b/docs/report/detailed_test_results/testpmd_performance_results/index.rst @@ -0,0 +1,6 @@ +Testpmd Performance Results
+===========================
+
+.. toctree::
+
+ testpmd_performance_results
diff --git a/docs/report/testpmd_performance_results_hw/testpmd_performance_results_hw.rst b/docs/report/detailed_test_results/testpmd_performance_results/testpmd_performance_results.rst index afcb6120b2..afcb6120b2 100644 --- a/docs/report/testpmd_performance_results_hw/testpmd_performance_results_hw.rst +++ b/docs/report/detailed_test_results/testpmd_performance_results/testpmd_performance_results.rst diff --git a/docs/report/detailed_test_results/vpp_functional_results/index.rst b/docs/report/detailed_test_results/vpp_functional_results/index.rst new file mode 100644 index 0000000000..abc3ae4cc5 --- /dev/null +++ b/docs/report/detailed_test_results/vpp_functional_results/index.rst @@ -0,0 +1,7 @@ +VPP Functional Results
+======================
+
+.. toctree::
+
+ vpp_functional_results
+
diff --git a/docs/report/vpp_functional_results_virl/functional_results_virl.rst b/docs/report/detailed_test_results/vpp_functional_results/vpp_functional_results.rst index 5e823f5f6d..dd7b607363 100644 --- a/docs/report/vpp_functional_results_virl/functional_results_virl.rst +++ b/docs/report/detailed_test_results/vpp_functional_results/vpp_functional_results.rst @@ -9,8 +9,8 @@ DHCP - Client and Proxy .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -80,8 +80,8 @@ GRE Overlay Tunnels .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -122,8 +122,8 @@ L2BD Ethernet Switching .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -246,8 +246,8 @@ L2XC Ethernet Switching .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -303,8 +303,8 @@ LISP Overlay Tunnels .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -867,8 +867,8 @@ Softwire Tunnels .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -965,8 +965,8 @@ Cop Address Security .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1060,8 +1060,8 @@ IPSec - Tunnels and Transport .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1227,8 +1227,8 @@ IPv6 Routed-Forwarding .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1295,8 +1295,8 @@ RPF Source Security .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1325,8 +1325,8 @@ Tap Interface .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1429,8 +1429,8 @@ IPFIX Netflow Statistics .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1502,8 +1502,8 @@ SPAN Port Mirroring .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1553,8 +1553,8 @@ VRF Routed-Forwarding .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1622,8 +1622,8 @@ iACL Security .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1720,8 +1720,8 @@ IPv4 Routed-Forwarding .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1811,8 +1811,8 @@ QoS Policer Metering .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1872,8 +1872,8 @@ VLAN Tag Translation .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -2088,8 +2088,8 @@ VXLAN Overlay Tunnels .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-functional-1701-virl
- <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/>`_
+ `csit-vpp-functional-1701-virl/18
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-functional-1701-virl/18>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
diff --git a/docs/report/detailed_test_results/vpp_performance_results/index.rst b/docs/report/detailed_test_results/vpp_performance_results/index.rst new file mode 100644 index 0000000000..63f0f7fcf6 --- /dev/null +++ b/docs/report/detailed_test_results/vpp_performance_results/index.rst @@ -0,0 +1,6 @@ +VPP Performance Results
+=======================
+
+.. toctree::
+
+ vpp_performance_results
diff --git a/docs/report/vpp_performance_results_hw/performance_results_hw.rst b/docs/report/detailed_test_results/vpp_performance_results/vpp_performance_results.rst index 594f8911a9..84fb2d4b31 100644 --- a/docs/report/vpp_performance_results_hw/performance_results_hw.rst +++ b/docs/report/detailed_test_results/vpp_performance_results/vpp_performance_results.rst @@ -9,10 +9,8 @@ L2 Ethernet Switching .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-perf-1701-all
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and
- `csit-vpp-perf-1701-long
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_
+ `csit-vpp-perf-1701-all/4
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -376,10 +374,8 @@ IPv4 Routed-Forwarding .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-perf-1701-all
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and
- `csit-vpp-perf-1701-long
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_
+ `csit-vpp-perf-1701-all/4
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -815,10 +811,8 @@ IPv6 Routed-Forwarding .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-perf-1701-all
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and
- `csit-vpp-perf-1701-long
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_
+ `csit-vpp-perf-1701-all/4
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1189,10 +1183,8 @@ IPv4 Overlay Tunnels .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-perf-1701-all
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and
- `csit-vpp-perf-1701-long
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_
+ `csit-vpp-perf-1701-all/4
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1422,10 +1414,8 @@ IPv6 Overlay Tunnels .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-perf-1701-all
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and
- `csit-vpp-perf-1701-long
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_
+ `csit-vpp-perf-1701-all/4
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
@@ -1545,10 +1535,8 @@ VM Vhost Connections .. note::
Data sources for reported test results: i) FD.io test executor jobs
- `csit-vpp-perf-1701-all
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/>`_ and
- `csit-vpp-perf-1701-long
- <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-long/>`_
+ `csit-vpp-perf-1701-all/4
+ <https://jenkins.fd.io/view/csit/job/csit-vpp-perf-1701-all/4>`_
, ii) archived FD.io jobs test result `output files
<../../_static/archive/>`_.
diff --git a/docs/report/detailed_test_results/vpp_unittest_results/index.rst b/docs/report/detailed_test_results/vpp_unittest_results/index.rst new file mode 100644 index 0000000000..f77153d6b9 --- /dev/null +++ b/docs/report/detailed_test_results/vpp_unittest_results/index.rst @@ -0,0 +1,7 @@ +VPP Unit Test Results
+=====================
+
+.. toctree::
+
+ vpp_unittest_results
+
diff --git a/docs/report/detailed_test_results/vpp_unittest_results/vpp_unittest_results.rst b/docs/report/detailed_test_results/vpp_unittest_results/vpp_unittest_results.rst new file mode 100644 index 0000000000..4a56b07d5e --- /dev/null +++ b/docs/report/detailed_test_results/vpp_unittest_results/vpp_unittest_results.rst @@ -0,0 +1,296 @@ +CRUD Loopback
+`````````````
+::
+
+ ==============================================================================
+ CRUD Loopback
+ ==============================================================================
+ test_crud (test_interface_crud.TestLoopbackInterfaceCRUD) OK
+ test_down (test_interface_crud.TestLoopbackInterfaceCRUD) OK
+
+Flow-per-packet plugin
+``````````````````````
+::
+
+ ==============================================================================
+ Flow-per-packet plugin: test both L2 and IP4 reporting
+ ==============================================================================
+ Flow per packet L3 test OK
+
+DHCP
+````
+::
+
+ ==============================================================================
+ DHCP Test Case
+ ==============================================================================
+ DHCPv6 Proxy OK
+ DHCPv4 Proxy OK
+
+IP4 VRF Multi-instance
+``````````````````````
+::
+
+ ==============================================================================
+ IP4 VRF Multi-instance Test Case
+ ==============================================================================
+ IP4 VRF Multi-instance test 1 - create 5 BDs OK
+ IP4 VRF Multi-instance test 2 - delete 2 VRFs OK
+ IP4 VRF Multi-instance 3 - add 2 VRFs OK
+ IP4 VRF Multi-instance test 4 - delete 4 VRFs OK
+
+Load Balancer
+`````````````
+::
+
+ ==============================================================================
+ Load Balancer Test Case
+ ==============================================================================
+ Load Balancer IP4 GRE4 OK
+ Load Balancer IP4 GRE6 OK
+ Load Balancer IP6 GRE4 OK
+ Load Balancer IP6 GRE6 OK
+
+BFD IPv4
+````````
+
+::
+
+ ==============================================================================
+ Bidirectional Forwarding Detection (BFD)
+ ==============================================================================
+ verify session goes down after inactivity OK
+ hold BFD session up OK
+ immediately honor remote min rx reduction OK
+ large remote RequiredMinRxInterval OK
+ bring BFD session down OK
+ bring BFD session up OK
+ verify slow periodic control frames while session down OK
+ no packets when zero BFD RemoteMinRxInterval OK
+
+BFD IPv6
+````````
+::
+
+ ==============================================================================
+ Bidirectional Forwarding Detection (BFD) (IPv6)
+ ==============================================================================
+ hold BFD session up OK
+ bring BFD session up OK
+
+BFD API tests
+`````````````
+::
+
+ ==============================================================================
+ Bidirectional Forwarding Detection (BFD) - API
+ ==============================================================================
+ activate SHA1 authentication OK
+ create BFD session using non-existent SHA1 (negative case) OK
+ create a BFD session OK
+ create IPv6 BFD session OK
+ create a BFD session (SHA1) OK
+ add SHA1 keys OK
+ test_change_key (test_bfd.BFDAPITestCase) OK
+ deactivate SHA1 authentication OK
+ create the same BFD session twice (negative case) OK
+ create the same BFD session twice (negative case) (SHA1) OK
+ share single SHA1 key between multiple BFD sessions OK
+
+BFD authorization
+`````````````````
+::
+
+ ==============================================================================
+ Bidirectional Forwarding Detection (BFD) (changing auth)
+ ==============================================================================
+ change auth key without disturbing session state (delayed) OK
+ change auth key without disturbing session state (immediate) OK
+ turn auth off without disturbing session state (delayed) OK
+ turn auth off without disturbing session state (immediate) OK
+ turn auth on without disturbing session state (delayed) OK
+ turn auth on without disturbing session state (immediate) OK
+
+BFD authentication
+``````````````````
+::
+
+ ==============================================================================
+ Bidirectional Forwarding Detection (BFD) (SHA1 auth)
+ ==============================================================================
+ hold BFD session up OK
+ hold BFD session up - meticulous auth OK
+ session is not brought down by unauthenticated msg OK
+ session is not brought down by msg with non-existent key-id OK
+ session is not brought down by msg with wrong auth type OK
+ simulate remote peer restart and resynchronization OK
+ session is not kept alive by msgs with bad seq numbers OK
+ bring BFD session up OK
+
+IPv6 Tests
+``````````
+::
+
+ ==============================================================================
+ IPv6 Test Case
+ ==============================================================================
+ IPv6 FIB test OK
+ IPv6 Neighbour Solicitation Exceptions OK
+ IPv6 Router Solicitation Exceptions OK
+
+SPAN Test
+`````````
+::
+
+ ==============================================================================
+ SPAN Test Case
+ ==============================================================================
+ SPAN test OK
+
+GRE Tests
+`````````
+::
+
+ ==============================================================================
+ GRE Test Case
+ ==============================================================================
+ GRE tunnel Tests OK
+ GRE tunnel L2 Tests OK
+ GRE tunnel VRF Tests OK
+
+SNAT Test Cases
+```````````````
+::
+
+ ==============================================================================
+ SNAT Test Cases
+ ==============================================================================
+ SNAT dynamic translation test OK
+ SNAT hairpinning OK
+ SNAT multiple inside interfaces with overlapping address space OK
+ Acquire SNAT addresses from interface OK
+ Static mapping with addresses from interface OK
+ S-NAT IPFIX logging NAT addresses exhausted OK
+ S-NAT IPFIX logging NAT44 session created/delted OK
+ MAX translations per user - recycle the least recently used OK
+ SNAT multiple inside interfaces (non-overlapping address space) OK
+ S-NAT add pool addresses to FIB OK
+ SNAT 1:1 NAT initialized from inside network OK
+ SNAT 1:1 NAT initialized from outside network OK
+ SNAT 1:1 NAT VRF awareness OK
+ SNAT 1:1 NAT with port initialized from inside network OK
+ SNAT 1:1 NAT with port initialized from outside network OK
+
+L2XC Multi-instance
+```````````````````
+::
+
+ ==============================================================================
+ L2XC Multi-instance Test Case
+ ==============================================================================
+ L2XC Multi-instance test 1 - create 10 cross-connects OK
+ L2XC Multi-instance test 2 - delete 4 cross-connects OK
+ L2BD Multi-instance 3 - add new 4 cross-connects OK
+ L2XC Multi-instance test 4 - delete 10 cross-connects OK
+
+IPv4 Tests
+``````````
+::
+
+ ==============================================================================
+ IPv4 Test Case
+ ==============================================================================
+ IPv4 FIB test OK
+
+IPv4 FIB CRUD
+`````````````
+::
+
+ ==============================================================================
+ FIB - add/update/delete - ip4 routes
+ ==============================================================================
+ Add 1k routes OK
+ Delete 100 routes OK
+ Add 1k routes OK
+ Delete 1.5k routes OK
+
+IRB Tests
+`````````
+::
+
+ ==============================================================================
+ IRB Test Case
+ ==============================================================================
+ IPv4 IRB test 1 OK
+ IPv4 IRB test 2 OK
+
+MPLS Tests
+``````````
+::
+
+ ==============================================================================
+ MPLS Test Case
+ ==============================================================================
+ MPLS Local Label Binding test OK
+ MPLS Deagg OK
+ MPLS label imposition test OK
+ MPLS label swap tests OK
+ MPLS Tunnel Tests OK
+ MPLS V4 Explicit NULL test OK
+ MPLS V6 Explicit NULL test OK
+
+L2XC Tests
+``````````
+::
+
+ ==============================================================================
+ L2XC Test Case
+ ==============================================================================
+ L2XC dual-loop test OK
+ L2XC single-loop test OK
+
+Classifier
+``````````
+::
+
+ ==============================================================================
+ Classifier Test Case
+ ==============================================================================
+ IP ACL test OK
+ MAC ACL test OK
+ IP PBR test OK
+
+VXLAN Tests
+```````````
+::
+
+ ==============================================================================
+ VXLAN Test Case
+ ==============================================================================
+ Decapsulation test OK
+ Encapsulation test OK
+ Multicast flood test OK
+ Multicast receive test OK
+ Unicast flood test OK
+
+L2 FIB CRUD
+```````````
+::
+
+ ==============================================================================
+ L2 FIB Test Case
+ ==============================================================================
+ L2 FIB test 1 - program 100 MAC addresses OK
+ L2 FIB test 2 - delete 12 MAC entries OK
+ L2 FIB test 3 - program new 100 MAC addresses OK
+ L2 FIB test 4 - delete 160 MAC entries OK
+
+L2BD Tests
+``````````
+::
+
+ ==============================================================================
+ L2BD Test Case
+ ==============================================================================
+ L2BD MAC learning dual-loop test OK
+ L2BD MAC learning single-loop test OK
diff --git a/docs/report/honeycomb_functional_results_virl/index.rst b/docs/report/honeycomb_functional_results_virl/index.rst deleted file mode 100644 index 4b825d5b59..0000000000 --- a/docs/report/honeycomb_functional_results_virl/index.rst +++ /dev/null @@ -1,7 +0,0 @@ -HoneyComb Functional Results VIRL
-=================================
-
-.. toctree::
-
- honeycomb_functional_results_virl
-
diff --git a/docs/report/honeycomb_functional_tests/csit_release_notes.rst b/docs/report/honeycomb_functional_tests/csit_release_notes.rst new file mode 100644 index 0000000000..68b5eb6fb5 --- /dev/null +++ b/docs/report/honeycomb_functional_tests/csit_release_notes.rst @@ -0,0 +1,35 @@ +CSIT Release Notes
+==================
+
+Changes in CSIT |release|
+-------------------------
+
+#. Naming change for all Honeycomb functional test suites in VIRL
+
+ - Honeycomb functional test case names stayed unchanged
+
+#. Added Honeycomb functional tests
+
+ - NSH_SFC
+ - LISP
+ - NAT
+ - SPAN
+
+Known Issues
+------------
+
+Here is the list of known issues in CSIT |release| for Honeycomb functional
+tests in VIRL:
+
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| # | Issue | Description |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 1 | Operational data for Vhost-user interfaces | Honeycomb Operational data reports Vhost-user interfaces |
+| | "server" flag | as client, even if they are server. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 2 | Persistence of VxLAN tunnels | Configuration persistence often fails to restore |
+| | | Honeycomb's internal naming context for VxLAN interfaces. |
+| | | The interface is renamed to "vxlan_tunnel0" but is otherwise |
+| | | configured correctly. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+
diff --git a/docs/report/honeycomb_functional_tests/documentation.rst b/docs/report/honeycomb_functional_tests/documentation.rst new file mode 100644 index 0000000000..e53052ddb1 --- /dev/null +++ b/docs/report/honeycomb_functional_tests/documentation.rst @@ -0,0 +1,6 @@ +Documentation
+=============
+
+`CSIT Honeycomb Functional Tests Documentation
+<https://docs.fd.io/csit/master/doc/tests.func.html>`_ contains detailed
+functional description and input parameters for each test case.
diff --git a/docs/report/honeycomb_functional_tests_virl/index.rst b/docs/report/honeycomb_functional_tests/index.rst index 6b58b06935..d5d1b6180b 100644 --- a/docs/report/honeycomb_functional_tests_virl/index.rst +++ b/docs/report/honeycomb_functional_tests/index.rst @@ -1,5 +1,5 @@ -HoneyComb Functional Tests VIRL -=============================== +HoneyComb Functional Tests +========================== .. toctree:: diff --git a/docs/report/honeycomb_functional_tests_virl/overview.rst b/docs/report/honeycomb_functional_tests/overview.rst index 2bfc89a814..d93de132a9 100644 --- a/docs/report/honeycomb_functional_tests_virl/overview.rst +++ b/docs/report/honeycomb_functional_tests/overview.rst @@ -1,8 +1,8 @@ Overview
========
-Tested Topologies VIRL
-----------------------
+Tested Virtual Topologies
+-------------------------
CSIT Honeycomb functional tests are executed on virtualized topologies created
using Virtual Internet Routing Lab (VIRL) simulation platform contributed by
@@ -45,8 +45,8 @@ test execution, all nodes are reachable thru the Management network connected to every node via dedicated virtual NICs and virtual links (not shown above
for clarity).
-Honeycomb Functional Tests Overview
------------------------------------
+Functional Tests Coverage
+-------------------------
The following Honeycomb functional test areas are included in the CSIT |release|
with results listed in this report:
@@ -135,3 +135,34 @@ Because of this, test cases follow this general pattern: Test cases involving network interfaces utilize the first two interfaces on
the DUT node.
+Functional Tests Naming
+-----------------------
+
+CSIT |release| introduced a common structured naming convention for all
+performance and functional tests. This change was driven by substantially
+growing number and type of CSIT test cases. Firstly, the original practice did
+not always follow any strict naming convention. Secondly test names did not
+always clearly capture tested packet encapsulations, and the actual type or
+content of the tests. Thirdly HW configurations in terms of NICs, ports and
+their locality were not captured either. These were but few reasons that drove
+the decision to change and define a new more complete and stricter test naming
+convention, and to apply this to all existing and new test cases.
+
+The new naming should be intuitive for majority of the tests. The complete
+description of CSIT test naming convention is provided on `CSIT test naming
+page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
+
+Here few illustrative examples of the new naming usage for functional test
+suites:
+
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
+
+ - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline
+ routed forwarding, functional tests.
+
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**
+
+ - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of
+ Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost
+ interfaces and one VM, functional tests.
diff --git a/docs/report/honeycomb_functional_tests_virl/test_environment.rst b/docs/report/honeycomb_functional_tests/test_environment.rst index e706aa5573..e706aa5573 100644 --- a/docs/report/honeycomb_functional_tests_virl/test_environment.rst +++ b/docs/report/honeycomb_functional_tests/test_environment.rst diff --git a/docs/report/honeycomb_functional_tests_virl/csit_release_notes.rst b/docs/report/honeycomb_functional_tests_virl/csit_release_notes.rst deleted file mode 100644 index 2708b05b0c..0000000000 --- a/docs/report/honeycomb_functional_tests_virl/csit_release_notes.rst +++ /dev/null @@ -1,68 +0,0 @@ -CSIT Release Notes
-==================
-
-Changes in CSIT |release|
--------------------------
-
-#. Naming change for all Honeycomb functional test suites in VIRL
-
- - Honeycomb functional test case names stayed unchanged
-
-#. Added Honeycomb functional tests
-
- - NSH_SFC
- - LISP
- - NAT
- - SPAN
-
-Functional Tests Naming
------------------------
-
-CSIT |release| introduced a common structured naming convention for all
-performance and functional tests. This change was driven by substantially
-growing number and type of CSIT test cases. Firstly, the original practice did
-not always follow any strict naming convention. Secondly test names did not
-always clearly capture tested packet encapsulations, and the actual type or
-content of the tests. Thirdly HW configurations in terms of NICs, ports and
-their locality were not captured either. These were but few reasons that drove
-the decision to change and define a new more complete and stricter test naming
-convention, and to apply this to all existing and new test cases.
-
-The new naming should be intuitive for majority of the tests. The complete
-description of CSIT test naming convention is provided on `CSIT test naming
-page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
-
-Here few illustrative examples of the new naming usage for functional test
-suites:
-
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
-
- - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline
- routed forwarding, functional tests.
-
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
- P2V2P, NIC2VMchain2NIC, P2V2V2P**
-
- - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of
- Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost
- interfaces and one VM, functional tests.
-
-
-Known Issues
-------------
-
-Here is the list of known issues in CSIT |release| for Honeycomb functional
-tests in VIRL:
-
-+-------------------------------------------------+-----------------------------------------------------------------+
-| Issue | Description |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| Operational data for Vhost-user interfaces | Honeycomb Operational data reports Vhost-user interfaces |
-| "server" flag | as client, even if they are server. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-| Persistence of VxLAN tunnels | Configuration persistence often fails to restore |
-| | Honeycomb's internal naming context for VxLAN interfaces. |
-| | The interface is renamed to "vxlan_tunnel0" but is otherwise |
-| | configured correctly. |
-+-------------------------------------------------+-----------------------------------------------------------------+
-
diff --git a/docs/report/honeycomb_functional_tests_virl/documentation.rst b/docs/report/honeycomb_functional_tests_virl/documentation.rst deleted file mode 100644 index 6904068516..0000000000 --- a/docs/report/honeycomb_functional_tests_virl/documentation.rst +++ /dev/null @@ -1,4 +0,0 @@ -Documentation
-=============
-
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.func.html.
diff --git a/docs/report/index.rst b/docs/report/index.rst index fe3fe55c5f..9eeabf685f 100644 --- a/docs/report/index.rst +++ b/docs/report/index.rst @@ -5,14 +5,9 @@ CSIT 17.01 :numbered: introduction/index - vpp_performance_tests_hw/index - testpmd_performance_tests_hw/index - vpp_functional_tests_virl/index - vpp_functional_tests_vpptest/index - honeycomb_functional_tests_virl/index - vpp_performance_results_hw/index - testpmd_performance_results_hw/index - vpp_functional_results_virl/index - vpp_functional_results_vpptest/index - honeycomb_functional_results_virl/index - + vpp_performance_tests/index + testpmd_performance_tests/index + vpp_functional_tests/index + honeycomb_functional_tests/index + vpp_unit_tests/index + detailed_test_results/index diff --git a/docs/report/introduction/general_notes.rst b/docs/report/introduction/general_notes.rst new file mode 100644 index 0000000000..c05401e4cc --- /dev/null +++ b/docs/report/introduction/general_notes.rst @@ -0,0 +1,57 @@ +General Notes
+=============
+
+All CSIT test results listed in this report are sourced and auto-generated
+from output.xml Robot Framework (RF) files resulting from LF FD.io Jenkins
+jobs execution against VPP-17.01 release artifacts. References are provided
+to the original LF FD.io Jenkins job results. However, as LF FD.io Jenkins
+infrastructure does not automatically archive all jobs (history record is
+provided for the last 30 days or 40 jobs only), additional references are
+provided to the RF result files that got archived in FD.io nexus online
+storage system.
+
+FD.io CSIT project currently covers multiple FD.io system and sub-system
+testing areas and this is reflected in this report, where each testing area
+is listed separately, as follows
+
+#. **VPP Performance Tests** - VPP performance are tests executed in physical
+ FD.io testbeds, focusing on VPP network data plane performance at this stage,
+ both for Phy-to-Phy (NIC-to-NIC) and Phy-to-VM-to-Phy (NIC-to-VM-to-NIC)
+ forwarding topologies. Tested across a range of NICs, 10GE and 40GE
+ interfaces, range of multi-thread and multi-core configurations. VPP
+ application runs in host user- mode. TRex is used as a traffic generator.
+
+#. **Testpmd Performance Tests** - VPP is using DPDK code to control and drive
+ the NICs and physical interfaces. Testpmd tests are used as a baseline to
+ profile the DPDK sub-system of VPP. Testpmd performance tests executed in
+ physical FD.io testbeds, focusing on Testpmd data plane performance for Phy-
+ to-Phy (NIC-to-NIC). Tests cover a range of NICs, 10GE and 40GE interfaces,
+ range of multi-thread and multi-core configurations. Testpmd application runs
+ in host user-mode. TRex is used as a traffic generator.
+
+#. **VPP Functional Tests** - VPP functional tests are executed in virtual
+ FD.io testbeds focusing on VPP packet processing functionality, including
+ network data plane and in -line control plane. Tests cover vNIC-to-vNIC
+ vNIC-to-VM-to-vNIC forwarding topologies. Scapy is used as a traffic
+ generator.
+
+#. **HoneyComb Functional Tests** - HoneyComb functional tests are executed in
+ virtual FD.io testbeds focusing on HoneyComb management and programming
+ functionality of VPP. Tests cover a range of CRUD operations executed
+ against VPP.
+
+CSIT |release| report does also include VPP unit test results. These tests
+have been developed within the FD.io VPP project, and not in CSIT, but they do
+complement tests done by CSIT. They are provided mainly as a reference, to
+give the reader a more complete view of automated testing executed against
+VPP-17.01 release.
+
+FD.io CSIT system is developed using two main coding platforms: Robot
+Framework (RF) and Python. CSIT |release| source code for the executed test
+suites is available in CSIT branch |release| in the directory
+"./tests/<name_of_the_test_suite>". A local copy of CSIT source code can be
+obtained by cloning CSIT git repository - "git clone
+https://gerrit.fd.io/r/csit". The CSIT testing virtual environment can be run
+on a local computer workstation (laptop, server) using Vagrant by following
+the instructions in `CSIT tutorials
+<https://wiki.fd.io/view/CSIT#Tutorials>`_.
\ No newline at end of file diff --git a/docs/report/introduction/general_remarks.rst b/docs/report/introduction/general_remarks.rst deleted file mode 100644 index 60fcbd882d..0000000000 --- a/docs/report/introduction/general_remarks.rst +++ /dev/null @@ -1,17 +0,0 @@ -General Remarks
-===============
-
-All CSIT test results listed in this report are sourced and auto-generated
-from output.xml Robot Framework (RF) files resulting from LF FD.io Jenkins
-jobs execution against VPP-17.01 release artifacts. References are provided
-to the original LF FD.io Jenkins job results. However, as LF FD.io Jenkins
-infrastructure does not automatically archive all jobs (history record is
-provided for the last 30 dayts or 40 jobs only), additional references are
-provided to the RF result files that got archived in FD.io nexus online
-storage system.
-
-CSIT |release| report does also include VPP make test results, mainly as
-a reference. Although these tests have not been developed FD.io CSIT project,
-but inthe FD.io VPP project, they have been provided in this report to give
-reader a more complete view of automated testing executed against VPP-17.01
-release, as they do complement tests done by CSIT.
diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst index 1aa603b692..1a485e2b1e 100644 --- a/docs/report/introduction/index.rst +++ b/docs/report/introduction/index.rst @@ -4,5 +4,5 @@ Introduction .. toctree:: overview + general_notes csit_test_naming - general_remarks diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index dc86f8600e..ee1a91c174 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -1,49 +1,68 @@ Overview
========
-This is the **F**\ast **D**\ata **I**/**O** Project (FD.io) **C**\ontinuous
-**S**\ystem **I**\ntegration and **T**\esting (CSIT) project report for CSIT
-|release| system testing of VPP-17.01 release. Initial sections of the report
-describe FD.io functional and performance tests and their continuous execution
-achieved in CSIT |release|. This is followed by high-level overview of FD.io
-test environments running in Linux Foundation (LF) FD.io Continuous
-Performance Labs), CSIT |release| release notes and result highlights.
-Subsequent sections list the functional and performance detailed test results
-generated by LF FD.io Jenkins system executing CSIT jobs running tests
-against the VPP-17.01 release and associated sub-systems (HoneyComb, DPDK).
-
-The exact structure of the report is as follows:
-
-#. ``Introduction`` - Includes: ``Overview`` - this section;
- ``CSIT Test Naming`` - describing CSIT naming convention for test suites and
- test cases, important to understand to correctly interpret the reported
- results; ``General Remarks`` - additional notes related to this report.
-#. ``VPP Performance Tests HW`` - Summary of VPP performance tests executed in
- physical Hardware testbed. Includes: ``Overview`` - containing topology
- description and list of executed tests; ``Performance Tests Naming`` - lists
- CSIT naming specifics for performance tests; ``CSIT Release Notes`` - with
- changes in CSIT |release| including major test additions, performance
- improvements, changes in testing environment and methodology and known
- issues; ``Performance Measurements Methodology`` - details testing
- methodology followed for multi-thread multi-core, packet throughput and
- latency, and KVM VM vhost tests; ``Report Addendum Targets`` - lists content
- to be added to this report as part of CSIT |release|.
-#. ``Testpmd Performance Tests HW`` - ...
-#. ``VPP Functional Tests VIRL`` - ...
-#. ``VPP Functional Tests VPPtest`` - ...
-#. ``HoneyComb Functional Tests VIRL`` - ..
-#. ``VPP Performance Results HW`` - ...
-#. ``Testpmd Performance Results HW`` - ...
-#. ``VPP Functional Results VIRL`` - ...
-#. ``VPP Functional Results VPPtest`` - ...
-#. ``HoneyComb Functional Results VIRL`` - ...
-
-FD.io CSIT system is developed using two main coding platforms: Robot
-Framework (RF) and Python. CSIT |release| source code for the executed test
-suites is available in CSIT branch |release| in the directory
-"./tests/<name_of_the_test_suite>". A local copy of CSIT source code can be
-obtained by cloning CSIT git repository - "git clone
-https://gerrit.fd.io/r/csit". The CSIT testing virtual environment can be run
-on a local computer workstation (laptop, server) using Vagrant by following
-the instructions in `CSIT tutorials
-<https://wiki.fd.io/view/CSIT#Tutorials>`_.
+This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous
+**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT
+|release| system testing of VPP-17.01 release.
+
+The report describes CSIT functional and performance tests and their
+continuous execution delivered in CSIT |release|. A high-level overview is
+provided for each CSIT test environment running in Linux Foundation (LF) FD.io
+Continuous Performance Labs. This is followed by summary of all executed tests
+against the VPP-17.01 release and associated sub-systems (HoneyComb, DPDK),
+CSIT |release| release notes, result highlights and known issues. More
+detailed description of each environment, pointers to CSIT test code
+documentation and detailed test resuls with links to the source data files are
+also provided.
+
+CSIT |release| report contains following main sections and sub-sections:
+
+#. **Introduction** - general introduction to CSIT project; *Overview* -
+ this section; *CSIT Test Naming* - CSIT general naming convention for test
+ suites and test cases, important to recognize the high-level test content
+ and interpret reported results; *General Notes* - general notes related to
+ this report.
+
+#. **VPP Performance Tests** - VPP performance tests executed in physical
+ FD.io testbeds; *Overview* - tested topologies, test coverage and naming
+ specifics, methodology for multi-core, packet throughput and latency, and
+ KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added
+ tests, performance changes, environment or methodology changes, known CSIT
+ issues, tests to be added; *Packet Throughput Graphs* and *Packet Latency
+ Graphs* - plotted NDR, PDR throughput and latency results from multiple
+ test job executions; *Test Environment* - environment description ;
+ *Documentation* - source code documentation for VPP performance tests.
+
+#. **Testpmd Performance Tests** - Testpmd performance tests executed in
+ physical FD.io testbeds; *Overview* - tested topologies, test coverage;
+ *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues;
+ *Tests to Be Added* - performance tests to be added in the next revision of
+ CSIT |release| report; *Packet Throughput Graphs* and *Packet Latency Graphs*
+ - plotted NDR, PDR throughput and latency results from multiple test job
+ executions; *Test Environment* - environment description; *Documentation* -
+ source code documentation for Testpmd performance tests.
+
+#. **VPP Functional Tests** - VPP functional tests executed in virtual
+ FD.io testbeds; *Overview* - tested virtual topologies, test coverage and
+ naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added
+ tests, environment or methodology changes, known CSIT issues, tests to be
+ added; *Test Environment* - environment description ; *Documentation* -
+ source code documentation for VPP functional tests.
+
+#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in
+ virtual FD.io testbeds; *Overview* - tested virtual topologies, test
+ coverage and naming specifics; *CSIT Release Notes* - changes in CSIT
+ |release|, added tests, environment or methodology changes, known CSIT issues,
+ tests to be added; *Test Environment* - environment description ;
+ *Documentation* - source code documentation for Honeycomb functional tests.
+
+#. **VPP Unit Tests** - refers to VPP functional unit tests executed as
+ part of vpp make test verify option within the FD.io VPP project; listed in
+ this report to give a more complete view about executed VPP functional tests;
+ *Overview* - short overview of unit test framework and executed tests;
+ *Documentation* - source code documentation of VPP unit tests.
+
+#. **Detailed Test Results** - auto-generated results from CSIT jobs
+ executions using CSIT Robot Framework output files as source data; *VPP
+ Performance Results*, *Testpmd Performance Results*, *VPP Functional
+ Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*.
diff --git a/docs/report/testpmd_performance_results_hw/index.rst b/docs/report/testpmd_performance_results_hw/index.rst deleted file mode 100644 index 5fa5a67fb9..0000000000 --- a/docs/report/testpmd_performance_results_hw/index.rst +++ /dev/null @@ -1,6 +0,0 @@ -Testpmd Performance Results HW
-==============================
-
-.. toctree::
-
- testpmd_performance_results_hw
diff --git a/docs/report/testpmd_performance_tests/csit_release_notes.rst b/docs/report/testpmd_performance_tests/csit_release_notes.rst new file mode 100644 index 0000000000..09d7f727da --- /dev/null +++ b/docs/report/testpmd_performance_tests/csit_release_notes.rst @@ -0,0 +1,16 @@ +CSIT Release Notes +================== + +Changes in CSIT |release| +------------------------- + +#. Added Testpmd tests + + - new NICs - Intel x520 + +Tests to be Added - Additional NICs +----------------------------------- + +Adding test cases with more NIC types. Once the results become available, they +will be published as an addendum to the current version of CSIT |release| +report. diff --git a/docs/report/testpmd_performance_tests/documentation.rst b/docs/report/testpmd_performance_tests/documentation.rst new file mode 100644 index 0000000000..737e46505b --- /dev/null +++ b/docs/report/testpmd_performance_tests/documentation.rst @@ -0,0 +1,6 @@ +Documentation
+=============
+
+`CSIT Testpmd Performance Tests Documentation
+<https://docs.fd.io/csit/master/doc/tests.perf.html>`_ contains detailed
+functional description and input parameters for each test case.
diff --git a/docs/report/testpmd_performance_tests_hw/index.rst b/docs/report/testpmd_performance_tests/index.rst index a2302a9460..23fa5d9d8d 100644 --- a/docs/report/testpmd_performance_tests_hw/index.rst +++ b/docs/report/testpmd_performance_tests/index.rst @@ -1,5 +1,5 @@ -Testpmd Performance Tests HW -============================ +Testpmd Performance Tests +========================= .. toctree:: diff --git a/docs/report/testpmd_performance_tests_hw/overview.rst b/docs/report/testpmd_performance_tests/overview.rst index 4ef3982e49..db1ac7849d 100644 --- a/docs/report/testpmd_performance_tests_hw/overview.rst +++ b/docs/report/testpmd_performance_tests/overview.rst @@ -1,8 +1,8 @@ Overview ======== -Tested Topologies HW --------------------- +Tested Physical Topologies +-------------------------- CSIT Testpmd performance tests are executed on physical baremetal servers hosted by LF FD.io project. Testbed physical topology is shown in the figure below. @@ -45,8 +45,8 @@ tested NIC models include: Detailed LF FD.io test bed specification and topology is described in `wiki CSIT LF testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_. -Testing Summary ---------------- +Performance Tests Coverage +-------------------------- Performance tests are split into the two main categories: @@ -78,3 +78,73 @@ Niantic) has been limited to few baseline tests. Over time we expect the HW testbed resources to grow, and will be adding complete set of performance tests for all models of hardware to be executed regularly and(or) continuously. + +Methodology: Multi-Thread and Multi-Core +---------------------------------------- + +**HyperThreading** - CSIT |release| performance tests are executed with SUT +servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS +settings). This is the simplest configuration used to establish baseline +single-thread single-core SW packet processing and forwarding performance. +Subsequent releases of CSIT will add performance tests with Intel +HyperThreading Enabled (requires BIOS settings change and hard reboot). + +**Multi-core Test** - CSIT |release| multi-core tests are executed in the +following Testpmd thread and core configurations: + +#. 1t1c - 1 Testpmd pmd thread on 1 CPU physical core. +#. 2t2c - 2 Testpmd pmd threads on 2 CPU physical cores. +#. 4t4c - 4 Testpmd pmd threads on 4 CPU physical cores. + +Note that in many tests running Testpmd reaches tested NIC I/O bandwidth +or packets-per-second limit. + +Methodology: Packet Throughput +------------------------------ + +Following values are measured and reported for packet throughput tests: + +- NDR binary search per RFC2544: + + - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps + (2x <per direction packets-per-second>)" + - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per + second> Gbps (untagged)" + +- PDR binary search per RFC2544: + + - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x + <per direction packets-per-second>)" + - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per + second> Gbps (untagged)" + - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets + lost at PDR rate>"" + +- NDR and PDR are measured for the following L2 frame sizes: + + - IPv4: 64B, 1518B, 9000B. + + +Methodology: Packet Latency +--------------------------- + +TRex Traffic Generator (TG) is used for measuring latency of Testpmd DUTs. +Reported latency values are measured using following methodology: + +- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate) + for each NDR throughput test and packet size (except IMIX). +- TG sends dedicated latency streams, one per direction, each at the rate of + 10kpps at the prescribed packet size; these are sent in addition to the main + load streams. +- TG reports min/avg/max latency values per stream direction, hence two sets + of latency values are reported per test case; future release of TRex is + expected to report latency percentiles. +- Reported latency values are aggregate across two SUTs due to three node + topology used for all performance tests; for per SUT latency, reported value + should be divided by two. +- 1usec is the measurement accuracy advertised by TRex TG for the setup used in + FD.io labs used by CSIT project. +- TRex setup introduces an always-on error of about 2*2usec per latency flow - + additonal Tx/Rx interface latency induced by TRex SW writing and reading + packet timestamps on CPU cores without HW acceleration on NICs closer to the + interface line.
\ No newline at end of file diff --git a/docs/report/testpmd_performance_tests_hw/packet_latency_graphs/index.rst b/docs/report/testpmd_performance_tests/packet_latency_graphs/index.rst index 4402794a1b..4402794a1b 100644 --- a/docs/report/testpmd_performance_tests_hw/packet_latency_graphs/index.rst +++ b/docs/report/testpmd_performance_tests/packet_latency_graphs/index.rst diff --git a/docs/report/testpmd_performance_tests_hw/packet_latency_graphs/l2.rst b/docs/report/testpmd_performance_tests/packet_latency_graphs/l2.rst index a323761133..a323761133 100644 --- a/docs/report/testpmd_performance_tests_hw/packet_latency_graphs/l2.rst +++ b/docs/report/testpmd_performance_tests/packet_latency_graphs/l2.rst diff --git a/docs/report/testpmd_performance_tests_hw/packet_throughput_graphs/index.rst b/docs/report/testpmd_performance_tests/packet_throughput_graphs/index.rst index 4e4e87a1ea..4e4e87a1ea 100644 --- a/docs/report/testpmd_performance_tests_hw/packet_throughput_graphs/index.rst +++ b/docs/report/testpmd_performance_tests/packet_throughput_graphs/index.rst diff --git a/docs/report/testpmd_performance_tests_hw/packet_throughput_graphs/l2.rst b/docs/report/testpmd_performance_tests/packet_throughput_graphs/l2.rst index 31a3d31b66..31a3d31b66 100644 --- a/docs/report/testpmd_performance_tests_hw/packet_throughput_graphs/l2.rst +++ b/docs/report/testpmd_performance_tests/packet_throughput_graphs/l2.rst diff --git a/docs/report/testpmd_performance_tests_hw/test_environment.rst b/docs/report/testpmd_performance_tests/test_environment.rst index e4f301daa0..e4f301daa0 100644 --- a/docs/report/testpmd_performance_tests_hw/test_environment.rst +++ b/docs/report/testpmd_performance_tests/test_environment.rst diff --git a/docs/report/testpmd_performance_tests_hw/csit_release_notes.rst b/docs/report/testpmd_performance_tests_hw/csit_release_notes.rst deleted file mode 100644 index cc1bf8dd1c..0000000000 --- a/docs/report/testpmd_performance_tests_hw/csit_release_notes.rst +++ /dev/null @@ -1,87 +0,0 @@ -CSIT Release Notes -================== - -Changes in CSIT |release| -------------------------- - -#. Added Testpmd tests - - - new NICs - Intel x520 - -Multi-Thread and Multi-Core Measurements ----------------------------------------- - -**HyperThreading** - CSIT |release| performance tests are executed with SUT -servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS -settings). This is the simplest configuration used to establish baseline -single-thread single-core SW packet processing and forwarding performance. -Subsequent releases of CSIT will add performance tests with Intel -HyperThreading Enabled (requires BIOS settings change and hard reboot). - -**Multi-core Test** - CSIT |release| multi-core tests are executed in the -following Testpmd thread and core configurations: - -#. 1t1c - 1 Testpmd pmd thread on 1 CPU physical core. -#. 2t2c - 2 Testpmd pmd threads on 2 CPU physical cores. -#. 4t4c - 4 Testpmd pmd threads on 4 CPU physical cores. - -Note that in many tests running Testpmd reaches tested NIC I/O bandwidth -or packets-per-second limit. - -Packet Throughput Measurements ------------------------------- - -Following values are measured and reported for packet throughput tests: - -- NDR binary search per RFC2544: - - - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps - (2x <per direction packets-per-second>)" - - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per - second> Gbps (untagged)" - -- PDR binary search per RFC2544: - - - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x - <per direction packets-per-second>)" - - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per - second> Gbps (untagged)" - - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets - lost at PDR rate>"" - -- NDR and PDR are measured for the following L2 frame sizes: - - - IPv4: 64B, 1518B, 9000B. - - -Packet Latency Measurements ---------------------------- - -TRex Traffic Generator (TG) is used for measuring latency of Testpmd DUTs. -Reported latency values are measured using following methodology: - -- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate) - for each NDR throughput test and packet size (except IMIX). -- TG sends dedicated latency streams, one per direction, each at the rate of - 10kpps at the prescribed packet size; these are sent in addition to the main - load streams. -- TG reports min/avg/max latency values per stream direction, hence two sets - of latency values are reported per test case; future release of TRex is - expected to report latency percentiles. -- Reported latency values are aggregate across two SUTs due to three node - topology used for all performance tests; for per SUT latency, reported value - should be divided by two. -- 1usec is the measurement accuracy advertised by TRex TG for the setup used in - FD.io labs used by CSIT project. -- TRex setup introduces an always-on error of about 2*2usec per latency flow - - additonal Tx/Rx interface latency induced by TRex SW writing and reading - packet timestamps on CPU cores without HW acceleration on NICs closer to the - interface line. - - -Report Addendum Tests - Additional NICs ---------------------------------------- - -Adding test cases with more NIC types. Once the results become available, they -will be published as an addendum to the current version of CSIT |release| -report. diff --git a/docs/report/testpmd_performance_tests_hw/documentation.rst b/docs/report/testpmd_performance_tests_hw/documentation.rst deleted file mode 100644 index 1bc1bcbcc9..0000000000 --- a/docs/report/testpmd_performance_tests_hw/documentation.rst +++ /dev/null @@ -1,4 +0,0 @@ -Documentation
-=============
-
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.perf.html.
diff --git a/docs/report/vpp_functional_results_virl/index.rst b/docs/report/vpp_functional_results_virl/index.rst deleted file mode 100644 index e0302e49b7..0000000000 --- a/docs/report/vpp_functional_results_virl/index.rst +++ /dev/null @@ -1,7 +0,0 @@ -VPP Functional Results VIRL
-===========================
-
-.. toctree::
-
- functional_results_virl
-
diff --git a/docs/report/vpp_functional_results_vpptest/functional_results_vpptest.rst b/docs/report/vpp_functional_results_vpptest/functional_results_vpptest.rst deleted file mode 100644 index 8132622a13..0000000000 --- a/docs/report/vpp_functional_results_vpptest/functional_results_vpptest.rst +++ /dev/null @@ -1,24 +0,0 @@ -DHCP - Client and Proxy
-```````````````````````
-
-to be added ..
-
-GRE Overlay Tunnels
-```````````````````
-
-to be added ..
-
-L2BD Ethernet Switching
-```````````````````````
-
-to be added ..
-
-L2XC Ethernet Switching
-```````````````````````
-
-to be added ..
-
-LISP Overlay Tunnels
-````````````````````
-
-to be added ..
diff --git a/docs/report/vpp_functional_results_vpptest/index.rst b/docs/report/vpp_functional_results_vpptest/index.rst deleted file mode 100644 index 2284d165d4..0000000000 --- a/docs/report/vpp_functional_results_vpptest/index.rst +++ /dev/null @@ -1,7 +0,0 @@ -VPP Functional Results VPPtest
-==============================
-
-.. toctree::
-
- functional_results_vpptest
-
diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst new file mode 100644 index 0000000000..6db4900046 --- /dev/null +++ b/docs/report/vpp_functional_tests/csit_release_notes.rst @@ -0,0 +1,76 @@ + +.. |br| raw:: html + + <br /> + +CSIT Release Notes +================== + +Changes in CSIT |release| +------------------------- + +#. Naming change for all VPP functional test suites in VIRL + + - VPP functional test case names stayed unchanged + +#. VPP functional test environment changes + + - upgrade to Ubuntu 16.04 + - VM and vhost-user test environment optimizations + +#. Introduction of Centos tests + +#. Added VPP functional tests + + - more VM vhost-user tests + - more LISP tests + - more IPSec crypto tests + - IPv4 and IPv6 Equal-Cost Multi-Path routing tests + - Telemetry: + - IPFIX tests + - SPAN tests + +Known Issues +------------ + +Here is the list of known issues in CSIT |release| for VPP functional tests in VIRL: + ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| # | Issue | Description | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 1 | DHCPv4 client: Client responses to DHCPv4 OFFER | Client replies with DHCPv4 REQUEST message when received DHCPv4 | +| | sent with different XID | OFFER message with different (wrong) XID. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 2 | Softwire - MAP-E: Incorrect calculation of IPv6 | IPv6 destination address is wrongly calculated in case that | +| | destination address when IPv4 prefix is 0 | IPv4 prefix is equal to 0 and IPv6 prefix is less than 40. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 3 | Softwire - MAP-E: Map domain is created when | Map domain is created in case that the sum of suffix length of | +| | incorrect parameters provided | IPv4 prefix and PSID length is greater than EA bits length. | +| | | IPv6 destination address contains bits writen with PSID over | +| | | the EA-bit length when IPv4 packet is sent. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 4 | IPv6 RA: Incorrect IPv6 destination address in | Wrong IPv6 destination address (ff02::1) is used in ICMPv6 | +| | response to ICMPv6 Router Solicitation | Router Advertisement packet sent as a response to received | +| | | ICMPv6 Router Solicitation packet. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 5 | IPFIX: IPv6_src key name reported instead of | The report contains IPv6_src key name instead of IPv6_dst when | +| | IPv6_dst | classify session is configured with IPv6 destination address. | +| | | Anyhow the correct IPv6 destination address is reported. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 6 | SPAN: Tx traffic is not mirrored | Tx traffic is not mirrored from SPAN source port to SPAN | +| | | destination port. |br| | +| | | NOTE: Fix is going to be merged in vpp stable/1701. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ +| 7 | SPAN: Packet trace always contains local0 as | There is reported wrong destination port in the traffic trace: | +| | destination port | |br| SPAN: mirrored GigabitEthernet0/5/0 -> local0 |br| | +| | | The (Rx) traffic is mirrored to correct destination port. | ++---+-------------------------------------------------+-----------------------------------------------------------------+ + +Tests to be Added - Centos +-------------------------- + +CSIT |release| added Centos functional test execution environment in FD.io +VIRL testbeds.However due to the factors beyond CSIT project control execution +of those tests within the LF FD.io test environment still need to complete. +Once the results become available, they will be published as an addendum to +the current version of CSIT |release| report. diff --git a/docs/report/vpp_functional_tests/documentation.rst b/docs/report/vpp_functional_tests/documentation.rst new file mode 100644 index 0000000000..e286e6cf19 --- /dev/null +++ b/docs/report/vpp_functional_tests/documentation.rst @@ -0,0 +1,7 @@ +Documentation
+=============
+
+`CSIT VPP Functional Tests Documentation
+<https://docs.fd.io/csit/master/doc/tests.func.html>`_ contains detailed
+functional description and input parameters for each test case.
+
diff --git a/docs/report/vpp_functional_tests_virl/index.rst b/docs/report/vpp_functional_tests/index.rst index 564783b750..5111296c78 100644 --- a/docs/report/vpp_functional_tests_virl/index.rst +++ b/docs/report/vpp_functional_tests/index.rst @@ -1,5 +1,5 @@ -VPP Functional Tests VIRL -========================= +VPP Functional Tests +==================== .. toctree:: diff --git a/docs/report/vpp_functional_tests_virl/overview.rst b/docs/report/vpp_functional_tests/overview.rst index 16c7ef7ba0..e85bb92f7b 100644 --- a/docs/report/vpp_functional_tests_virl/overview.rst +++ b/docs/report/vpp_functional_tests/overview.rst @@ -1,8 +1,8 @@ Overview
========
-Tested Topologies VIRL
-----------------------
+Tested Virtual Topologies
+-------------------------
CSIT VPP functional tests are executed on virtualized topologies created using
Virtual Internet Routing Lab (VIRL) simulation platform contributed by Cisco.
@@ -94,10 +94,10 @@ is shown in the figure below including the applicable packet flow thru the VM ******************* | |********************
+-----------+
-VPP Functional Tests Overview
------------------------------
+Functional Tests Coverage
+-------------------------
-Following VPP functional test areas are included in the CSIT |release| with
+Following VPP functional test areas are covered in the CSIT |release| with
results listed in this report:
- **DHCP - Client and Proxy** - Dynamic Host Control Protocol Client and Proxy
@@ -127,3 +127,35 @@ results listed in this report: IPv6.
- **VLAN Tag Translation** - L2 VLAN tag translation 2to2, 2to1, 1to2, 1to1.
- **VXLAN Overlay Tunnels** - VXLAN tunneling for L2-over-IP, for IPv4, IPv6.
+
+Functional Tests Naming
+-----------------------
+
+CSIT |release| introduced a common structured naming convention for all
+performance and functional tests. This change was driven by substantially
+growing number and type of CSIT test cases. Firstly, the original practice did
+not always follow any strict naming convention. Secondly test names did not
+always clearly capture tested packet encapsulations, and the actual type or
+content of the tests. Thirdly HW configurations in terms of NICs, ports and
+their locality were not captured either. These were but few reasons that drove
+the decision to change and define a new more complete and stricter test naming
+convention, and to apply this to all existing and new test cases.
+
+The new naming should be intuitive for majority of the tests. The complete
+description of CSIT test naming convention is provided on `CSIT test naming
+page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
+
+Here few illustrative examples of the new naming usage for functional test
+suites:
+
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
+
+ - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline
+ routed forwarding, functional tests.
+
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**
+
+ - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of
+ Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost
+ interfaces and one VM, functional tests.
diff --git a/docs/report/vpp_functional_tests_virl/test_environment.rst b/docs/report/vpp_functional_tests/test_environment.rst index fc7c63351a..fc7c63351a 100644 --- a/docs/report/vpp_functional_tests_virl/test_environment.rst +++ b/docs/report/vpp_functional_tests/test_environment.rst diff --git a/docs/report/vpp_functional_tests_virl/csit_release_notes.rst b/docs/report/vpp_functional_tests_virl/csit_release_notes.rst deleted file mode 100644 index 6035ddb9ce..0000000000 --- a/docs/report/vpp_functional_tests_virl/csit_release_notes.rst +++ /dev/null @@ -1,99 +0,0 @@ - -.. |br| raw:: html - - <br /> - -CSIT Release Notes -================== - -Changes in CSIT |release| -------------------------- - -#. Naming change for all VPP functional test suites in VIRL - - - VPP functional test case names stayed unchanged - -#. VPP functional test environment changes - - - upgrade to Ubuntu 16.04 - - VM and vhost-user test environment optimizations - -#. Introduction of Centos tests - -#. Added VPP functional tests - - - more VM vhost-user tests - - more LISP tests - - more IPSec crypto tests - - IPv4 and IPv6 Equal-Cost Multi-Path routing tests - - Telemetry: - - IPFIX tests - - SPAN tests - -Functional Tests Naming ------------------------ - -CSIT |release| introduced a common structured naming convention for all -performance and functional tests. This change was driven by substantially -growing number and type of CSIT test cases. Firstly, the original practice did -not always follow any strict naming convention. Secondly test names did not -always clearly capture tested packet encapsulations, and the actual type or -content of the tests. Thirdly HW configurations in terms of NICs, ports and -their locality were not captured either. These were but few reasons that drove -the decision to change and define a new more complete and stricter test naming -convention, and to apply this to all existing and new test cases. - -The new naming should be intuitive for majority of the tests. The complete -description of CSIT test naming convention is provided on `CSIT test naming -page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_. - -Here few illustrative examples of the new naming usage for functional test -suites: - -#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** - - - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline - routed forwarding, functional tests. - -#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, - P2V2P, NIC2VMchain2NIC, P2V2V2P** - - - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of - Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost - interfaces and one VM, functional tests. - -Known Issues ------------- - -Here is the list of known issues in CSIT |release| for VPP functional tests in VIRL: - -+-------------------------------------------------+-----------------------------------------------------------------+ -| ISSUE | DESCRIPTION | -+-------------------------------------------------+-----------------------------------------------------------------+ -| DHCPv4 client: Client responses to DHCPv4 OFFER | Client replies with DHCPv4 REQUEST message when received DHCPv4 | -| sent with different XID | OFFER message with different (wrong) XID. | -+-------------------------------------------------+-----------------------------------------------------------------+ -| Softwire - MAP-E: Incorrect calculation of IPv6 | IPv6 destination address is wrongly calculated in case that | -| destination address when IPv4 prefix is 0 | IPv4 prefix is equal to 0 and IPv6 prefix is less than 40. | -+-------------------------------------------------+-----------------------------------------------------------------+ -| Softwire - MAP-E: Map domain is created when | Map domain is created in case that the sum of suffix length of | -| incorrect parameters provided | IPv4 prefix and PSID length is greater than EA bits length. | -| | IPv6 destination address contains bits writen with PSID over | -| | the EA-bit length when IPv4 packet is sent. | -+-------------------------------------------------+-----------------------------------------------------------------+ -| IPv6 RA: Incorrect IPv6 destination address in | Wrong IPv6 destination address (ff02::1) is used in ICMPv6 | -| response to ICMPv6 Router Solicitation | Router Advertisement packet sent as a response to received | -| | ICMPv6 Router Solicitation packet. | -+-------------------------------------------------+-----------------------------------------------------------------+ -| IPFIX: IPv6_src key name reported instead of | The report contains IPv6_src key name instead of IPv6_dst when | -| IPv6_dst | classify session is configured with IPv6 destination address. | -| | Anyhow the correct IPv6 destination address is reported. | -+-------------------------------------------------+-----------------------------------------------------------------+ -| SPAN: Tx traffic is not mirrored | Tx traffic is not mirrored from SPAN source port to SPAN | -| | destination port. |br| | -| | NOTE: Fix is going to be merged in vpp stable/1701. | -+-------------------------------------------------+-----------------------------------------------------------------+ -| SPAN: Packet trace always contains local0 as | There is reported wrong destination port in the traffic trace: | -| destination port | |br| SPAN: mirrored GigabitEthernet0/5/0 -> local0 |br| | -| | The (Rx) traffic is mirrored to correct destination port. | -+-------------------------------------------------+-----------------------------------------------------------------+ diff --git a/docs/report/vpp_functional_tests_virl/documentation.rst b/docs/report/vpp_functional_tests_virl/documentation.rst deleted file mode 100644 index 6904068516..0000000000 --- a/docs/report/vpp_functional_tests_virl/documentation.rst +++ /dev/null @@ -1,4 +0,0 @@ -Documentation
-=============
-
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.func.html.
diff --git a/docs/report/vpp_functional_tests_vpptest/documentation.rst b/docs/report/vpp_functional_tests_vpptest/documentation.rst deleted file mode 100644 index 416fe8af46..0000000000 --- a/docs/report/vpp_functional_tests_vpptest/documentation.rst +++ /dev/null @@ -1,4 +0,0 @@ -Documentation
-=============
-
-Link
diff --git a/docs/report/vpp_functional_tests_vpptest/index.rst b/docs/report/vpp_functional_tests_vpptest/index.rst deleted file mode 100644 index 5cf36a9999..0000000000 --- a/docs/report/vpp_functional_tests_vpptest/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -VPP Functional Tests VPPtest -============================ - -.. toctree:: - - overview - test_environment - documentation - diff --git a/docs/report/vpp_functional_tests_vpptest/overview.rst b/docs/report/vpp_functional_tests_vpptest/overview.rst deleted file mode 100644 index 570ca4e3de..0000000000 --- a/docs/report/vpp_functional_tests_vpptest/overview.rst +++ /dev/null @@ -1,12 +0,0 @@ -Overview
-========
-
-VPP Functional Test Topologies in VPPtest
------------------------------------------
-
-to be added ..
-
-VPP Functional Tests Overview
------------------------------
-
-to be added ..
diff --git a/docs/report/vpp_functional_tests_vpptest/test_environment.rst b/docs/report/vpp_functional_tests_vpptest/test_environment.rst deleted file mode 100644 index db19dce75b..0000000000 --- a/docs/report/vpp_functional_tests_vpptest/test_environment.rst +++ /dev/null @@ -1,6 +0,0 @@ -Test Environment
-================
-
-VPPtest functional tests are currently executed in ...
-
-to be added
diff --git a/docs/report/vpp_performance_results_hw/index.rst b/docs/report/vpp_performance_results_hw/index.rst deleted file mode 100644 index 42f79bfd16..0000000000 --- a/docs/report/vpp_performance_results_hw/index.rst +++ /dev/null @@ -1,6 +0,0 @@ -VPP Performance Results HW
-==========================
-
-.. toctree::
-
- performance_results_hw
diff --git a/docs/report/vpp_performance_tests_hw/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index dc7f36db09..54fa73dfee 100644 --- a/docs/report/vpp_performance_tests_hw/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -11,92 +11,38 @@ Changes in CSIT |release| - VM and vhost-user test environment optimizations
- addition of HW cryptodev devices in LFD FD.io physical testbed
-
#. Added tests
- - new NICs - Intel x710, Cisco VIC1385, Cisco VIC1227
- - more VM vhost-user tests
- - more LISP tests
+ - NICs
+
+ - Intel x710 (L2 Bridge Domain switched-forwarding, vhost-user)
+ - Cisco VIC1385 (L2 Bridge Domain switched-forwarding)
+ - Cisco VIC1227 (L2 Bridge Domain switched-forwarding)
+
+ - vhost-user tests with VM
+
+ - L2 Bridge Domain switched-forwarding with Intel x710 NIC
+ - L2 Bridge Domain switched-forwarding with VxLAN and Intel x520 NIC
+ - L2 Bridge Domain switched-forwarding with Intel xl710 NIC
+ - Tests with VxLAN
-Performance Tests Naming
+ - L2 Bridge Domain switched-forwarding with Intel x520 NIC
+ - L2 Bridge Domain switched-forwarding with vhost-user, VM and Intel x520
+ NIC
+
+Performance Improvements
------------------------
-CSIT |release| introduced a common structured naming convention for all
-performance and functional tests. This change was driven by substantially
-growing number and type of CSIT test cases. Firstly, the original practice did
-not always follow any strict naming convention. Secondly test names did not
-always clearly capture tested packet encapsulations, and the actual type or
-content of the tests. Thirdly HW configurations in terms of NICs, ports and
-their locality were not captured either. These were but few reasons that drove
-the decision to change and define a new more complete and stricter test naming
-convention, and to apply this to all existing and new test cases.
-
-The new naming should be intuitive for majority of the tests. The complete
-description of CSIT test naming convention is provided on `CSIT test naming wiki
-<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
-
-Here few illustrative examples of the new naming usage for performance test
-suites:
-
-#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
-
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
- PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
- Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
- with MAC learning, NDR throughput discovery.
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
- on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
- switching with MAC learning, NDR throughput discovery.
- - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
- x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
- - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
- Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
- discovery.
-
-#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
- P2V2P, NIC2VMchain2NIC, P2V2V2P**
-
- - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
- PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
- VirtPortConfig-VMconfig-TestType*
- - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
- of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
- switching to/from two vhost interfaces and one VM, NDR throughput
- discovery.
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
- switching to/from two vhost interfaces and one VM, NDR throughput
- discovery.
- - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
- switching to/from four vhost interfaces and two VMs, NDR throughput
- discovery.
-
-Measured Performance Improvements
----------------------------------
-
-Substantial improvements in measured packet throughput have been observed
-in VPP-17.01 for the following CSIT |release| tests:
-
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Improvement |
-+===================+================================================================+===========+=================+======================+
-| L2XC | 10ge2p1x520:64B-1t1c-eth-l2xcbase-ndrdisc | 9.4 Mpps | 12.6..12.9 Mpps | 34..37% |
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-| L2XC | 10ge2p1xl710:64B-1t1c-eth-l2xcbase-ndrdisc | 9.5 Mpps | 12.1..12.4 Mpps | 27..30% |
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-| L2BD | 10ge2p1x520:64B-1t1c-eth-l2bdbasemaclrn-ndrdisc | 7.8 Mpps | 10.6 Mpps | 36% |
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-| L2BD-vhost-VM | 10ge2p1x520:64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4 Mpps | 2.8 Mpps | 600% |
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-| L2XC-vhost-VM | 10ge2p1x520:64B-1t1c-eth-l2xcbase-eth-2vhost-1vm-ndrdisc | 0.4 Mpps | 3.2 Mpps | 700% |
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 | 10ge2p1x520:64B-1t1c-ethip4-ip4base-ndrdisc | 8.7 Mpps | 9.7 Mpps | 12% |
-+-------------------+----------------------------------------------------------------+-----------+-----------------+----------------------+
-
-Non-drop rate search:
+Substantial improvements in measured packet throughput, with relative increase
+of double-digit percentage points, have been observed in a number of CSIT
+|release| tests listed below. Relative improvements are calculated against the
+test results in CSIT rls1609 report.
+
+NDR Throughput
+~~~~~~~~~~~~~~
+
+Non-Drop Rate Throughput discovery tests:
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Improvement |
@@ -117,46 +63,27 @@ Non-drop rate search: +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| L2BD-vhost-VM | 10ge2p1x520: 64B-1t1c-eth-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc | 0.4 Mpps | 2.7..2.8 Mpps | 575..600% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc | 8.7 Mpps | 9.7 Mpps | 11% |
+| IPv4 | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ndrdisc | 8.7 Mpps | 9.7 Mpps | 12% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv4 COP | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-ndrdisc | 7.1 Mpps | 8.3..8.5 Mpps | 17..20% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 FIB 200k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc | 8.5 Mpps | 9.0 Mpps | 6% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 FIB 20k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc | 8.5 Mpps | 9.0..9.2 Mpps | 6..8% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 FIB 2M | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc | 8.5 Mpps | 7.8..8.1 Mpps | -8..-5% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv4 iAcl | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-iacldstbase-ndrdisc | 6.9 Mpps | 7.6..7.8 Mpps | 10..13% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 Policer | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc | 6.9 Mpps | 7.4..7.6 Mpps | 7..10% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 LISP | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc | 4.4 Mpps | 4.8 Mpps | 9% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv4 vhost | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-ndrdisc | 0.3 Mpps | 2.6 Mpps | 767% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv6 | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-ndrdisc | 3.0 Mpps | 7.3..7.7 Mpps | 143..157% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv6 COP | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc | 6.1 Mpps | 6.1..6.5 Mpps | 0..7% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv6 FIB 200k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc | 6.5 Mpps | 5.3..5.7 Mpps | -18..-12% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv6 FIB 20k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc | 6.9 Mpps | 6.5 Mpps | -6% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv6 FIB 2M | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc | 5.3 Mpps | 4.2 Mpps | -21% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv6 iAcl | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc | 6.5 Mpps | 6.1..6.5 Mpps | -6..0% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-Partial drop rate search, LT = 0.5%:
+PDR Throughput
+~~~~~~~~~~~~~~
+
+Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Improvement |
+===================+=================================================================+===========+=================+======================+
| L2XC | 10ge2p1x520: 64B-1t1c-eth-l2xcbase-pdrdisc | 9.4 Mpps | 12.7..12.9 Mpps | 35..37% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| L2XC | 10ge2p1xl710: 64B-1t1c-eth-l2xcbase-pdrdisc | no data | no data | |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| L2XC dot1ad | 10ge2p1x520: 64B-1t1c-dot1ad-l2xcbase-pdrdisc | 7.4 Mpps | 8.8..9.1 Mpps | 19..23% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| L2XC dot1q | 10ge2p1x520: 64B-1t1c-dot1q-l2xcbase-pdrdisc | 7.5 Mpps | 8.8..9.0 Mpps | 17..20% |
@@ -173,6 +100,64 @@ Partial drop rate search, LT = 0.5%: +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv4 COP | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-copwhtlistbase-pdrdisc | 7.1 Mpps | 8.3..8.5 Mpps | 17..20% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv4 vhost | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc | 2.0 Mpps | 2.7 Mpps | 35% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+
+Measured improvements are in line with VPP code optimizations listed in
+`VPP-17.01 release notes
+<https://docs.fd.io/vpp/17.01/release_notes_1701.html>`_.
+
+Additionally, vhost-VM performance improvements are due to both VPP code
+optimizations as well as due to the FD.io CSIT Linux KVM test environment
+optimizations for vhost-VM tests - see section "2.1.7. Methodology: KVM VM
+vhost".
+
+
+Other Performance Changes
+-------------------------
+
+Other changes in measured packet throughput, with either minor relative
+increase or decrease, have been observed in a number of CSIT |release| tests
+listed below. Relative changes are calculated against the test results in CSIT
+rls1609 report.
+
+NDR Throughput
+~~~~~~~~~~~~~~
+
+Non-Drop Rate Throughput discovery tests:
+
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Change |
++===================+=================================================================+===========+=================+======================+
+| IPv4 FIB 200k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-ndrdisc | 8.5 Mpps | 9.0 Mpps | 6% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv4 FIB 20k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-ndrdisc | 8.5 Mpps | 9.0..9.2 Mpps | 6..8% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv4 FIB 2M | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale2m-ndrdisc | 8.5 Mpps | 7.8..8.1 Mpps | -8..-5% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv4 Policer | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-ipolicemarkbase-ndrdisc | 6.9 Mpps | 7.4..7.6 Mpps | 7..10% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv4 LISP | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-ndrdisc | 4.4 Mpps | 4.8 Mpps | 9% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv6 COP | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-ndrdisc | 6.1 Mpps | 6.1..6.5 Mpps | 0..7% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv6 FIB 200k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale200k-ndrdisc | 6.5 Mpps | 5.3..5.7 Mpps | -18..-12% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv6 FIB 20k | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale20k-ndrdisc | 6.9 Mpps | 6.5 Mpps | -6% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv6 FIB 2M | 10ge2p1x520: 78B-1t1c-ethip6-ip6scale2m-ndrdisc | 5.3 Mpps | 4.2 Mpps | -21% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| IPv6 iAcl | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-ndrdisc | 6.5 Mpps | 6.1..6.5 Mpps | -6..0% |
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+
+PDR Throughput
+~~~~~~~~~~~~~~
+
+Partial Drop Rate thoughput discovery tests with packet Loss Tolerance of 0.5%:
+
++-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
+| VPP Functionality | Test Name | VPP-16.09 | VPP-17.01 | Relative Change |
++===================+=================================================================+===========+=================+======================+
| IPv4 FIB 200k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale200k-pdrdisc | 8.5 Mpps | 9.0 Mpps | 6% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv4 FIB 20k | 10ge2p1x520: 64B-1t1c-ethip4-ip4scale20k-pdrdisc | 8.5 Mpps | 9.0..9.2 Mpps | 6..8% |
@@ -185,8 +170,6 @@ Partial drop rate search, LT = 0.5%: +-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv4 LISP | 10ge2p1x520: 64B-1t1c-ethip4lispip4-ip4base-pdrdisc | 4.6 Mpps | 4.8 Mpps | 9% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-| IPv4 vhost | 10ge2p1x520: 64B-1t1c-ethip4-ip4base-eth-2vhost-1vm-pdrdisc | 2.0 Mpps | 2.7 Mpps | 35% |
-+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv6 | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-pdrdisc | 7.7 Mpps | 7.3..7.7 Mpps | -5..0% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
| IPv6 COP | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-copwhtlistbase-pdrdisc | 6.1 Mpps | 6.1..6.5 Mpps | 0..7% |
@@ -200,107 +183,33 @@ Partial drop rate search, LT = 0.5%: | IPv6 iAcl | 10ge2p1x520: 78B-1t1c-ethip6-ip6base-iacldstbase-pdrdisc | 6.5 Mpps | 6.1..6.5 Mpps | -6..0% |
+-------------------+-----------------------------------------------------------------+-----------+-----------------+----------------------+
-This is in line with VPP code optimizations listed in `VPP-17.01 release notes
-<https://docs.fd.io/vpp/17.01/release_notes_1701.html>`_.
-
-vhost-VM performance improvement is also due to Linux KVM test environment
-optimization for vhost-VM tests - see section below "VM vhost-user
-Throughput Measurements".
-
-Multi-Thread and Multi-Core Measurements
-----------------------------------------
-
-**HyperThreading** - CSIT |release| performance tests are executed with SUT
-servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS
-settings). This is the simplest configuration used to establish baseline
-single-thread single-core SW packet processing and forwarding performance.
-Subsequent releases of CSIT will add performance tests with Intel
-HyperThreading Enabled (requires BIOS settings change and hard reboot).
-
-**Multi-core Test** - CSIT |release| multi-core tests are executed in the
-following VPP thread and core configurations:
-
-#. 1t1c - 1 VPP worker thread on 1 CPU physical core.
-#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.
-#. 4t4c - 4 VPP threads on 4 CPU physical cores.
-
-Note that in quite a few test cases running VPP on 2 or 4 physical cores hits
-the tested NIC I/O bandwidth or packets-per-second limit.
-
-Packet Throughput Measurements
-------------------------------
-
-Following values are measured and reported for packet throughput tests:
-
-- NDR binary search per RFC2544:
-
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
- (2x <per direction packets-per-second>)"
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
- second> Gbps (untagged)"
-
-- PDR binary search per RFC2544:
-
- - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
- <per direction packets-per-second>)"
- - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
- second> Gbps (untagged)"
- - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
- lost at PDR rate>""
-
-- NDR and PDR are measured for the following L2 frame sizes:
-
- - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.
- - IPv6: 78B, 1518B, 9000B.
-
-
-Packet Latency Measurements
----------------------------
-
-TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported
-latency values are measured using following methodology:
-
-- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
- for each NDR throughput test and packet size (except IMIX).
-- TG sends dedicated latency streams, one per direction, each at the rate of
- 10kpps at the prescribed packet size; these are sent in addition to the main
- load streams.
-- TG reports min/avg/max latency values per stream direction, hence two sets
- of latency values are reported per test case; future release of TRex is
- expected to report latency percentiles.
-- Reported latency values are aggregate across two SUTs due to three node
- topology used for all performance tests; for per SUT latency, reported value
- should be divided by two.
-- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
- FD.io labs used by CSIT project.
-- TRex setup introduces an always-on error of about 2*2usec per latency flow -
- additonal Tx/Rx interface latency induced by TRex SW writing and reading
- packet timestamps on CPU cores without HW acceleration on NICs closer to the
- interface line.
-
-
-KVM VM vhost Measurements
--------------------------
-
-CSIT |release| introduced environment configuration changes to KVM Qemu vhost-
-user tests in order to more representatively measure VPP-17.01 performance in
-configurations with vhost-user interfaces and VMs.
-
-Current setup of CSIT FD.io performance lab is using tuned settings for more
-optimal performance of KVM Qemu:
+Known Issues
+------------
-- Default Qemu virtio queue size of 256 descriptors.
-- Adjusted Linux kernel CFS scheduler settings, as detailed on this CSIT wiki
- page: https://wiki.fd.io/view/CSIT/VM-vhost-env-tuning.
+Here is the list of known issues in CSIT |release| for VPP performance tests:
-Adjusted Linux kernel CFS settings make the NDR and PDR throughput performance
-of VPP+VM system less sensitive to other Linux OS system tasks by reducing
-their interference on CPU cores that are designated for critical software
-tasks under test, namely VPP worker threads in host and Testpmd threads in
-guest dealing with data plan.
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| # | Issue | Description |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 1 | Sporadic IPv4 routed-forwarding NDR discovery | Suspected issue with DPDK IPv4 checksum calculation. |
+| | test failures for 1518B frame size | Observed frequency: sporadic, ca. 20% to 30% of test runs. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 2 | Vic1385 and Vic1227 NICs jumbo frames test | Suspected issue with VIC DPDK drivers that do not support jumbo |
+| | failures (9000B) | frames (dropped rx-miss). Observed frequency: 100%. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 3 | Vic1385 and Vic1227 performance | Low performance of NDR results. Big difference between NDR and |
+| | | PDR. . |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 4 | Sporadic NDR discovery test failures on x520 | Suspected issue with HW settings (BIOS, FW) in LF |
+| | | infrastructure. Issue can't be replicated outside LF. |
++---+-------------------------------------------------+-----------------------------------------------------------------+
+| 5 | Non-repeatible zig-zagging NDR throughput | Suspected NIC driver issue in multi-thread multi-core operation |
+| | results for multi-thread multi-core tests | affectin NDR throughput. Applies to XL710 and X710 NICs tested. |
+| | - 2t2c, 4t4c - for some tested NICs. | |
++---+-------------------------------------------------+-----------------------------------------------------------------+
-Report Addendum Tests - Cryptodev
----------------------------------
+Tests to be Added - Cryptodev
+-----------------------------
DPDK Cryptodev functionality support for both SW and HW crypto devices has
been introduced in VPP-17.01 release. CSIT functional and performance tests
@@ -309,36 +218,11 @@ project control execution of those tests within the LF FD.io test environment still need to complete. Once the results become available, they will be
published as an addendum to the current version of CSIT |release| report.
-Report Addendum Tests - Centos
-------------------------------
-
-CSIT |release| added Centos functional test execution environment in FD.io
-VIRL testbeds.However due to the factors beyond CSIT project control execution
-of those tests within the LF FD.io test environment still need to complete.
-Once the results become available, they will be published as an addendum to
-the current version of CSIT |release| report.
-
-Report Addendum Tests - SNAT44
-------------------------------
+Tests to be Added - SNAT44
+--------------------------
VPP SNAT44 functionality has been introduced in VPP-17.01 release. CSIT
performance tests are still in development and integration into LFD FD.io test
environment. Once the tests are fully integrated and results become available,
they will be published as an addendum to the current version of CSIT |release|
report.
-
-Known Issues
-------------
-
-Here is the list of known issues in CSIT |release| for VPP performance tests in physical HW testbed:
-
-+---+-------------------------------------------------+-----------------------------------------------------------------+
-| # | Issue | Description |
-+---+-------------------------------------------------+-----------------------------------------------------------------+
-| 1 | Sporadic IPv4 routed-forwwarding NDR discovery | Suspected issue with DPDK IPv4 checksum calculation, VPP jira # |
-| | test failures for 1518B frame size | Observed frequency: sporadic, ca. 20% to 30% of test runs |
-+---+-------------------------------------------------+-----------------------------------------------------------------+
-| | | |
-+---+-------------------------------------------------+-----------------------------------------------------------------+
-| | | |
-+---+-------------------------------------------------+-----------------------------------------------------------------+
diff --git a/docs/report/vpp_performance_tests/documentation.rst b/docs/report/vpp_performance_tests/documentation.rst new file mode 100644 index 0000000000..5017580ca0 --- /dev/null +++ b/docs/report/vpp_performance_tests/documentation.rst @@ -0,0 +1,6 @@ +Documentation
+=============
+
+`CSIT VPP Performance Tests Documentation
+<https://docs.fd.io/csit/master/doc/tests.perf.html>`_ contains detailed
+functional description and input parameters for each test case.
diff --git a/docs/report/vpp_performance_tests_hw/index.rst b/docs/report/vpp_performance_tests/index.rst index 5ffb5d8837..bf58d8e92b 100644 --- a/docs/report/vpp_performance_tests_hw/index.rst +++ b/docs/report/vpp_performance_tests/index.rst @@ -1,5 +1,5 @@ -VPP Performance Tests HW -======================== +VPP Performance Tests +===================== .. toctree:: diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst new file mode 100644 index 0000000000..d0ce5e7bcf --- /dev/null +++ b/docs/report/vpp_performance_tests/overview.rst @@ -0,0 +1,325 @@ +Overview
+========
+
+Tested Physical Topologies
+--------------------------
+
+CSIT VPP performance tests are executed on physical baremetal servers hosted by LF
+FD.io project. Testbed physical topology is shown in the figure below.
+
+::
+
+ +------------------------+ +------------------------+
+ | | | |
+ | +------------------+ | | +------------------+ |
+ | | | | | | | |
+ | | <-----------------> | |
+ | | DUT1 | | | | DUT2 | |
+ | +--^---------------+ | | +---------------^--+ |
+ | | | | | |
+ | | SUT1 | | SUT2 | |
+ +------------------------+ +------------------^-----+
+ | |
+ | |
+ | +-----------+ |
+ | | | |
+ +------------------> TG <------------------+
+ | |
+ +-----------+
+
+SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two
+Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with
+two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a
+Device Under Test (DUT). TG runs TRex SW application as a packet Traffic
+Generator. Physical connectivity between SUTs and to TG is provided using
+different NIC models that need to be tested for performance. Currently
+installed and tested NIC models include:
+
+#. 2port10GE X520-DA2 Intel.
+#. 2port10GE X710 Intel.
+#. 2port10GE VIC1227 Cisco.
+#. 2port40GE VIC1385 Cisco.
+#. 2port40GE XL710 Intel.
+
+From SUT and DUT perspective, all performance tests involve forwarding packets
+between two physical Ethernet ports (10GE or 40GE). Due to the number of
+listed NIC models tested and available PCI slot capacity in SUT servers, in
+all of the above cases both physical ports are located on the same NIC. In
+some test cases this results in measured packet throughput being limited not
+by VPP DUT but by either the physical interface or the NIC capacity.
+
+Going forward CSIT project will be looking to add more hardware into FD.io
+performance labs to address larger scale multi-interface and multi-NIC
+performance testing scenarios.
+
+For test cases that require DUT (VPP) to communicate with VM over vhost-user
+interfaces, a VM is created on SUT1 and SUT2. DUT (VPP) test topology with VM
+is shown in the figure below including applicable packet flow thru the VM
+(marked in the figure with ``***``).
+
+::
+
+ +------------------------+ +------------------------+
+ | +----------+ | | +----------+ |
+ | | VM | | | | VM | |
+ | | ****** | | | | ****** | |
+ | +--^----^--+ | | +--^----^--+ |
+ | *| |* | | *| |* |
+ | +------v----v------+ | | +------v----v------+ |
+ | | * * |**|***********|**| * * | |
+ | | ***** *******<----------------->******* ***** | |
+ | | * DUT1 | | | | DUT2 * | |
+ | +--^---------------+ | | +---------------^--+ |
+ | *| | | |* |
+ | *| SUT1 | | SUT2 |* |
+ +------------------------+ +------------------^-----+
+ *| |*
+ *| |*
+ *| +-----------+ |*
+ *| | | |*
+ *+------------------> TG <------------------+*
+ ******************* | |********************
+ +-----------+
+
+For VM tests, packets are switched by DUT (VPP) twice, hence the
+throughput rates measured by TG (and listed in this report) must be multiplied
+by two to represent the actual DUT aggregate packet forwarding rate.
+
+Note that reported VPP performance results are specific to the SUT tested.
+Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with
+other CPUs are likely to yield different results. A good rule of thumb, that
+can be applied to estimate VPP packet thoughput for Phy-to-Phy (NIC-to-NIC,
+PCI-to-PCI) topology, is to expect the forwarding performance to be
+proportional to CPU core frequency, assuming CPU is the only limiting factor
+and all other SUT aspects equal to FD.io CSIT environment. The same rule of
+thumb can be also applied for Phy-to-VM-to-Phy (NIC-to-VM-to-NIC) topology,
+but due to much higher dependency on very high frequency memory operations and
+sensitivity to Linux kernel scheduler settings and behaviour, this estimation
+may not always yield good enough accuracy.
+
+Detailed LF FD.io test bed specification and physical topology are described
+in `wiki CSIT LF FDio testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
+
+Performance Tests Coverage
+--------------------------
+
+Performance tests are split into the two main categories:
+
+- Throughput discovery - discovery of packet forwarding rate using binary search
+ in accordance to RFC2544.
+
+ - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss;
+ followed by packet one-way latency measurements at 10%, 50% and 100% of
+ discovered NDR throughput.
+ - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss
+ currently set to 0.5%; followed by packet one-way latency measurements at
+ 100% of discovered PDR throughput.
+
+- Throughput verification - verification of packet forwarding rate against
+ previously discovered throughput rate. These tests are currently done against
+ 0.9 of reference NDR, with reference rates updated periodically.
+
+CSIT |release| includes following performance test suites, listed per NIC type:
+
+- 2port10GE X520-DA2 Intel
+
+ - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad
+ VLAN tagged Ethernet frames.
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning; disabled MAC learning i.e. static MAC tests to be added.
+ - **IPv4** - IPv4 routed-forwarding.
+ - **IPv6** - IPv6 routed-forwarding.
+ - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries.
+ - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries.
+ - **VM with vhost-user** - switching between NIC ports and VM over vhost-user
+ interfaces in different switching modes incl. L2 Cross-Connect, L2
+ Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.
+ - **COP** - IPv4 and IPv6 routed-forwarding with COP address security.
+ - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security.
+ - **LISP** - LISP overlay tunneling for IPv4-over-IPV4, IPv6-over-IPv4,
+ IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes.
+ - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.
+ - **QoS Policer** - ingress packet rate measuring, marking and limiting
+ (IPv4).
+
+- 2port40GE XL710 Intel
+
+ - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames.
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+ - **IPv4** - IPv4 routed-forwarding.
+ - **IPv6** - IPv6 routed-forwarding.
+ - **VM with vhost-user** - switching between NIC ports and VM over vhost-user
+ interfaces in different switching modes incl. L2 Bridge-Domain.
+
+- 2port10GE X710 Intel
+
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+ - **VM with vhost-user** - switching between NIC ports and VM over vhost-user
+ interfaces in different switching modes incl. L2 Bridge-Domain.
+
+- 2port10GE VIC1227 Cisco
+
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+
+- 2port40GE VIC1385 Cisco
+
+ - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
+ with MAC learning.
+
+Execution of performance tests takes time, especially the throughput discovery
+tests. Due to limited HW testbed resources available within FD.io labs hosted
+by Linux Foundation, the number of tests for NICs other than X520 (a.k.a.
+Niantic) has been limited to few baseline tests. Over time we expect the HW
+testbed resources to grow, and will be adding complete set of performance
+tests for all models of hardware to be executed regularly and(or)
+continuously.
+
+Performance Tests Naming
+------------------------
+
+CSIT |release| introduced a common structured naming convention for all
+performance and functional tests. This change was driven by substantially
+growing number and type of CSIT test cases. Firstly, the original practice did
+not always follow any strict naming convention. Secondly test names did not
+always clearly capture tested packet encapsulations, and the actual type or
+content of the tests. Thirdly HW configurations in terms of NICs, ports and
+their locality were not captured either. These were but few reasons that drove
+the decision to change and define a new more complete and stricter test naming
+convention, and to apply this to all existing and new test cases.
+
+The new naming should be intuitive for majority of the tests. The complete
+description of CSIT test naming convention is provided on `CSIT test naming wiki
+<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
+
+Here few illustrative examples of the new naming usage for performance test
+suites:
+
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
+
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
+ with MAC learning, NDR throughput discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
+ on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
+ switching with MAC learning, NDR throughput discovery.
+ - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
+ x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
+ - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
+ discovery.
+
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**
+
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
+ VirtPortConfig-VMconfig-TestType*
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
+ of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput
+ discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput
+ discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from four vhost interfaces and two VMs, NDR throughput
+ discovery.
+
+Methodology: Multi-Thread and Multi-Core
+----------------------------------------
+
+**HyperThreading** - CSIT |release| performance tests are executed with SUT
+servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS
+settings). This is the simplest configuration used to establish baseline
+single-thread single-core SW packet processing and forwarding performance.
+Subsequent releases of CSIT will add performance tests with Intel
+HyperThreading Enabled (requires BIOS settings change and hard reboot).
+
+**Multi-core Test** - CSIT |release| multi-core tests are executed in the
+following VPP thread and core configurations:
+
+#. 1t1c - 1 VPP worker thread on 1 CPU physical core.
+#. 2t2c - 2 VPP worker threads on 2 CPU physical cores.
+#. 4t4c - 4 VPP threads on 4 CPU physical cores.
+
+Note that in quite a few test cases running VPP on 2 or 4 physical cores hits
+the tested NIC I/O bandwidth or packets-per-second limit.
+
+Methodology: Packet Throughput
+------------------------------
+
+Following values are measured and reported for packet throughput tests:
+
+- NDR binary search per RFC2544:
+
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps
+ (2x <per direction packets-per-second>)"
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+ second> Gbps (untagged)"
+
+- PDR binary search per RFC2544:
+
+ - Packet rate: "RATE: <aggregate packet rate in packets-per-second> pps (2x
+ <per direction packets-per-second>)"
+ - Aggregate bandwidth: "BANDWIDTH: <aggregate bandwidth in Gigabits per
+ second> Gbps (untagged)"
+ - Packet loss tolerance: "LOSS_ACCEPTANCE <accepted percentage of packets
+ lost at PDR rate>""
+
+- NDR and PDR are measured for the following L2 frame sizes:
+
+ - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B.
+ - IPv6: 78B, 1518B, 9000B.
+
+
+Methodology: Packet Latency
+---------------------------
+
+TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported
+latency values are measured using following methodology:
+
+- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate)
+ for each NDR throughput test and packet size (except IMIX).
+- TG sends dedicated latency streams, one per direction, each at the rate of
+ 10kpps at the prescribed packet size; these are sent in addition to the main
+ load streams.
+- TG reports min/avg/max latency values per stream direction, hence two sets
+ of latency values are reported per test case; future release of TRex is
+ expected to report latency percentiles.
+- Reported latency values are aggregate across two SUTs due to three node
+ topology used for all performance tests; for per SUT latency, reported value
+ should be divided by two.
+- 1usec is the measurement accuracy advertised by TRex TG for the setup used in
+ FD.io labs used by CSIT project.
+- TRex setup introduces an always-on error of about 2*2usec per latency flow -
+ additonal Tx/Rx interface latency induced by TRex SW writing and reading
+ packet timestamps on CPU cores without HW acceleration on NICs closer to the
+ interface line.
+
+
+Methodology: KVM VM vhost
+-------------------------
+
+CSIT |release| introduced environment configuration changes to KVM Qemu vhost-
+user tests in order to more representatively measure VPP-17.01 performance in
+configurations with vhost-user interfaces and VMs.
+
+Current setup of CSIT FD.io performance lab is using tuned settings for more
+optimal performance of KVM Qemu:
+
+- Default Qemu virtio queue size of 256 descriptors.
+- Adjusted Linux kernel CFS scheduler settings, as detailed on this CSIT wiki
+ page: https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604.
+
+Adjusted Linux kernel CFS settings make the NDR and PDR throughput performance
+of VPP+VM system less sensitive to other Linux OS system tasks by reducing
+their interference on CPU cores that are designated for critical software
+tasks under test, namely VPP worker threads in host and Testpmd threads in
+guest dealing with data plan.
diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/index.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/index.rst index 8c3e941d41..8c3e941d41 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/index.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/index.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv4.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv4.rst index c46f0141a2..c46f0141a2 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv4.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv4.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv4_tunnels.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv4_tunnels.rst index d3293cc6aa..d3293cc6aa 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv4_tunnels.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv4_tunnels.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv6.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv6.rst index 38bcf71de6..38bcf71de6 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv6.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv6.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv6_tunnels.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv6_tunnels.rst index 94bf9a811f..94bf9a811f 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/ipv6_tunnels.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/ipv6_tunnels.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/l2.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst index a487306dad..a487306dad 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/l2.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/l2.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/vm_vhost.rst b/docs/report/vpp_performance_tests/packet_latency_graphs/vm_vhost.rst index 6421db1c59..6421db1c59 100644 --- a/docs/report/vpp_performance_tests_hw/packet_latency_graphs/vm_vhost.rst +++ b/docs/report/vpp_performance_tests/packet_latency_graphs/vm_vhost.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/index.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst index bf2a87fab7..bf2a87fab7 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/index.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/index.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv4.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv4.rst index c413adb51e..c413adb51e 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv4.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv4.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv4_tunnels.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv4_tunnels.rst index 23ac3fc001..23ac3fc001 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv4_tunnels.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv4_tunnels.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv6.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv6.rst index e04f2c6822..e04f2c6822 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv6.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv6.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv6_tunnels.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv6_tunnels.rst index 1a57998018..1a57998018 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/ipv6_tunnels.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/ipv6_tunnels.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/l2.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/l2.rst index 89429a6192..89429a6192 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/l2.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/l2.rst diff --git a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/vm_vhost.rst b/docs/report/vpp_performance_tests/packet_throughput_graphs/vm_vhost.rst index b5bc0dd5a3..b5bc0dd5a3 100644 --- a/docs/report/vpp_performance_tests_hw/packet_throughput_graphs/vm_vhost.rst +++ b/docs/report/vpp_performance_tests/packet_throughput_graphs/vm_vhost.rst diff --git a/docs/report/vpp_performance_tests_hw/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst index 0fd22213b9..0fd22213b9 100644 --- a/docs/report/vpp_performance_tests_hw/test_environment.rst +++ b/docs/report/vpp_performance_tests/test_environment.rst diff --git a/docs/report/vpp_performance_tests_hw/documentation.rst b/docs/report/vpp_performance_tests_hw/documentation.rst deleted file mode 100644 index 1bc1bcbcc9..0000000000 --- a/docs/report/vpp_performance_tests_hw/documentation.rst +++ /dev/null @@ -1,4 +0,0 @@ -Documentation
-=============
-
-Documentation is located on nexus repository https://docs.fd.io/csit/master/doc/tests.perf.html.
diff --git a/docs/report/vpp_performance_tests_hw/overview.rst b/docs/report/vpp_performance_tests_hw/overview.rst deleted file mode 100644 index e1afabf1d8..0000000000 --- a/docs/report/vpp_performance_tests_hw/overview.rst +++ /dev/null @@ -1,156 +0,0 @@ -Overview
-========
-
-Tested Topologies HW
---------------------
-
-CSIT VPP performance tests are executed on physical baremetal servers hosted by LF
-FD.io project. Testbed physical topology is shown in the figure below.
-
-::
-
- +------------------------+ +------------------------+
- | | | |
- | +------------------+ | | +------------------+ |
- | | | | | | | |
- | | <-----------------> | |
- | | DUT1 | | | | DUT2 | |
- | +--^---------------+ | | +---------------^--+ |
- | | | | | |
- | | SUT1 | | SUT2 | |
- +------------------------+ +------------------^-----+
- | |
- | |
- | +-----------+ |
- | | | |
- +------------------> TG <------------------+
- | |
- +-----------+
-
-SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two
-Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with
-two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a
-Device Under Test (DUT). TG runs TRex SW application as a packet Traffic
-Generator. Physical connectivity between SUTs and to TG is provided using
-different NIC models that need to be tested for performance. Currently
-installed and tested NIC models include:
-
-#. 2port10GE X520-DA2 Intel.
-#. 2port10GE X710 Intel.
-#. 2port10GE VIC1227 Cisco.
-#. 2port40GE VIC1385 Cisco.
-#. 2port40GE XL710 Intel.
-
-Detailed LF FD.io test bed specification and physical topology are described
-in `wiki CSIT LF testbed <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
-
-For test cases that require DUT (VPP) to communicate with VM over vhost-user
-interfaces, a VM is created on SUT1 and SUT2. DUT (VPP) test topology with VM
-is shown in the figure below including applicable packet flow thru the VM
-(marked in the figure with ``***``).
-
-::
-
- +------------------------+ +------------------------+
- | +----------+ | | +----------+ |
- | | VM | | | | VM | |
- | | ****** | | | | ****** | |
- | +--^----^--+ | | +--^----^--+ |
- | *| |* | | *| |* |
- | +------v----v------+ | | +------v----v------+ |
- | | * * |**|***********|**| * * | |
- | | ***** *******<----------------->******* ***** | |
- | | * DUT1 | | | | DUT2 * | |
- | +--^---------------+ | | +---------------^--+ |
- | *| | | |* |
- | *| SUT1 | | SUT2 |* |
- +------------------------+ +------------------^-----+
- *| |*
- *| |*
- *| +-----------+ |*
- *| | | |*
- *+------------------> TG <------------------+*
- ******************* | |********************
- +-----------+
-
-Note that for VM tests, packets are switched by DUT (VPP) twice, hence the
-throughput rates measured by TG (and listed in this report) must be multiplied
-by two to represent the actual DUT aggregate packet forwarding rate.
-
-VPP Performance Tests Overview
-------------------------------
-
-Performance tests are split into the two main categories:
-
-- Throughput discovery - discovery of packet forwarding rate using binary search
- in accordance to RFC2544.
-
- - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss;
- followed by packet one-way latency measurements at 10%, 50% and 100% of
- discovered NDR throughput.
- - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss
- currently set to 0.5%; followed by packet one-way latency measurements at
- 100% of discovered PDR throughput.
-
-- Throughput verification - verification of packet forwarding rate against
- previously discovered throughput rate. These tests are currently done against
- 0.9 of reference NDR, with reference rates updated periodically.
-
-CSIT |release| includes following performance test suites, listed per NIC type:
-
-- 2port10GE X520-DA2 Intel
-
- - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad
- VLAN tagged Ethernet frames.
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
- with MAC learning; disabled MAC learning i.e. static MAC tests to be added.
- - **IPv4** - IPv4 routed-forwarding.
- - **IPv6** - IPv6 routed-forwarding.
- - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries.
- - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries.
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user
- interfaces in different switching modes incl. L2 Cross-Connect, L2
- Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding.
- - **COP** - IPv4 and IPv6 routed-forwarding with COP address security.
- - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security.
- - **LISP** - LISP overlay tunneling for IPv4-over-IPV4, IPv6-over-IPv4,
- IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes.
- - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD.
- - **QoS Policer** - ingress packet rate measuring, marking and limiting
- (IPv4).
-
-- 2port40GE XL710 Intel
-
- - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames.
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
- with MAC learning.
- - **IPv4** - IPv4 routed-forwarding.
- - **IPv6** - IPv6 routed-forwarding.
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user
- interfaces in different switching modes incl. L2 Bridge-Domain.
-
-- 2port10GE X710 Intel
-
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
- with MAC learning.
- - **VM with vhost-user** - switching between NIC ports and VM over vhost-user
- interfaces in different switching modes incl. L2 Bridge-Domain.
-
-- 2port10GE VIC1227 Cisco
-
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
- with MAC learning.
-
-- 2port40GE VIC1385 Cisco
-
- - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames
- with MAC learning.
-
-Execution of performance tests takes time, especially the throughput discovery
-tests. Due to limited HW testbed resources available within FD.io labs hosted
-by Linux Foundation, the number of tests for NICs other than X520 (a.k.a.
-Niantic) has been limited to few baseline tests. Over time we expect the HW
-testbed resources to grow, and will be adding complete set of performance
-tests for all models of hardware to be executed regularly and(or)
-continuously.
-
diff --git a/docs/report/vpp_unit_tests/documentation.rst b/docs/report/vpp_unit_tests/documentation.rst new file mode 100644 index 0000000000..dc10704725 --- /dev/null +++ b/docs/report/vpp_unit_tests/documentation.rst @@ -0,0 +1,6 @@ +Documentation
+=============
+
+For complete description of the VPP Test Framework including anatomy of a test
+case and detailed documentation of existing VPP unit test cases please refer
+to the `VPP Unit Test Documentation <link>`_
diff --git a/docs/report/vpp_unit_tests/index.rst b/docs/report/vpp_unit_tests/index.rst new file mode 100644 index 0000000000..525eb1230d --- /dev/null +++ b/docs/report/vpp_unit_tests/index.rst @@ -0,0 +1,8 @@ +VPP Unit Tests +============== + +.. toctree:: + + overview + documentation + diff --git a/docs/report/vpp_unit_tests/overview.rst b/docs/report/vpp_unit_tests/overview.rst new file mode 100644 index 0000000000..7c8210bb7f --- /dev/null +++ b/docs/report/vpp_unit_tests/overview.rst @@ -0,0 +1,61 @@ +Overview
+========
+
+.. note::
+
+ This section includes an abbreviated version of the VPP Test Framework
+ overview maintained within the VPP project. Complete overview can be found
+ in `VPP unit test documentation <link>`_
+
+VPP Unit Test Framework
+-----------------------
+
+VPP Test Framework is used to ease writing, running and debugging unit tests
+for the VPP. It is based on python as a high level language to allow rapid
+test development. scapy\_ is used as a tool for creating and dissecting
+packets.
+
+VPP Test Framework does not send any packets to VPP directly. Traffic is
+instead injected using VPP packet-generator interfaces. Packets are written
+into a temporary .pcap file, which is then read by the VPP code with packets
+getting injected into the VPP processing nodes.
+
+Similarly, VPP does not send any packets to VPP Test Framework directly.
+Instead, VPP packet capture feature is used to capture and write packets to a
+temporary .pcap file, which is then read and analyzed by the VPP Test
+Framework.
+
+For complete description of the VPP Test Framework including anatomy of a test
+case and detailed documentation of existing VPP unit test cases please refer
+to the `VPP unit test documentation <link>`_
+
+Unit Tests Coverage
+-------------------
+
+Following VPP functional test areas are covered in VPP unit test code included
+in VPP rls1701 with results listed in this report:
+
+- CRUD Loopback - create, read, update, delete Loopback interfaces.
+- Flow-per-packet plugin - collect and report L2 and IP4 flow statistics.
+- DHCP - DHCPv4 and DHCPv6 Proxy.
+- IP4 VRF Multi-instance - create, read, update, delete and verify IPv4 VRFs.
+- Load Balancer - IP4 GRE4, IP4 GRE6, IP6 GRE4, IP6 GRE6
+- Bidirectional Forwarding Detection (BFD) IPv4 - baseline BFD session operation.
+- Bidirectional Forwarding Detection (BFD) IPv6 - baseline BFD session operation.
+- Bidirectional Forwarding Detection (BFD) - API tests.
+- Bidirectional Forwarding Detection (BFD) - changing authorization.
+- Bidirectional Forwarding Detection (BFD) - SHA1 authentication.
+- IPv6 Tests - baseline FIB operations, NS/RS exception handling.
+- SPAN Test - Switched Port Analyzer packet mirroring.
+- GRE Tests - GRE tunnel tests.
+- SNAT Test Cases - SNAT44 tests.
+- L2XC Multi-instance - L2 cross-connect multi-instance tests.
+- IPv4 Tests - baseline FIB tests.
+- IPv4 FIB CRUD - add/update/delete IPv4 routes.
+- IRB Tests - Integrated Routing and Bridging tests.
+- MPLS Tests - MPLS baseline tests.
+- L2XC Tests - L2 cross-connect baseline tests incl. single- and dual-loop.
+- Classifier - classification with IP ACL, MAC ACL, IP PBR.
+- VXLAN Tests - baseline VXLAN tunneling.
+- L2 FIB CRUD - add/update/delete L2 MAC entries.
+- L2BD Tests - L2 Bridge-Domain baseline tests incl. single- and dual-loop.
diff --git a/resources/tools/report_gen/run_report.sh b/resources/tools/report_gen/run_report.sh index 2a88750464..2400b6bd24 100755 --- a/resources/tools/report_gen/run_report.sh +++ b/resources/tools/report_gen/run_report.sh @@ -64,34 +64,34 @@ JEN_URL='https://jenkins.fd.io/view/csit/job' JEN_FILE_PERF='output_perf_data.xml' JEN_JOB='csit-vpp-perf-1701-all' -JEN_BUILD=(3 4 7) +JEN_BUILD=(3 4 7 8) for i in "${JEN_BUILD[@]}"; do - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -O ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -o ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip done JEN_JOB='csit-vpp-perf-1701-long' -JEN_BUILD=(2 4) +JEN_BUILD=(2 4 5 6) for i in "${JEN_BUILD[@]}"; do - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -O ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -o ${STATIC_DIR_VPP}/${JEN_JOB}-${i}.xml + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip done JEN_JOB='csit-dpdk-perf-1701-all' JEN_BUILD=(2 3) for i in "${JEN_BUILD[@]}"; do - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -O ${STATIC_DIR_TESTPMD}/${JEN_JOB}-${i}.xml - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/${JEN_FILE_PERF} -o ${STATIC_DIR_TESTPMD}/${JEN_JOB}-${i}.xml + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip done JEN_JOB='csit-vpp-functional-1701-virl' JEN_BUILD=(18) for i in "${JEN_BUILD[@]}"; do - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip done JEN_URL='https://jenkins.fd.io/view/hc2vpp/job' @@ -99,7 +99,7 @@ JEN_JOB='hc2vpp-csit-integration-1701-ubuntu1404' JEN_BUILD=(1) for i in "${JEN_BUILD[@]}"; do - wget -q ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -O ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip + curl -fs ${JEN_URL}/${JEN_JOB}/${i}/artifact/\*zip\*/archive.zip -o ${STATIC_DIR_ARCH}/${JEN_JOB}-${i}.zip done |