diff options
7 files changed, 104 insertions, 132 deletions
diff --git a/docs/report/dmm_functional_tests/csit_release_notes.rst b/docs/report/dmm_functional_tests/csit_release_notes.rst index 352809c038..1987e218a5 100644 --- a/docs/report/dmm_functional_tests/csit_release_notes.rst +++ b/docs/report/dmm_functional_tests/csit_release_notes.rst @@ -6,7 +6,7 @@ Changes in |csit-release| #. **DMM Functional Tests** - - Added DMM base vs. epoll test case. + - Added DMM base vs. epoll test case. Known Issues ------------ diff --git a/docs/report/dmm_functional_tests/overview.rst b/docs/report/dmm_functional_tests/overview.rst index aa9e853817..562fa09eb7 100644 --- a/docs/report/dmm_functional_tests/overview.rst +++ b/docs/report/dmm_functional_tests/overview.rst @@ -49,6 +49,5 @@ environment: | | testcase demonstrates single server[DUT1] | | | and single client[DUT2] scenario using DMM | | | framework and kernel tcp/ip stack. | -| | | | | Test case count: 1. | +-----------------------+----------------------------------------------+ diff --git a/docs/report/dpdk_performance_tests/csit_release_notes.rst b/docs/report/dpdk_performance_tests/csit_release_notes.rst index d84b4e259a..bd89657135 100644 --- a/docs/report/dpdk_performance_tests/csit_release_notes.rst +++ b/docs/report/dpdk_performance_tests/csit_release_notes.rst @@ -6,33 +6,33 @@ Changes in |csit-release| #. **DPDK performance tests** - - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now - configurable trial duration and number of consecutive executions. - Coverage of MRR tests has been extended across more test - scenarios. MRR tests are used for continuous performance trending - and for comparison between DPDK releases. + - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now + configurable trial duration and number of consecutive executions. + Coverage of MRR tests has been extended across more test + scenarios. MRR tests are used for continuous performance trending + and for comparison between DPDK releases. - - **MLRsearch Throughput**: MLRsearch algorithm has been introduced - for all NDR and PDR throughput tests. All tests that previously - used binary search got converted to MLRsearch. Coverage of NDR/PDR - tests has been extended across more test scenarios. + - **MLRsearch Throughput**: MLRsearch algorithm has been introduced + for all NDR and PDR throughput tests. All tests that previously + used binary search got converted to MLRsearch. Coverage of NDR/PDR + tests has been extended across more test scenarios. - - **2-Node Tests**: A new set of 2-node tests covering DPDK Testpmd - and L3fwd, running on new Xeon Skylake testbeds. + - **2-Node Tests**: A new set of 2-node tests covering DPDK Testpmd + and L3fwd, running on new Xeon Skylake testbeds. - - **More NICs**: Increased coverage of NIC models tested: Intel- - xxv710-da2, Intel-x710-da4. + - **More NICs**: Increased coverage of NIC models tested: Intel- + xxv710-da2, Intel-x710-da4. - - **Generated tests**: Simplified and unified test structure, semi- - autogenerated by generator script. Test generator is currently - able to create test combinations with various frame size and - cores combinations. All existing test cases were converted to new - format. + - **Generated tests**: Simplified and unified test structure, semi- + autogenerated by generator script. Test generator is currently + able to create test combinations with various frame size and + cores combinations. All existing test cases were converted to new + format. #. **Presentation and Analytics Layer** - - **Performance trending**: Added continuous performance trending with - anomaly detection and analysis for DPDK. + - **Performance trending**: Added continuous performance trending with + anomaly detection and analysis for DPDK. Performance Changes ------------------- diff --git a/docs/report/honeycomb_functional_tests/overview.rst b/docs/report/honeycomb_functional_tests/overview.rst index b57d2b4094..29bd671ae3 100644 --- a/docs/report/honeycomb_functional_tests/overview.rst +++ b/docs/report/honeycomb_functional_tests/overview.rst @@ -31,7 +31,7 @@ figure below. SUT (System Under Test) is a VM running Ubuntu Linux (or Centos, depending on the test suite), TG (Traffic Generator) is another VM -running Ubuntu Linux. SUT VMs run HoneyComb management agent and VPP in +running Ubuntu Linux. SUT VMs run HoneyComb management agent and VPP in Linux user-mode as a combined DUT (Device Under Test). TG runs Scapy application as a packet Traffic Generator. Virtual connectivity between SUT and TG is provided using virtual NICs using VMs' virtio drivers. @@ -45,9 +45,8 @@ virtual VM environment: +-----------------------+----------------------------------------------+ | Functionality | Description | +=======================+==============================================+ -| ACL | CRD for low-level classifiers: table and | -| | session management, interface assignment. | -| | | +| ACL | - CRD for low-level classifiers: table and | +| | session management, interface assignment. | | | - Configure up to 2 classify tables. | | | - Configure up to 2 classify sessions on one | | | table. | @@ -57,8 +56,7 @@ virtual VM environment: | | assignments. | | | - Test case count: 9. | +-----------------------+----------------------------------------------+ -| ACL-PLUGIN | CRD for high-level classifier. | -| | | +| ACL-PLUGIN | - CRD for high-level classifier. | | | - MAC + IP address classification. | | | - IPv4, IPv6 address classification. | | | - TCP, UDP, ICMP, ICMPv6 protocol and | @@ -67,36 +65,31 @@ virtual VM environment: | | - ICMP, ICMPv6 code and type classification. | | | - Test case count: 15. | +-----------------------+----------------------------------------------+ -| Basic interface | CRUD for interface state. | -| management | | -| | - ipv4/ipv6 address, ipv4 neighbor, MTU | +| Basic interface | - CRUD for interface state. | +| management | - ipv4/ipv6 address, ipv4 neighbor, MTU | | | value. | | | - Test case count: 14. | +-----------------------+----------------------------------------------+ -| Border Gateway | CRUD and functional tests for BGP. | -| Protocol | | -| | - Configure peers and routes | +| Border Gateway | - CRUD and functional tests for BGP. | +| Protocol | - Configure peers and routes | | | - Check interactions with another BGP peer. | | | - Test case count: 13. | +-----------------------+----------------------------------------------+ -| DHCP Relay | CRD for DHCP relay feature. | -| | | +| DHCP Relay | - CRD for DHCP relay feature. | | | - Configure DHCP Relays. | | | - IPv4 and IPv6 variants. | | | - Test case count: 4. | +-----------------------+----------------------------------------------+ -| Honeycomb | Configuration persistence. | -| Infractructure | | -| | - Netconf notifications for interface | +| Honeycomb | - Configuration persistence. | +| Infractructure | - Netconf notifications for interface | | | events. | | | - Netconf negative tests aimed at specific | | | issues. | | | - Netconf/Restconf northbound over IPv6. | | | - Test case count: 12. | +-----------------------+----------------------------------------------+ -| L2BD | CRUD for L2 Bridge-Domain, interface | -| | assignment. | -| | | +| L2BD | - CRUD for L2 Bridge-Domain, interface | +| | assignment. | | | - Create up to two bridge domains with all | | | implemented functions turned on: | | | flooding, unknown-unicast flooding, | @@ -107,17 +100,15 @@ virtual VM environment: | | bridge domains. | | | - Test case count: 5. | +-----------------------+----------------------------------------------+ -| L2FIB | CRD for L2-FIB entries. | -| | | +| L2FIB | - CRD for L2-FIB entries. | | | - Create 4 FIB entries: | | | one of each for filter/forward, | | | static/dynamic combinations. | | | - Remove FIB entries. | | | - Test case count: 7. | +-----------------------+----------------------------------------------+ -| LISP | CRD for Lisp: mapping, locator set, | -| | adjacency, mapresolver. | -| | | +| LISP | - CRD for Lisp: mapping, locator set, | +| | adjacency, mapresolver. | | | - Toggle Lisp feature status. | | | - Configure and delete Lisp mapping as local | | | and remote. | @@ -127,83 +118,72 @@ virtual VM environment: | | proxy ITR. | | | - Test case count: 18. | +-----------------------+----------------------------------------------+ -| LISP GPE | CRUD for LISP GPE mappings. | -| | | +| LISP GPE | - CRUD for LISP GPE mappings. | | | - Toggle Lisp GPE feature status. | | | - Configure Lisp GPE mappings. | | | - Traffic test verifying encapsulation. | | | - Test case count: 12. | +-----------------------+----------------------------------------------+ -| NAT | CRD for NAT entries, interface assignment. | -| | | +| NAT | - CRD for NAT entries, interface assignment. | | | - Configure and delete up to two NAT | | | entries. | | | - Assign NAT entries to a physical | | | interface. | | | - Test case count: 6. | +-----------------------+----------------------------------------------+ -| NSH_SFC | CRD for NSH maps and entries, using NSH_SFC | -| | plugin. | -| | | +| NSH_SFC | - CRD for NSH maps and entries, using | +| | NSH_SFC plugin. | | | - Configure up to 2 NSH entries. | | | - Configure up to 2 NSH maps. | | | - Modify and delete NSH maps and entries. | | | - Test case count: 8. | +-----------------------+----------------------------------------------+ -| PBB | CRD for provider backbone bridge | -| | sub-interface. | -| | | +| PBB | - CRD for provider backbone bridge | +| | sub-interface. | | | - Configure, modify and remove a PBB | | | sub-interface over a physical interface. | | | - Test case count: 8. | +-----------------------+----------------------------------------------+ -| Policer | CRD for traffic policing feature. | -| | | +| Policer | - CRD for traffic policing feature. | | | - Configure Policing rules. | | | - Assign to interface. | | | - Test case count: 6. | +-----------------------+----------------------------------------------+ -| Port mirroring | CRD for SPAN port mirroring, interface | -| | assignment. | -| | | +| Port mirroring | - CRD for SPAN port mirroring, interface | +| | assignment. | | | - Configure SPAN port mirroring on a | | | physical interface, mirroring. | | | - up to 2 interfaces. | | | - Remove SPAN configuration from interfaces. | | | - Test case count: 14. | +-----------------------+----------------------------------------------+ -| ProxyARP | CRD for proxyARP feature. | -| | | +| ProxyARP | - CRD for proxyARP feature. | | | - Configure proxyARP. | | | - Assign to interface. | | | - Test case count: 3. | +-----------------------+----------------------------------------------+ -| ProxyND6 | CRD for Neighbor Discovery Proxy. | -| | | +| ProxyND6 | - CRD for Neighbor Discovery Proxy. | | | - Configure ProxyND6 feature on interface. | | | - Test case count: 4. | +-----------------------+----------------------------------------------+ -| Routing | CRD for routing. | -| | | +| Routing | - CRD for routing. | | | - Configure single-hop route. | | | - Configure multi-hop routes. | | | - Configure blackhole route. | | | - IPv4 and IPv6 variants. | | | - Test case count: 6. | +-----------------------+----------------------------------------------+ -| SLAAC | CRD for Stateless Address AutoConfiguration. | -| | | +| SLAAC | - CRD for Stateless Address | +| | AutoConfiguration. | | | - Configure SLAAC feature on interfaces. | | | - Test case count: 7. | +-----------------------+----------------------------------------------+ -| Vhost-user | CRUD for Vhost-user interfaces. | -| | | +| Vhost-user | - CRUD for Vhost-user interfaces. | | | - Create, modify and delete Vhost-user | | | interface, as client and server. | | | - Test case count: 8. | +-----------------------+----------------------------------------------+ -| VLAN | CRUD for VLAN sub-interface management. | -| | | +| VLAN | - CRUD for VLAN sub-interface management. | | | - Create VLAN sub-interface over a physical | | | interface. | | | - Toggle interface state separately for | @@ -214,22 +194,19 @@ virtual VM environment: | | sub-interface. | | | - Test case count: 24. | +-----------------------+----------------------------------------------+ -| VxLAN | CRD for VxLAN tunnels. | -| | | +| VxLAN | - CRD for VxLAN tunnels. | | | - Create VxLAN interface. | | | - Disable VxLAN interface. | | | - Re-create a disabled VxLAN interface. | | | - Test case count: 6. | +-----------------------+----------------------------------------------+ -| VxLAN-GPE | CRD for VxLAN GPE tunnels. | -| | | +| VxLAN-GPE | - CRD for VxLAN GPE tunnels. | | | - Create VxLAN GPE interface. | | | - Disable VxLAN interface. | | | - Re-create a disabled VxLAN interface. | | | - Test case count: 7. | +-----------------------+----------------------------------------------+ -| TAP | CRUD for Tap interface management. | -| | | +| TAP | - CRUD for Tap interface management. | | | - Create, modify and delete TAP interface. | | | - Test case count: 3. | +-----------------------+----------------------------------------------+ diff --git a/docs/report/nsh_sfc_functional_tests/overview.rst b/docs/report/nsh_sfc_functional_tests/overview.rst index 6d36f377ce..39b9dc6eac 100644 --- a/docs/report/nsh_sfc_functional_tests/overview.rst +++ b/docs/report/nsh_sfc_functional_tests/overview.rst @@ -50,7 +50,6 @@ virtual VM environment: | | these packets from one NIC and loopback the | | | VXLAN-GPE-NSH encapsulated packets to the TG | | | from other NIC. | -| | | | | Test case count: 7. | +-----------------------+----------------------------------------------+ | NSH SFC Proxy Inbound | TG sends some VXLAN-GPE-NSH encapsulated | @@ -59,7 +58,6 @@ virtual VM environment: | | from one NIC and loopback the VXLAN | | | encapsulated packets to the TG from other | | | NIC. | -| | | | | Test case count: 6. | +-----------------------+----------------------------------------------+ | NSH SFC Proxy | TG sends some VXLAN encapsulated packets to | @@ -67,7 +65,6 @@ virtual VM environment: | | will receive these packets from one NIC and | | | loopback the VXLAN-GPE-NSH encapsulated | | | packets to the TG from other NIC. | -| | | | | Test case count: 6. | +-----------------------+----------------------------------------------+ | NSH SFC Service | TG sends some VXLAN-GPE-NSH | @@ -77,7 +74,6 @@ virtual VM environment: | | the VXLAN-GPE-NSH header, after that DUT1 | | | loopback the VXLAN-GPE-NSH encapsulated | | | packets to the TG from other NIC. | -| | | | | Test case count: 6. | +-----------------------+----------------------------------------------+ diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst index 7110cdab7e..84cb3b671a 100644 --- a/docs/report/vpp_functional_tests/csit_release_notes.rst +++ b/docs/report/vpp_functional_tests/csit_release_notes.rst @@ -6,17 +6,17 @@ Changes in |csit-release| #. **CSIT Framework** - - Few test case bug fixes. + - Few test case bug fixes. #. **Test Case Porting to VPP_make_test** - - Implementation of VIRL functional integration tests in - VPP_make_test. + - Implementation of VIRL functional integration tests in + VPP_make_test. - - All VIRL tests identified by FD.io VPP and CSIT projects as - Priority-0 ported to VPP_make_test. - Ref. `CSIT_VIRL migration progress - <https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing>`_. + - All VIRL tests identified by FD.io VPP and CSIT projects as + Priority-0 ported to VPP_make_test. + Ref. `CSIT_VIRL migration progress + <https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing>`_. Known Issues ------------ diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 03fca8f9a0..f6887e138f 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -6,53 +6,53 @@ Changes in |csit-release| #. **VPP Performance Tests** - - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now - configurable trial duration and number of consecutive executions. - Coverage of MRR tests has been extended across more test - scenarios. MRR tests are used for continuous performance trending - and for comparison between VPP releases. - - - **MLRsearch Throughput**: MLRsearch algorithm has been introduced - for all NDR and PDR throughput tests. All tests that previously - used binary search got converted to MLRsearch. Coverage of NDR/PDR - tests has been extended across more test scenarios. - - - **L2patch Tests**: Tests measure performance of VPP L2patch, the - fastest L2 forwarding path implemented in VPP, that cross-links - RX and TX of two physical interfaces. - - - **2-Node Tests**: A new baseline set of 2-node tests covering base - ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake - testbeds. - - - **Generated tests**: Simplified and unified test structure, semi- - autogenerated by generator script. Test generator is currently - able to create test combinations with various frame size and - cores combinations. All existing test cases were converted to new - format. - - - **Simultaneous Multi-Threading**: SMT-aware detection of server - processor operation mode (HyperThreading enabled/disabled) with - associated compute resource configuration including thread - affinity, number of Rx queues and DPDK I/O mbufs. Tests are - automatically tagged during execution to indicate executed thread - configuration. - - - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node - physical testbed topologies based on the new SuperMirco servers - each with two Intel Xeon Skylake Platinum processors. Full - Ansible playbooks refactor for quick server (re)installation and - reference pointers of configuration. + - **MRR Throughput**: MRR (Maximum Receive Rate) test code has now + configurable trial duration and number of consecutive executions. + Coverage of MRR tests has been extended across more test + scenarios. MRR tests are used for continuous performance trending + and for comparison between VPP releases. + + - **MLRsearch Throughput**: MLRsearch algorithm has been introduced + for all NDR and PDR throughput tests. All tests that previously + used binary search got converted to MLRsearch. Coverage of NDR/PDR + tests has been extended across more test scenarios. + + - **L2patch Tests**: Tests measure performance of VPP L2patch, the + fastest L2 forwarding path implemented in VPP, that cross-links + RX and TX of two physical interfaces. + + - **2-Node Tests**: A new baseline set of 2-node tests covering base + ip4, ip6, l2patch, l2bd, l2xc, running on new Xeon Skylake + testbeds. + + - **Generated tests**: Simplified and unified test structure, semi- + autogenerated by generator script. Test generator is currently + able to create test combinations with various frame size and + cores combinations. All existing test cases were converted to new + format. + + - **Simultaneous Multi-Threading**: SMT-aware detection of server + processor operation mode (HyperThreading enabled/disabled) with + associated compute resource configuration including thread + affinity, number of Rx queues and DPDK I/O mbufs. Tests are + automatically tagged during execution to indicate executed thread + configuration. + + - **Intel Xeon Skylake Support**: Support for 2-Node and 3-Node + physical testbed topologies based on the new SuperMirco servers + each with two Intel Xeon Skylake Platinum processors. Full + Ansible playbooks refactor for quick server (re)installation and + reference pointers of configuration. #. **Presentation and Analytics Layer** - - **Performance trending**: Further improved continuous performance - trending with anomaly detection and analysis. + - **Performance trending**: Further improved continuous performance + trending with anomaly detection and analysis. #. **Test Framework Optimizations** - - **General Code Housekeeping**: Ongoing RF keywords optimizations, - removal of redundant RF keywords. + - **General Code Housekeeping**: Ongoing RF keywords optimizations, + removal of redundant RF keywords. Performance Changes ------------------- |