diff options
Diffstat (limited to 'docs/report/honeycomb_functional_tests/overview.rst')
| -rw-r--r-- | docs/report/honeycomb_functional_tests/overview.rst | 394 |
1 files changed, 223 insertions, 171 deletions
diff --git a/docs/report/honeycomb_functional_tests/overview.rst b/docs/report/honeycomb_functional_tests/overview.rst index 3ff8a6e2e9..b57d2b4094 100644 --- a/docs/report/honeycomb_functional_tests/overview.rst +++ b/docs/report/honeycomb_functional_tests/overview.rst @@ -1,15 +1,16 @@ Overview ======== -Tested Virtual Topologies -------------------------- +Virtual Topologies +------------------ + +CSIT HoneyComb functional tests are executed in VM-based virtual topologies +created on demand using :abbr:`VIRL (Virtual Internet Routing Lab)` +simulation platform contributed by Cisco. VIRL runs on physical +baremetal servers hosted by LF FD.io project. -CSIT Honeycomb functional tests are executed on virtualized topologies created -using :abbr:`VIRL (Virtual Internet Routing Lab)` simulation platform -contributed by Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io -project. All tests are executed in two node logical test topology - Traffic -Generator (TG) node and Systems Under Test (SUT1) node connected in a loop. -Logical test topology is shown in the figure below. +All tests are executed in two-node virtual test topology shown in the +figure below. .. only:: latex @@ -17,153 +18,226 @@ Logical test topology is shown in the figure below. \begin{figure}[H] \centering - \graphicspath{{../_tmp/src/vpp_performance_tests/}} - \includegraphics[width=0.90\textwidth]{logical-2n-nic2nic} - \label{fig:logical-2n-nic2nic} + \graphicspath{{../_tmp/src/vpp_functional_tests/}} + \includegraphics[width=0.90\textwidth]{virtual-2n-nic2nic} + \label{fig:virtual-2n-nic2nic} \end{figure} .. only:: html - .. figure:: logical-2n-nic2nic.svg - :alt: logical-2n-nic2nic + .. figure:: ../vpp_functional_tests/virtual-2n-nic2nic.svg + :alt: virtual-2n-nic2nic :align: center -SUT1 is a VM (Ubuntu or Centos, depending on the test suite), TG is a Traffic -Generator (TG, another Ubuntu VM). SUTs run Honeycomb and VPP SW applications -in Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy -SW application as a packet Traffic Generator. Logical connectivity between -SUTs and to TG is provided using virtual NICs using VMs' virtio driver. - -Virtual testbeds are created on-demand whenever a verification job is started -(e.g. triggered by the gerrit patch submission) and destroyed upon completion -of all functional tests. Each node is a Virtual Machine and each connection -that is drawn on the diagram is available for use in any test case. During the -test execution, all nodes are reachable thru the Management network connected -to every node via dedicated virtual NICs and virtual links (not shown above -for clarity). +SUT (System Under Test) is a VM running Ubuntu Linux (or Centos, +depending on the test suite), TG (Traffic Generator) is another VM +running Ubuntu Linux. SUT VMs run HoneyComb management agent and VPP in +Linux user-mode as a combined DUT (Device Under Test). TG runs Scapy +application as a packet Traffic Generator. Virtual connectivity between +SUT and TG is provided using virtual NICs using VMs' virtio drivers. Functional Tests Coverage ------------------------- -The following Honeycomb functional test areas are included in the |csit-release| -with results listed in this report: - -- **Basic interface management** - CRUD for interface state, - - ipv4/ipv6 address, ipv4 neighbor, MTU value. - - Test case count: 14 -- **L2BD** - CRUD for L2 Bridge-Domain, interface assignment. - - Create up to two bridge domains with all implemented functions turned on. - - (flooding, unknown-unicast flooding, forwarding, learning, arp-termination) - - Assign up to two physical interfaces to a single bridge domain. - - Remove interface assignments, remove bridge domains. - - Test case count: 5 -- **L2FIB** - CRD for L2-FIB entries. - - Create 4 FIB entries - - (one of each for filter/forward, static/dynamic combinations). - - Remove FIB entries. - - Test case count: 7 -- **VxLAN** - CRD for VxLAN tunnels. - - Create VxLAN interface. - - Disable VxLAN interface. - - Re-create a disabled VxLAN interface. - - Test case count: 6 -- **VxLAN-GPE** - CRD for VxLAN GPE tunnels. - - Create VxLAN GPE interface. - - Disable VxLAN interface. - - Re-create a disabled VxLAN interface. - - Test case count: 7 -- **Vhost-user** - CRUD for Vhost-user interfaces. - - Create, modify and delete Vhost-user interface, as client and server. - - Test case count: 8 -- **TAP** - CRUD for Tap interface management. - - Create, modify and delete TAP interface. - - Test case count: 3 -- **VLAN** - CRUD for VLAN sub-interface management. - - Create VLAN sub-interface over a physical interface. - - Toggle interface state separately for super-interface and sub-interface. - - Configure IP address and bridge domain assignment on sub-interface. - - Configure VLAN tag rewrite on sub-interface. - - Test case count: 24 -- **ACL** - CRD for low-level classifiers: table and session management, - - interface assignment. - - Configure up to 2 classify tables. - - Configure up to 2 classify sessions on one table. - - Assign classify session to a physical interface. - - Remove tables, sessions, interface assignments. - - Test case count: 9 -- **PBB** - CRD for provider backbone bridge sub-interface. - - Configure, modify and remove a PBB sub-interface over a physical interface. - - Test case count: 8 -- **NSH_SFC** - CRD for NSH maps and entries, using NSH_SFC plugin. - - Configure up to 2 NSH entries. - - Configure up to 2 NSH maps. - - Modify and delete NSH maps and entries. - - Test case count: 8 -- **LISP** - CRD for Lisp: mapping, locator set, adjacency, map resolver. - - Toggle Lisp feature status. - - Configure and delete Lisp mapping as local and remote. - - Configure and delete Lisp adjacency mapping - - Configure and delete Lisp map resolver, proxy ITR. - - Test case count: 18 -- **LISP GPE** - CRUD for LISP GPE mappings. - - Toggle Lisp GPE feature status. - - Configure Lisp GPE mappings. - - Traffic test verifying encapsulation. - - Test case count: 12 -- **NAT** - CRD for NAT entries, interface assignment. - - Configure and delete up to two NAT entries. - - Assign NAT entries to a physical interface. - - Test case count: 6 -- **Port mirroring** - CRD for SPAN port mirroring, interface assignment. - - Configure SPAN port mirroring on a physical interface, mirroring - - up to 2 interfaces. - - Remove SPAN configuration from interfaces. - - Test case count: 14 -- **ACL-PLUGIN** - CRD for high-level classifier - - MAC + IP address classification. - - IPv4, IPv6 address classification. - - TCP, UDP, ICMP, ICMPv6 protocol/next-header classification. - - port number classification. - - ICMP, ICMPv6 code and type classification. - - Test case count: 15 -- **ProxyARP** - CRD for proxyARP feature. - - Configure proxyARP. - - Assign to interface. - - Test case count: 3 -- **ProxyND6** - CRD for Neighbor Discovery Proxy. - - Configure ProxyND6 feature on interface. - - Test case count: 4 -- **DHCP Relay** - CRD for DHCP relay feature. - - Configure DHCP Relays. - - IPv4 and IPv6 variants. - - Test case count: 4 -- **SLAAC** - CRD for Stateless Address AutoConfiguration. - - Configure SLAAC feature on interfaces. - - Test case count: 7 -- **Routing** - CRD for routing. - - Configure single-hop route. - - Configure multi-hop routes. - - Configure blackhole route. - - IPv4 and IPv6 variants. - - Test case count: 6 -- **Policer** - CRD for traffic policing feature. - - Configure Policing rules. - - Assign to interface. - - Test case count: 6 -- **Border Gateway Protocol** - CRUD and functional tests for BGP. - - Configure peers and routes - - Check interactions with another BGP peer. - - Test case count: 13 -- **Honeycomb Infractructure** - configuration persistence, - - Netconf notifications for interface events, - - Netconf negative tests aimed at specific issues - - Netconf/Restconf northbound over IPv6 - - Test case count: 12 +|csit-release| includes following HoneyComb functionality tested in +virtual VM environment: + ++-----------------------+----------------------------------------------+ +| Functionality | Description | ++=======================+==============================================+ +| ACL | CRD for low-level classifiers: table and | +| | session management, interface assignment. | +| | | +| | - Configure up to 2 classify tables. | +| | - Configure up to 2 classify sessions on one | +| | table. | +| | - Assign classify session to a physical | +| | interface. | +| | - Remove tables, sessions, interface | +| | assignments. | +| | - Test case count: 9. | ++-----------------------+----------------------------------------------+ +| ACL-PLUGIN | CRD for high-level classifier. | +| | | +| | - MAC + IP address classification. | +| | - IPv4, IPv6 address classification. | +| | - TCP, UDP, ICMP, ICMPv6 protocol and | +| | next-header classification. | +| | - port number classification. | +| | - ICMP, ICMPv6 code and type classification. | +| | - Test case count: 15. | ++-----------------------+----------------------------------------------+ +| Basic interface | CRUD for interface state. | +| management | | +| | - ipv4/ipv6 address, ipv4 neighbor, MTU | +| | value. | +| | - Test case count: 14. | ++-----------------------+----------------------------------------------+ +| Border Gateway | CRUD and functional tests for BGP. | +| Protocol | | +| | - Configure peers and routes | +| | - Check interactions with another BGP peer. | +| | - Test case count: 13. | ++-----------------------+----------------------------------------------+ +| DHCP Relay | CRD for DHCP relay feature. | +| | | +| | - Configure DHCP Relays. | +| | - IPv4 and IPv6 variants. | +| | - Test case count: 4. | ++-----------------------+----------------------------------------------+ +| Honeycomb | Configuration persistence. | +| Infractructure | | +| | - Netconf notifications for interface | +| | events. | +| | - Netconf negative tests aimed at specific | +| | issues. | +| | - Netconf/Restconf northbound over IPv6. | +| | - Test case count: 12. | ++-----------------------+----------------------------------------------+ +| L2BD | CRUD for L2 Bridge-Domain, interface | +| | assignment. | +| | | +| | - Create up to two bridge domains with all | +| | implemented functions turned on: | +| | flooding, unknown-unicast flooding, | +| | forwarding, learning, arp-termination. | +| | - Assign up to two physical interfaces to a | +| | single bridge domain. | +| | - Remove interface assignments, remove | +| | bridge domains. | +| | - Test case count: 5. | ++-----------------------+----------------------------------------------+ +| L2FIB | CRD for L2-FIB entries. | +| | | +| | - Create 4 FIB entries: | +| | one of each for filter/forward, | +| | static/dynamic combinations. | +| | - Remove FIB entries. | +| | - Test case count: 7. | ++-----------------------+----------------------------------------------+ +| LISP | CRD for Lisp: mapping, locator set, | +| | adjacency, mapresolver. | +| | | +| | - Toggle Lisp feature status. | +| | - Configure and delete Lisp mapping as local | +| | and remote. | +| | - Configure and delete Lisp adjacency | +| | mapping. | +| | - Configure and delete Lisp map resolver, | +| | proxy ITR. | +| | - Test case count: 18. | ++-----------------------+----------------------------------------------+ +| LISP GPE | CRUD for LISP GPE mappings. | +| | | +| | - Toggle Lisp GPE feature status. | +| | - Configure Lisp GPE mappings. | +| | - Traffic test verifying encapsulation. | +| | - Test case count: 12. | ++-----------------------+----------------------------------------------+ +| NAT | CRD for NAT entries, interface assignment. | +| | | +| | - Configure and delete up to two NAT | +| | entries. | +| | - Assign NAT entries to a physical | +| | interface. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| NSH_SFC | CRD for NSH maps and entries, using NSH_SFC | +| | plugin. | +| | | +| | - Configure up to 2 NSH entries. | +| | - Configure up to 2 NSH maps. | +| | - Modify and delete NSH maps and entries. | +| | - Test case count: 8. | ++-----------------------+----------------------------------------------+ +| PBB | CRD for provider backbone bridge | +| | sub-interface. | +| | | +| | - Configure, modify and remove a PBB | +| | sub-interface over a physical interface. | +| | - Test case count: 8. | ++-----------------------+----------------------------------------------+ +| Policer | CRD for traffic policing feature. | +| | | +| | - Configure Policing rules. | +| | - Assign to interface. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| Port mirroring | CRD for SPAN port mirroring, interface | +| | assignment. | +| | | +| | - Configure SPAN port mirroring on a | +| | physical interface, mirroring. | +| | - up to 2 interfaces. | +| | - Remove SPAN configuration from interfaces. | +| | - Test case count: 14. | ++-----------------------+----------------------------------------------+ +| ProxyARP | CRD for proxyARP feature. | +| | | +| | - Configure proxyARP. | +| | - Assign to interface. | +| | - Test case count: 3. | ++-----------------------+----------------------------------------------+ +| ProxyND6 | CRD for Neighbor Discovery Proxy. | +| | | +| | - Configure ProxyND6 feature on interface. | +| | - Test case count: 4. | ++-----------------------+----------------------------------------------+ +| Routing | CRD for routing. | +| | | +| | - Configure single-hop route. | +| | - Configure multi-hop routes. | +| | - Configure blackhole route. | +| | - IPv4 and IPv6 variants. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| SLAAC | CRD for Stateless Address AutoConfiguration. | +| | | +| | - Configure SLAAC feature on interfaces. | +| | - Test case count: 7. | ++-----------------------+----------------------------------------------+ +| Vhost-user | CRUD for Vhost-user interfaces. | +| | | +| | - Create, modify and delete Vhost-user | +| | interface, as client and server. | +| | - Test case count: 8. | ++-----------------------+----------------------------------------------+ +| VLAN | CRUD for VLAN sub-interface management. | +| | | +| | - Create VLAN sub-interface over a physical | +| | interface. | +| | - Toggle interface state separately for | +| | super-interface and sub-interface. | +| | - Configure IP address and bridge domain | +| | assignment on sub-interface. | +| | - Configure VLAN tag rewrite on | +| | sub-interface. | +| | - Test case count: 24. | ++-----------------------+----------------------------------------------+ +| VxLAN | CRD for VxLAN tunnels. | +| | | +| | - Create VxLAN interface. | +| | - Disable VxLAN interface. | +| | - Re-create a disabled VxLAN interface. | +| | - Test case count: 6. | ++-----------------------+----------------------------------------------+ +| VxLAN-GPE | CRD for VxLAN GPE tunnels. | +| | | +| | - Create VxLAN GPE interface. | +| | - Disable VxLAN interface. | +| | - Re-create a disabled VxLAN interface. | +| | - Test case count: 7. | ++-----------------------+----------------------------------------------+ +| TAP | CRUD for Tap interface management. | +| | | +| | - Create, modify and delete TAP interface. | +| | - Test case count: 3. | ++-----------------------+----------------------------------------------+ Total 219 Honeycomb functional tests in the |csit-release|. -Operational data in Honeycomb should mirror configuration data at all times. -Because of this, test cases follow this general pattern: +Operational data in Honeycomb should mirror configuration data at all +times. Because of this, test cases follow this general pattern: #. read operational data of the feature using restconf. #. read status of the feature using VPP API dump. @@ -172,37 +246,15 @@ Because of this, test cases follow this general pattern: #. verify changes using VPP API dump, OR #. send a packet to VPP node and observe behaviour to verify configuration. -Test cases involving network interfaces utilize the first two interfaces on -the DUT node. +Test cases involving network interfaces utilize the first two interfaces +on the DUT node. Functional Tests Naming ----------------------- -CSIT-17.01 introduced a common structured naming convention for all -performance and functional tests. This change was driven by substantially -growing number and type of CSIT test cases. Firstly, the original practice did -not always follow any strict naming convention. Secondly test names did not -always clearly capture tested packet encapsulations, and the actual type or -content of the tests. Thirdly HW configurations in terms of NICs, ports and -their locality were not captured either. These were but few reasons that drove -the decision to change and define a new more complete and stricter test naming -convention, and to apply this to all existing and new test cases. - -The new naming should be intuitive for majority of the tests. The complete -description of CSIT test naming convention is provided on `CSIT test naming -page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_. - -Here few illustrative examples of the new naming usage for functional test -suites: - -#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** - - - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline - routed forwarding, functional tests. - -#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, - P2V2P, NIC2VMchain2NIC, P2V2V2P** +|csit-release| follows a common structured naming convention for all +performance and system functional tests, introduced in CSIT-17.01. - - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of - Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost - interfaces and one VM, functional tests. +The naming should be intuitive for majority of the tests. Complete +description of CSIT test naming convention is provided on +:ref:`csit_test_naming`. |