From d9a08e59581aafe44d446b5bc1ce6ff86b2b173b Mon Sep 17 00:00:00 2001 From: Peter Mikus Date: Thu, 13 Jul 2017 14:42:38 +0200 Subject: CSIT-618 CSIT rls1707 Report - Update I Change-Id: I7c3af55db6cc89f03004db6ddf6fcf67965132a7 Signed-off-by: Peter Mikus --- docs/report/introduction/csit_design.png | Bin 0 -> 106902 bytes docs/report/introduction/csit_design.rst | 178 ++++++++++++++++++++++++++ docs/report/introduction/csit_test_naming.rst | 120 ++++++++--------- docs/report/introduction/general_notes.rst | 6 +- docs/report/introduction/index.rst | 1 + docs/report/introduction/overview.rst | 7 + 6 files changed, 249 insertions(+), 63 deletions(-) create mode 100644 docs/report/introduction/csit_design.png create mode 100644 docs/report/introduction/csit_design.rst (limited to 'docs/report/introduction') diff --git a/docs/report/introduction/csit_design.png b/docs/report/introduction/csit_design.png new file mode 100644 index 0000000000..175c2f597a Binary files /dev/null and b/docs/report/introduction/csit_design.png differ diff --git a/docs/report/introduction/csit_design.rst b/docs/report/introduction/csit_design.rst new file mode 100644 index 0000000000..8c6c87f319 --- /dev/null +++ b/docs/report/introduction/csit_design.rst @@ -0,0 +1,178 @@ +CSIT Design +=========== + +FD.io CSIT system design needs to meet continuously expanding requirements of +FD.io projects including VPP, related sub-systems (e.g. plugin applications, +DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as +growing number of compute platforms running those applications. With CSIT +project scope and charter including both FD.io continuous testing AND +performance trending/comparisons, those evolving requirements further amplify +the need for CSIT framework modularity, flexibility and usability. + +Design Hierarchy +---------------- + +CSIT follows a hierarchical system design with SUTs and DUTs at the bottom +level, and presentation level at the top level, with a number of functional +layers in-between. The current CSIT design including CSIT framework is depicted +in the diagram below. + +.. figure:: csit_design.png + :alt: FD.io CSIT system design + :align: center + + *Figure 1. FD.io CSIT system design* + +A brief bottom-up description is provided here: + +#. SUTs, DUTs, TGs: + + - SUTs - Systems Under Test + - DUTs - Devices Under Test + - TGs - Traffic Generators + +#. Level-1 libraries - Robot and Python: + + - Lowest level CSIT libraries abstracting underlying test environment, SUT, + DUT and TG specifics + - Used commonly across multiple L2 KWs + - Performance and functional tests: + + - L1 KWs (KeyWords) are implemented as RF libraries and Python + libraries + + - Performance TG L1 KWs: + + - All L1 KWs are implemented as Python libraries + + - Support for TRex only today + - Need to add IXIA + + - Performance data plane traffic profiles: + + - TG-specific stream profiles provide full control of: + + - Packet definition – layers, MACs, IPs, ports, combinations thereof + e.g. IPs and UDP ports + - Stream definitions - different streams can run together, delayed, + one after each other + - Stream profiles are independent of CSIT framework and can be used + in any T-rex setup, can be sent anywhere to repeat tests with + exactly the same setup + - Easily extensible – one can create a new stream profile that meets + tests requirements + - Same stream profile can be used for different tests with the same + traffic needs + + - Sunctional data plane traffic scripts: + + - Scapy specific traffic scripts + +#. Level-2 libraries - Robot resource files: + + - Higher level CSIT libraries abstracting required functions for executing + tests + - L2 KWs are classified into the following functional categories: + + - Configuration, test, verification, state report + - Suite setup, suite teardown + - Test setup, test teardown + +#. Tests - Robot: + + - Test suites with test cases; + - Functional tests using VIRL environment: + + - VPP + - HoneyComb + + - Performance tests using physical testbed environment: + + - VPP + - Testpmd + + - Tools: + + - Documentation generator + - Report generator + - Testbed environment setup ansible playbooks + - Operational debugging scripts + +Test Lifecycle Abstraction +-------------------------- + +A well coded test must follow a disciplined abstraction of the test lifecycles +that includes setup, configuration, test and verification. In addition to +improve test execution efficiency, the commmon aspects of test setup and +configuration shared across multiple test cases should be done only once. +Translating these high-level guidelines into the Robot Framework one arrives to +definition of a well coded RF tests for FD.io CSIT. +Anatomy of Good Tests for CSIT: + +#. Suite Setup - Suite startup Configuration common to all Test Cases in suite: + uses Configuration KWs, Verification KWs, StateReport KWs +#. Test Setup - Test startup Configuration common to multiple Test Cases: uses + Configuration KWs, StateReport KWs +#. Test Case - uses L2 KWs with RF Gherkin style: + + - prefixed with {Given} - Verification of Test setup, reading state: uses + Configuration KWs, Verification KWs, StateReport KWs + - prefixed with {When} - Test execution: Configuration KWs, Test KWs + - prefixed with {Then} - Verification of Test execution, reading state: uses + Verification KWs, StateReport KWs + +#. Test Teardown - post Test teardown with Configuration cleanup and + Verification common to multiple Test Cases - uses: Configuration KWs, + Verification KWs, StateReport KWs +#. Suite Teardown - Suite post-test Configuration cleanup: uses Configuration + KWs, Verification KWs, StateReport KWs + +RF Keywords Functional Classification +------------------------------------- + +CSIT RF KWs are classified into the functional categories matching the test +lifecycle events described earlier. All CSIT RF L2 and L1 KWs have been grouped +into the following functional categories: + +#. Configuration +#. Test +#. Verification +#. StateReport +#. SuiteSetup +#. TestSetup +#. SuiteTeardown +#. TestTeardown + +RF Keywords Naming Guidelines +----------------------------- + +Readability counts: "..code is read much more often than it is written." Hence +following a good and consistent grammar practice is important when writing RF +KeyWords and Tests. +All CSIT test cases are coded using Gherkin style and include only L2 KWs +references. L2 KWs are coded using simple style and include L2 KWs, L1 KWs, and +L1 python references. To improve readability, the proposal is to use the same +grammar for both RF KW styles, and to formalize the grammar of English sentences +used for naming the RF KWs. +RF KWs names are short sentences expressing functional description of the +command. They must follow English sentence grammar in one of the following +forms: + +#. **Imperative** - verb-object(s): *"Do something"*, verb in base form. +#. **Declarative** - subject–verb–object(s): *"Subject does something"*, verb in + a third-person singular present tense form. +#. **Affirmative** - modal_verb-verb-object(s): *"Subject should be something"*, + *"Object should exist"*, verb in base form. +#. **Negative** - modal_verb-Not-verb-object(s): *"Subject should not be + something"*, *"Object should not exist"*, verb in base form. + +Passive form MUST NOT be used. However a usage of past participle as an +adjective is okay. See usage examples. +Following sections list applicability of the above grammar forms to different +RF KW categories. Usage examples are provided, both good and bad. + +Coding guidelines +----------------- + +Coding guidelines can be found on `Design optimizations wiki page +`_. diff --git a/docs/report/introduction/csit_test_naming.rst b/docs/report/introduction/csit_test_naming.rst index 682fcd941a..13eab06df5 100644 --- a/docs/report/introduction/csit_test_naming.rst +++ b/docs/report/introduction/csit_test_naming.rst @@ -19,26 +19,26 @@ Naming Convention The CSIT approach is to use tree naming convention and to encode following testing information into test suite and test case names: -1. packet network port configuration +#. packet network port configuration - * port type, physical or virtual; - * number of ports; - * NIC model, if applicable; - * port-NIC locality, if applicable; + * port type, physical or virtual; + * number of ports; + * NIC model, if applicable; + * port-NIC locality, if applicable; -2. packet encapsulations; +#. packet encapsulations; -3. VPP packet processing +#. VPP packet processing - * packet forwarding mode; - * packet processing function(s); + * packet forwarding mode; + * packet processing function(s); -4. packet forwarding path +#. packet forwarding path - * if present, network functions (processes, containers, VMs) and their - topology within the computer; + * if present, network functions (processes, containers, VMs) and their + topology within the computer; -5. main measured variable, type of test. +#. main measured variable, type of test. Proposed convention is to encode ports and NICs on the left (underlay), followed by outer-most frame header, then other stacked headers up to the @@ -58,61 +58,61 @@ topologies: 1. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** - * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- - PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* - * *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on Intel - x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with - MAC learning, NDR throughput discovery. - * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE on - Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching - with MAC learning, NDR throughput discovery. - * *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel x520 - NIC, IPv4 baseline routed forwarding, NDR throughput discovery. - * *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on Intel - x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery. - * *10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot* => 2 ports of 10GE on - Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control - Lists baseline matching on destination, NDR throughput discovery. - * *40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot* => 2 ports of 40GE on Cisco - vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. - * *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline - routed forwarding, functional tests. + * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* + * *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on Intel + x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with + MAC learning, NDR throughput discovery. + * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE on + Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching + with MAC learning, NDR throughput discovery. + * *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel x520 + NIC, IPv4 baseline routed forwarding, NDR throughput discovery. + * *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on Intel + x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery. + * *10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot* => 2 ports of 10GE on + Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control + Lists baseline matching on destination, NDR throughput discovery. + * *40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot* => 2 ports of 40GE on Cisco + vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. + * *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline + routed forwarding, functional tests. 2. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, P2V2P, NIC2VMchain2NIC, P2V2V2P** - * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- - PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- - VirtPortConfig-VMconfig-TestType* - * *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports - of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching - to/from two vhost interfaces and one VM, NDR throughput discovery. - * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 - ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain - switching to/from two vhost interfaces and one VM, NDR throughput discovery. - * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 - ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain - switching to/from four vhost interfaces and two VMs, NDR throughput - discovery. - * *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of - Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost - interfaces and one VM, functional tests. + * *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- + VirtPortConfig-VMconfig-TestType* + * *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports + of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching + to/from two vhost interfaces and one VM, NDR throughput discovery. + * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from two vhost interfaces and one VM, NDR throughput discovery. + * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from four vhost interfaces and two VMs, NDR throughput + discovery. + * *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of + Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost + interfaces and one VM, functional tests. 3. **API CRUD tests - Create (Write), Read (Retrieve), Update (Modify), Delete (Destroy) operations for configuration and operational data** - * *ManagementTestKeyword-ManagementOperation-ManagedFunction1-...- - ManagedFunctionN-ManagementAPI1-ManagementAPIN-TestType* - * *mgmt-cfg-lisp-apivat-func* => configuration of LISP with VAT API calls, - functional tests. - * *mgmt-cfg-l2bd-apihc-apivat-func* => configuration of L2 Bridge-Domain with - HoneyComb API and VAT API calls, functional tests. - * *mgmt-oper-int-apihcnc-func* => reading status and operational data of - interface with HoneyComb NetConf API calls, functional tests. - * *mgmt-cfg-int-tap-apihcnc-func* => configuration of tap interfaces with - HoneyComb NetConf API calls, functional tests. - * *mgmt-notif-int-subint-apihcnc-func* => notifications of interface and - sub-interface events with HoneyComb NetConf Notifications, functional tests. + * *ManagementTestKeyword-ManagementOperation-ManagedFunction1-...- + ManagedFunctionN-ManagementAPI1-ManagementAPIN-TestType* + * *mgmt-cfg-lisp-apivat-func* => configuration of LISP with VAT API calls, + functional tests. + * *mgmt-cfg-l2bd-apihc-apivat-func* => configuration of L2 Bridge-Domain with + HoneyComb API and VAT API calls, functional tests. + * *mgmt-oper-int-apihcnc-func* => reading status and operational data of + interface with HoneyComb NetConf API calls, functional tests. + * *mgmt-cfg-int-tap-apihcnc-func* => configuration of tap interfaces with + HoneyComb NetConf API calls, functional tests. + * *mgmt-notif-int-subint-apihcnc-func* => notifications of interface and + sub-interface events with HoneyComb NetConf Notifications, functional tests. For complete description of CSIT test naming convention please refer to `CSIT test naming wiki page `_. diff --git a/docs/report/introduction/general_notes.rst b/docs/report/introduction/general_notes.rst index d96dc00101..f91d96138f 100644 --- a/docs/report/introduction/general_notes.rst +++ b/docs/report/introduction/general_notes.rst @@ -41,9 +41,9 @@ is listed separately, as follows: functionality of VPP. Tests cover a range of CRUD operations executed against VPP. -#. **TLDK Tests** - TODO - -#. **NSH_SFC Tests** - TODO +#. **NSH_SFC Functional Tests** - NSH_SFC functional tests are executed in + virtual FD.io testbeds focusing onNSH_SFC of VPP. Tests cover a range of + CRUD operations executed against VPP. In addition to above, CSIT |release| report does also include VPP unit test results. VPP unit tests are developed within the FD.io VPP project and as they diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst index 7c74f46cd1..eabbf7b7ae 100644 --- a/docs/report/introduction/index.rst +++ b/docs/report/introduction/index.rst @@ -5,5 +5,6 @@ Introduction overview general_notes + csit_design csit_test_naming csit_tag_description diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index 1356ca6795..7954686ad3 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -61,6 +61,13 @@ CSIT |release| report contains following main sections and sub-sections: *Overview* - short overview of unit test framework and executed tests; *Documentation* - source code documentation of VPP unit tests. +#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for NSH_SFC functional tests. + #. **Detailed Test Results** - auto-generated results from CSIT jobs executions using CSIT Robot Framework output files as source data; *VPP Performance Results*, *DPDK Performance Results*, *VPP Functional -- cgit 1.2.3-korg