diff options
Diffstat (limited to 'docs/report/introduction')
| -rw-r--r-- | docs/report/introduction/csit_design.png | bin | 0 -> 106902 bytes | |||
| -rw-r--r-- | docs/report/introduction/csit_design.rst | 178 | ||||
| -rw-r--r-- | docs/report/introduction/csit_test_naming.rst | 120 | ||||
| -rw-r--r-- | docs/report/introduction/general_notes.rst | 6 | ||||
| -rw-r--r-- | docs/report/introduction/index.rst | 1 | ||||
| -rw-r--r-- | docs/report/introduction/overview.rst | 7 |
6 files changed, 249 insertions, 63 deletions
diff --git a/docs/report/introduction/csit_design.png b/docs/report/introduction/csit_design.png Binary files differnew file mode 100644 index 0000000000..175c2f597a --- /dev/null +++ b/docs/report/introduction/csit_design.png diff --git a/docs/report/introduction/csit_design.rst b/docs/report/introduction/csit_design.rst new file mode 100644 index 0000000000..8c6c87f319 --- /dev/null +++ b/docs/report/introduction/csit_design.rst @@ -0,0 +1,178 @@ +CSIT Design
+===========
+
+FD.io CSIT system design needs to meet continuously expanding requirements of
+FD.io projects including VPP, related sub-systems (e.g. plugin applications,
+DPDK drivers) and FD.io applications (e.g. DPDK applications), as well as
+growing number of compute platforms running those applications. With CSIT
+project scope and charter including both FD.io continuous testing AND
+performance trending/comparisons, those evolving requirements further amplify
+the need for CSIT framework modularity, flexibility and usability.
+
+Design Hierarchy
+----------------
+
+CSIT follows a hierarchical system design with SUTs and DUTs at the bottom
+level, and presentation level at the top level, with a number of functional
+layers in-between. The current CSIT design including CSIT framework is depicted
+in the diagram below.
+
+.. figure:: csit_design.png
+ :alt: FD.io CSIT system design
+ :align: center
+
+ *Figure 1. FD.io CSIT system design*
+
+A brief bottom-up description is provided here:
+
+#. SUTs, DUTs, TGs:
+
+ - SUTs - Systems Under Test
+ - DUTs - Devices Under Test
+ - TGs - Traffic Generators
+
+#. Level-1 libraries - Robot and Python:
+
+ - Lowest level CSIT libraries abstracting underlying test environment, SUT,
+ DUT and TG specifics
+ - Used commonly across multiple L2 KWs
+ - Performance and functional tests:
+
+ - L1 KWs (KeyWords) are implemented as RF libraries and Python
+ libraries
+
+ - Performance TG L1 KWs:
+
+ - All L1 KWs are implemented as Python libraries
+
+ - Support for TRex only today
+ - Need to add IXIA
+
+ - Performance data plane traffic profiles:
+
+ - TG-specific stream profiles provide full control of:
+
+ - Packet definition – layers, MACs, IPs, ports, combinations thereof
+ e.g. IPs and UDP ports
+ - Stream definitions - different streams can run together, delayed,
+ one after each other
+ - Stream profiles are independent of CSIT framework and can be used
+ in any T-rex setup, can be sent anywhere to repeat tests with
+ exactly the same setup
+ - Easily extensible – one can create a new stream profile that meets
+ tests requirements
+ - Same stream profile can be used for different tests with the same
+ traffic needs
+
+ - Sunctional data plane traffic scripts:
+
+ - Scapy specific traffic scripts
+
+#. Level-2 libraries - Robot resource files:
+
+ - Higher level CSIT libraries abstracting required functions for executing
+ tests
+ - L2 KWs are classified into the following functional categories:
+
+ - Configuration, test, verification, state report
+ - Suite setup, suite teardown
+ - Test setup, test teardown
+
+#. Tests - Robot:
+
+ - Test suites with test cases;
+ - Functional tests using VIRL environment:
+
+ - VPP
+ - HoneyComb
+
+ - Performance tests using physical testbed environment:
+
+ - VPP
+ - Testpmd
+
+ - Tools:
+
+ - Documentation generator
+ - Report generator
+ - Testbed environment setup ansible playbooks
+ - Operational debugging scripts
+
+Test Lifecycle Abstraction
+--------------------------
+
+A well coded test must follow a disciplined abstraction of the test lifecycles
+that includes setup, configuration, test and verification. In addition to
+improve test execution efficiency, the commmon aspects of test setup and
+configuration shared across multiple test cases should be done only once.
+Translating these high-level guidelines into the Robot Framework one arrives to
+definition of a well coded RF tests for FD.io CSIT.
+Anatomy of Good Tests for CSIT:
+
+#. Suite Setup - Suite startup Configuration common to all Test Cases in suite:
+ uses Configuration KWs, Verification KWs, StateReport KWs
+#. Test Setup - Test startup Configuration common to multiple Test Cases: uses
+ Configuration KWs, StateReport KWs
+#. Test Case - uses L2 KWs with RF Gherkin style:
+
+ - prefixed with {Given} - Verification of Test setup, reading state: uses
+ Configuration KWs, Verification KWs, StateReport KWs
+ - prefixed with {When} - Test execution: Configuration KWs, Test KWs
+ - prefixed with {Then} - Verification of Test execution, reading state: uses
+ Verification KWs, StateReport KWs
+
+#. Test Teardown - post Test teardown with Configuration cleanup and
+ Verification common to multiple Test Cases - uses: Configuration KWs,
+ Verification KWs, StateReport KWs
+#. Suite Teardown - Suite post-test Configuration cleanup: uses Configuration
+ KWs, Verification KWs, StateReport KWs
+
+RF Keywords Functional Classification
+-------------------------------------
+
+CSIT RF KWs are classified into the functional categories matching the test
+lifecycle events described earlier. All CSIT RF L2 and L1 KWs have been grouped
+into the following functional categories:
+
+#. Configuration
+#. Test
+#. Verification
+#. StateReport
+#. SuiteSetup
+#. TestSetup
+#. SuiteTeardown
+#. TestTeardown
+
+RF Keywords Naming Guidelines
+-----------------------------
+
+Readability counts: "..code is read much more often than it is written." Hence
+following a good and consistent grammar practice is important when writing RF
+KeyWords and Tests.
+All CSIT test cases are coded using Gherkin style and include only L2 KWs
+references. L2 KWs are coded using simple style and include L2 KWs, L1 KWs, and
+L1 python references. To improve readability, the proposal is to use the same
+grammar for both RF KW styles, and to formalize the grammar of English sentences
+used for naming the RF KWs.
+RF KWs names are short sentences expressing functional description of the
+command. They must follow English sentence grammar in one of the following
+forms:
+
+#. **Imperative** - verb-object(s): *"Do something"*, verb in base form.
+#. **Declarative** - subject–verb–object(s): *"Subject does something"*, verb in
+ a third-person singular present tense form.
+#. **Affirmative** - modal_verb-verb-object(s): *"Subject should be something"*,
+ *"Object should exist"*, verb in base form.
+#. **Negative** - modal_verb-Not-verb-object(s): *"Subject should not be
+ something"*, *"Object should not exist"*, verb in base form.
+
+Passive form MUST NOT be used. However a usage of past participle as an
+adjective is okay. See usage examples.
+Following sections list applicability of the above grammar forms to different
+RF KW categories. Usage examples are provided, both good and bad.
+
+Coding guidelines
+-----------------
+
+Coding guidelines can be found on `Design optimizations wiki page
+<https://wiki.fd.io/view/CSIT/Design_Optimizations>`_.
diff --git a/docs/report/introduction/csit_test_naming.rst b/docs/report/introduction/csit_test_naming.rst index 682fcd941a..13eab06df5 100644 --- a/docs/report/introduction/csit_test_naming.rst +++ b/docs/report/introduction/csit_test_naming.rst @@ -19,26 +19,26 @@ Naming Convention The CSIT approach is to use tree naming convention and to encode following
testing information into test suite and test case names:
-1. packet network port configuration
+#. packet network port configuration
- * port type, physical or virtual;
- * number of ports;
- * NIC model, if applicable;
- * port-NIC locality, if applicable;
+ * port type, physical or virtual;
+ * number of ports;
+ * NIC model, if applicable;
+ * port-NIC locality, if applicable;
-2. packet encapsulations;
+#. packet encapsulations;
-3. VPP packet processing
+#. VPP packet processing
- * packet forwarding mode;
- * packet processing function(s);
+ * packet forwarding mode;
+ * packet processing function(s);
-4. packet forwarding path
+#. packet forwarding path
- * if present, network functions (processes, containers, VMs) and their
- topology within the computer;
+ * if present, network functions (processes, containers, VMs) and their
+ topology within the computer;
-5. main measured variable, type of test.
+#. main measured variable, type of test.
Proposed convention is to encode ports and NICs on the left (underlay),
followed by outer-most frame header, then other stacked headers up to the
@@ -58,61 +58,61 @@ topologies: 1. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
- * *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
- PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
- * *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on Intel
- x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with
- MAC learning, NDR throughput discovery.
- * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE on
- Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching
- with MAC learning, NDR throughput discovery.
- * *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel x520
- NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
- * *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on Intel
- x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery.
- * *10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot* => 2 ports of 10GE on
- Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control
- Lists baseline matching on destination, NDR throughput discovery.
- * *40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot* => 2 ports of 40GE on Cisco
- vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
- * *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline
- routed forwarding, functional tests.
+ * *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
+ * *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on Intel
+ x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching with
+ MAC learning, NDR throughput discovery.
+ * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE on
+ Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline switching
+ with MAC learning, NDR throughput discovery.
+ * *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel x520
+ NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
+ * *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on Intel
+ x520 NIC, IPv6 scaled up routed forwarding, NDR throughput discovery.
+ * *10ge2p1x520-ethip4-ip4base-iacldstbase-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, IPv4 baseline routed forwarding, ingress Access Control
+ Lists baseline matching on destination, NDR throughput discovery.
+ * *40ge2p1vic1385-ethip4-ip4base-ndrdisc.robot* => 2 ports of 40GE on Cisco
+ vic1385 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
+ * *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline
+ routed forwarding, functional tests.
2. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
P2V2P, NIC2VMchain2NIC, P2V2V2P**
- * *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
- PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
- VirtPortConfig-VMconfig-TestType*
- * *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
- of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching
- to/from two vhost interfaces and one VM, NDR throughput discovery.
- * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
- switching to/from two vhost interfaces and one VM, NDR throughput discovery.
- * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
- ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
- switching to/from four vhost interfaces and two VMs, NDR throughput
- discovery.
- * *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of
- Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost
- interfaces and one VM, functional tests.
+ * *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
+ VirtPortConfig-VMconfig-TestType*
+ * *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
+ of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain switching
+ to/from two vhost interfaces and one VM, NDR throughput discovery.
+ * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput discovery.
+ * *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from four vhost interfaces and two VMs, NDR throughput
+ discovery.
+ * *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of
+ Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost
+ interfaces and one VM, functional tests.
3. **API CRUD tests - Create (Write), Read (Retrieve), Update (Modify), Delete
(Destroy) operations for configuration and operational data**
- * *ManagementTestKeyword-ManagementOperation-ManagedFunction1-...-
- ManagedFunctionN-ManagementAPI1-ManagementAPIN-TestType*
- * *mgmt-cfg-lisp-apivat-func* => configuration of LISP with VAT API calls,
- functional tests.
- * *mgmt-cfg-l2bd-apihc-apivat-func* => configuration of L2 Bridge-Domain with
- HoneyComb API and VAT API calls, functional tests.
- * *mgmt-oper-int-apihcnc-func* => reading status and operational data of
- interface with HoneyComb NetConf API calls, functional tests.
- * *mgmt-cfg-int-tap-apihcnc-func* => configuration of tap interfaces with
- HoneyComb NetConf API calls, functional tests.
- * *mgmt-notif-int-subint-apihcnc-func* => notifications of interface and
- sub-interface events with HoneyComb NetConf Notifications, functional tests.
+ * *ManagementTestKeyword-ManagementOperation-ManagedFunction1-...-
+ ManagedFunctionN-ManagementAPI1-ManagementAPIN-TestType*
+ * *mgmt-cfg-lisp-apivat-func* => configuration of LISP with VAT API calls,
+ functional tests.
+ * *mgmt-cfg-l2bd-apihc-apivat-func* => configuration of L2 Bridge-Domain with
+ HoneyComb API and VAT API calls, functional tests.
+ * *mgmt-oper-int-apihcnc-func* => reading status and operational data of
+ interface with HoneyComb NetConf API calls, functional tests.
+ * *mgmt-cfg-int-tap-apihcnc-func* => configuration of tap interfaces with
+ HoneyComb NetConf API calls, functional tests.
+ * *mgmt-notif-int-subint-apihcnc-func* => notifications of interface and
+ sub-interface events with HoneyComb NetConf Notifications, functional tests.
For complete description of CSIT test naming convention please refer to `CSIT
test naming wiki page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
diff --git a/docs/report/introduction/general_notes.rst b/docs/report/introduction/general_notes.rst index d96dc00101..f91d96138f 100644 --- a/docs/report/introduction/general_notes.rst +++ b/docs/report/introduction/general_notes.rst @@ -41,9 +41,9 @@ is listed separately, as follows: functionality of VPP. Tests cover a range of CRUD operations executed
against VPP.
-#. **TLDK Tests** - TODO
-
-#. **NSH_SFC Tests** - TODO
+#. **NSH_SFC Functional Tests** - NSH_SFC functional tests are executed in
+ virtual FD.io testbeds focusing onNSH_SFC of VPP. Tests cover a range of
+ CRUD operations executed against VPP.
In addition to above, CSIT |release| report does also include VPP unit test
results. VPP unit tests are developed within the FD.io VPP project and as they
diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst index 7c74f46cd1..eabbf7b7ae 100644 --- a/docs/report/introduction/index.rst +++ b/docs/report/introduction/index.rst @@ -5,5 +5,6 @@ Introduction overview general_notes + csit_design csit_test_naming csit_tag_description diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index 1356ca6795..7954686ad3 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -61,6 +61,13 @@ CSIT |release| report contains following main sections and sub-sections: *Overview* - short overview of unit test framework and executed tests;
*Documentation* - source code documentation of VPP unit tests.
+#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in
+ virtual FD.io testbeds; *Overview* - tested virtual topologies, test
+ coverage and naming specifics; *CSIT Release Notes* - changes in CSIT
+ |release|, added tests, environment or methodology changes, known CSIT issues;
+ *Test Environment* - environment description ;
+ *Documentation* - source code documentation for NSH_SFC functional tests.
+
#. **Detailed Test Results** - auto-generated results from CSIT jobs
executions using CSIT Robot Framework output files as source data; *VPP
Performance Results*, *DPDK Performance Results*, *VPP Functional
|
