diff options
Diffstat (limited to 'docs/report')
| -rw-r--r-- | docs/report/introduction/general_notes.rst | 74 | ||||
| -rw-r--r-- | docs/report/introduction/index.rst | 4 | ||||
| -rw-r--r-- | docs/report/introduction/introduction.rst | 146 | ||||
| -rw-r--r-- | docs/report/introduction/overview.rst | 93 | ||||
| -rw-r--r-- | docs/report/introduction/test_scenarios_overview.rst | 86 |
5 files changed, 234 insertions, 169 deletions
diff --git a/docs/report/introduction/general_notes.rst b/docs/report/introduction/general_notes.rst deleted file mode 100644 index 91cd46e678..0000000000 --- a/docs/report/introduction/general_notes.rst +++ /dev/null @@ -1,74 +0,0 @@ -General Notes -============= - -All CSIT test results listed in this report are sourced and auto-generated -from :file:`output.xml` :abbr:`RF (Robot Framework)` files resulting from -:abbr:`LF (Linux Foundation)` FD.io Jenkins jobs execution against |vpp-release| -release artifacts. References are provided to the original :abbr:`LF (Linux -Foundation)` FD.io Jenkins job results. Additional references are provided to -the :abbr:`RF (Robot Framework)` result files that got archived in FD.io nexus -online storage system. - -FD.io CSIT project currently covers multiple FD.io system and sub-system -testing areas and this is reflected in this report, where each testing area -is listed separately, as follows: - -#. **VPP - Performance** - VPP benchmarking tests are executed in physical - FD.io testbeds, focusing on VPP network data plane performance at this stage, - both for Phy-to-Phy (NIC-to-NIC) and Phy-to-VM-to-Phy (NIC-to-VM-to-NIC) - forwarding topologies. Tested across a range of NICs, 10GE, 25GE and 40GE - interfaces, range of multi-thread and multi-core configurations. VPP - application runs in host user-mode. TRex is used as a traffic generator. - -#. **LXC and Docker Containers VPP memif - Performance** - VPP memif - virtual interface tests interconnect multiple VPP instances running in - containers. VPP vswitch instance runs in bare-metal user-mode - handling Intel x520 NIC 10GbE, Intel x710 NIC 10GbE, Intel xl710 NIC 40GbE - interfaces and connecting over memif (Slave side) virtual interfaces to more - instances of VPP running in LXC or in Docker Containers, both with memif - virtual interfaces (Master side). Tested across a range of multi-thread and - multi-core configurations. TRex is used as a traffic generator. - -#. **Container Topologies Orchestrated by K8s - Performance** - CSIT Container - topologies connected over the memif virtual interface (shared memory - interface). For these tests VPP vswitch instance runs in a Docker Container - handling Intel x520 NIC 10GbE, Intel x710 NIC 10GbE interfaces and connecting - over memif virtual interfaces to more instances of VPP running in Docker - Containers with memif virtual interfaces. All containers are - orchestrated by Kubernetes, with `Ligato <https://github.com/ligato>`_ for - container networking. TRex is used as a traffic generator. - -#. **DPDK Performance** - VPP is using DPDK code to control and drive - the NICs and physical interfaces. Tests are used as a baseline to - profile performance of the DPDK sub-system. DPDK tests are executed in - physical FD.io testbeds, focusing on Testpmd/L3FWD data plane performance for - Phy-to-Phy (NIC-to-NIC). Tested across a range of NICs, 10GE, 25GE and 40GE - interfaces, range of multi-thread and multi-core configurations. - Testpmd/L3FWD application runs in host user-mode. TRex is used as a traffic - generator. - -#. **VPP Functional** - VPP functional tests are executed in virtual - FD.io testbeds focusing on VPP packet processing functionality, including - network data plane and in -line control plane. Tests cover vNIC-to-vNIC - vNIC-to-VM-to-vNIC forwarding topologies. Scapy is used as a traffic - generator. - -#. **Honeycomb Functional** - Honeycomb functional tests are executed in - virtual FD.io testbeds, focusing on Honeycomb management and programming - functionality of VPP. Tests cover a range of CRUD operations executed - against VPP. - -#. **NSH_SFC Functional** - NSH_SFC functional tests are executed in - virtual FD.io testbeds focusing on NSH_SFC of VPP. Tests cover a range of - CRUD operations executed against VPP. - -#. **DMM Functional** - DMM functional tests are executed in virtual FD.io - testbeds demonstrates single server[DUT1] and single client[DUT2] scenario - using DMM framework and kernel tcp/ip stack. - -FD.io CSIT system is developed using two main coding platforms :abbr:`RF (Robot -Framework)` and Python2.7. |csit-release| source code for the executed test -suites is available in CSIT branch |release| in the directory -:file:`./tests/<name_of_the_test_suite>`. A local copy of CSIT source code -can be obtained by cloning CSIT git repository - :command:`git clone -https://gerrit.fd.io/r/csit`. diff --git a/docs/report/introduction/index.rst b/docs/report/introduction/index.rst index 7d4f4c0d77..ba6599cfc8 100644 --- a/docs/report/introduction/index.rst +++ b/docs/report/introduction/index.rst @@ -3,7 +3,7 @@ Introduction .. toctree:: - overview - general_notes + introduction + test_scenarios_overview physical_testbeds methodology diff --git a/docs/report/introduction/introduction.rst b/docs/report/introduction/introduction.rst new file mode 100644 index 0000000000..c88aeb4e30 --- /dev/null +++ b/docs/report/introduction/introduction.rst @@ -0,0 +1,146 @@ +Introduction +============ + +This is FD.io CSIT (Fast Data i/o Continuous System Integration and +Testing) project report for |csit-release| system performance and +functional testing of |vpp-release|. + +There is also a downloadable `PDF version of this report`_. + +This report describes CSIT performance and functional tests delivered in +|csit-release|. A high-level overview is provided for each CSIT test +environment running in :abbr:`LF (Linux Foundation)` FD.io Continuous +Performance Labs. This is followed by summary of all executed tests +against the |vpp-release| and associated FD.io projects and sub-systems +(Honeycomb, DPDK, NSH_SFC, DMM), |csit-release| release notes, result +highlights and known issues discovered in CSIT. More detailed +description of each environment, pointers to CSIT test code +documentation and detailed test resuls with links to the source data +files are also provided. + +|csit-release| report contains following main sections and sub-sections: + +#. **Introduction**: general introduction to CSIT project. + + - **Introduction**: this section. + - **Test Scenarios Overview**: a brief overview of test scenarios + covered in this report. + - **Physical Testbeds**: description of physical testbeds. + - **Performance Test Methodology**: benchmarking methodologies. + +#. **VPP Performance**: VPP performance tests executed in physical + FD.io testbeds. + + - **Overview**: tested logical topologies, test coverage and naming + specifics. + - **Release Notes**: changes in |csit-release|, added tests, + performance changes, environment or methodology changes, known + issues. + - **Packet Throughput**: NDR, PDR throughput graphs based on results + from repeated same test job executions to verify repeatibility of + measurements. + - **Packet Latency**: Latency graphs based on results from test job + executions. + - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup + graphs based on results from test job executions. + - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance + graphs. + - **Test Environment**: performance test environment configuration. + - **Documentation**: documentation of K8s Pod/Container orchestration + in CSIT and pointers to CSIT source code documentation for VPP + performance tests. + +#. **DPDK Performance**: DPDK performance tests executed in physical + FD.io testbeds. + + - **Overview**: tested logical topologies, test coverage. + - **Release Notes**: changes in |csit-release|, known issues. + - **Packet Throughput**: NDR, PDR throughput graphs based on results + from repeated same test job executions to verify repeatibility of + measurements. + - **Packet Latency**: Latency graphs based on results from test job + executions. + - **Test Environment**: performance test environment configuration. + - **Documentation**: pointers to CSIT source code documentation for + DPDK performance tests. + +#. **VPP Functional**: VPP functional tests executed in virtual FD.io + testbeds. + + - **Overview**: tested virtual topologies, test coverage and naming + specifics; + - **Release Notes** - changes in |csit-release|, added tests, + environment or methodology changes, known issues. + - **Test Environment**: functional test environment configuration. + - **Documentation**: pointers to CSIT source code documentation for + VPP functional tests. + +#. **Honeycomb Functional**: Honeycomb functional tests executed in + virtual FD.io testbeds. + + - **Overview**: tested virtual topologies, test coverage and naming + specifics; + - **Release Notes** - changes in |csit-release|, known issues. + - **Test Environment**: functional test environment configuration. + - **Documentation**: pointers to CSIT source code documentation for + Honeycomb functional tests. + +#. **NSH_SFC Functional**: NSH_SFC functional tests executed in + virtual FD.io testbeds. + + - **Overview**: tested virtual topologies, test coverage and naming + specifics; + - **Release Notes** - changes in |csit-release|, known issues. + - **Test Environment**: functional test environment configuration. + - **Documentation**: pointers to CSIT source code documentation for + NSH_SFC functional tests. + +#. **DMM Functional**: DMM functional tests executed in + virtual FD.io testbeds. + + - **Overview**: tested virtual topologies, test coverage and naming + specifics; + - **Release Notes** - changes in |csit-release|, known issues. + - **Test Environment**: functional test environment configuration. + - **Documentation**: pointers to CSIT source code documentation for + DMM functional tests. + +#. **Detailed Results**: detailed result tables auto-generated from CSIT + test job executions using RF (Robot Framework) output files as + sources. + + - **VPP Performance NDR/PDR**: VPP NDR/PDR throughput and latency. + - **VPP Performance MRR**: VPP MRR throughput. + - **VPP K8s Container Memif**: VPP K8s Container/Pod topologies + NDR/PDR throughput. + - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput + and latency. + - **VPP Functional**: detailed VPP functional results. + - **Honeycomb Functional**: detailed HoneyComb functional results. + - **NSH_SFC Functional**: detailed nsh-plugin functional results. + - **DMM Functional**: detailed DMM functional results. + +#. **Test Configuration**: VPP DUT configuration data based on VPP API + Test (VAT) Commands History auto-generated from CSIT test job + executions using RF output files as sources. + + - **VPP Performance NDR/PDR**: configuration data. + - **VPP Performance MRR**: configuration data. + - **VPP K8s Container Memif**: configuration data. + - **VPP Functional**: configuration data. + +#. **Test Operational Data**: VPP DUT operational data auto-generated + from CSIT test job executions using RFoutput files as sources. + + - **VPP Performance NDR/PDR**: VPP `show run` outputs under test + load. + +#. **CSIT Framework Documentation**: description of the overall FD.io + CSIT framework. + + - **Design**: framework modular design hierarchy. + - **Test naming**: test naming convention. + - **Presentation and Analytics Layer**: description of PAL CSIT + analytics module. + - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and + test case grouping and selection. diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst deleted file mode 100644 index c3537a9f66..0000000000 --- a/docs/report/introduction/overview.rst +++ /dev/null @@ -1,93 +0,0 @@ -Overview -======== - -This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous -**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for -|csit-release| system testing of |vpp-release|. - -This is the full html version, there is also a reduced -`PDF version of this report`_. - -The report describes CSIT functional and performance tests and their -continuous execution delivered in |csit-release|. A high-level overview is -provided for each CSIT test environment running in :abbr:`LF (Linux Foundation)` -FD.io Continuous Performance Labs. This is followed by summary of all executed -tests against the |vpp-release| and associated FD.io projects and sub-systems -(Honeycomb, DPDK, NSH_SFC), |csit-release| release notes, result highlights and -known issues discovered in CSIT. More detailed description of each environment, -pointers to CSIT test code documentation and detailed test resuls with links to -the source data files are also provided. - -|csit-release| report contains following main sections and sub-sections: - -#. **Introduction** - general introduction to CSIT project; *Overview* - - this section; *General Notes* - general notes related to this report; - *Physical Testbeds* - Description of physical testbeds used in CSIT; - *Performance Test Methodology* - Methodologies used in CSIT. - -#. **VPP Performance** - VPP performance tests executed in physical - FD.io testbeds; *Overview* - tested topologies, test coverage and naming - specifics, methodology for multi-core, packet throughput and latency, and - KVM VM vhost tests; *CSIT Release Notes* - changes in |csit-release|, added - tests, performance changes, environment or methodology changes, known CSIT - issues; *Packet Throughput Graphs* and *Packet Latency - Graphs* - plotted NDR, PDR throughput and latency results from multiple - test job executions; *Throughput Speedup Multi-Core* - plotted core - configuration speedup comparision; *Test Environment* - environment - description; *VPP HTTP Server Performance Results* - plotted HTTP Server - performance; *Documentation* - CSIT source code documentation for VPP - performance tests. - -#. **DPDK Performance** - DPDK performance tests executed in - physical FD.io testbeds; *Overview* - tested topologies, test coverage; - *CSIT Release Notes* - changes in |csit-release|, any known CSIT issues; - *Packet Throughput Graphs* and *Packet Latency Graphs* - - plotted NDR, PDR throughput and latency results from multiple test job - executions; *Test Environment* - environment description; *Documentation* - - CSIT source code documentation for DPDK performance tests. - -#. **VPP Functional** - VPP functional tests executed in virtual - FD.io testbeds; *Overview* - tested virtual topologies, test coverage and - naming specifics; *CSIT Release Notes* - changes in |csit-release|, added - tests, environment or methodology changes, known CSIT issues, tests to be - added; *Test Environment* - environment description ; *Documentation* - - source code documentation for VPP functional tests. - -#. **Honeycomb Functional** - Honeycomb functional tests executed in - virtual FD.io testbeds; *Overview* - tested virtual topologies, test - coverage and naming specifics; *CSIT Release Notes* - changes in CSIT - |release|, added tests, environment or methodology changes, known CSIT issues; - *Test Environment* - environment description; - *Documentation* - source code documentation for Honeycomb functional tests. - -#. **NSH_SFC Functional** - NSH_SFC functional tests executed in - virtual FD.io testbeds; *Overview* - tested virtual topologies, test - coverage and naming specifics; *CSIT Release Notes* - changes in CSIT - |release|, added tests, environment or methodology changes, known CSIT issues; - *Test Environment* - environment description; - *Documentation* - source code documentation for NSH_SFC functional tests. - -#. **DMM Functional** - DMM functional tests executed in - virtual FD.io testbeds; *Overview* - tested virtual topologies, test - coverage and naming specifics; *CSIT Release Notes* - changes in CSIT - |release|, added tests, environment or methodology changes, known CSIT issues; - *Test Environment* - environment description; - *Documentation* - source code documentation for DMM functional tests. - -#. **Detailed Results** - auto-generated results from CSIT jobs - executions using CSIT Robot Framework output files as source data; *VPP - Performance Results*, *DPDK Performance Results*, *VPP Functional - Results*, *Honeycomb Functional Results*, *VPPtest Functional Results*. - -#. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs - executions using CSIT Robot Framework output files as source data; *VPP - Performance Test Configs*, *VPP Functional Test Configs*. - -#. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs - executions using CSIT Robot Framework output files as source data; *VPP - Performance Operational Data*. - -#. **CSIT Framework Documentation** - description of the overall CSIT - framework design hierarchy, CSIT test naming convention, followed by - description of Presentation and Analytics Layer (PAL) introduced in - CSIT-17.07 and description of CSIT RF Tags. diff --git a/docs/report/introduction/test_scenarios_overview.rst b/docs/report/introduction/test_scenarios_overview.rst new file mode 100644 index 0000000000..8b7266d0a4 --- /dev/null +++ b/docs/report/introduction/test_scenarios_overview.rst @@ -0,0 +1,86 @@ +Test Scenarios Overview +======================= + +FD.io CSIT report includes multiple test scenarios of a number of VPP +centric applications, topologies and use cases. In addition it also +covers baseline tests of DPDK sample applications. Tests are executed +executed in physical (performance tests) and virtual environments +(functional tests). + +Following list provides a brief overview of test scenarios covered in +this report: + +#. **VPP Performance**: VPP performance tests are executed in physical + FD.io testbeds, focusing on VPP network data plane performance in + NIC-to-NIC switching topologies. Tested across Intel Xeon Haswell + and Skylake servers, range of NICs (10GE, 25GE, 40GE) and multi- + thread/multi-core configurations. VPP application runs in bare-metal + host user-mode handling NICs. TRex is used as a traffic generator. + +#. **VPP Vhostuser Performance with KVM VMs**: VPP VM service switching + performance tests using vhostuser virtual interface for + interconnecting multiple Testpmd-in-VM instances. VPP vswitch + instance runs in bare-metal user-mode handling NICs and connecting + over vhost-user interfaces to VM instances each running DPDK + Testpmd with virtio virtual interfaces. Similarly to VPP + Performance, tests are run across a range of configurations. TRex + is used as a traffic generator. + +#. **VPP Memif Performance with LXC and Docker Containers**: VPP + Container service switching performance tests using memif virtual + interface for interconnecting multiple VPP-in-container instances. + VPP vswitch instance runs in bare-metal user-mode handling NICs and + connecting over memif (Slave side) interfaces to more instances of + VPP running in LXC or in Docker Containers, both with memif + interfaces (Master side). Similarly to VPP Performance, tests are + run across a range of configurations. TRex is used as a traffic + generator. + +#. **K8s Container/Pod Topologies Performance**: VPP container + performance tests using memif for interconnecting VPP-in- + Container/Pod instances orchestrated by K8s integrated with `Ligato + <https://github.com/ligato>`_ for container networking. TRex is + used as a traffic generator. + +#. **DPDK Performance**: VPP uses DPDK to drive the NICs and physical + interfaces. DPDK performance tests are used as a baseline to + profile performance of the DPDK sub-system. Two DPDK applications + are tested: Testpmd and L3fwd. DPDK tests are executed in the same + testing environment as VPP tests. DPDK Testpmd and L3fwd + applications run in host user-mode. TRex is used as a traffic + generator. + +#. **VPP Functional**: VPP functional tests are executed in virtual + FD.io testbeds, focusing on VPP packet processing functionality, + including both network data plane and in-line control plane. Tests + cover vNIC-to-vNIC vNIC-to-nestedVM-to-vNIC forwarding topologies. + Scapy is used as a traffic generator. + +#. **Honeycomb Functional**: Honeycomb functional tests are executed in + virtual FD.io testbeds, focusing on Honeycomb management and + programming functionality of VPP. Tests cover a range of CRUD + operations executed against VPP. + +#. **NSH_SFC Functional**: NSH_SFC functional tests are executed in + virtual FD.io testbeds focusing on VPP nsh-plugin data plane + functionality. Scapy is used as a traffic generator. + +#. **DMM Functional**: DMM functional tests are executed in virtual + FD.io testbeds demonstrating a single server (DUT1) and single + client (DUT2) scenario using DMM framework and Linux kernel TCP/IP + stack. + +All CSIT test results listed in this report are sourced and auto- +generated from :abbr:`RF (Robot Framework)` :file:`output.xml` files +resulting from :abbr:`LF (Linux Foundation)` FD.io Jenkins jobs executed +against |vpp-release| release artifacts. References are provided to the +original FD.io Jenkins job results. Additional references are provided +to the :abbr:`RF (Robot Framework)` result files that got archived in +FD.io Nexus online storage system. + +FD.io CSIT system is developed using two main coding platforms: :abbr:`RF (Robot +Framework)` and Python2.7. |csit-release| source code for the executed test +suites is available in CSIT branch |release| in the directory +:file:`./tests/<name_of_the_test_suite>`. A local copy of CSIT source code +can be obtained by cloning CSIT git repository - :command:`git clone +https://gerrit.fd.io/r/csit`. |