diff options
Diffstat (limited to 'docs/report')
| -rw-r--r-- | docs/report/detailed_test_results/index.rst | 1 | ||||
| -rw-r--r-- | docs/report/detailed_test_results/nshsfc_functional_results/index.rst | 14 | ||||
| -rw-r--r-- | docs/report/introduction/overview.rst | 164 |
3 files changed, 97 insertions, 82 deletions
diff --git a/docs/report/detailed_test_results/index.rst b/docs/report/detailed_test_results/index.rst index a609ef40d4..9a986bcd65 100644 --- a/docs/report/detailed_test_results/index.rst +++ b/docs/report/detailed_test_results/index.rst @@ -7,4 +7,5 @@ Detailed Test Results dpdk_performance_results/index vpp_functional_results/index honeycomb_functional_results/index + nshsfc_functional_results/index vpp_unittest_results/index diff --git a/docs/report/detailed_test_results/nshsfc_functional_results/index.rst b/docs/report/detailed_test_results/nshsfc_functional_results/index.rst new file mode 100644 index 0000000000..11c168a92c --- /dev/null +++ b/docs/report/detailed_test_results/nshsfc_functional_results/index.rst @@ -0,0 +1,14 @@ +NSH SFC Functional Results +========================== + +.. note:: + + Data sources for reported test results: i) FD.io test executor jobs + `###JOB### + <###LINK###>`_ + , ii) archived FD.io jobs test result `output files + <../../_static/archive/>`_. + +.. toctree:: + + nshsfc_functional_results diff --git a/docs/report/introduction/overview.rst b/docs/report/introduction/overview.rst index 7954686ad3..4aac352d0c 100644 --- a/docs/report/introduction/overview.rst +++ b/docs/report/introduction/overview.rst @@ -1,82 +1,82 @@ -Overview
-========
-
-This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous
-**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT
-|release| system testing of |vpp-release|.
-
-The report describes CSIT functional and performance tests and their
-continuous execution delivered in CSIT |release|. A high-level overview is
-provided for each CSIT test environment running in Linux Foundation (LF) FD.io
-Continuous Performance Labs. This is followed by summary of all executed tests
-against the |vpp-release| and associated FD.io projects and sub-systems
-(HoneyComb, DPDK), CSIT |release| release notes, result highlights and known
-issues discovered in CSIT. More detailed description of each environment,
-pointers to CSIT test code documentation and detailed test resuls with links to
-the source data files are also provided.
-
-CSIT |release| report contains following main sections and sub-sections:
-
-#. **Introduction** - general introduction to CSIT project; *Overview* -
- this section; *CSIT Test Naming* - CSIT general naming convention for test
- suites and test cases, important to recognize the high-level test content
- and interpret reported results; *General Notes* - general notes related to
- this report.
-
-#. **VPP Performance Tests** - VPP performance tests executed in physical
- FD.io testbeds; *Overview* - tested topologies, test coverage and naming
- specifics, methodology for multi-core, packet throughput and latency, and
- KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added
- tests, performance changes, environment or methodology changes, known CSIT
- issues; *Packet Throughput Graphs* and *Packet Latency
- Graphs* - plotted NDR, PDR throughput and latency results from multiple
- test job executions; *Test Environment* - environment description;
- *Documentation* - CSIT source code documentation for VPP performance tests.
-
-#. **DPDK Performance Tests** - DPDK performance tests executed in
- physical FD.io testbeds; *Overview* - tested topologies, test coverage;
- *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues;
- *Packet Throughput Graphs* and *Packet Latency Graphs*
- - plotted NDR, PDR throughput and latency results from multiple test job
- executions; *Test Environment* - environment description; *Documentation* -
- CSIT source code documentation for DPDK performance tests.
-
-#. **VPP Functional Tests** - VPP functional tests executed in virtual
- FD.io testbeds; *Overview* - tested virtual topologies, test coverage and
- naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added
- tests, environment or methodology changes, known CSIT issues, tests to be
- added; *Test Environment* - environment description ; *Documentation* -
- source code documentation for VPP functional tests.
-
-#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in
- virtual FD.io testbeds; *Overview* - tested virtual topologies, test
- coverage and naming specifics; *CSIT Release Notes* - changes in CSIT
- |release|, added tests, environment or methodology changes, known CSIT issues;
- *Test Environment* - environment description ;
- *Documentation* - source code documentation for Honeycomb functional tests.
-
-#. **VPP Unit Tests** - refers to VPP functional unit tests executed as
- part of vpp make test verify option within the FD.io VPP project; listed in
- this report to give a more complete view about executed VPP functional tests;
- *Overview* - short overview of unit test framework and executed tests;
- *Documentation* - source code documentation of VPP unit tests.
-
-#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in
- virtual FD.io testbeds; *Overview* - tested virtual topologies, test
- coverage and naming specifics; *CSIT Release Notes* - changes in CSIT
- |release|, added tests, environment or methodology changes, known CSIT issues;
- *Test Environment* - environment description ;
- *Documentation* - source code documentation for NSH_SFC functional tests.
-
-#. **Detailed Test Results** - auto-generated results from CSIT jobs
- executions using CSIT Robot Framework output files as source data; *VPP
- Performance Results*, *DPDK Performance Results*, *VPP Functional
- Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*.
-
-#. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs
- executions using CSIT Robot Framework output files as source data; *VPP
- Performance Test Configs*, *VPP Functional Test Configs*.
-
-#. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs
- executions using CSIT Robot Framework output files as source data; *VPP
- Performance Operational Data*.
+Overview +======== + +This is the **F**\ast **D**\ata **I**/**O** Project (**FD.io**) **C**\ontinuous +**S**\ystem **I**\ntegration and **T**\esting (**CSIT**) project report for CSIT +|release| system testing of |vpp-release|. + +The report describes CSIT functional and performance tests and their +continuous execution delivered in CSIT |release|. A high-level overview is +provided for each CSIT test environment running in Linux Foundation (LF) FD.io +Continuous Performance Labs. This is followed by summary of all executed tests +against the |vpp-release| and associated FD.io projects and sub-systems +(HoneyComb, DPDK, NSH_SFC), CSIT |release| release notes, result highlights and +known issues discovered in CSIT. More detailed description of each environment, +pointers to CSIT test code documentation and detailed test resuls with links to +the source data files are also provided. + +CSIT |release| report contains following main sections and sub-sections: + +#. **Introduction** - general introduction to CSIT project; *Overview* - + this section; *CSIT Test Naming* - CSIT general naming convention for test + suites and test cases, important to recognize the high-level test content + and interpret reported results; *General Notes* - general notes related to + this report. + +#. **VPP Performance Tests** - VPP performance tests executed in physical + FD.io testbeds; *Overview* - tested topologies, test coverage and naming + specifics, methodology for multi-core, packet throughput and latency, and + KVM VM vhost tests; *CSIT Release Notes* - changes in CSIT |release|, added + tests, performance changes, environment or methodology changes, known CSIT + issues; *Packet Throughput Graphs* and *Packet Latency + Graphs* - plotted NDR, PDR throughput and latency results from multiple + test job executions; *Test Environment* - environment description; + *Documentation* - CSIT source code documentation for VPP performance tests. + +#. **DPDK Performance Tests** - DPDK performance tests executed in + physical FD.io testbeds; *Overview* - tested topologies, test coverage; + *CSIT Release Notes* - changes in CSIT |release|, any known CSIT issues; + *Packet Throughput Graphs* and *Packet Latency Graphs* + - plotted NDR, PDR throughput and latency results from multiple test job + executions; *Test Environment* - environment description; *Documentation* - + CSIT source code documentation for DPDK performance tests. + +#. **VPP Functional Tests** - VPP functional tests executed in virtual + FD.io testbeds; *Overview* - tested virtual topologies, test coverage and + naming specifics; *CSIT Release Notes* - changes in CSIT |release|, added + tests, environment or methodology changes, known CSIT issues, tests to be + added; *Test Environment* - environment description ; *Documentation* - + source code documentation for VPP functional tests. + +#. **HoneyComb Functional Tests** - HoneyComb functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for Honeycomb functional tests. + +#. **VPP Unit Tests** - refers to VPP functional unit tests executed as + part of vpp make test verify option within the FD.io VPP project; listed in + this report to give a more complete view about executed VPP functional tests; + *Overview* - short overview of unit test framework and executed tests; + *Documentation* - source code documentation of VPP unit tests. + +#. **NSH_SFC Functional Tests** - NSH_SFC functional tests executed in + virtual FD.io testbeds; *Overview* - tested virtual topologies, test + coverage and naming specifics; *CSIT Release Notes* - changes in CSIT + |release|, added tests, environment or methodology changes, known CSIT issues; + *Test Environment* - environment description ; + *Documentation* - source code documentation for NSH_SFC functional tests. + +#. **Detailed Test Results** - auto-generated results from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Results*, *DPDK Performance Results*, *VPP Functional + Results*, *HoneyComb Functional Results*, *VPPtest Functional Results*. + +#. **Test Configuration** - auto-generated DUT configuration data from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Test Configs*, *VPP Functional Test Configs*. + +#. **Test Operational Data** - auto-generated DUT operational data from CSIT jobs + executions using CSIT Robot Framework output files as source data; *VPP + Performance Operational Data*. |