aboutsummaryrefslogtreecommitdiffstats
path: root/docs/report/introduction/introduction.rst
blob: d7fc0610516c83365ab36a188e7dc43a71492eb5 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
Report Structure
================

FD.io |csit-release| report contains system performance and functional
testing data of |vpp-release|. `PDF version of this report`_ is
available for download.

|csit-release| report is structured as follows:

#. INTRODUCTION: General introduction to FD.io |csit-release|.

   - **Introduction**: This section.
   - **Test Scenarios Overview**: A brief overview of test scenarios
     covered in this report.
   - **Physical Testbeds**: Description of physical testbeds.
   - **Test Methodology**: Performance benchmarking and functional test
     methodologies.

#. VPP PERFORMANCE: VPP performance tests executed in physical
   FD.io testbeds.

   - **Overview**: Tested logical topologies, test coverage and naming
     specifics.
   - **Release Notes**: Changes in |csit-release|, added tests,
     environment or methodology changes, known issues.
   - **Packet Throughput**: NDR, PDR throughput graphs based on results
     from repeated same test job executions to verify repeatibility of
     measurements.
   - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup
     graphs based on results from test job executions.
   - **Packet Latency**: Latency graphs based on results from test job
     executions.
   - **Soak Tests**: Long duration soak tests are executed using PLRsearch
     algorithm.
   - **NFV Service Density**: Network Function Virtualization (NFV) service
     density tests focus on measuring total per server throughput at varied NFV
     service "packing" densities with vswitch providing host dataplane.
   - **Comparisons**: Performance comparisons between VPP releases and
     between different testbed types.
   - **Throughput Trending**: References to continuous VPP performance
     trending.
   - **Test Environment**: Performance test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for VPP
     performance tests.

#. DPDK PERFORMANCE: DPDK performance tests executed in physical
   FD.io testbeds.

   - **Overview**: Tested logical topologies, test coverage.
   - **Release Notes**: Changes in |csit-release|, known issues.
   - **Packet Throughput**: NDR, PDR throughput graphs based on results
     from repeated same test job executions to verify repeatibility of
     measurements.
   - **Packet Latency**: Latency graphs based on results from test job
     executions.
   - **Comparisons**: Performance comparisons between DPDK releases and
     between different testbed types.
   - **Throughput Trending**: References to regular DPDK performance
     trending.
   - **Test Environment**: Performance test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     DPDK performance tests.

#. VPP DEVICE: VPP functional tests executed in physical FD.io
   testbeds using containers.

   - **Overview**: Tested virtual topologies, test coverage and naming
     specifics;
   - **Release Notes**: Changes in |csit-release|, added tests,
     environment or methodology changes, known issues.
   - **Integration Tests**: Functional test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     VPP functional tests.

#. DETAILED RESULTS: Detailed result tables auto-generated from CSIT
   test job executions using RF (Robot Framework) output files as
   sources.

   - **VPP Performance NDR/PDR**: VPP NDR/PDR throughput and latency.
   - **VPP Performance MRR**: VPP MRR throughput.
   - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput
     and latency.

#. TEST CONFIGURATION: VPP DUT configuration data based on VPP API
   Test (VAT) Commands History auto-generated from CSIT test job
   executions using RF output files as sources.

   - **VPP Performance NDR/PDR**: Configuration data.
   - **VPP Performance MRR**: Configuration data.

#. TEST OPERATIONAL DATA: VPP DUT operational data auto-generated
   from CSIT test job executions using RFoutput files as sources.

   - **VPP Performance NDR/PDR**: VPP `show run` outputs under test
     load.

#. CSIT FRAMEWORK DOCUMENTATION: Description of the overall FD.io
   CSIT framework.

   - **Design**: Framework modular design hierarchy.
   - **Test naming**: Test naming convention.
   - **Presentation and Analytics Layer**: Description of PAL CSIT
     analytics module.
   - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and
     test case grouping and selection.

..
    #. DMM FUNCTIONAL: DMM functional tests executed in
       virtual FD.io testbeds.

       - **Overview**: Tested virtual topologies, test coverage and naming
         specifics;
       - **Release Notes**: Changes in |csit-release|, known issues.
       - **Test Environment**: Functional test environment configuration.
       - **Documentation**: Pointers to CSIT source code documentation for
         DMM functional tests.