diff options
Diffstat (limited to 'docs/report')
5 files changed, 49 insertions, 68 deletions
diff --git a/docs/report/introduction/introduction.rst b/docs/report/introduction/introduction.rst index c15df8d9bc..80f926e83a 100644 --- a/docs/report/introduction/introduction.rst +++ b/docs/report/introduction/introduction.rst @@ -26,10 +26,15 @@ available for download. - **Packet Throughput**: NDR, PDR throughput graphs based on results from repeated same test job executions to verify repeatibility of measurements. - - **Packet Latency**: Latency graphs based on results from test job - executions. - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup graphs based on results from test job executions. + - **Packet Latency**: Latency graphs based on results from test job + executions. + - **Soak Tests**: Long duration soak tests are executed using PLRsearch + algorithm. + - **NFV Service Density**: Network Function Virtualization (NFV) service + density tests focus on measuring total per server throughput at varied NFV + service "packing" densities with vswitch providing host dataplane. - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance graphs. - **Comparisons**: Performance comparisons between VPP releases and @@ -59,6 +64,17 @@ available for download. - **Documentation**: Pointers to CSIT source code documentation for DPDK performance tests. +#. VPP DEVICE: VPP functional tests executed in physical FD.io + testbeds using containers. + + - **Overview**: Tested virtual topologies, test coverage and naming + specifics; + - **Release Notes**: Changes in |csit-release|, added tests, + environment or methodology changes, known issues. + - **Integration Tests**: Functional test environment configuration. + - **Documentation**: Pointers to CSIT source code documentation for + VPP functional tests. + #. VPP FUNCTIONAL: VPP functional tests executed in virtual FD.io testbeds. @@ -70,7 +86,7 @@ available for download. - **Documentation**: Pointers to CSIT source code documentation for VPP functional tests. -#. HONEYCOMBE FUNCTIONAL: Honeycomb functional tests executed in +#. HONEYCOMB FUNCTIONAL: Honeycomb functional tests executed in virtual FD.io testbeds. - **Overview**: Tested virtual topologies, test coverage and naming diff --git a/docs/report/introduction/methodology_k8s_container_memif.rst b/docs/report/introduction/methodology_k8s_container_memif.rst index 2e5ce0e017..ec7a3c9ccc 100644 --- a/docs/report/introduction/methodology_k8s_container_memif.rst +++ b/docs/report/introduction/methodology_k8s_container_memif.rst @@ -1,7 +1,7 @@ K8s Container Memif ------------------- -|csit-release| includes tests of VPP topologies running in K8s +CSIT includes tests of VPP topologies running in K8s orchestrated Pods/Containers and connected over memif virtual interfaces. In order to provide simple topology coding flexibility and extensibility container orchestration is done with `Kubernetes diff --git a/docs/report/introduction/methodology_kvm_vms_vhost_user.rst b/docs/report/introduction/methodology_kvm_vms_vhost_user.rst index 34e3bc0447..0a0ea1be1e 100644 --- a/docs/report/introduction/methodology_kvm_vms_vhost_user.rst +++ b/docs/report/introduction/methodology_kvm_vms_vhost_user.rst @@ -4,14 +4,13 @@ KVM VMs vhost-user FD.io CSIT performance lab is testing VPP vhost with KVM VMs using following environment settings: -- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr256] - default 256 descriptors, [vr1024] 1024 descriptors to optimize for - packet throughput. +- Tests with varying Qemu virtio queue (a.k.a. vring) sizes: [vr1024] 1024 + descriptors to optimize for packet throughput. - Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)` settings: [cfs] default settings, [cfsrr1] CFS RoundRobin(1) policy applied to all data plane threads handling test packet path including all VPP worker threads and all Qemu testpmd poll-mode threads. -- Resulting test cases are all combinations with [vr256,vr1024] and +- Resulting test cases are all combinations with [vr1024] and [cfs,cfsrr1] settings. - Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)` scheduler policy for data plane threads used in CSIT is documented in diff --git a/docs/report/introduction/methodology_lxc_drc_container_memif.rst b/docs/report/introduction/methodology_lxc_drc_container_memif.rst index 56f052550c..125a71ccc7 100644 --- a/docs/report/introduction/methodology_lxc_drc_container_memif.rst +++ b/docs/report/introduction/methodology_lxc_drc_container_memif.rst @@ -1,7 +1,7 @@ LXC/DRC Container Memif ----------------------- -|csit-release| includes tests taking advantage of VPP memif virtual +CSIT includes tests taking advantage of VPP memif virtual interface (shared memory interface) to interconnect VPP running in Containers. VPP vswitch instance runs in bare-metal user-mode handling NIC interfaces and connecting over memif (Slave side) to VPPs running in diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 5d87b85c74..50dce62687 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -6,36 +6,13 @@ Changes in |csit-release| #. VPP PERFORMANCE TESTS - - **BMRR Throughput**: MRR (Maximum Receive Rate) test code has been - updated with a configurable burst MRR parameters: trial duration - and number of trials in a single burst. Enables a new Burst MRR - (BMRR for short) methodology for more precise performance - trending. See updated :ref:`test_methodology` section - for more details. - - - **2n-skx tests**: Added performamce tests for 2n-skx (2-Node Xeon - Skylake) testbeds: focus on baseline and scale tests, including - VM vhost and Container memif tests. - - - **3n-skx tests**: Added performamce tests for 3n-skx (3-Node Xeon - Skylake) testbeds: VM vhost and Container memif tests. - - - **VXLAN Scale Tests**: Added performamce tests for VXLAN scale with - dot1q and VPP L2BD. - - - **AVF Driver Tests**: Added performamce tests for i40e AVF driver - on VPP, no DPDK required. - - - **QAT**: Fixed reoccuring issues with QAT crypto accelerator cards. - - - **VM Vhost Virtio Params Combinations**: Added performance tests - for VM vhost with different virtio parameters combinations: - indirect buffers, mergeable buffers. - - - **K8s/Ligato in Trending**: Added K8s/Ligato Container memif tests - to daily trending. - -#. TEST FRAMEWORK + - **Service density 2n-skx tests**: Network Function Virtualization (NFV) + service density tests focus on measuring total per server throughput at + varied NFV service *packing* densities with vswitch providing host + dataplane. The goal is to compare and contrast performance of a shared + vswitch for different network topologies and virtualization technologies, + and their impact on vswitch performance and efficiency in a range of NFV + service configurations. - **Experimental Soak Tests**: Added performamce soak tests framework code for extended time duration tests and throughput discovery @@ -43,23 +20,14 @@ Changes in |csit-release| days, weeks, months, years. See updated :ref:`test_methodology` section for more details. - - **Trending Tests BMRR**: Used new Burst MRR (BMRR) tests for daily - trending. - - - **Per VPP Patch Performance Checks**: Per VPP gerrit patch vs. - parent performance tests, anomaly detection and no verify voting - (-1/0/+1) yet. Manual trigger only. Not "marketed" to FD.io - community yet to avoid excessive LFN FD.io physical performance - testbed blocking. +#. TEST FRAMEWORK - - **Patch-on-Patch Infra**: Added capability to run performance tests - using CSIT gerrit patch code testing VPP gerrit patch code, i.e. - before any VPP and/or CSIT code is merged into git branch. + - **Container code optimizations**: Optimized container library allows to + run containre_memif tests faster. - - **CSIT PAPI Support**: Initial implementation of PAPI L1 KWs in - CSIT using VPP Python bindings. Required for migraing away from - VAT. Very few L1 KWs implemented ("show version", "show - interfaces"). + - **CSIT PAPI Support**: Continue converting existing VAT L1 keywords to + PAPI L1 KWs in CSIT using VPP Python bindings. Required for migrating away + from VAT. - **General Code Housekeeping**: Ongoing RF keywords optimizations, removal of redundant RF keywords. @@ -68,11 +36,7 @@ Changes in |csit-release| - **Graphs Layout Improvements**: Improved performance graphs layout for better readibility and maintenance: test grouping, axis - labels, descriptions, other informative decoration. Master report - generated. 744 graphs(!) - - - **Performance Trending**: Further improvements of continuous - performance trending, anomaly detection and analysis. + labels, descriptions, other informative decoration. #. MISCELLANEOUS @@ -100,16 +64,18 @@ List of known issues in |csit-release| for VPP performance tests: | 1 | `CSIT-570 | Sporadic (1 in 200) NDR discovery test failures on x520. DPDK reporting rx-errors, indicating L1 issue. | | | <https://jira.fd.io/browse/CSIT-570>`_ | Suspected issue with HW combination of X710-X520 in LF testbeds. Not observed outside of LF testbeds. | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 2 | `CSIT-1234 | VPP IPSecHW scale interface mode 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | <https://jira.fd.io/browse/CSIT-1234>`_ | ip4ipsecscale1000tnl-ip4base-int 1core CSIT-18.07/18.04 relative change: NDR -31%, PDR -32%, MRR -38%. | +| 2 | `VPP-1562 | Link bonding (mode LACP, transmit policy l34) test are failing due to VPP crashing producing core dump. | +| | <https://jira.fd.io/browse/VPP-1562>`_ | | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 3 | `CSIT-1242 | VPP xl710 ip4base test 1core, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-18.07 vs. CSIT-18.04. | -| | <https://jira.fd.io/browse/CSIT-1242>`_ | xl710 ip4base 1core CSIT-18.07/18.04 relative change: NDR -29%, high stdev. | +| 3 | `CSIT-1427 | Scale HW IPsec are failing due to issue with selecting ipsec backend in VPP. | +| | <https://jira.fd.io/browse/CSIT-1427>`_ | | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 4 | `CSIT-1243 | VPP nat44 base test 2core, low NDR and PDR 64B throughput in 3n-skx testbeds, compared to 3n-hsw testbeds. | -| | <https://jira.fd.io/browse/CSIT-1243>`_ | ip4base-nat44 2core 3n-skx/3n-hsw relative change: NDR -19%, PDR -22%. | +| 4 | `VPP-1563 | AVF L2patch tests are failing for all packet size and core combination. Reason: null-node blackholed packets in show error. | +| | <https://jira.fd.io/browse/VPP-1563>`_ | | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-1246 | Ligato K8S orchestrated tests are failing due to incompatibility of the latest released Ligato vpp-agent with VPP-18.07. | -| | <https://jira.fd.io/browse/CSIT-1246>`_ | Past vpp-agent releases are not compatible either. | +| 5 | `CSIT-1234 | VPP IPSecHW/IPSecsW scale interface mode, low NDR and PDR 64B throughput in 3n-hsw testbeds, in CSIT-19.01 vs. CSIT-18.10. | +| | <https://jira.fd.io/browse/CSIT-1234>`_ | | ++----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ +| 6 | `CSIT-1428 | VM vhost tests with RXQ more than 2 are failing on 3n-skx. Suspecting RXQ configuration issues. | +| | <https://jira.fd.io/browse/CSIT-1428>`_ | | +----+-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------+ - |