From da2aa127ee071e37bc4d6060dd39476179e65365 Mon Sep 17 00:00:00 2001 From: Maciek Konstantynowicz Date: Sun, 5 Feb 2017 16:33:28 +0000 Subject: csit rls1701 report edits: - simplified section structure for clarity and readability, - updated overview sections, - moved not rls related content from rls_notes to overview sections, - removed section title suffixes: HW, VIRL. - completed vpp_unit_tests and vpp_unittest_results sections. - updated all documentation sections. - updated rls_notes sections for vpp performance and vpp functional. Change-Id: Id2c2abbf9d3531ec9f63ecd353f385a0b55ae1ba Signed-off-by: Maciek Konstantynowicz Signed-off-by: pmikus Signed-off-by: Maciek Konstantynowicz Signed-off-by: pmikus Signed-off-by: Maciek Konstantynowicz --- docs/report/vpp_performance_tests/overview.rst | 325 +++++++++++++++++++++++++ 1 file changed, 325 insertions(+) create mode 100644 docs/report/vpp_performance_tests/overview.rst (limited to 'docs/report/vpp_performance_tests/overview.rst') diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst new file mode 100644 index 0000000000..d0ce5e7bcf --- /dev/null +++ b/docs/report/vpp_performance_tests/overview.rst @@ -0,0 +1,325 @@ +Overview +======== + +Tested Physical Topologies +-------------------------- + +CSIT VPP performance tests are executed on physical baremetal servers hosted by LF +FD.io project. Testbed physical topology is shown in the figure below. + +:: + + +------------------------+ +------------------------+ + | | | | + | +------------------+ | | +------------------+ | + | | | | | | | | + | | <-----------------> | | + | | DUT1 | | | | DUT2 | | + | +--^---------------+ | | +---------------^--+ | + | | | | | | + | | SUT1 | | SUT2 | | + +------------------------+ +------------------^-----+ + | | + | | + | +-----------+ | + | | | | + +------------------> TG <------------------+ + | | + +-----------+ + +SUT1 and SUT2 are two System Under Test servers (Cisco UCS C240, each with two +Intel XEON CPUs), TG is a Traffic Generator (TG, another Cisco UCS C240, with +two Intel XEON CPUs). SUTs run VPP SW application in Linux user-mode as a +Device Under Test (DUT). TG runs TRex SW application as a packet Traffic +Generator. Physical connectivity between SUTs and to TG is provided using +different NIC models that need to be tested for performance. Currently +installed and tested NIC models include: + +#. 2port10GE X520-DA2 Intel. +#. 2port10GE X710 Intel. +#. 2port10GE VIC1227 Cisco. +#. 2port40GE VIC1385 Cisco. +#. 2port40GE XL710 Intel. + +From SUT and DUT perspective, all performance tests involve forwarding packets +between two physical Ethernet ports (10GE or 40GE). Due to the number of +listed NIC models tested and available PCI slot capacity in SUT servers, in +all of the above cases both physical ports are located on the same NIC. In +some test cases this results in measured packet throughput being limited not +by VPP DUT but by either the physical interface or the NIC capacity. + +Going forward CSIT project will be looking to add more hardware into FD.io +performance labs to address larger scale multi-interface and multi-NIC +performance testing scenarios. + +For test cases that require DUT (VPP) to communicate with VM over vhost-user +interfaces, a VM is created on SUT1 and SUT2. DUT (VPP) test topology with VM +is shown in the figure below including applicable packet flow thru the VM +(marked in the figure with ``***``). + +:: + + +------------------------+ +------------------------+ + | +----------+ | | +----------+ | + | | VM | | | | VM | | + | | ****** | | | | ****** | | + | +--^----^--+ | | +--^----^--+ | + | *| |* | | *| |* | + | +------v----v------+ | | +------v----v------+ | + | | * * |**|***********|**| * * | | + | | ***** *******<----------------->******* ***** | | + | | * DUT1 | | | | DUT2 * | | + | +--^---------------+ | | +---------------^--+ | + | *| | | |* | + | *| SUT1 | | SUT2 |* | + +------------------------+ +------------------^-----+ + *| |* + *| |* + *| +-----------+ |* + *| | | |* + *+------------------> TG <------------------+* + ******************* | |******************** + +-----------+ + +For VM tests, packets are switched by DUT (VPP) twice, hence the +throughput rates measured by TG (and listed in this report) must be multiplied +by two to represent the actual DUT aggregate packet forwarding rate. + +Note that reported VPP performance results are specific to the SUT tested. +Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with +other CPUs are likely to yield different results. A good rule of thumb, that +can be applied to estimate VPP packet thoughput for Phy-to-Phy (NIC-to-NIC, +PCI-to-PCI) topology, is to expect the forwarding performance to be +proportional to CPU core frequency, assuming CPU is the only limiting factor +and all other SUT aspects equal to FD.io CSIT environment. The same rule of +thumb can be also applied for Phy-to-VM-to-Phy (NIC-to-VM-to-NIC) topology, +but due to much higher dependency on very high frequency memory operations and +sensitivity to Linux kernel scheduler settings and behaviour, this estimation +may not always yield good enough accuracy. + +Detailed LF FD.io test bed specification and physical topology are described +in `wiki CSIT LF FDio testbed `_. + +Performance Tests Coverage +-------------------------- + +Performance tests are split into the two main categories: + +- Throughput discovery - discovery of packet forwarding rate using binary search + in accordance to RFC2544. + + - NDR - discovery of Non Drop Rate packet throughput, at zero packet loss; + followed by packet one-way latency measurements at 10%, 50% and 100% of + discovered NDR throughput. + - PDR - discovery of Partial Drop Rate, with specified non-zero packet loss + currently set to 0.5%; followed by packet one-way latency measurements at + 100% of discovered PDR throughput. + +- Throughput verification - verification of packet forwarding rate against + previously discovered throughput rate. These tests are currently done against + 0.9 of reference NDR, with reference rates updated periodically. + +CSIT |release| includes following performance test suites, listed per NIC type: + +- 2port10GE X520-DA2 Intel + + - **L2XC** - L2 Cross-Connect switched-forwarding of untagged, dot1q, dot1ad + VLAN tagged Ethernet frames. + - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames + with MAC learning; disabled MAC learning i.e. static MAC tests to be added. + - **IPv4** - IPv4 routed-forwarding. + - **IPv6** - IPv6 routed-forwarding. + - **IPv4 Scale** - IPv4 routed-forwarding with 20k, 200k and 2M FIB entries. + - **IPv6 Scale** - IPv6 routed-forwarding with 20k, 200k and 2M FIB entries. + - **VM with vhost-user** - switching between NIC ports and VM over vhost-user + interfaces in different switching modes incl. L2 Cross-Connect, L2 + Bridge-Domain, VXLAN with L2BD, IPv4 routed-forwarding. + - **COP** - IPv4 and IPv6 routed-forwarding with COP address security. + - **iACL** - IPv4 and IPv6 routed-forwarding with iACL address security. + - **LISP** - LISP overlay tunneling for IPv4-over-IPV4, IPv6-over-IPv4, + IPv6-over-IPv6, IPv4-over-IPv6 in IPv4 and IPv6 routed-forwarding modes. + - **VXLAN** - VXLAN overlay tunnelling integration with L2XC and L2BD. + - **QoS Policer** - ingress packet rate measuring, marking and limiting + (IPv4). + +- 2port40GE XL710 Intel + + - **L2XC** - L2 Cross-Connect switched-forwarding of untagged Ethernet frames. + - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames + with MAC learning. + - **IPv4** - IPv4 routed-forwarding. + - **IPv6** - IPv6 routed-forwarding. + - **VM with vhost-user** - switching between NIC ports and VM over vhost-user + interfaces in different switching modes incl. L2 Bridge-Domain. + +- 2port10GE X710 Intel + + - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames + with MAC learning. + - **VM with vhost-user** - switching between NIC ports and VM over vhost-user + interfaces in different switching modes incl. L2 Bridge-Domain. + +- 2port10GE VIC1227 Cisco + + - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames + with MAC learning. + +- 2port40GE VIC1385 Cisco + + - **L2BD** - L2 Bridge-Domain switched-forwarding of untagged Ethernet frames + with MAC learning. + +Execution of performance tests takes time, especially the throughput discovery +tests. Due to limited HW testbed resources available within FD.io labs hosted +by Linux Foundation, the number of tests for NICs other than X520 (a.k.a. +Niantic) has been limited to few baseline tests. Over time we expect the HW +testbed resources to grow, and will be adding complete set of performance +tests for all models of hardware to be executed regularly and(or) +continuously. + +Performance Tests Naming +------------------------ + +CSIT |release| introduced a common structured naming convention for all +performance and functional tests. This change was driven by substantially +growing number and type of CSIT test cases. Firstly, the original practice did +not always follow any strict naming convention. Secondly test names did not +always clearly capture tested packet encapsulations, and the actual type or +content of the tests. Thirdly HW configurations in terms of NICs, ports and +their locality were not captured either. These were but few reasons that drove +the decision to change and define a new more complete and stricter test naming +convention, and to apply this to all existing and new test cases. + +The new naming should be intuitive for majority of the tests. The complete +description of CSIT test naming convention is provided on `CSIT test naming wiki +`_. + +Here few illustrative examples of the new naming usage for performance test +suites: + +#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** + + - *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* + - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on + Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching + with MAC learning, NDR throughput discovery. + - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE + on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline + switching with MAC learning, NDR throughput discovery. + - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel + x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. + - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on + Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput + discovery. + +#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, + P2V2P, NIC2VMchain2NIC, P2V2V2P** + + - *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- + VirtPortConfig-VMconfig-TestType* + - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports + of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain + switching to/from two vhost interfaces and one VM, NDR throughput + discovery. + - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from two vhost interfaces and one VM, NDR throughput + discovery. + - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from four vhost interfaces and two VMs, NDR throughput + discovery. + +Methodology: Multi-Thread and Multi-Core +---------------------------------------- + +**HyperThreading** - CSIT |release| performance tests are executed with SUT +servers' Intel XEON CPUs configured in HyperThreading Disabled mode (BIOS +settings). This is the simplest configuration used to establish baseline +single-thread single-core SW packet processing and forwarding performance. +Subsequent releases of CSIT will add performance tests with Intel +HyperThreading Enabled (requires BIOS settings change and hard reboot). + +**Multi-core Test** - CSIT |release| multi-core tests are executed in the +following VPP thread and core configurations: + +#. 1t1c - 1 VPP worker thread on 1 CPU physical core. +#. 2t2c - 2 VPP worker threads on 2 CPU physical cores. +#. 4t4c - 4 VPP threads on 4 CPU physical cores. + +Note that in quite a few test cases running VPP on 2 or 4 physical cores hits +the tested NIC I/O bandwidth or packets-per-second limit. + +Methodology: Packet Throughput +------------------------------ + +Following values are measured and reported for packet throughput tests: + +- NDR binary search per RFC2544: + + - Packet rate: "RATE: pps + (2x )" + - Aggregate bandwidth: "BANDWIDTH: Gbps (untagged)" + +- PDR binary search per RFC2544: + + - Packet rate: "RATE: pps (2x + )" + - Aggregate bandwidth: "BANDWIDTH: Gbps (untagged)" + - Packet loss tolerance: "LOSS_ACCEPTANCE "" + +- NDR and PDR are measured for the following L2 frame sizes: + + - IPv4: 64B, IMIX_v4_1 (28x64B,16x570B,4x1518B), 1518B, 9000B. + - IPv6: 78B, 1518B, 9000B. + + +Methodology: Packet Latency +--------------------------- + +TRex Traffic Generator (TG) is used for measuring latency of VPP DUTs. Reported +latency values are measured using following methodology: + +- Latency tests are performed at 10%, 50% of discovered NDR rate (non drop rate) + for each NDR throughput test and packet size (except IMIX). +- TG sends dedicated latency streams, one per direction, each at the rate of + 10kpps at the prescribed packet size; these are sent in addition to the main + load streams. +- TG reports min/avg/max latency values per stream direction, hence two sets + of latency values are reported per test case; future release of TRex is + expected to report latency percentiles. +- Reported latency values are aggregate across two SUTs due to three node + topology used for all performance tests; for per SUT latency, reported value + should be divided by two. +- 1usec is the measurement accuracy advertised by TRex TG for the setup used in + FD.io labs used by CSIT project. +- TRex setup introduces an always-on error of about 2*2usec per latency flow - + additonal Tx/Rx interface latency induced by TRex SW writing and reading + packet timestamps on CPU cores without HW acceleration on NICs closer to the + interface line. + + +Methodology: KVM VM vhost +------------------------- + +CSIT |release| introduced environment configuration changes to KVM Qemu vhost- +user tests in order to more representatively measure VPP-17.01 performance in +configurations with vhost-user interfaces and VMs. + +Current setup of CSIT FD.io performance lab is using tuned settings for more +optimal performance of KVM Qemu: + +- Default Qemu virtio queue size of 256 descriptors. +- Adjusted Linux kernel CFS scheduler settings, as detailed on this CSIT wiki + page: https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604. + +Adjusted Linux kernel CFS settings make the NDR and PDR throughput performance +of VPP+VM system less sensitive to other Linux OS system tasks by reducing +their interference on CPU cores that are designated for critical software +tasks under test, namely VPP worker threads in host and Testpmd threads in +guest dealing with data plan. -- cgit 1.2.3-korg