From 9c459a2b303eb8d776ac040bc82830d850a67623 Mon Sep 17 00:00:00 2001 From: pmikus Date: Tue, 5 May 2020 12:30:44 +0000 Subject: Report: Infra update Signed-off-by: pmikus Change-Id: Ie39585e88460d9a5e3764726d93dc065f5cbacfb --- .../dpdk_performance_tests/csit_release_notes.rst | 52 ++------- docs/report/dpdk_performance_tests/overview.rst | 9 +- .../dpdk_performance_tests/test_environment.rst | 17 ++- .../introduction/methodology_mellanox_nic.rst | 129 +++++++++++++++++++++ .../methodology_trex_traffic_generator.rst | 37 ++++-- docs/report/introduction/physical_testbeds.rst | 8 +- .../introduction/test_environment_sut_conf_1.rst | 6 +- docs/report/introduction/test_environment_tg.rst | 35 ++++-- .../vpp_performance_tests/test_environment.rst | 17 ++- 9 files changed, 217 insertions(+), 93 deletions(-) create mode 100644 docs/report/introduction/methodology_mellanox_nic.rst (limited to 'docs/report') diff --git a/docs/report/dpdk_performance_tests/csit_release_notes.rst b/docs/report/dpdk_performance_tests/csit_release_notes.rst index 0c207e5a52..ceacd89eb3 100644 --- a/docs/report/dpdk_performance_tests/csit_release_notes.rst +++ b/docs/report/dpdk_performance_tests/csit_release_notes.rst @@ -6,54 +6,19 @@ Changes in |csit-release| #. DPDK PERFORMANCE TESTS - - **Intel Xeon 2n-skx, 3n-skx testbeds**: Testpmd and L3fwd - performance test data is not included in this report version. - This is due to the lower performance and behaviour inconsistency - of these systems following the upgrade of processor microcode - packages (skx ucode 0x2000064) as part of updating Ubuntu 18.04 - LTS kernel version. Tested VPP and DPDK applications (L3fwd) are - affected. Skx test data will be added in subsequent maintenance - report version(s) once the issue is resolved. See - :ref:`dpdk_known_issues`. + - Refactor of CSIT initialization helper scripts for DPDK testpmd + and l3fwd. Removing obsolete parameters from DPDK testpmd and l3fwd + command line. - - **Intel Xeon 2n-clx testbeds**: DPDK performance test data is now - included in this report, after resolving the issue of lower - performance and behaviour inconsistency of these systems due to - the Linux kernel driven upgrade of processor microcode packages - to 0x500002c. The resolution is to use latest SuperMicro BIOS 3.2 - (for X11DPG-QT motherboards used) that upgrades processor - microcode to 0x500002c, AND NOT kernel provided ucode package as - it does put THE system into sub-optimal state. + - Fixed 9000B L2 packet size tests not passing for all NICs. + - Fixed DPDK compilation and test initialization of Mellanox NICs using + CONFIG_RTE_LIBRTE_MLX5_PMD=y in compile configuration. #. DPDK RELEASE VERSION CHANGE - |csit-release| tested |dpdk-release|, as used by |vpp-release|. -#. TEST ENVIRONMENT - - - **TRex Fortville NIC Performance**: Received FVL fix from Intel - resolving TRex low throughput issue. TRex per FVL NIC throughput - increased from ~27 Mpps to the nominal ~37 Mpps. For detail see - `CSIT-1503 `_ and `TRex-519 - `_]. - - - **New Intel Xeon Cascadelake Testbeds**: Added performance tests - for 2-Node-Cascadelake (2n-clx) testbeds with x710, xxv710 and - cx556a-edat NIC cards. - -.. - // Alternative Note for 1st Bullet when bad microcode Skx, Clx results are published - - **Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds**: Testpmd and - L3fwd performance test data is included in this report version, - but it shows lower performance and behaviour inconsistency of - these systems following the upgrade of processor microcode - packages (skx ucode 0x2000064, clx ucode 0x500002c) as part of - updating Ubuntu 18.04 LTS kernel version. Tested VPP and DPDK - applications (L3fwd) are affected. Skx and Clx test data will be - corrected in subsequent maintenance report version(s) once the - issue is resolved. See :ref:`vpp_known_issues`. - .. _dpdk_known_issues: Known Issues @@ -64,7 +29,6 @@ List of known issues in |csit-release| for DPDK performance tests: +----+------------------------------------------+----------------------------------------------------------------------------------------------------------+ | # | JiraID | Issue Description | +====+==========================================+==========================================================================================================+ -| 1 | `CSIT-1675 | Intel Xeon 2n-skx, 3n-skx and 2n-clx testbeds behaviour and performance became inconsistent following | -| | `_ | the upgrade to the latest Ubuntu 18.04 LTS kernel version (4.15.0-72-generic) and associated microcode | -| | | packages (skx ucode 0x2000064, clx ucode 0x500002c). VPP as well as DPDK L3fwd tests are affected. | +| 1 | `CSIT-1701 | DPDK L3fwd tests with 9000B L2 packet size are not passing with Mellanox NICs. | +| | `_ | L3fwd application does not accept parameter for increasing -mbuf-size in same way DPDK testpmd does. | +----+------------------------------------------+----------------------------------------------------------------------------------------------------------+ diff --git a/docs/report/dpdk_performance_tests/overview.rst b/docs/report/dpdk_performance_tests/overview.rst index 212e202c13..7f3e930324 100644 --- a/docs/report/dpdk_performance_tests/overview.rst +++ b/docs/report/dpdk_performance_tests/overview.rst @@ -1,11 +1,10 @@ Overview ======== -DPDK performance test results are reported for all three physical -testbed types present in FD.io labs: 3-Node Xeon Haswell (3n-hsw), -3-Node Xeon Skylake (3n-skx), 2-Node Xeon Skylake (2n-skx) and installed -NIC models. For description of physical testbeds used for DPDK -performance tests please refer to :ref:`tested_physical_topologies`. +DPDK performance test results are reported for all physical testbed types +present in FD.io labs and installed NIC models. For description of physical +testbeds used for DPDK performance tests please refer to +:ref:`tested_physical_topologies`. Logical Topologies ------------------ diff --git a/docs/report/dpdk_performance_tests/test_environment.rst b/docs/report/dpdk_performance_tests/test_environment.rst index 03013a4981..3ea7bdc2a8 100644 --- a/docs/report/dpdk_performance_tests/test_environment.rst +++ b/docs/report/dpdk_performance_tests/test_environment.rst @@ -33,29 +33,28 @@ DPDK Compile Parameters .. code-block:: bash - make install T=-native-linuxapp-gcc -j + make install T=--linuxapp-gcc -j Testpmd Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Testpmd startup configuration changes per test case with different -settings for `$$CORES`, `$$RXQ` and max-pkt-len parameter if test is +Testpmd startup configuration changes per test case with different settings for +`$$INT`, `$$CORES`, `$$RXQ`, `$$RXD` and max-pkt-len parameter if test is sending jumbo frames. Startup command template: .. code-block:: bash - testpmd -c $$CORE_MASK -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=$$CORES [--max-pkt-len=9000] --txqflags=0 --forward-mode=io --rxq=$$RXQ --txq=$$TXQ --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start + testpmd -v -l $$CORE_LIST -w $$INT1 -w $$INT2 --master-lcore 0 --in-memory -- --forward-mode=io --burst=64 --txd=$$TXD --rxd=$$RXD --txq=$$TXQ --rxq=$$RXQ --tx-offloads=0x0 --numa --auto-start --total-num-mbufs=16384 --nb-ports=2 --portmask=0x3 --disable-link-check --max-pkt-len=$$PKT_LEN [--mbuf-size=16384] --nb-cores=$$CORES L3FWD Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -L3FWD startup configuration changes per test case with different -settings for `$$CORES` and enable-jumbo parameter if test is sending -jumbo frames. Startup command template: +L3FWD startup configuration changes per test case with different settings for +`$$INT`, `$$CORES` and enable-jumbo parameter if test is sending jumbo frames. +Startup command template: .. code-block:: bash - l3fwd -l $$CORE_LIST -n 4 -- -P -L -p 0x3 --config='${port_config}' [--enable-jumbo --max-pkt-len=9000] --eth-dest=0,${adj_mac0} --eth-dest=1,${adj_mac1} --parse-ptype - + l3fwd -v -l $$CORE_LIST -w $$INT1 -w $$INT2 --master-lcore 0 --in-memory -- --parse-ptype --eth-dest="0,${adj_mac0}" --eth-dest="1,${adj_mac1}" --config="${port_config}" [--enable-jumbo] -P -L -p 0x3 .. include:: ../introduction/test_environment_tg.rst diff --git a/docs/report/introduction/methodology_mellanox_nic.rst b/docs/report/introduction/methodology_mellanox_nic.rst new file mode 100644 index 0000000000..40fcb08aab --- /dev/null +++ b/docs/report/introduction/methodology_mellanox_nic.rst @@ -0,0 +1,129 @@ +Mellanox NIC +------------ + +Performance test results using Mellanox ConnectX5 2p100GE are reported for +2-Node Xeon Cascade Lake physical testbed type present in FD.io labs. For +description of physical testbeds used please refer to +:ref:`tested_physical_topologies`. + +Mellanox NIC settings +~~~~~~~~~~~~~~~~~~~~~ + +Mellanox ConnectX5 NIC settings are following recommendations from +[DpdkPerformanceReport]_, [MellanoxDpdkGuide]_ and [MellanoxDpdkBits]_. +Specifically: + +- Flow Control OFF: + :: + + $ ethtool -A $netdev rx off tx off + +- Change PCI MaxReadReq to 1024B for each port of each NIC: + :: + + $ setpci -s $PORT_PCI_ADDRESS 68.w=3BCD + +- Set CQE COMPRESSION to "AGGRESSIVE": + :: + + $ mlxconfig -d $PORT_PCI_ADDRESS set CQE_COMPRESSION=1 + +Mellanox :abbr:`OFED (OpenFabrics Enterprise Distribution)` driver of version +4.6-1.0.1.1 is installed and used to manage the NIC settings. + +:: + + $ sudo ethtool -i enp94s0f0 + driver: mlx5_core + version: 4.6-1.0.1 + firmware-version: 16.25.1020 (MT_0000000009) + expansion-rom-version: + bus-info: 0000:5e:00.0 + supports-statistics: yes + supports-test: yes + supports-eeprom-access: no + supports-register-dump: no + supports-priv-flags: yes + +:: + + $ sudo lspci -vvvs 5e:00.0 + 5e:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] + Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] + Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ + Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- + Capabilities: [40] Power Management version 3 + Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+) + Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- + Capabilities: [100 v1] Advanced Error Reporting + UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- + UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- + UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- + CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- + CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ + AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn- + Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) + ARICap: MFVC- ACS-, Next Function: 1 + ARICtl: MFVC- ACS-, Function Group: 0 + Capabilities: [1c0 v1] #19 + Capabilities: [230 v1] Access Control Services + ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- + ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- + Capabilities: [320 v1] #27 + Capabilities: [370 v1] #26 + Capabilities: [420 v1] #25 + Kernel driver in use: mlx5_core + Kernel modules: mlx5_core + +TG and SUT settings +~~~~~~~~~~~~~~~~~~~ + +For the TG and SUT environment settings please refer to +:ref:`_vpp_test_environment` and :ref:`_dpdk_test_environment`. + +Links +~~~~~ + +.. [DpdkPerformanceReport] `DPDK 19.11 performance report ` +.. [MellanoxDpdkGuide] `Mellanox DPDK guide ` +.. [MellanoxDpdkBits] `Mellanox DPDK bits ` diff --git a/docs/report/introduction/methodology_trex_traffic_generator.rst b/docs/report/introduction/methodology_trex_traffic_generator.rst index d9e7df57d3..cae65ab1c8 100644 --- a/docs/report/introduction/methodology_trex_traffic_generator.rst +++ b/docs/report/introduction/methodology_trex_traffic_generator.rst @@ -4,27 +4,42 @@ TRex Traffic Generator Usage ~~~~~ -`TRex traffic generator `_ is used for all -CSIT performance tests. TRex stateless mode is used to measure NDR and -PDR throughputs using MLRsearch and to measure maximum transer rate -in MRR tests. +`TRex traffic generator `_ is used for majority of +CSIT performance tests. TRex stateless mode is used to measure NDR and PDR +throughputs using MLRsearch and to measure maximum transfer rate in MRR tests. TRex is installed and run on the TG compute node. The typical procedure is: -- If the TRex is not already installed on TG, it is installed in the - suite setup phase - see `TRex installation`_. - TRex configuration is set in its configuration file :: - /etc/trex_cfg.yaml + $ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml + - version: 2 + c: 15 + limit_memory: 8192 + interfaces: ["${pci1}","${pci2}"] + port_info: + - dest_mac: [${dest_mac1}] + src_mac: [${src_mac1}] + - dest_mac: [${dest_mac2}] + src_mac: [${src_mac2}] + platform : + master_thread_id: 0 + latency_thread_id: 16 + dual_if: + - socket: 0 + threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] + EOF' -- TRex is started in the background mode +- TRex is started in the interactive mode as a background service :: - $ sh -c 'cd /scripts/ && sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server > /tmp/trex.log 2>&1 &' > /dev/null + $ sh -c 'cd /scripts/ && \ + sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \ + > /tmp/trex.log 2>&1 &' > /dev/null - There are traffic streams dynamically prepared for each test, based on traffic - profiles. The traffic is sent and the statistics obtained using + profiles. The traffic is sent and the statistics obtained using API :command:`trex.stl.api.STLClient`. Measuring Packet Loss @@ -48,4 +63,4 @@ Measuring Latency If measurement of latency is requested, two more packet streams are created (one for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In that case, returned statistics will also include -min/avg/max latency values and encoded HDRHstogram data. +min/avg/max latency values and encoded HDRHistogram data. diff --git a/docs/report/introduction/physical_testbeds.rst b/docs/report/introduction/physical_testbeds.rst index 24061f9737..3776c03d72 100644 --- a/docs/report/introduction/physical_testbeds.rst +++ b/docs/report/introduction/physical_testbeds.rst @@ -71,7 +71,7 @@ SUT servers are populated with the following NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. @@ -81,7 +81,7 @@ NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) @@ -118,7 +118,7 @@ SUT servers are populated with the following NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) +#. NIC-3: empty, future expansion. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. @@ -128,7 +128,7 @@ NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) +#. NIC-3: empty, future expansion. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) diff --git a/docs/report/introduction/test_environment_sut_conf_1.rst b/docs/report/introduction/test_environment_sut_conf_1.rst index 29baeab8b1..63ff50205d 100644 --- a/docs/report/introduction/test_environment_sut_conf_1.rst +++ b/docs/report/introduction/test_environment_sut_conf_1.rst @@ -7,14 +7,14 @@ install and Below a subset of the running configuration: -1. Ubuntu 18.04.x LTS +1. Ubuntu 18.04.2 LTS :: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu - Description: Ubuntu 18.04.3 LTS + Description: Ubuntu 18.04.2 LTS Release: 18.04 Codename: bionic @@ -60,7 +60,7 @@ Linux Boot Parameters Hugepages Configuration ~~~~~~~~~~~~~~~~~~~~~~~ -Huge pages are namaged via sysctl configuration located in +Huge pages are managed via sysctl configuration located in `/etc/sysctl.d/90-csit.conf` on each testbed. Default huge page size is 2M. The exact amount of huge pages depends on testbed. All the values are defined in `Ansible inventory - hosts` files. diff --git a/docs/report/introduction/test_environment_tg.rst b/docs/report/introduction/test_environment_tg.rst index 38dfb90070..190c7c3f5b 100644 --- a/docs/report/introduction/test_environment_tg.rst +++ b/docs/report/introduction/test_environment_tg.rst @@ -11,31 +11,42 @@ DPDK Version DPDK v19.05 -TG Build Script Used -~~~~~~~~~~~~~~~~~~~~ +TG Installation +~~~~~~~~~~~~~~~ -`TRex installation`_ +T-Rex installation is managed via Ansible role. TG Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~ :: - $ cat /etc/trex_cfg.yaml - - version : 2 - interfaces : ["0000:0d:00.0","0000:0d:00.1"] - port_info : - - dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5] - src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4] - - dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4] - src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5] + $ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml + - version: 2 + c: 15 + limit_memory: 8192 + interfaces: ["${pci1}","${pci2}"] + port_info: + - dest_mac: [${dest_mac1}] + src_mac: [${src_mac1}] + - dest_mac: [${dest_mac2}] + src_mac: [${src_mac2}] + platform : + master_thread_id: 0 + latency_thread_id: 16 + dual_if: + - socket: 0 + threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] + EOF' TG Startup Command ~~~~~~~~~~~~~~~~~~ :: - $ sh -c 'cd /scripts/ && sudo nohup ./t-rex-64 -i -c 7 --prefix $(hostname) --hdrh > /tmp/trex.log 2>&1 &'> /dev/null + $ sh -c 'cd /scripts/ && \ + sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \ + > /tmp/trex.log 2>&1 &' > /dev/null TG API Driver ~~~~~~~~~~~~~ diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst index 632b9a9fb4..5ef248903f 100644 --- a/docs/report/vpp_performance_tests/test_environment.rst +++ b/docs/report/vpp_performance_tests/test_environment.rst @@ -44,9 +44,9 @@ VPP Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~ VPP startup configuration vary per test case, with different settings -for `$$CORELIST_WORKERS`, `$$NUM_RX_QUEUES`, `$$UIO_DRIVER`, `$$NUM- -MBUFS` and `$$NO_MULTI_SEG` parameter. Default template is provided -below: +for `$$CORELIST_WORKERS`, `$$NUM_RX_QUEUES`, `$$UIO_DRIVER`, and +`$$NO_MULTI_SEG` parameter. List of plugins to enable is driven by test +requirements. Default template is provided below: :: @@ -57,12 +57,14 @@ below: statseg { size 4G + per-node-counters on } unix { cli-listen /run/vpp/cli.sock log /tmp/vpe.log nodaemon + full-coredump } socksvr { socket-name /run/vpp/api.sock @@ -79,7 +81,7 @@ below: { disable } - plugin dpdk_plugin.so + plugin <$$test_requirement>_plugin.so { enable } @@ -89,9 +91,14 @@ below: corelist-workers $$CORELIST_WORKERS main-core 1 } + buffers + { + buffers-per-numa 215040 + } + + # Below: in case of dpdk based drivers (vfio-pci) only dpdk { - num-mbufs $$NUM-MBUFS uio-driver $$UIO_DRIVER $$NO_MULTI_SEG log-level debug -- cgit 1.2.3-korg