diff options
Diffstat (limited to 'docs/report/introduction')
5 files changed, 185 insertions, 30 deletions
diff --git a/docs/report/introduction/methodology_mellanox_nic.rst b/docs/report/introduction/methodology_mellanox_nic.rst new file mode 100644 index 0000000000..40fcb08aab --- /dev/null +++ b/docs/report/introduction/methodology_mellanox_nic.rst @@ -0,0 +1,129 @@ +Mellanox NIC +------------ + +Performance test results using Mellanox ConnectX5 2p100GE are reported for +2-Node Xeon Cascade Lake physical testbed type present in FD.io labs. For +description of physical testbeds used please refer to +:ref:`tested_physical_topologies`. + +Mellanox NIC settings +~~~~~~~~~~~~~~~~~~~~~ + +Mellanox ConnectX5 NIC settings are following recommendations from +[DpdkPerformanceReport]_, [MellanoxDpdkGuide]_ and [MellanoxDpdkBits]_. +Specifically: + +- Flow Control OFF: + :: + + $ ethtool -A $netdev rx off tx off + +- Change PCI MaxReadReq to 1024B for each port of each NIC: + :: + + $ setpci -s $PORT_PCI_ADDRESS 68.w=3BCD + +- Set CQE COMPRESSION to "AGGRESSIVE": + :: + + $ mlxconfig -d $PORT_PCI_ADDRESS set CQE_COMPRESSION=1 + +Mellanox :abbr:`OFED (OpenFabrics Enterprise Distribution)` driver of version +4.6-1.0.1.1 is installed and used to manage the NIC settings. + +:: + + $ sudo ethtool -i enp94s0f0 + driver: mlx5_core + version: 4.6-1.0.1 + firmware-version: 16.25.1020 (MT_0000000009) + expansion-rom-version: + bus-info: 0000:5e:00.0 + supports-statistics: yes + supports-test: yes + supports-eeprom-access: no + supports-register-dump: no + supports-priv-flags: yes + +:: + + $ sudo lspci -vvvs 5e:00.0 + 5e:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] + Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] + Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ + Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- + Latency: 0, Cache Line Size: 32 bytes + Interrupt: pin A routed to IRQ 37 + NUMA node: 0 + Region 0: Memory at 38fffe000000 (64-bit, prefetchable) [size=32M] + Expansion ROM at c5e00000 [disabled] [size=1M] + Capabilities: [60] Express (v2) Endpoint, MSI 00 + DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited + ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W + DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- + RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- + MaxPayload 256 bytes, MaxReadReq 4096 bytes + DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- + LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited + ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ + LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ + ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- + LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- + DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported + DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled + LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- + Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- + Compliance De-emphasis: -6dB + LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+ + EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- + Capabilities: [48] Vital Product Data + Product Name: CX556A - ConnectX-5 QSFP28 + Read-only fields: + [PN] Part number: MCX556A-EDAT + [EC] Engineering changes: AA + [V2] Vendor specific: MCX556A-EDAT + [SN] Serial number: MT1945X00360 + [V3] Vendor specific: f8d15ef7e701ea118000b8599ffe4aa8 + [VA] Vendor specific: MLX:MODL=CX556A:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0 + [V0] Vendor specific: PCIeGen4 x16 + [RV] Reserved: checksum good, 2 byte(s) reserved + End + Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- + Vector table: BAR=0 offset=00002000 + PBA: BAR=0 offset=00003000 + Capabilities: [c0] Vendor Specific Information: Len=18 <?> + Capabilities: [40] Power Management version 3 + Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+) + Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- + Capabilities: [100 v1] Advanced Error Reporting + UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- + UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- + UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- + CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- + CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ + AERCap: First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn- + Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) + ARICap: MFVC- ACS-, Next Function: 1 + ARICtl: MFVC- ACS-, Function Group: 0 + Capabilities: [1c0 v1] #19 + Capabilities: [230 v1] Access Control Services + ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- + ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- + Capabilities: [320 v1] #27 + Capabilities: [370 v1] #26 + Capabilities: [420 v1] #25 + Kernel driver in use: mlx5_core + Kernel modules: mlx5_core + +TG and SUT settings +~~~~~~~~~~~~~~~~~~~ + +For the TG and SUT environment settings please refer to +:ref:`_vpp_test_environment` and :ref:`_dpdk_test_environment`. + +Links +~~~~~ + +.. [DpdkPerformanceReport] `DPDK 19.11 performance report <http://static.dpdk.org/doc/perf/DPDK_19_11_Mellanox_NIC_performance_report.pdf>` +.. [MellanoxDpdkGuide] `Mellanox DPDK guide <https://www.mellanox.com/related-docs/prod_software/MLNX_DPDK_Quick_Start_Guide_v16.11_3.0.pdf>` +.. [MellanoxDpdkBits] `Mellanox DPDK bits <https://community.mellanox.com/s/article/mellanox-dpdk>` diff --git a/docs/report/introduction/methodology_trex_traffic_generator.rst b/docs/report/introduction/methodology_trex_traffic_generator.rst index d9e7df57d3..cae65ab1c8 100644 --- a/docs/report/introduction/methodology_trex_traffic_generator.rst +++ b/docs/report/introduction/methodology_trex_traffic_generator.rst @@ -4,27 +4,42 @@ TRex Traffic Generator Usage ~~~~~ -`TRex traffic generator <https://trex-tgn.cisco.com>`_ is used for all -CSIT performance tests. TRex stateless mode is used to measure NDR and -PDR throughputs using MLRsearch and to measure maximum transer rate -in MRR tests. +`TRex traffic generator <https://trex-tgn.cisco.com>`_ is used for majority of +CSIT performance tests. TRex stateless mode is used to measure NDR and PDR +throughputs using MLRsearch and to measure maximum transfer rate in MRR tests. TRex is installed and run on the TG compute node. The typical procedure is: -- If the TRex is not already installed on TG, it is installed in the - suite setup phase - see `TRex installation`_. - TRex configuration is set in its configuration file :: - /etc/trex_cfg.yaml + $ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml + - version: 2 + c: 15 + limit_memory: 8192 + interfaces: ["${pci1}","${pci2}"] + port_info: + - dest_mac: [${dest_mac1}] + src_mac: [${src_mac1}] + - dest_mac: [${dest_mac2}] + src_mac: [${src_mac2}] + platform : + master_thread_id: 0 + latency_thread_id: 16 + dual_if: + - socket: 0 + threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] + EOF' -- TRex is started in the background mode +- TRex is started in the interactive mode as a background service :: - $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server > /tmp/trex.log 2>&1 &' > /dev/null + $ sh -c 'cd <t-rex-install-dir>/scripts/ && \ + sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \ + > /tmp/trex.log 2>&1 &' > /dev/null - There are traffic streams dynamically prepared for each test, based on traffic - profiles. The traffic is sent and the statistics obtained using + profiles. The traffic is sent and the statistics obtained using API :command:`trex.stl.api.STLClient`. Measuring Packet Loss @@ -48,4 +63,4 @@ Measuring Latency If measurement of latency is requested, two more packet streams are created (one for each direction) with TRex flow_stats parameter set to STLFlowLatencyStats. In that case, returned statistics will also include -min/avg/max latency values and encoded HDRHstogram data. +min/avg/max latency values and encoded HDRHistogram data. diff --git a/docs/report/introduction/physical_testbeds.rst b/docs/report/introduction/physical_testbeds.rst index 24061f9737..3776c03d72 100644 --- a/docs/report/introduction/physical_testbeds.rst +++ b/docs/report/introduction/physical_testbeds.rst @@ -71,7 +71,7 @@ SUT servers are populated with the following NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. @@ -81,7 +81,7 @@ NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28) +#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) @@ -118,7 +118,7 @@ SUT servers are populated with the following NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) +#. NIC-3: empty, future expansion. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: empty, future expansion. @@ -128,7 +128,7 @@ NIC models: #. NIC-1: x710-DA4 4p10GE Intel. #. NIC-2: xxv710-DA2 2p25GE Intel. -#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.) +#. NIC-3: empty, future expansion. #. NIC-4: empty, future expansion. #. NIC-5: empty, future expansion. #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.) diff --git a/docs/report/introduction/test_environment_sut_conf_1.rst b/docs/report/introduction/test_environment_sut_conf_1.rst index 29baeab8b1..63ff50205d 100644 --- a/docs/report/introduction/test_environment_sut_conf_1.rst +++ b/docs/report/introduction/test_environment_sut_conf_1.rst @@ -7,14 +7,14 @@ install and Below a subset of the running configuration: -1. Ubuntu 18.04.x LTS +1. Ubuntu 18.04.2 LTS :: $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu - Description: Ubuntu 18.04.3 LTS + Description: Ubuntu 18.04.2 LTS Release: 18.04 Codename: bionic @@ -60,7 +60,7 @@ Linux Boot Parameters Hugepages Configuration ~~~~~~~~~~~~~~~~~~~~~~~ -Huge pages are namaged via sysctl configuration located in +Huge pages are managed via sysctl configuration located in `/etc/sysctl.d/90-csit.conf` on each testbed. Default huge page size is 2M. The exact amount of huge pages depends on testbed. All the values are defined in `Ansible inventory - hosts` files. diff --git a/docs/report/introduction/test_environment_tg.rst b/docs/report/introduction/test_environment_tg.rst index 38dfb90070..190c7c3f5b 100644 --- a/docs/report/introduction/test_environment_tg.rst +++ b/docs/report/introduction/test_environment_tg.rst @@ -11,31 +11,42 @@ DPDK Version DPDK v19.05 -TG Build Script Used -~~~~~~~~~~~~~~~~~~~~ +TG Installation +~~~~~~~~~~~~~~~ -`TRex installation`_ +T-Rex installation is managed via Ansible role. TG Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~ :: - $ cat /etc/trex_cfg.yaml - - version : 2 - interfaces : ["0000:0d:00.0","0000:0d:00.1"] - port_info : - - dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5] - src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4] - - dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4] - src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5] + $ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml + - version: 2 + c: 15 + limit_memory: 8192 + interfaces: ["${pci1}","${pci2}"] + port_info: + - dest_mac: [${dest_mac1}] + src_mac: [${src_mac1}] + - dest_mac: [${dest_mac2}] + src_mac: [${src_mac2}] + platform : + master_thread_id: 0 + latency_thread_id: 16 + dual_if: + - socket: 0 + threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] + EOF' TG Startup Command ~~~~~~~~~~~~~~~~~~ :: - $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --prefix $(hostname) --hdrh > /tmp/trex.log 2>&1 &'> /dev/null + $ sh -c 'cd <t-rex-install-dir>/scripts/ && \ + sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \ + > /tmp/trex.log 2>&1 &' > /dev/null TG API Driver ~~~~~~~~~~~~~ |