Report: Infra update

Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ie39585e88460d9a5e3764726d93dc065f5cbacfb

5 files changed, 185 insertions, 30 deletions

diff --git a/docs/report/introduction/methodology_mellanox_nic.rst b/docs/report/introduction/methodology_mellanox_nic.rst
new file mode 100644
index 0000000000..40fcb08aab
--- /dev/null
+++ b/docs/report/introduction/methodology_mellanox_nic.rst
@@ -0,0 +1,129 @@
+Mellanox NIC
+------------
+
+Performance test results using Mellanox ConnectX5 2p100GE are reported for
+2-Node Xeon Cascade Lake physical testbed type present in FD.io labs. For
+description of physical testbeds used please refer to
+:ref:`tested_physical_topologies`.
+
+Mellanox NIC settings
+~~~~~~~~~~~~~~~~~~~~~
+
+Mellanox ConnectX5 NIC settings are following recommendations from
+[DpdkPerformanceReport]_, [MellanoxDpdkGuide]_ and [MellanoxDpdkBits]_.
+Specifically:
+
+- Flow Control OFF:
+  ::
+
+      $ ethtool -A $netdev rx off tx off
+
+- Change PCI MaxReadReq to 1024B for each port of each NIC:
+  ::
+
+      $ setpci -s $PORT_PCI_ADDRESS 68.w=3BCD
+
+- Set CQE COMPRESSION to "AGGRESSIVE":
+  ::
+
+      $ mlxconfig -d $PORT_PCI_ADDRESS set CQE_COMPRESSION=1
+
+Mellanox :abbr:`OFED (OpenFabrics Enterprise Distribution)` driver of version
+4.6-1.0.1.1 is installed and used to manage the NIC settings.
+
+::
+
+    $ sudo ethtool -i enp94s0f0
+    driver: mlx5_core
+    version: 4.6-1.0.1
+    firmware-version: 16.25.1020 (MT_0000000009)
+    expansion-rom-version:
+    bus-info: 0000:5e:00.0
+    supports-statistics: yes
+    supports-test: yes
+    supports-eeprom-access: no
+    supports-register-dump: no
+    supports-priv-flags: yes
+
+::
+
+    $ sudo lspci -vvvs 5e:00.0
+    5e:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
+	    Subsystem: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
+	    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
+	    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
+	    Latency: 0, Cache Line Size: 32 bytes
+	    Interrupt: pin A routed to IRQ 37
+	    NUMA node: 0
+	    Region 0: Memory at 38fffe000000 (64-bit, prefetchable) [size=32M]
+	    Expansion ROM at c5e00000 [disabled] [size=1M]
+	    Capabilities: [60] Express (v2) Endpoint, MSI 00
+		    DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
+			    ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
+		    DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
+			    RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
+			    MaxPayload 256 bytes, MaxReadReq 4096 bytes
+		    DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
+		    LnkCap:	Port #0, Speed 16GT/s, Width x16, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
+			    ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
+		    LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
+			    ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
+		    LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
+		    DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
+		    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
+		    LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
+			     Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
+			     Compliance De-emphasis: -6dB
+		    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
+			     EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
+	    Capabilities: [48] Vital Product Data
+		    Product Name: CX556A - ConnectX-5 QSFP28
+		    Read-only fields:
+			    [PN] Part number: MCX556A-EDAT
+			    [EC] Engineering changes: AA
+			    [V2] Vendor specific: MCX556A-EDAT
+			    [SN] Serial number: MT1945X00360
+			    [V3] Vendor specific: f8d15ef7e701ea118000b8599ffe4aa8
+			    [VA] Vendor specific: MLX:MODL=CX556A:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0
+			    [V0] Vendor specific: PCIeGen4 x16
+			    [RV] Reserved: checksum good, 2 byte(s) reserved
+		    End
+	    Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
+		    Vector table: BAR=0 offset=00002000
+		    PBA: BAR=0 offset=00003000
+	    Capabilities: [c0] Vendor Specific Information: Len=18 <?>
+	    Capabilities: [40] Power Management version 3
+		    Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)
+		    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
+	    Capabilities: [100 v1] Advanced Error Reporting
+		    UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
+		    UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
+		    UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
+		    CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
+		    CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
+		    AERCap:	First Error Pointer: 04, GenCap+ CGenEn- ChkCap+ ChkEn-
+	    Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
+		    ARICap:	MFVC- ACS-, Next Function: 1
+		    ARICtl:	MFVC- ACS-, Function Group: 0
+	    Capabilities: [1c0 v1] #19
+	    Capabilities: [230 v1] Access Control Services
+		    ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
+		    ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
+	    Capabilities: [320 v1] #27
+	    Capabilities: [370 v1] #26
+	    Capabilities: [420 v1] #25
+	    Kernel driver in use: mlx5_core
+	    Kernel modules: mlx5_core
+
+TG and SUT settings
+~~~~~~~~~~~~~~~~~~~
+
+For the TG and SUT environment settings please refer to
+:ref:`_vpp_test_environment` and :ref:`_dpdk_test_environment`.
+
+Links
+~~~~~
+
+.. [DpdkPerformanceReport] `DPDK 19.11 performance report <http://static.dpdk.org/doc/perf/DPDK_19_11_Mellanox_NIC_performance_report.pdf>`
+.. [MellanoxDpdkGuide] `Mellanox DPDK guide <https://www.mellanox.com/related-docs/prod_software/MLNX_DPDK_Quick_Start_Guide_v16.11_3.0.pdf>`
+.. [MellanoxDpdkBits] `Mellanox DPDK bits <https://community.mellanox.com/s/article/mellanox-dpdk>`
diff --git a/docs/report/introduction/methodology_trex_traffic_generator.rst b/docs/report/introduction/methodology_trex_traffic_generator.rst
index d9e7df57d3..cae65ab1c8 100644
--- a/docs/report/introduction/methodology_trex_traffic_generator.rst
+++ b/docs/report/introduction/methodology_trex_traffic_generator.rst
@@ -4,27 +4,42 @@ TRex Traffic Generator
 Usage
 ~~~~~
 
-`TRex traffic generator <https://trex-tgn.cisco.com>`_ is used for all
-CSIT performance tests. TRex stateless mode is used to measure NDR and
-PDR throughputs using MLRsearch and to measure maximum transer rate
-in MRR tests.
+`TRex traffic generator <https://trex-tgn.cisco.com>`_ is used for majority of
+CSIT performance tests. TRex stateless mode is used to measure NDR and PDR
+throughputs using MLRsearch and to measure maximum transfer rate in MRR tests.
 
 TRex is installed and run on the TG compute node. The typical procedure is:
 
-- If the TRex is not already installed on TG, it is installed in the
-  suite setup phase - see `TRex installation`_.
 - TRex configuration is set in its configuration file
   ::
 
-  /etc/trex_cfg.yaml
+  $ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml
+  - version: 2
+    c: 15
+    limit_memory: 8192
+    interfaces: ["${pci1}","${pci2}"]
+    port_info:
+      - dest_mac: [${dest_mac1}]
+        src_mac: [${src_mac1}]
+      - dest_mac: [${dest_mac2}]
+        src_mac: [${src_mac2}]
+    platform :
+      master_thread_id: 0
+      latency_thread_id: 16
+      dual_if:
+          - socket: 0
+            threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
+  EOF'
 
-- TRex is started in the background mode
+- TRex is started in the interactive mode as a background service
   ::
 
-  $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server > /tmp/trex.log 2>&1 &' > /dev/null
+  $ sh -c 'cd <t-rex-install-dir>/scripts/ && \
+    sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \
+    > /tmp/trex.log 2>&1 &' > /dev/null
 
 - There are traffic streams dynamically prepared for each test, based on traffic
-  profiles. The traffic is sent and the statistics obtained using
+  profiles. The traffic is sent and the statistics obtained using API
   :command:`trex.stl.api.STLClient`.
 
 Measuring Packet Loss
@@ -48,4 +63,4 @@ Measuring Latency
 If measurement of latency is requested, two more packet streams are
 created (one for each direction) with TRex flow_stats parameter set to
 STLFlowLatencyStats. In that case, returned statistics will also include
-min/avg/max latency values and encoded HDRHstogram data.
+min/avg/max latency values and encoded HDRHistogram data.
diff --git a/docs/report/introduction/physical_testbeds.rst b/docs/report/introduction/physical_testbeds.rst
index 24061f9737..3776c03d72 100644
--- a/docs/report/introduction/physical_testbeds.rst
+++ b/docs/report/introduction/physical_testbeds.rst
@@ -71,7 +71,7 @@ SUT servers are populated with the following NIC models:
 
 #. NIC-1: x710-DA4 4p10GE Intel.
 #. NIC-2: xxv710-DA2 2p25GE Intel.
-#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28)
+#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox.
 #. NIC-4: empty, future expansion.
 #. NIC-5: empty, future expansion.
 #. NIC-6: empty, future expansion.
@@ -81,7 +81,7 @@ NIC models:
 
 #. NIC-1: x710-DA4 4p10GE Intel.
 #. NIC-2: xxv710-DA2 2p25GE Intel.
-#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Only testbed t27, t28)
+#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox.
 #. NIC-4: empty, future expansion.
 #. NIC-5: empty, future expansion.
 #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.)
@@ -118,7 +118,7 @@ SUT servers are populated with the following NIC models:
 
 #. NIC-1: x710-DA4 4p10GE Intel.
 #. NIC-2: xxv710-DA2 2p25GE Intel.
-#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.)
+#. NIC-3: empty, future expansion.
 #. NIC-4: empty, future expansion.
 #. NIC-5: empty, future expansion.
 #. NIC-6: empty, future expansion.
@@ -128,7 +128,7 @@ NIC models:
 
 #. NIC-1: x710-DA4 4p10GE Intel.
 #. NIC-2: xxv710-DA2 2p25GE Intel.
-#. NIC-3: cx556a-edat ConnectX5 2p100GE Mellanox. (Not used yet.)
+#. NIC-3: empty, future expansion.
 #. NIC-4: empty, future expansion.
 #. NIC-5: empty, future expansion.
 #. NIC-6: x710-DA4 4p10GE Intel. (For self-tests.)
diff --git a/docs/report/introduction/test_environment_sut_conf_1.rst b/docs/report/introduction/test_environment_sut_conf_1.rst
index 29baeab8b1..63ff50205d 100644
--- a/docs/report/introduction/test_environment_sut_conf_1.rst
+++ b/docs/report/introduction/test_environment_sut_conf_1.rst
@@ -7,14 +7,14 @@ install and
 
 Below a subset of the running configuration:
 
-1. Ubuntu 18.04.x LTS
+1. Ubuntu 18.04.2 LTS
 
 ::
 
     $ lsb_release -a
     No LSB modules are available.
     Distributor ID: Ubuntu
-    Description:    Ubuntu 18.04.3 LTS
+    Description:    Ubuntu 18.04.2 LTS
     Release:        18.04
     Codename:       bionic
 
@@ -60,7 +60,7 @@ Linux Boot Parameters
 Hugepages Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~
 
-Huge pages are namaged via sysctl configuration located in
+Huge pages are managed via sysctl configuration located in
 `/etc/sysctl.d/90-csit.conf` on each testbed. Default huge page size is 2M.
 The exact amount of huge pages depends on testbed. All the values are defined
 in `Ansible inventory - hosts` files.
diff --git a/docs/report/introduction/test_environment_tg.rst b/docs/report/introduction/test_environment_tg.rst
index 38dfb90070..190c7c3f5b 100644
--- a/docs/report/introduction/test_environment_tg.rst
+++ b/docs/report/introduction/test_environment_tg.rst
@@ -11,31 +11,42 @@ DPDK Version
 
 DPDK v19.05
 
-TG Build Script Used
-~~~~~~~~~~~~~~~~~~~~
+TG Installation
+~~~~~~~~~~~~~~~
 
-`TRex installation`_
+T-Rex installation is managed via Ansible role.
 
 TG Startup Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
 
-    $ cat /etc/trex_cfg.yaml
-    - version         : 2
-      interfaces      : ["0000:0d:00.0","0000:0d:00.1"]
-      port_info       :
-        - dest_mac        :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf5]
-          src_mac         :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf4]
-        - dest_mac        :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf4]
-          src_mac         :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf5]
+  $ sudo -E -S sh -c 'cat << EOF > /etc/trex_cfg.yaml
+  - version: 2
+    c: 15
+    limit_memory: 8192
+    interfaces: ["${pci1}","${pci2}"]
+    port_info:
+      - dest_mac: [${dest_mac1}]
+        src_mac: [${src_mac1}]
+      - dest_mac: [${dest_mac2}]
+        src_mac: [${src_mac2}]
+    platform :
+      master_thread_id: 0
+      latency_thread_id: 16
+      dual_if:
+          - socket: 0
+            threads: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
+  EOF'
 
 TG Startup Command
 ~~~~~~~~~~~~~~~~~~
 
 ::
 
-    $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --prefix $(hostname) --hdrh > /tmp/trex.log 2>&1 &'> /dev/null
+  $ sh -c 'cd <t-rex-install-dir>/scripts/ && \
+    sudo nohup ./t-rex-64 -i --prefix $(hostname) --hdrh --no-scapy-server \
+    > /tmp/trex.log 2>&1 &' > /dev/null
 
 TG API Driver
 ~~~~~~~~~~~~~