Test Environment ================ To execute performance tests, there are three identical testbeds, each testbed consists of two SUTs and one TG. SUT Configuration - Host HW --------------------------- Host hardware details (CPU, memory, NIC layout) and physical topology are described in detail in `LF FDio CSIT testbed wiki page `_. **Host configuration** - All hosts are Cisco UCS C240-M4 (2x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz, 18c, 512GB RAM), - BIOS settings are default except the following: - Hyperthreading disabled. - SpeedStep disabled. - TurboBoost disabled. - Power Technology: Performance. - Hosts run Ubuntu 16.04.1, kernel 4.4.0-42-generic. - Linux kernel boot command line option "intel_pstate=disable" is applied to both SUTs and TG. In addition, on SUTs, only cores 0 and 18 (the first core on each socket) are available to the Linux operating system and generic tasks, all other CPU cores are isolated and reserved for VPP. - In addition to CIMC and Management, each TG has 4x Intel X710 10GB NIC (=8 ports) and 2x Intel XL710 40GB NIC (=4 ports), whereas each SUT has: - 1x Intel X520 NIC (10GB, 2 ports), - 1x Cisco VIC 1385 (40GB, 2 ports), - 1x Intel XL710 NIC (40GB, 2 ports), - 1x Intel X710 NIC (10GB, 2 ports), - 1x Cisco VIC 1227 (10GB, 2 ports). - This allows for a total of five ring topologies, each using ports on specific NIC model, enabling per NIC model benchmarking. **NIC models and placement** - 0a:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 - 0a:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 - 06:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) Subsystem: Cisco Systems Inc VIC 1227 PCIe Ethernet NIC - 07:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) Subsystem: Cisco Systems Inc VIC 1227 PCIe Ethernet NIC - 13:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) Subsystem: Cisco Systems Inc VIC 1385 PCIe Ethernet NIC - 15:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) Subsystem: Cisco Systems Inc VIC 1385 PCIe Ethernet NIC - 85:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2 - 85:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2 - 87:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2 - 87:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2 SUT Configuration - Host OS Linux --------------------------------- Software details (OS, configuration) are described in `CSIT/CSIT_LF_testbed `_. Below a subset of the configuration: .. code-block:: xml

1 1 0 0 0 0 0 0 0

127 0 0 0 0 0 1 0 Jan 27 2017 05:58:38 PST 0 127 0 0 0 1 127 0 /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8" GRUB_CMDLINE_LINUX="isolcpus=1-17,19-35 nohz_full=1-17,19-35 rcu_nocbs=1-17,19-35 intel_pstate=disable" # Uncomment to enable BadRAM filtering, modify to suit your needs # This works with Linux (no patch required) and with any kernel that obtains # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef" # Uncomment to disable graphical terminal (grub-pc only) #GRUB_TERMINAL=console GRUB_TERMINAL=serial GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" # The resolution used on graphical terminal # note that you can use only modes which your graphic card supports via VBE # you can see them in real GRUB with the command `vbeinfo' #GRUB_GFXMODE=640x480 # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux #GRUB_DISABLE_LINUX_UUID=true # Uncomment to disable generation of recovery mode menu entries #GRUB_DISABLE_RECOVERY="true" # Uncomment to get a beep at grub start #GRUB_INIT_TUNE="480 440 1" ]]> 1 0 mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 6: enp23s0f0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:fe:c8:e5:68:32 brd ff:ff:ff:ff:ff:ff ]]> 0 0

127 0

0

1 127 1

DUT Configuration - VPP ----------------------- **VPP Version** 17.01-release_amd64 **VPP Compile Parameters** VPP Compile Job: https://jenkins.fd.io/view/vpp/job/vpp-merge-1701-ubuntu1604/ **VPP Install Parameters** :: $ dpkg -i --force-all **VPP Startup Configuration** VPP startup configuration changes per test case with different settings for CPU cores, rx-queues and no-multi-seg parameter. Startup config is aligned with applied test case tag: Tagged by **1T1C**:: $ cat /etc/vpp/startup.conf unix { nodaemon log /tmp/vpe.log cli-listen localhost:5002 full-coredump } api-trace { on } cpu { main-core 0 corelist-workers 1 } dpdk { socket-mem 1024,1024 dev default { num-rx-queues 1 } dev 0000:0a:00.1 dev 0000:0a:00.0 no-multi-seg } ip6 { hash-buckets 2000000 heap-size 3G } Tagged by **2T1C**:: $ cat /etc/vpp/startup.conf unix { nodaemon log /tmp/vpe.log cli-listen localhost:5002 full-coredump } api-trace { on } cpu { main-core 0 corelist-workers 1,2 } dpdk { socket-mem 1024,1024 dev default { num-rx-queues 1 } dev 0000:0a:00.1 dev 0000:0a:00.0 no-multi-seg } ip6 { hash-buckets 2000000 heap-size 3G } Tagged by **4T4C**:: $ cat /etc/vpp/startup.conf unix { nodaemon log /tmp/vpe.log cli-listen localhost:5002 full-coredump } api-trace { on } cpu { main-core 0 corelist-workers 1,2,3,4 } dpdk { socket-mem 1024,1024 dev default { num-rx-queues 1 } dev 0000:0a:00.1 dev 0000:0a:00.0 no-multi-seg } ip6 { hash-buckets 2000000 heap-size 3G } TG Configuration - TRex ----------------------- **TG Version** TRex v2.22 **DPDK version** DPDK v17.02 (f4decdc59e9323ecff5ddb5de7ebf0c79d50a960 - in DPDK repo) **TG Build Script used** https://gerrit.fd.io/r/gitweb?p=csit.git;a=blob;f=resources/tools/t-rex/t-rex-installer.sh;h=8090b7568327ac5f869e82664bc51b24f89f603f;hb=refs/heads/rls1704 **TG Startup Configuration** :: $ cat /etc/trex_cfg.yaml - port_limit : 2 version : 2 interfaces : ["0000:0d:00.0","0000:0d:00.1"] port_info : - dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5] src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4] - dest_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf4] src_mac : [0x3c,0xfd,0xfe,0x9c,0xee,0xf5] **TG common API - pointer to driver** https://gerrit.fd.io/r/gitweb?p=csit.git;a=blob;f=resources/tools/t-rex/t-rex-stateless.py;h=ae8d18767013ebecb0bec6c732ac66e483408661;hb=refs/heads/rls1704