diff options
| author |   Peter Mikus <pmikus@cisco.com> | 2019-09-10 11:27:11 +0000 |
|---|---|---|
| committer | Peter Mikus <pmikus@cisco.com> | 2019-09-11 13:07:15 +0000 |
| commit | b82474874d4329d3e82ea8a22754b7b04cf969ee (patch) | |
| tree | 684a3a5ccf06639139221f15423aed4f633a5579 | |
| parent | 2fe2a2c140bffa10678e3e217c8a5cba0fd4dbd0 (diff) | |
Ansible: Cascadelake include
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Iecb18e9d94ff715e40152564fb778650d43a48d3
18 files changed, 1901 insertions, 227 deletions
diff --git a/docs/lab/testbeds_sm_clx_hw_bios_cfg.md b/docs/lab/testbeds_sm_clx_hw_bios_cfg.md new file mode 100644 index 0000000000..30146cda81 --- /dev/null +++ b/docs/lab/testbeds_sm_clx_hw_bios_cfg.md @@ -0,0 +1,1131 @@ +# SuperMicro Cascadelake Servers - HW and BIOS Configuration + +1. [Linux lscpu](#linux-lscpu) +1. [Linux dmidecode](#dmidecode) +1. [Linux dmidecode pci](#linux-dmidecode-pci) +1. [Linux dmidecode memory](#linux-dmidecode-memory) +1. [Xeon Clx Server BIOS Configuration](#xeon-clx-server-bios-configuration) + 1. [Boot Feature](#boot-feature) + 1. [CPU Configuration](#cpu-configuration) + 1. [Advanced Power Management Configuration](#advanced-power-management-configuration) + 1. [CPU P State Control](#cpu-p-state-control) + 1. [Hardware PM State Control](#hardware-pm-state-control) + 1. [CPU C State Control](#cpu-c-state-control) + 1. [Package C State Control](#package-c-state-control) + 1. [CPU T State Control](#cpu-t-state-control) + 1. [Chipset Configuration](#chipset-configuration) + 1. [North Bridge](#north-bridge) + 1. [UPI Configuration](#upi-configuration) + 1. [Memory Configuration](#memory-configuration) + 1. [IIO Configuration](#iio-configuration) + 1. [CPU1 Configuration](#cpu1-configuration) + 1. [CPU2 Configuration](#cpu2-configuration) + 1. [South Bridge](#south-bridge) + 1. [PCIe/PCI/PnP Configuration](#pciepcipnp-configuration) + 1. [ACPI Settings](#acpi-settings) +1. [Xeon Clx Server Firmware Inventory](#xeon-clx-server-firmware-inventory) + +## Linux lscpu + +``` +$ lscpu +Architecture: x86_64 +CPU op-mode(s): 32-bit, 64-bit +Byte Order: Little Endian +CPU(s): 112 +On-line CPU(s) list: 0-111 +Thread(s) per core: 2 +Core(s) per socket: 28 +Socket(s): 2 +NUMA node(s): 2 +Vendor ID: GenuineIntel +CPU family: 6 +Model: 85 +Model name: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz +Stepping: 7 +CPU MHz: 3299.609 +BogoMIPS: 5400.00 +Virtualization: VT-x +L1d cache: 32K +L1i cache: 32K +L2 cache: 1024K +L3 cache: 39424K +NUMA node0 CPU(s): 0-27,56-83 +NUMA node1 CPU(s): 28-55,84-111 +Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca +cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx +pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology +nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est +tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt +tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch +cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp +ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 +hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx +smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 +xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts +pku ospke avx512_vnni md_clear flush_l1d arch_capabilities +``` + +``` +$ lscpu +Architecture: x86_64 +CPU op-mode(s): 32-bit, 64-bit +Byte Order: Little Endian +CPU(s): 96 +On-line CPU(s) list: 0-95 +Thread(s) per core: 2 +Core(s) per socket: 24 +Socket(s): 2 +NUMA node(s): 2 +Vendor ID: GenuineIntel +CPU family: 6 +Model: 85 +Model name: Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz +Stepping: 7 +CPU MHz: 3000.989 +BogoMIPS: 4600.00 +Virtualization: VT-x +L1d cache: 32K +L1i cache: 32K +L2 cache: 1024K +L3 cache: 36608K +NUMA node0 CPU(s): 0-23,48-71 +NUMA node1 CPU(s): 24-47,72-95 +Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca +cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx +pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology +nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 +ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt +tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch +cpuid_fault epb cat_l3 cdp_l3 invpcid_single ssbd mba ibrs ibpb stibp +ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle +avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap +clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 +xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts +pku ospke avx512_vnni md_clear flush_l1d arch_capabilities +``` + +## Linux dmidecode + +``` + # dmidecode 3.1 + Getting SMBIOS data from sysfs. + SMBIOS 3.1.2 present. + Table at 0x6EB92000. + + Handle 0x0000, DMI type 0, 26 bytes + BIOS Information + Vendor: American Megatrends Inc. + Version: 3.0c + Release Date: 03/27/2019 + Address: 0xF0000 + Runtime Size: 64 kB + ROM Size: 32 MB + Characteristics: + PCI is supported + BIOS is upgradeable + BIOS shadowing is allowed + Boot from CD is supported + Selectable boot is supported + BIOS ROM is socketed + EDD is supported + 5.25"/1.2 MB floppy services are supported (int 13h) + 3.5"/720 kB floppy services are supported (int 13h) + 3.5"/2.88 MB floppy services are supported (int 13h) + Print screen service is supported (int 5h) + Serial services are supported (int 14h) + Printer services are supported (int 17h) + ACPI is supported + USB legacy is supported + BIOS boot specification is supported + Targeted content distribution is supported + UEFI is supported + BIOS Revision: 5.14 + + Handle 0x0001, DMI type 1, 27 bytes + System Information + Manufacturer: Supermicro + Product Name: SYS-7049GP-TRT + Version: 0123456789 + Serial Number: S291427X9525476 + UUID: 00000000-0000-0000-0000-AC1F6BACD7BA + Wake-up Type: Power Switch + SKU Number: To be filled by O.E.M. + Family: To be filled by O.E.M. + + Handle 0x0002, DMI type 2, 15 bytes + Base Board Information + Manufacturer: Supermicro + Product Name: X11DPG-QT + Version: 1.10A + Serial Number: VM189S007860 + Asset Tag: To be filled by O.E.M. + Features: + Board is a hosting board + Board is replaceable + Location In Chassis: To be filled by O.E.M. + Chassis Handle: 0x0003 + Type: Motherboard + Contained Object Handles: 0 + + Handle 0x0003, DMI type 3, 22 bytes + Chassis Information + Manufacturer: Supermicro + Type: Other + Lock: Not Present + Version: 0123456789 + Serial Number: C7470KH37A30566 + Asset Tag: To be filled by O.E.M. + Boot-up State: Safe + Power Supply State: Safe + Thermal State: Safe + Security Status: None + OEM Information: 0x00000000 + Height: Unspecified + Number Of Power Cords: 1 + Contained Elements: 0 + SKU Number: To be filled by O.E.M. + + Handle 0x0055, DMI type 4, 48 bytes + Processor Information + Socket Designation: CPU1 + Type: Central Processor + Family: Xeon + Manufacturer: Intel(R) Corporation + ID: 57 06 05 00 FF FB EB BF + Signature: Type 0, Family 6, Model 85, Stepping 7 + Flags: + FPU (Floating-point unit on-chip) + VME (Virtual mode extension) + DE (Debugging extension) + PSE (Page size extension) + TSC (Time stamp counter) + MSR (Model specific registers) + PAE (Physical address extension) + MCE (Machine check exception) + CX8 (CMPXCHG8 instruction supported) + APIC (On-chip APIC hardware supported) + SEP (Fast system call) + MTRR (Memory type range registers) + PGE (Page global enable) + MCA (Machine check architecture) + CMOV (Conditional move instruction supported) + PAT (Page attribute table) + PSE-36 (36-bit page size extension) + CLFSH (CLFLUSH instruction supported) + DS (Debug store) + ACPI (ACPI supported) + MMX (MMX technology supported) + FXSR (FXSAVE and FXSTOR instructions supported) + SSE (Streaming SIMD extensions) + SSE2 (Streaming SIMD extensions 2) + SS (Self-snoop) + HTT (Multi-threading) + TM (Thermal monitor supported) + PBE (Pending break enabled) + Version: Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz + Voltage: 1.6 V + External Clock: 100 MHz + Max Speed: 4500 MHz + Current Speed: 2300 MHz + Status: Populated, Enabled + Upgrade: Socket LGA3647-1 + L1 Cache Handle: 0x0052 + L2 Cache Handle: 0x0053 + L3 Cache Handle: 0x0054 + Serial Number: Not Specified + Asset Tag: UNKNOWN + Part Number: Not Specified + Core Count: 24 + Core Enabled: 24 + Thread Count: 48 + Characteristics: + 64-bit capable + Multi-Core + Hardware Thread + Execute Protection + Enhanced Virtualization + Power/Performance Control + + Handle 0x0059, DMI type 4, 48 bytes + Processor Information + Socket Designation: CPU2 + Type: Central Processor + Family: Xeon + Manufacturer: Intel(R) Corporation + ID: 57 06 05 00 FF FB EB BF + Signature: Type 0, Family 6, Model 85, Stepping 7 + Flags: + FPU (Floating-point unit on-chip) + VME (Virtual mode extension) + DE (Debugging extension) + PSE (Page size extension) + TSC (Time stamp counter) + MSR (Model specific registers) + PAE (Physical address extension) + MCE (Machine check exception) + CX8 (CMPXCHG8 instruction supported) + APIC (On-chip APIC hardware supported) + SEP (Fast system call) + MTRR (Memory type range registers) + PGE (Page global enable) + MCA (Machine check architecture) + CMOV (Conditional move instruction supported) + PAT (Page attribute table) + PSE-36 (36-bit page size extension) + CLFSH (CLFLUSH instruction supported) + DS (Debug store) + ACPI (ACPI supported) + MMX (MMX technology supported) + FXSR (FXSAVE and FXSTOR instructions supported) + SSE (Streaming SIMD extensions) + SSE2 (Streaming SIMD extensions 2) + SS (Self-snoop) + HTT (Multi-threading) + TM (Thermal monitor supported) + PBE (Pending break enabled) + Version: Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz + Voltage: 1.6 V + External Clock: 100 MHz + Max Speed: 4500 MHz + Current Speed: 2300 MHz + Status: Populated, Enabled + Upgrade: Socket LGA3647-1 + L1 Cache Handle: 0x0056 + L2 Cache Handle: 0x0057 + L3 Cache Handle: 0x0058 + Serial Number: Not Specified + Asset Tag: UNKNOWN + Part Number: Not Specified + Core Count: 24 + Core Enabled: 24 + Thread Count: 48 + Characteristics: + 64-bit capable + Multi-Core + Hardware Thread + Execute Protection + Enhanced Virtualization + Power/Performance Control +``` + +## Linux dmidecode pci + +``` + $ dmidecode -t slot + Handle 0x000B, DMI type 9, 17 bytes + System Slot Information + Designation: CPU1 SLOT2 PCI-E 3.0 X16 + Type: x16 PCI Express 3 x16 + Current Usage: In Use + Length: Long + ID: 2 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:18:00.0 + + Handle 0x000C, DMI type 9, 17 bytes + System Slot Information + Designation: CPU1 SLOT4 PCI-E 3.0 X16 + Type: x16 PCI Express 3 x16 + Current Usage: In Use + Length: Short + ID: 4 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:3b:00.0 + + Handle 0x000D, DMI type 9, 17 bytes + System Slot Information + Designation: CPU2 SLOT6 PCI-E 3.0 X16 + Type: x16 PCI Express 3 x16 + Current Usage: Available + Length: Short + ID: 6 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:ff:00.0 + + Handle 0x000E, DMI type 9, 17 bytes + System Slot Information + Designation: CPU2 SLOT8 PCI-E 3.0 X16 + Type: x16 PCI Express 3 x16 + Current Usage: Available + Length: Short + ID: 8 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:ff:00.0 + + Handle 0x000F, DMI type 9, 17 bytes + System Slot Information + Designation: CPU1 SLOT9 PCI-E 3.0 X16 + Type: x16 PCI Express 3 x16 + Current Usage: Available + Length: Short + ID: 9 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:ff:00.0 + + Handle 0x0010, DMI type 9, 17 bytes + System Slot Information + Designation: CPU2 SLOT10 PCI-E 3.0 X16 + Type: x16 PCI Express 3 x16 + Current Usage: Available + Length: Short + ID: 10 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:ff:00.0 + + Handle 0x0011, DMI type 9, 17 bytes + System Slot Information + Designation: CPU2 SLOT11 PCI-E 3.0 X4(IN X8) + Type: x4 PCI Express 3 x8 + Current Usage: Available + Length: Short + ID: 11 + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:ff:00.0 + + Handle 0x0012, DMI type 9, 17 bytes + System Slot Information + Designation: M.2 CONNECTOR + Type: x4 M.2 Socket 2 + Current Usage: Available + Length: Short + Characteristics: + 3.3 V is provided + Opening is shared + PME signal is supported + Bus Address: 0000:ff:00.0 +``` + +## Linux dmidecode memory + +``` + $ dmidecode -t memory + Handle 0x0021, DMI type 16, 23 bytes + Physical Memory Array + Location: System Board Or Motherboard + Use: System Memory + Error Correction Type: Single-bit ECC + Maximum Capacity: 2304 GB + Error Information Handle: Not Provided + Number Of Devices: 4 + + Handle 0x0023, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0021 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P1-DIMMA1 + Bank Locator: P0_Node0_Channel0_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275F0E + Asset Tag: P1-DIMMA1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0024, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0021 + Error Information Handle: Not Provided + Total Width: Unknown + Data Width: Unknown + Size: No Module Installed + Form Factor: DIMM + Set: None + Locator: P1-DIMMA2 + Bank Locator: P0_Node0_Channel0_Dimm1 + Type: Unknown + Type Detail: Unknown + Speed: Unknown + Manufacturer: NO DIMM + Serial Number: NO DIMM + Asset Tag: NO DIMM + Part Number: NO DIMM + Rank: Unknown + Configured Clock Speed: Unknown + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0025, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0021 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P1-DIMMB1 + Bank Locator: P0_Node0_Channel1_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275F1F + Asset Tag: P1-DIMMB1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0027, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0021 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P1-DIMMC1 + Bank Locator: P0_Node0_Channel2_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275F07 + Asset Tag: P1-DIMMC1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x002B, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0029 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P1-DIMMD1 + Bank Locator: P0_Node1_Channel0_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275F02 + Asset Tag: P1-DIMMD1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x002C, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0029 + Error Information Handle: Not Provided + Total Width: Unknown + Data Width: Unknown + Size: No Module Installed + Form Factor: DIMM + Set: None + Locator: P1-DIMMD2 + Bank Locator: P0_Node1_Channel0_Dimm1 + Type: Unknown + Type Detail: Unknown + Speed: Unknown + Manufacturer: NO DIMM + Serial Number: NO DIMM + Asset Tag: NO DIMM + Part Number: NO DIMM + Rank: Unknown + Configured Clock Speed: Unknown + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x002D, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0029 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P1-DIMME1 + Bank Locator: P0_Node1_Channel1_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275F19 + Asset Tag: P1-DIMME1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x002F, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0029 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P1-DIMMF1 + Bank Locator: P0_Node1_Channel2_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275FD3 + Asset Tag: P1-DIMMF1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0031, DMI type 16, 23 bytes + Physical Memory Array + Location: System Board Or Motherboard + Use: System Memory + Error Correction Type: Single-bit ECC + Maximum Capacity: 2304 GB + Error Information Handle: Not Provided + Number Of Devices: 4 + + Handle 0x0033, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0031 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P2-DIMMA1 + Bank Locator: P1_Node0_Channel0_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275FE2 + Asset Tag: P2-DIMMA1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0034, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0031 + Error Information Handle: Not Provided + Total Width: Unknown + Data Width: Unknown + Size: No Module Installed + Form Factor: DIMM + Set: None + Locator: P2-DIMMA2 + Bank Locator: P1_Node0_Channel0_Dimm1 + Type: Unknown + Type Detail: Unknown + Speed: Unknown + Manufacturer: NO DIMM + Serial Number: NO DIMM + Asset Tag: NO DIMM + Part Number: NO DIMM + Rank: Unknown + Configured Clock Speed: Unknown + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0035, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0031 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P2-DIMMB1 + Bank Locator: P1_Node0_Channel1_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93276001 + Asset Tag: P2-DIMMB1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0037, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0031 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P2-DIMMC1 + Bank Locator: P1_Node0_Channel2_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93276005 + Asset Tag: P2-DIMMC1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x0039, DMI type 16, 23 bytes + Physical Memory Array + Location: System Board Or Motherboard + Use: System Memory + Error Correction Type: Single-bit ECC + Maximum Capacity: 2304 GB + Error Information Handle: Not Provided + Number Of Devices: 4 + + Handle 0x003B, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0039 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P2-DIMMD1 + Bank Locator: P1_Node1_Channel0_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275F44 + Asset Tag: P2-DIMMD1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x003C, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0039 + Error Information Handle: Not Provided + Total Width: Unknown + Data Width: Unknown + Size: No Module Installed + Form Factor: DIMM + Set: None + Locator: P2-DIMMD2 + Bank Locator: P1_Node1_Channel0_Dimm1 + Type: Unknown + Type Detail: Unknown + Speed: Unknown + Manufacturer: NO DIMM + Serial Number: NO DIMM + Asset Tag: NO DIMM + Part Number: NO DIMM + Rank: Unknown + Configured Clock Speed: Unknown + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x003D, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0039 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P2-DIMME1 + Bank Locator: P1_Node1_Channel1_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275FDF + Asset Tag: P2-DIMME1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V + + Handle 0x003F, DMI type 17, 84 bytes + Memory Device + Array Handle: 0x0039 + Error Information Handle: Not Provided + Total Width: 72 bits + Data Width: 64 bits + Size: 16384 MB + Form Factor: DIMM + Set: None + Locator: P2-DIMMF1 + Bank Locator: P1_Node1_Channel2_Dimm0 + Type: DDR4 + Type Detail: Synchronous + Speed: 2933 MT/s + Manufacturer: SK Hynix + Serial Number: 93275FDD + Asset Tag: P2-DIMMF1_AssetTag (date:19/22) + Part Number: HMA82GR7CJR8N-WM + Rank: 2 + Configured Clock Speed: 2934 MT/s + Minimum Voltage: 1.2 V + Maximum Voltage: 1.2 V + Configured Voltage: 1.2 V +``` + +## Xeon Clx Server BIOS Configuration + +### Boot Feature + +``` + | Quiet Boot [Enabled] |Boot option | + | | | + | Option ROM Messages [Force BIOS] | | + | Bootup NumLock State [On] | | + | Wait For "F1" If Error [Enabled] | | + | INT19 Trap Response [Immediate] | | + | Re-try Boot [Disabled] | | + | Install Windows 7 USB support [Disabled] | | + | Port 61h Bit-4 Emulation [Disabled] | | + | | | + | Power Configuration | | + | Watch Dog Function [Disabled] | | + | Restore on AC Power Loss [Last State] | | + | Power Button Function [Instant Off] | | +``` + +### CPU Configuration + +``` + | Processor Configuration |Enables Hyper Threading | + | -------------------------------------------------- |(Software Method to | + | Processor BSP Revision 50657 - CLX B1 |Enable/Disable Logical | + | Processor Socket CPU1 | CPU2 |Processor threads. | + | Processor ID 00050657* | 00050657 | | + | Processor Frequency 2.300GHz | 2.300GHz | | + | Processor Max Ratio 17H | 17H | | + | Processor Min Ratio 0AH | 0AH | | + | Microcode Revision 05000021 | 05000021 | | + | L1 Cache RAM 64KB | 64KB | | + | L2 Cache RAM 1024KB | 1024KB | | + | L3 Cache RAM 36608KB | 36608KB | | + | Processor 0 Version | | + | Intel(R) Xeon(R) Platinum 6252 CPU @ 2.30GHz | | + | Processor 1 Version | | + | Intel(R) Xeon(R) Platinum 6252 CPU @ 2.30GHz | | + | | | + | Hyper-Threading [ALL] [Enable] | | + | Core Enabled 0 | | + | Monitor/MWAIT [Auto] | | + | Execute Disable Bit [Enable] | | + | Intel Virtualization Technology [Enable] | | + | PPIN Control [Unlock/Enable] | | + | Hardware Prefetcher [Enable] | | + | Adjacent Cache Prefetch [Enable] | | + | DCU Streamer Prefetcher [Enable] | | + | DCU IP Prefetcher [Enable] | | + | LLC Prefetch [Disable] | | + | Extended APIC [Disable] | | + | AES-NI [Enable] | | + |> Advanced Power Management Configuration | | +``` + +#### Advanced Power Management Configuration + +``` + | Advanced Power Management Configuration |Switch CPU Power Management | + | -------------------------------------------------- |profile | + | Power Technology [Custom] | | + | Power Performance Tuning [BIOS Controls EPB] | | + | ENERGY_PERF_BIAS_CFG mode [Maximum Performance] | | + |> CPU P State Control | | + |> Hardware PM State Control | | + |> CPU C State Control | | + |> Package C State Control | | + |> CPU T State Control | | +``` + +##### CPU P State Control + +``` + | CPU P State Control |Enable/Disable EIST | + | |(P-States) | + | SpeedStep (Pstates) [Disable] | | + | Activate PBF [Disable] | | + | Configure PBF | | + | EIST PSD Function | | +``` + +##### Hardware PM State Control + +``` + | Hardware PM State Control |Disable: Hardware chooses a | + | |P-state based on OS Request | + | Hardware P-States [Disable] |(Legacy P-States) | + | |Native Mode:Hardware | + | |chooses a P-state based on | + | |OS guidance | + | |Out of Band Mode:Hardware | + | |autonomously chooses a | + | |P-state (no OS guidance) | +``` + +##### CPU C State Control + +``` + | CPU C State Control |Autonomous Core C-State | + | |Control | + | Autonomous Core C-State [Disable] | | + | CPU C6 report [Disable] | | + | Enhanced Halt State (C1E) [Disable] | | +``` + +##### Package C State Control + +``` + | Package C State Control |Package C State limit | + | | | + | Package C State [C0/C1 state] | | +``` + +##### CPU T State Control + +``` + | CPU T State Control |Enable/Disable Software | + | |Controlled T-States | + | Software Controlled T-States [Disable] | | +``` + +#### Chipset Configuration + +``` + | WARNING: Setting wrong values in below sections may cause |North Bridge Parameters | + | system to malfunction. | | + |> North Bridge | | + |> South Bridge | | +``` + +##### North Bridge + +``` + |> UPI Configuration |Displays and provides | + |> Memory Configuration |option to change the UPI | + |> IIO Configuration |Settings | +``` + +##### UPI Configuration + +``` + | UPI Configuration |Choose Topology Precedence | + | -------------------------------------------------- |to degrade features if | + | Number of CPU 2 |system options are in | + | Number of Active UPI Link 3 |conflict or choose Feature | + | Current UPI Link Speed Fast |Precedence to degrade | + | Current UPI Link Frequency 10.4 GT/s |topology if system options | + | UPI Global MMIO Low Base / Limit 90000000 / FBFFFFFF |are in conflict. | + | UPI Global MMIO High Base / Limit 0000000000000000 / ... | | + | UPI Pci-e Configuration Base / Size 80000000 / 10000000 | | + | Degrade Precedence [Topology Precedence] | | + | Link L0p Enable [Disable] | | + | Link L1 Enable [Disable] | | + | IO Directory Cache (IODC) [Auto] | | + | SNC [Disable] | | + | XPT Prefetch [Disable] | | + | KTI Prefetch [Enable] | | + | Local/Remote Threshold [Auto] | | + | Stale AtoS [Auto] | | + | LLC dead line alloc [Enable] | | + | Isoc Mode [Auto] | | +``` + +##### Memory Configuration + +``` + | |POR - Enforces Plan Of | + | -------------------------------------------------- |Record restrictions for | + | Integrated Memory Controller (iMC) |DDR4 frequency and voltage | + | -------------------------------------------------- |programming. Disable - | + | |Disables this feature. | + | Enforce POR [Disable] | | + | PPR Type [Auto] | | + | Memory Frequency [2933] | | + | Data Scrambling for DDR4 [Auto] | | + | tCCD_L Relaxation [Auto] | | + | tRWSR Relaxation [Disable] | | + | 2x Refresh [Auto] | | + | Page Policy [Auto] | | + | IMC Interleaving [2-way Interleave] | | + |> Memory Topology | | + |> Memory RAS Configuration | | +``` + +##### IIO Configuration + +``` + | IIO Configuration |Expose IIO DFX devices and | + | -------------------------------------------------- |other CPU devices like PMON | + | | | + | EV DFX Features [Disable] | | + |> CPU1 Configuration | | + |> CPU2 Configuration | | + |> IOAT Configuration | | + |> Intel. VT for Directed I/O (VT-d) | | + |> Intel. VMD technology | | + | | | + | IIO-PCIE Express Global Options | | + | ======================================== | | + | PCI-E Completion Timeout Disable [No] | | +``` + +##### CPU1 Configuration + +``` + | IOU0 (IIO PCIe Br1) [Auto] |Selects PCIe port | + | IOU1 (IIO PCIe Br2) [Auto] |Bifurcation for selected | + | IOU2 (IIO PCIe Br3) [Auto] |slot(s) | + |> CPU1 SLOT2 PCI-E 3.0 X16 | | + |> CPU1 SLOT4 PCI-E 3.0 X16 | | + |> CPU1 SLOT9 PCI-E 3.0 X16 | | +``` + +##### CPU2 Configuration + +``` + | IOU0 (IIO PCIe Br1) [Auto] |Selects PCIe port | + | IOU1 (IIO PCIe Br2) [Auto] |Bifurcation for selected | + | IOU2 (IIO PCIe Br3) [Auto] |slot(s) | + |> CPU2 SLOT6 PCI-E 3.0 X16 | | + |> CPU2 SLOT8 PCI-E 3.0 X16 | | + |> CPU2 SLOT10 PCI-E 3.0 X16 | | +``` + +#### South Bridge + +``` + | |Enables Legacy USB support. | + | USB Module Version 21 |AUTO option disables legacy | + | |support if no USB devices | + | USB Devices: |are connected. DISABLE | + | 1 Keyboard, 1 Mouse, 1 Hub |option will keep USB | + | |devices available only for | + | Legacy USB Support [Enabled] |EFI applications. | + | XHCI Hand-off [Enabled] | | + | Port 60/64 Emulation [Enabled] | | + | PCIe PLL SSC [Disable] | | + | Real USB Wake Up [Enabled] | | + | Front USB Wake Up [Enabled] | | + | | | + | Azalia [Auto] | | + | Azalia PME Enable [Disabled] | | +``` + +### PCIe/PCI/PnP Configuration + +``` + | PCI Bus Driver Version A5.01.18 |Enables or Disables 64bit | + | |capable Devices to be | + | PCI Devices Common Settings: |Decoded in Above 4G Address | + | Above 4G Decoding [Enabled] |Space (Only if System | + | SR-IOV Support [Enabled] |Supports 64 bit PCI | + | MMIO High Base [56T] |Decoding). | + | MMIO High Granularity Size [256G] | | + | Maximum Read Request [Auto] | | + | MMCFG Base [2G] | | + | NVMe Firmware Source [Vendor Defined Fi...] | | + | VGA Priority [Onboard] | | + | CPU1 SLOT2 PCI-E 3.0 X16 OPROM [EFI] | | + | CPU1 SLOT4 PCI-E 3.0 X16 OPROM [EFI] | | + | CPU2 SLOT6 PCI-E 3.0 X16 OPROM [EFI] | | + | CPU2 SLOT8 PCI-E 3.0 X16 OPROM [EFI] | | + | CPU1 SLOT9 PCI-E 3.0 X16 OPROM [EFI] | | + | CPU2 SLOT10 PCI-E 3.0 X16 OPROM [EFI] | | + | CPU2 SLOT11 PCI-E 3.0 X4(IN X8) OPROM [EFI] | | + | M.2 CONNECTOR OPROM [EFI] | | + | Bus Master Enable [Enabled] | | + | Onboard LAN1 Option ROM [EFI] | | + | Onboard Video Option ROM [EFI] | | + |> Network Stack Configuration | | +``` + +### ACPI Settings + +``` + | ACPI Settings |Enable or Disable Non | + | |uniform Memory Access | + | NUMA [Enabled] |(NUMA). | + | WHEA Support [Enabled] | | + | High Precision Event Timer [Enabled] | | +``` + +## Xeon Clx Server Firmware Inventory + +``` +Host. IPMI IP. BIOS. CPLD. CPU Microcode. PCI Bus. X710 Firmware. XXV710 Firmware. i40e. +s32-t14-sut1. 10.30.55.17. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +s33-t27-sut1. 10.30.55.18. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +s34-t27-tg1. 10.30.55.19. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +s35-t28-sut1. 10.30.55.20. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +s36-t28-tg1. 10.30.55.21. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +s37-t29-sut1. 10.30.55.22. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +s38-t29-tg1. 10.30.55.23. 3.0c. 03.B1.05. 05000021. A5.01.18. 6.01 0x80003554 1.1747.0. 6.01 0x80003554 1.1747.0. 2.1.14-k. +``` diff --git a/docs/lab/testbeds_sm_skx_hw_bios_cfg.md b/docs/lab/testbeds_sm_skx_hw_bios_cfg.md index 8871ec781c..d1e0ea30fa 100644 --- a/docs/lab/testbeds_sm_skx_hw_bios_cfg.md +++ b/docs/lab/testbeds_sm_skx_hw_bios_cfg.md @@ -1,8 +1,7 @@ # SuperMicro Skylake Servers - HW and BIOS Configuration -1. [Linux lscpu TODO](#linux-lscpu-todo) -1. [Linux dmidecode pci TODO](#linux-dmidecode-pci-todo) -1. [Linux dmidecode memory TODO](#linux-dmidecode-memory-todo) +1. [Linux lscpu](#linux-lscpu) +1. [Linux dmidecode](#dmidecode) 1. [Xeon Skx Server BIOS Configuration](#xeon-skx-server-bios-configuration) 1. [Boot Feature](#boot-feature) 1. [CPU Configuration](#cpu-configuration) @@ -22,14 +21,249 @@ 1. [South Bridge](#south-bridge) 1. [PCIe/PCI/PnP Configuration](#pciepcipnp-configuration) 1. [ACPI Settings](#acpi-settings) - 1. [DMIDECODE](#dmidecode) 1. [Xeon Skx Server Firmware Inventory](#xeon-skx-server-firmware-inventory) -## Linux lscpu TODO +## Linux lscpu + +``` +$ lscpu +Architecture: x86_64 +CPU op-mode(s): 32-bit, 64-bit +Byte Order: Little Endian +CPU(s): 112 +On-line CPU(s) list: 0-111 +Thread(s) per core: 2 +Core(s) per socket: 28 +Socket(s): 2 +NUMA node(s): 2 +Vendor ID: GenuineIntel +CPU family: 6 +Model: 85 +Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz +Stepping: 4 +CPU MHz: 2500.550 +BogoMIPS: 5000.00 +Virtualization: VT-x +L1d cache: 32K +L1i cache: 32K +L2 cache: 1024K +L3 cache: 39424K +NUMA node0 CPU(s): 0-27,56-83 +NUMA node1 CPU(s): 28-55,84-111 +Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca +cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx +pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology +nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 +ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt +tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch +cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp +tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjustbmi1 hle avx2 smep bmi2 +erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb +intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc +cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke +md_clear flush_l1d +``` + +### Linux dmidecode -## Linux dmidecode pci TODO +``` + # dmidecode 3.1 + Getting SMBIOS data from sysfs. + SMBIOS 3.1.1 present. + Table at 0x000E89C0. + + Handle 0x0000, DMI type 0, 26 bytes + BIOS Information + Vendor: American Megatrends Inc. + Version: 2.0 + Release Date: 11/29/2017 + Address: 0xF0000 + Runtime Size: 64 kB + ROM Size: 64 MB + Characteristics: + PCI is supported + BIOS is upgradeable + BIOS shadowing is allowed + Boot from CD is supported + Selectable boot is supported + BIOS ROM is socketed + EDD is supported + 5.25"/1.2 MB floppy services are supported (int 13h) + 3.5"/720 kB floppy services are supported (int 13h) + 3.5"/2.88 MB floppy services are supported (int 13h) + Print screen service is supported (int 5h) + Serial services are supported (int 14h) + Printer services are supported (int 17h) + ACPI is supported + USB legacy is supported + BIOS boot specification is supported + Targeted content distribution is supported + UEFI is supported + BIOS Revision: 5.12 + + Handle 0x0001, DMI type 1, 27 bytes + System Information + Manufacturer: Supermicro + Product Name: SYS-7049GP-TRT + Version: 0123456789 + Serial Number: S291427X8332242 + UUID: 00000000-0000-0000-0000-AC1F6B8A8DB6 + Wake-up Type: Power Switch + SKU Number: To be filled by O.E.M. + Family: To be filled by O.E.M. + + Handle 0x0002, DMI type 2, 15 bytes + Base Board Information + Manufacturer: Supermicro + Product Name: X11DPG-QT + Version: 1.02 + Serial Number: VM183S014930 + Asset Tag: To be filled by O.E.M. + Features: + Board is a hosting board + Board is replaceable + Location In Chassis: To be filled by O.E.M. + Chassis Handle: 0x0003 + Type: Motherboard + Contained Object Handles: 0 -## Linux dmidecode memory TODO + Handle 0x0003, DMI type 3, 22 bytes + Chassis Information + Manufacturer: Supermicro + Type: Other + Lock: Not Present + Version: 0123456789 + Serial Number: C7470KH06A20167 + Asset Tag: To be filled by O.E.M. + Boot-up State: Safe + Power Supply State: Safe + Thermal State: Safe + Security Status: None + OEM Information: 0x00000000 + + Handle 0x0050, DMI type 4, 48 bytes + Processor Information + Socket Designation: CPU1 + Type: Central Processor + Family: Xeon + Manufacturer: Intel(R) Corporation + ID: 54 06 05 00 FF FB EB BF + Signature: Type 0, Family 6, Model 85, Stepping 4 + Flags: + FPU (Floating-point unit on-chip) + VME (Virtual mode extension) + DE (Debugging extension) + PSE (Page size extension) + TSC (Time stamp counter) + MSR (Model specific registers) + PAE (Physical address extension) + MCE (Machine check exception) + CX8 (CMPXCHG8 instruction supported) + APIC (On-chip APIC hardware supported) + SEP (Fast system call) + MTRR (Memory type range registers) + PGE (Page global enable) + MCA (Machine check architecture) + CMOV (Conditional move instruction supported) + PAT (Page attribute table) + PSE-36 (36-bit page size extension) + CLFSH (CLFLUSH instruction supported) + DS (Debug store) + ACPI (ACPI supported) + MMX (MMX technology supported) + FXSR (FXSAVE and FXSTOR instructions supported) + SSE (Streaming SIMD extensions) + SSE2 (Streaming SIMD extensions 2) + SS (Self-snoop) + HTT (Multi-threading) + TM (Thermal monitor supported) + PBE (Pending break enabled) + Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz + Voltage: 1.6 V + External Clock: 100 MHz + Max Speed: 4000 MHz + Current Speed: 2500 MHz + Status: Populated, Enabled + Upgrade: Other + L1 Cache Handle: 0x004D + L2 Cache Handle: 0x004E + L3 Cache Handle: 0x004F + Serial Number: Not Specified + Asset Tag: UNKNOWN + Part Number: Not Specified + Core Count: 28 + Core Enabled: 28 + Thread Count: 56 + Characteristics: + 64-bit capable + Multi-Core + Hardware Thread + Execute Protection + Enhanced Virtualization + Power/Performance Control + + + Handle 0x0054, DMI type 4, 48 bytes + Processor Information + Socket Designation: CPU2 + Type: Central Processor + Family: Xeon + Manufacturer: Intel(R) Corporation + ID: 54 06 05 00 FF FB EB BF + Signature: Type 0, Family 6, Model 85, Stepping 4 + Flags: + FPU (Floating-point unit on-chip) + VME (Virtual mode extension) + DE (Debugging extension) + PSE (Page size extension) + TSC (Time stamp counter) + MSR (Model specific registers) + PAE (Physical address extension) + MCE (Machine check exception) + CX8 (CMPXCHG8 instruction supported) + APIC (On-chip APIC hardware supported) + SEP (Fast system call) + MTRR (Memory type range registers) + PGE (Page global enable) + MCA (Machine check architecture) + CMOV (Conditional move instruction supported) + PAT (Page attribute table) + PSE-36 (36-bit page size extension) + CLFSH (CLFLUSH instruction supported) + DS (Debug store) + ACPI (ACPI supported) + MMX (MMX technology supported) + FXSR (FXSAVE and FXSTOR instructions supported) + SSE (Streaming SIMD extensions) + SSE2 (Streaming SIMD extensions 2) + SS (Self-snoop) + HTT (Multi-threading) + TM (Thermal monitor supported) + PBE (Pending break enabled) + Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz + Voltage: 1.6 V + External Clock: 100 MHz + Max Speed: 4000 MHz + Current Speed: 2500 MHz + Status: Populated, Enabled + Upgrade: Other + L1 Cache Handle: 0x0051 + L2 Cache Handle: 0x0052 + L3 Cache Handle: 0x0053 + Serial Number: Not Specified + Asset Tag: UNKNOWN + Part Number: Not Specified + Core Count: 28 + Core Enabled: 28 + Thread Count: 56 + Characteristics: + 64-bit capable + Multi-Core + Hardware Thread + Execute Protection + Enhanced Virtualization + Power/Performance Control +``` ## Xeon Skx Server BIOS Configuration @@ -314,207 +548,6 @@ | ACPI Sleep State [S3 (Suspend to RAM)] | | ``` -### DMIDECODE - -``` - # dmidecode 3.1 - Getting SMBIOS data from sysfs. - SMBIOS 3.1.1 present. - Table at 0x000E89C0. - - Handle 0x0000, DMI type 0, 26 bytes - BIOS Information - Vendor: American Megatrends Inc. - Version: 2.0 - Release Date: 11/29/2017 - Address: 0xF0000 - Runtime Size: 64 kB - ROM Size: 64 MB - Characteristics: - PCI is supported - BIOS is upgradeable - BIOS shadowing is allowed - Boot from CD is supported - Selectable boot is supported - BIOS ROM is socketed - EDD is supported - 5.25"/1.2 MB floppy services are supported (int 13h) - 3.5"/720 kB floppy services are supported (int 13h) - 3.5"/2.88 MB floppy services are supported (int 13h) - Print screen service is supported (int 5h) - Serial services are supported (int 14h) - Printer services are supported (int 17h) - ACPI is supported - USB legacy is supported - BIOS boot specification is supported - Targeted content distribution is supported - UEFI is supported - BIOS Revision: 5.12 - - Handle 0x0001, DMI type 1, 27 bytes - System Information - Manufacturer: Supermicro - Product Name: SYS-7049GP-TRT - Version: 0123456789 - Serial Number: S291427X8332242 - UUID: 00000000-0000-0000-0000-AC1F6B8A8DB6 - Wake-up Type: Power Switch - SKU Number: To be filled by O.E.M. - Family: To be filled by O.E.M. - - Handle 0x0002, DMI type 2, 15 bytes - Base Board Information - Manufacturer: Supermicro - Product Name: X11DPG-QT - Version: 1.02 - Serial Number: VM183S014930 - Asset Tag: To be filled by O.E.M. - Features: - Board is a hosting board - Board is replaceable - Location In Chassis: To be filled by O.E.M. - Chassis Handle: 0x0003 - Type: Motherboard - Contained Object Handles: 0 - - Handle 0x0003, DMI type 3, 22 bytes - Chassis Information - Manufacturer: Supermicro - Type: Other - Lock: Not Present - Version: 0123456789 - Serial Number: C7470KH06A20167 - Asset Tag: To be filled by O.E.M. - Boot-up State: Safe - Power Supply State: Safe - Thermal State: Safe - Security Status: None - OEM Information: 0x00000000 - - Handle 0x0050, DMI type 4, 48 bytes - Processor Information - Socket Designation: CPU1 - Type: Central Processor - Family: Xeon - Manufacturer: Intel(R) Corporation - ID: 54 06 05 00 FF FB EB BF - Signature: Type 0, Family 6, Model 85, Stepping 4 - Flags: - FPU (Floating-point unit on-chip) - VME (Virtual mode extension) - DE (Debugging extension) - PSE (Page size extension) - TSC (Time stamp counter) - MSR (Model specific registers) - PAE (Physical address extension) - MCE (Machine check exception) - CX8 (CMPXCHG8 instruction supported) - APIC (On-chip APIC hardware supported) - SEP (Fast system call) - MTRR (Memory type range registers) - PGE (Page global enable) - MCA (Machine check architecture) - CMOV (Conditional move instruction supported) - PAT (Page attribute table) - PSE-36 (36-bit page size extension) - CLFSH (CLFLUSH instruction supported) - DS (Debug store) - ACPI (ACPI supported) - MMX (MMX technology supported) - FXSR (FXSAVE and FXSTOR instructions supported) - SSE (Streaming SIMD extensions) - SSE2 (Streaming SIMD extensions 2) - SS (Self-snoop) - HTT (Multi-threading) - TM (Thermal monitor supported) - PBE (Pending break enabled) - Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz - Voltage: 1.6 V - External Clock: 100 MHz - Max Speed: 4000 MHz - Current Speed: 2500 MHz - Status: Populated, Enabled - Upgrade: Other - L1 Cache Handle: 0x004D - L2 Cache Handle: 0x004E - L3 Cache Handle: 0x004F - Serial Number: Not Specified - Asset Tag: UNKNOWN - Part Number: Not Specified - Core Count: 28 - Core Enabled: 28 - Thread Count: 56 - Characteristics: - 64-bit capable - Multi-Core - Hardware Thread - Execute Protection - Enhanced Virtualization - Power/Performance Control - - - Handle 0x0054, DMI type 4, 48 bytes - Processor Information - Socket Designation: CPU2 - Type: Central Processor - Family: Xeon - Manufacturer: Intel(R) Corporation - ID: 54 06 05 00 FF FB EB BF - Signature: Type 0, Family 6, Model 85, Stepping 4 - Flags: - FPU (Floating-point unit on-chip) - VME (Virtual mode extension) - DE (Debugging extension) - PSE (Page size extension) - TSC (Time stamp counter) - MSR (Model specific registers) - PAE (Physical address extension) - MCE (Machine check exception) - CX8 (CMPXCHG8 instruction supported) - APIC (On-chip APIC hardware supported) - SEP (Fast system call) - MTRR (Memory type range registers) - PGE (Page global enable) - MCA (Machine check architecture) - CMOV (Conditional move instruction supported) - PAT (Page attribute table) - PSE-36 (36-bit page size extension) - CLFSH (CLFLUSH instruction supported) - DS (Debug store) - ACPI (ACPI supported) - MMX (MMX technology supported) - FXSR (FXSAVE and FXSTOR instructions supported) - SSE (Streaming SIMD extensions) - SSE2 (Streaming SIMD extensions 2) - SS (Self-snoop) - HTT (Multi-threading) - TM (Thermal monitor supported) - PBE (Pending break enabled) - Version: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz - Voltage: 1.6 V - External Clock: 100 MHz - Max Speed: 4000 MHz - Current Speed: 2500 MHz - Status: Populated, Enabled - Upgrade: Other - L1 Cache Handle: 0x0051 - L2 Cache Handle: 0x0052 - L3 Cache Handle: 0x0053 - Serial Number: Not Specified - Asset Tag: UNKNOWN - Part Number: Not Specified - Core Count: 28 - Core Enabled: 28 - Thread Count: 56 - Characteristics: - 64-bit capable - Multi-Core - Hardware Thread - Execute Protection - Enhanced Virtualization - Power/Performance Control -``` - ## Xeon Skx Server Firmware Inventory ``` diff --git a/docs/report/introduction/test_environment_intro.rst b/docs/report/introduction/test_environment_intro.rst index 721b4142e5..e0df3b64ff 100644 --- a/docs/report/introduction/test_environment_intro.rst +++ b/docs/report/introduction/test_environment_intro.rst @@ -15,8 +15,8 @@ topology types are used: server as TG both connected in ring topology. Tested SUT servers are based on a range of processors including Intel -Xeon Haswell-SP, Intel Xeon Skylake-SP, Arm, Intel Atom. More detailed -description is provided in +Xeon Haswell-SP, Intel Xeon Skylake-SP, Intel Xeon Cascadelake-SP, Arm, Intel +Atom. More detailed description is provided in :ref:`tested_physical_topologies`. Tested logical topologies are described in :ref:`tested_logical_topologies`. @@ -25,6 +25,7 @@ Server Specifications Complete technical specifications of compute servers used in CSIT physical testbeds are maintained in FD.io CSIT repository: +`FD.io CSIT testbeds - Xeon Cascadelake`_, `FD.io CSIT testbeds - Xeon Skylake, Arm, Atom`_ and `FD.io CSIT Testbeds - Xeon Haswell`_. diff --git a/docs/report/introduction/test_environment_sut_calib_clx.rst b/docs/report/introduction/test_environment_sut_calib_clx.rst new file mode 100644 index 0000000000..43e2f599ea --- /dev/null +++ b/docs/report/introduction/test_environment_sut_calib_clx.rst @@ -0,0 +1,224 @@ +Calibration Data - Cascadelake +------------------------------ + +Following sections include sample calibration data measured on +s32-t27-sut1 server running in one of the Intel Xeon Skylake testbeds as +specified in `FD.io CSIT testbeds - Xeon Cascadelake`_. + +Calibration data obtained from all other servers in Cascadelake testbeds +shows the same or similar values. + + +Linux cmdline +~~~~~~~~~~~~~ + +:: + + $ cat /proc/cmdline + BOOT_IMAGE=/boot/vmlinuz-4.15.0-60-generic root=UUID=1d03969e-a2a0-41b2-a97e-1cc171b07e88 ro isolcpus=1-23,25-47,49-71,73-95 nohz_full=1-23,25-47,49-71,73-95 rcu_nocbs=1-23,25-47,49-71,73-95 numa_balancing=disable intel_pstate=disable intel_iommu=on iommu=pt nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable tsc=reliable mce=off console=tty0 console=ttyS0,115200n8 + +Linux uname +~~~~~~~~~~~ + +:: + + $ uname -a + Linux s32-t27-sut1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux + + +System-level Core Jitter +~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ sudo taskset -c 3 /home/testuser/pma_tools/jitter/jitter -i 30 + Linux Jitter testing program version 1.9 + Iterations=30 + The pragram will execute a dummy function 80000 times + Display is updated every 20000 displayUpdate intervals + Thread affinity will be set to core_id:7 + Timings are in CPU Core cycles + Inst_Min: Minimum Excution time during the display update interval(default is ~1 second) + Inst_Max: Maximum Excution time during the display update interval(default is ~1 second) + Inst_jitter: Jitter in the Excution time during rhe display update interval. This is the value of interest + last_Exec: The Excution time of last iteration just before the display update + Abs_Min: Absolute Minimum Excution time since the program started or statistics were reset + Abs_Max: Absolute Maximum Excution time since the program started or statistics were reset + tmp: Cumulative value calcualted by the dummy function + Interval: Time interval between the display updates in Core Cycles + Sample No: Sample number + + Inst_Min,Inst_Max,Inst_jitter,last_Exec,Abs_min,Abs_max,tmp,Interval,Sample No + 160022,167590,7568,160026,160022,167590,2057568256,3203711852,1 + 160022,170628,10606,160024,160022,170628,4079222784,3204010824,2 + 160022,169824,9802,160024,160022,170628,1805910016,3203812064,3 + 160022,168832,8810,160030,160022,170628,3827564544,3203792594,4 + 160022,168248,8226,160026,160022,170628,1554251776,3203765920,5 + 160022,167834,7812,160028,160022,170628,3575906304,3203761114,6 + 160022,167442,7420,160024,160022,170628,1302593536,3203769250,7 + 160022,169120,9098,160028,160022,170628,3324248064,3203853340,8 + 160022,170710,10688,160024,160022,170710,1050935296,3203985878,9 + 160022,167952,7930,160024,160022,170710,3072589824,3203733756,10 + 160022,168314,8292,160030,160022,170710,799277056,3203741152,11 + 160022,169672,9650,160024,160022,170710,2820931584,3203739910,12 + 160022,168684,8662,160024,160022,170710,547618816,3203727336,13 + 160022,168246,8224,160024,160022,170710,2569273344,3203739052,14 + 160022,168134,8112,160030,160022,170710,295960576,3203735874,15 + 160022,170230,10208,160024,160022,170710,2317615104,3203996356,16 + 160022,167190,7168,160024,160022,170710,44302336,3203713628,17 + 160022,167304,7282,160024,160022,170710,2065956864,3203717954,18 + 160022,167500,7478,160024,160022,170710,4087611392,3203706674,19 + 160022,167302,7280,160024,160022,170710,1814298624,3203726452,20 + 160022,167266,7244,160024,160022,170710,3835953152,3203702804,21 + 160022,167820,7798,160022,160022,170710,1562640384,3203719138,22 + 160022,168100,8078,160024,160022,170710,3584294912,3203716636,23 + 160022,170408,10386,160024,160022,170710,1310982144,3203946958,24 + 160022,167276,7254,160024,160022,170710,3332636672,3203706236,25 + 160022,167052,7030,160024,160022,170710,1059323904,3203696444,26 + 160022,170322,10300,160024,160022,170710,3080978432,3203747514,27 + 160022,167332,7310,160024,160022,170710,807665664,3203716210,28 + 160022,167426,7404,160026,160022,170710,2829320192,3203700630,29 + 160022,168840,8818,160024,160022,170710,556007424,3203727658,30 + + +Memory Bandwidth +~~~~~~~~~~~~~~~~ + +:: + + $ sudo /home/testuser/mlc --bandwidth_matrix + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --bandwidth_matrix + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + Measuring Memory Bandwidths between nodes within system + Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) + Using all the threads from each core if Hyper-threading is enabled + Using Read-only traffic type + Numa node + Numa node 0 1 + 0 122097.7 51327.9 + 1 51309.2 122005.5 + +:: + + $ sudo /home/testuser/mlc --peak_injection_bandwidth + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --peak_injection_bandwidth + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + + Measuring Peak Injection Memory Bandwidths for the system + Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) + Using all the threads from each core if Hyper-threading is enabled + Using traffic with the following read-write ratios + ALL Reads : 243159.4 + 3:1 Reads-Writes : 219132.5 + 2:1 Reads-Writes : 216603.1 + 1:1 Reads-Writes : 203713.0 + Stream-triad like: 193790.8 + +:: + + $ sudo /home/testuser/mlc --max_bandwidth + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --max_bandwidth + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + + Measuring Maximum Memory Bandwidths for the system + Will take several minutes to complete as multiple injection rates will be tried to get the best bandwidth + Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) + Using all the threads from each core if Hyper-threading is enabled + Using traffic with the following read-write ratios + ALL Reads : 244114.27 + 3:1 Reads-Writes : 219441.97 + 2:1 Reads-Writes : 216603.72 + 1:1 Reads-Writes : 203679.09 + Stream-triad like: 214902.80 + + +Memory Latency +~~~~~~~~~~~~~~ + +:: + + $ sudo /home/testuser/mlc --latency_matrix + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --latency_matrix + + Using buffer size of 2000.000MiB + Measuring idle latencies (in ns)... + Numa node + Numa node 0 1 + 0 81.2 130.2 + 1 130.2 81.1 + +:: + + $ sudo /home/testuser/mlc --idle_latency + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --idle_latency + + Using buffer size of 2000.000MiB + Each iteration took 186.1 core clocks ( 80.9 ns) + +:: + + $ sudo /home/testuser/mlc --loaded_latency + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --loaded_latency + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + + Measuring Loaded Latencies for the system + Using all the threads from each core if Hyper-threading is enabled + Using Read-only traffic type + Inject Latency Bandwidth + Delay (ns) MB/sec + ========================== + 00000 233.86 243421.9 + 00002 230.61 243544.1 + 00008 232.56 243394.5 + 00015 229.52 244076.6 + 00050 225.82 244290.6 + 00100 161.65 236744.8 + 00200 100.63 133844.0 + 00300 96.84 90548.2 + 00400 95.71 68504.3 + 00500 95.68 55139.0 + 00700 88.77 39798.4 + 01000 84.74 28200.1 + 01300 83.08 21915.5 + 01700 82.27 16969.3 + 02500 81.66 11810.6 + 03500 81.98 8662.9 + 05000 81.48 6306.8 + 09000 81.17 3857.8 + 20000 80.19 2179.9 + + +L1/L2/LLC Latency +~~~~~~~~~~~~~~~~~ + +:: + + $ sudo /home/testuser/mlc --c2c_latency + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --c2c_latency + + Measuring cache-to-cache transfer latency (in ns)... + Local Socket L2->L2 HIT latency 55.5 + Local Socket L2->L2 HITM latency 55.6 + Remote Socket L2->L2 HITM latency (data address homed in writer socket) + Reader Numa Node + Writer Numa Node 0 1 + 0 - 115.6 + 1 115.6 - + Remote Socket L2->L2 HITM latency (data address homed in reader socket) + Reader Numa Node + Writer Numa Node 0 1 + 0 - 178.2 + 1 178.4 - + +.. include:: ../introduction/test_environment_sut_meltspec_clx.rst diff --git a/docs/report/introduction/test_environment_sut_meltspec_clx.rst b/docs/report/introduction/test_environment_sut_meltspec_clx.rst new file mode 100644 index 0000000000..9056be839b --- /dev/null +++ b/docs/report/introduction/test_environment_sut_meltspec_clx.rst @@ -0,0 +1,154 @@ +Spectre and Meltdown Checks +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Following section displays the output of a running shell script to tell if +system is vulnerable against the several speculative execution CVEs that were +made public in 2018. Script is available on `Spectre & Meltdown Checker Github +<https://github.com/speed47/spectre-meltdown-checker>`_. + +:: + + Spectre and Meltdown mitigation detection tool v0.42 + + Checking for vulnerabilities on current system + Kernel is Linux 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 + CPU is Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz + + Hardware check + * Hardware support (CPU microcode) for mitigation techniques + * Indirect Branch Restricted Speculation (IBRS) + * SPEC_CTRL MSR is available: YES + * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) + * Indirect Branch Prediction Barrier (IBPB) + * PRED_CMD MSR is available: YES + * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) + * Single Thread Indirect Branch Predictors (STIBP) + * SPEC_CTRL MSR is available: YES + * CPU indicates STIBP capability: YES (Intel STIBP feature bit) + * Speculative Store Bypass Disable (SSBD) + * CPU indicates SSBD capability: YES (Intel SSBD) + * L1 data cache invalidation + * FLUSH_CMD MSR is available: YES + * CPU indicates L1D flush capability: YES (L1D flush feature bit) + * Microarchitecture Data Sampling + * VERW instruction is available: YES (MD_CLEAR feature bit) + * Enhanced IBRS (IBRS_ALL) + * CPU indicates ARCH_CAPABILITIES MSR availability: YES + * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES + * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES + * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO + * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES + * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO + * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): YES + * CPU supports Software Guard Extensions (SGX): NO + * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x7 ucode 0x5000021 cpuid 0x50657) + * CPU microcode is the latest known available version: awk: fatal: cannot open file `bash' for reading (No file or directory) + UNKNOWN (latest microcode version for your CPU model is unknown) + * CPU vulnerability to the speculative execution attack variants + * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES + * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES + * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO + * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES + * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES + * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO + * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO + * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO + * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): NO + * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): NO + * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): NO + * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)): NO + + CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass' + * Mitigated according to the /sys interface: YES (Mitigation: usercopy/swapgs barriers and __user pointer saniation) + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has mask_nospec64 (arm64): NO + > STATUS: NOT VULNERABLE (Mitigation: usercopy/swapgs barriers and __user pointer sanitization) + + CVE-2017-5715 aka 'Spectre Variant 2, branch target injection' + * Mitigated according to the /sys interface: YES (Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling) + * Mitigation 1 + * Kernel is compiled with IBRS support: YES + * IBRS enabled and active: YES (Enhanced flavor, performance impact will be greatly reduced) + * Kernel is compiled with IBPB support: YES + * IBPB enabled and active: YES + * Mitigation 2 + * Kernel has branch predictor hardening (arm): NO + * Kernel compiled with retpoline option: YES + * Kernel supports RSB filling: YES + > STATUS: NOT VULNERABLE (Enhanced IBRS + IBPB are mitigating the vulnerability) + + CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports Page Table Isolation (PTI): YES + * PTI enabled and active: NO + * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced) + * Running as a Xen PV DomU: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3640 aka 'Variant 3a, rogue system register read' + * CPU microcode mitigates the vulnerability: YES + > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) + + CVE-2018-3639 aka 'Variant 4, speculative store bypass' + * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + * Kernel supports disabling speculative store bypass (SSB): YES (found in /proc/self/status) + * SSB mitigation is enabled and active: YES (per-thread through prctl) + * SSB mitigation currently active for selected processes: YES ((deleted) systemd-journald systemd-logind systemd-networkd systemd-resolved systemd-timesyncd systemd-udevd) + > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + + CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault' + * CPU microcode mitigates the vulnerability: N/A + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports PTE inversion: YES (found in kernel image) + * PTE inversion enabled and active: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault' + * Information from the /sys interface: Not affected + * This system is a host running a hypervisor: NO + * Mitigation 1 (KVM) + * EPT is disabled: NO + * Mitigation 2 + * L1D flush is supported by kernel: YES (found flush_l1d in /proc/cpuinfo) + * L1D flush enabled: NO + * Hardware-backed L1D flush supported: YES (performance impact of the mitigation will be greatly reduced) + + * Hyper-Threading (SMT) is enabled: YES + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12126 aka 'Fallout, microarchitectural store buffer data sampling (MSBDS)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12130 aka 'ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12127 aka 'RIDL, microarchitectural load port data sampling (MLPDS)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2019-11091 aka 'RIDL, microarchitectural data sampling uncacheable memory (MDSUM)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:OK CVE-2018-12130:OK CVE-2018-12127:OK CVE-2019-11091:OK + diff --git a/resources/tools/presentation/conf.py b/resources/tools/presentation/conf.py index 5083850d73..a4131033cb 100644 --- a/resources/tools/presentation/conf.py +++ b/resources/tools/presentation/conf.py @@ -90,6 +90,7 @@ rst_epilog = """ .. _VPP test framework documentation: https://docs.fd.io/vpp/{vpprelease}/vpp_make_test/html/ .. _FD.io CSIT Testbeds - Xeon Haswell: https://git.fd.io/csit/tree/docs/lab/testbeds_ucs_hsw_hw_bios_cfg.md?h={release} .. _FD.io CSIT testbeds - Xeon Skylake, Arm, Atom: https://git.fd.io/csit/tree/docs/lab/testbeds_sm_skx_hw_bios_cfg.md?h={release} +.. _FD.io CSIT testbeds - Xeon Cascadelake: https://git.fd.io/csit/tree/docs/lab/testbeds_sm_clx_hw_bios_cfg.md?h={release} .. _Ansible inventory - hosts: https://git.fd.io/csit/tree/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars?h={release} .. _FD.io test executor dpdk performance job 3n-hsw: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-verify-{srelease}-3n-hsw .. _FD.io test executor dpdk performance job 3n-skx: https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-verify-{srelease}-3n-skx diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.17.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.17.yaml new file mode 100644 index 0000000000..88ebbfccf6 --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.17.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.17.yaml + +hostname: "s32-t14-sut1" +grub: + isolcpus: "1-27,29-55,57-83,85-111" + nohz_full: "1-27,29-55,57-83,85-111" + rcu_nocbs: "1-27,29-55,57-83,85-111" +sysctl: + kernel: + watchdog_cpumask: "0,28,56,84" + vm: + nr_hugepages: 65536 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.17' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.18.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.18.yaml new file mode 100644 index 0000000000..84da961557 --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.18.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.18.yaml + +hostname: "s33-t27-sut1" +grub: + isolcpus: "1-23,25-47,49-71,73-95" + nohz_full: "1-23,25-47,49-71,73-95" + rcu_nocbs: "1-23,25-47,49-71,73-95" +sysctl: + kernel: + watchdog_cpumask: "0,24,48,72" + vm: + nr_hugepages: 65536 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.18' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.19.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.19.yaml new file mode 100644 index 0000000000..e641f184d6 --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.19.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.19.yaml + +hostname: "s34-t27-tg1" +grub: + isolcpus: "1-27,29-55,57-83,85-111" + nohz_full: "1-27,29-55,57-83,85-111" + rcu_nocbs: "1-27,29-55,57-83,85-111" +sysctl: + kernel: + watchdog_cpumask: "0,28,56,84" + vm: + nr_hugepages: 4096 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.19' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.20.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.20.yaml new file mode 100644 index 0000000000..52a48be962 --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.20.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.20.yaml + +hostname: "s35-t28-sut1" +grub: + isolcpus: "1-23,25-47,49-71,73-95" + nohz_full: "1-23,25-47,49-71,73-95" + rcu_nocbs: "1-23,25-47,49-71,73-95" +sysctl: + kernel: + watchdog_cpumask: "0,24,48,72" + vm: + nr_hugepages: 65536 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.20' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.21.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.21.yaml new file mode 100644 index 0000000000..1543052eed --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.21.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.21.yaml + +hostname: "s36-t28-tg1" +grub: + isolcpus: "1-27,29-55,57-83,85-111" + nohz_full: "1-27,29-55,57-83,85-111" + rcu_nocbs: "1-27,29-55,57-83,85-111" +sysctl: + kernel: + watchdog_cpumask: "0,28,56,84" + vm: + nr_hugepages: 4096 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.21' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.22.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.22.yaml new file mode 100644 index 0000000000..681912dc41 --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.22.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.22.yaml + +hostname: "s37-t29-sut1" +grub: + isolcpus: "1-23,25-47,49-71,73-95" + nohz_full: "1-23,25-47,49-71,73-95" + rcu_nocbs: "1-23,25-47,49-71,73-95" +sysctl: + kernel: + watchdog_cpumask: "0,24,48,72" + vm: + nr_hugepages: 65536 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.22' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.23.yaml b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.23.yaml new file mode 100644 index 0000000000..186944f9a3 --- /dev/null +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/host_vars/10.32.8.23.yaml @@ -0,0 +1,18 @@ +--- +# file: host_vars/10.32.8.23.yaml + +hostname: "s38-t29-tg1" +grub: + isolcpus: "1-27,29-55,57-83,85-111" + nohz_full: "1-27,29-55,57-83,85-111" + rcu_nocbs: "1-27,29-55,57-83,85-111" +sysctl: + kernel: + watchdog_cpumask: "0,28,56,84" + vm: + nr_hugepages: 4096 + max_map_count: 20000 + +inventory_ipmi_hostname: '10.30.55.23' +cobbler_profile: 'ubuntu-18.04.2-server-x86_64' +cpu_microarchitecture: "cascadelake" diff --git a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/hosts b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/hosts index af29bea2d2..fe15007370 100644 --- a/resources/tools/testbed-setup/ansible/inventories/lf_inventory/hosts +++ b/resources/tools/testbed-setup/ansible/inventories/lf_inventory/hosts @@ -13,6 +13,9 @@ all: 10.30.51.57: #s10-t24-tg1 - skylake 10.30.51.60: #s16-t32-tg1 - skylake 10.32.8.10: #s28-t26t35-tg1 - skylake + 10.32.8.19: #s34-t27-tg1 - cascadelake + 10.32.8.21: #s36-t28-tg1 - cascadelake + 10.32.8.23: #s38-t29-tg1 - cascadelake sut: hosts: 10.30.51.17: #t1-sut1 - haswell @@ -34,7 +37,10 @@ all: 10.32.8.11: #s29-t26-sut1 - denverton 10.32.8.12: #s30-t35-sut1 - denverton 10.32.8.13: #s31-t35-sut2 - denverton - + 10.32.8.17: #s32-t14-sut1 - cascadelake + 10.32.8.18: #s33-t27-sut1 - cascadelake + 10.32.8.20: #s35-t28-sut1 - cascadelake + 10.32.8.22: #s37-t29-sut1 - cascadelake vpp_device: hosts: 10.30.51.50: #s1-t11-sut1 - skylake diff --git a/resources/tools/testbed-setup/ansible/inventories/sample_inventory/hosts b/resources/tools/testbed-setup/ansible/inventories/sample_inventory/hosts index da5d7f1634..e8e1439db4 100644 --- a/resources/tools/testbed-setup/ansible/inventories/sample_inventory/hosts +++ b/resources/tools/testbed-setup/ansible/inventories/sample_inventory/hosts @@ -1,11 +1,9 @@ all: children: - skylake: # Architecture: (options) skylake, haswell, taishan... - children: - tg: - hosts: - 1.1.1.1: null #t1-tg - sut: - hosts: - 2.2.2.2: null #t1-sut1 - 3.3.3.3: null #t1-sut2 + tg: + hosts: + 1.1.1.1: #t1-tg + sut: + hosts: + 2.2.2.2: #t1-sut1 + 3.3.3.3: #t1-sut2 diff --git a/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/main.yaml b/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/main.yaml index 7f74beec51..a2d2c2061a 100644 --- a/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/main.yaml +++ b/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/main.yaml @@ -9,9 +9,9 @@ include_tasks: '{{ ansible_machine }}.yaml' tags: [ disable-turbo-boost, set-grub, install-pip ] -- name: Skylake specific - import_tasks: skylake.yaml - when: cpu_microarchitecture == "skylake" +- name: Turbo boost specific + import_tasks: turbo_boost.yaml + when: cpu_microarchitecture == "skylake" or cpu_microarchitecture == "cascadelake" tags: [ install-msr, disable-turbo-boost ] - name: Copy netplan network config file diff --git a/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/skylake.yaml b/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/turbo_boost.yaml index 33e061e724..1002792aa8 100644 --- a/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/skylake.yaml +++ b/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/turbo_boost.yaml @@ -1,5 +1,5 @@ --- -# file: roles/tg_sut/tasks/skylake.yaml +# file: roles/tg_sut/tasks/turbo_boost.yaml - name: Install msr-tools apt: @@ -9,7 +9,7 @@ install_recommends: False tags: install-msr -- name: Disable Skylake Turbo-Boost by service +- name: Disable Turbo-Boost by service copy: src: 'files/disable-turbo-boost.service' dest: '/etc/systemd/system/disable-turbo-boost.service' @@ -18,7 +18,7 @@ mode: '0644' tags: disable-turbo-boost -- name: Disable Skylake Turbo-Boost by service on startup +- name: Disable Turbo-Boost by service on startup service: name: disable-turbo-boost enabled: yes diff --git a/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/ubuntu_bionic.yaml b/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/ubuntu_bionic.yaml index 6e8dee6cf1..a8c2266201 100644 --- a/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/ubuntu_bionic.yaml +++ b/resources/tools/testbed-setup/ansible/roles/tg_sut/tasks/ubuntu_bionic.yaml @@ -5,7 +5,7 @@ apt: name: - 'python-dev' - - 'python-virtualenv' + - 'virtualenv' - 'python-pip' - 'libpcap-dev' - 'cpufrequtils' |