diff options
Diffstat (limited to 'docs/report')
3 files changed, 381 insertions, 2 deletions
diff --git a/docs/report/introduction/test_environment_intro.rst b/docs/report/introduction/test_environment_intro.rst index 721b4142e5..e0df3b64ff 100644 --- a/docs/report/introduction/test_environment_intro.rst +++ b/docs/report/introduction/test_environment_intro.rst @@ -15,8 +15,8 @@ topology types are used: server as TG both connected in ring topology. Tested SUT servers are based on a range of processors including Intel -Xeon Haswell-SP, Intel Xeon Skylake-SP, Arm, Intel Atom. More detailed -description is provided in +Xeon Haswell-SP, Intel Xeon Skylake-SP, Intel Xeon Cascadelake-SP, Arm, Intel +Atom. More detailed description is provided in :ref:`tested_physical_topologies`. Tested logical topologies are described in :ref:`tested_logical_topologies`. @@ -25,6 +25,7 @@ Server Specifications Complete technical specifications of compute servers used in CSIT physical testbeds are maintained in FD.io CSIT repository: +`FD.io CSIT testbeds - Xeon Cascadelake`_, `FD.io CSIT testbeds - Xeon Skylake, Arm, Atom`_ and `FD.io CSIT Testbeds - Xeon Haswell`_. diff --git a/docs/report/introduction/test_environment_sut_calib_clx.rst b/docs/report/introduction/test_environment_sut_calib_clx.rst new file mode 100644 index 0000000000..43e2f599ea --- /dev/null +++ b/docs/report/introduction/test_environment_sut_calib_clx.rst @@ -0,0 +1,224 @@ +Calibration Data - Cascadelake +------------------------------ + +Following sections include sample calibration data measured on +s32-t27-sut1 server running in one of the Intel Xeon Skylake testbeds as +specified in `FD.io CSIT testbeds - Xeon Cascadelake`_. + +Calibration data obtained from all other servers in Cascadelake testbeds +shows the same or similar values. + + +Linux cmdline +~~~~~~~~~~~~~ + +:: + + $ cat /proc/cmdline + BOOT_IMAGE=/boot/vmlinuz-4.15.0-60-generic root=UUID=1d03969e-a2a0-41b2-a97e-1cc171b07e88 ro isolcpus=1-23,25-47,49-71,73-95 nohz_full=1-23,25-47,49-71,73-95 rcu_nocbs=1-23,25-47,49-71,73-95 numa_balancing=disable intel_pstate=disable intel_iommu=on iommu=pt nmi_watchdog=0 audit=0 nosoftlockup processor.max_cstate=1 intel_idle.max_cstate=1 hpet=disable tsc=reliable mce=off console=tty0 console=ttyS0,115200n8 + +Linux uname +~~~~~~~~~~~ + +:: + + $ uname -a + Linux s32-t27-sut1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux + + +System-level Core Jitter +~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ sudo taskset -c 3 /home/testuser/pma_tools/jitter/jitter -i 30 + Linux Jitter testing program version 1.9 + Iterations=30 + The pragram will execute a dummy function 80000 times + Display is updated every 20000 displayUpdate intervals + Thread affinity will be set to core_id:7 + Timings are in CPU Core cycles + Inst_Min: Minimum Excution time during the display update interval(default is ~1 second) + Inst_Max: Maximum Excution time during the display update interval(default is ~1 second) + Inst_jitter: Jitter in the Excution time during rhe display update interval. This is the value of interest + last_Exec: The Excution time of last iteration just before the display update + Abs_Min: Absolute Minimum Excution time since the program started or statistics were reset + Abs_Max: Absolute Maximum Excution time since the program started or statistics were reset + tmp: Cumulative value calcualted by the dummy function + Interval: Time interval between the display updates in Core Cycles + Sample No: Sample number + + Inst_Min,Inst_Max,Inst_jitter,last_Exec,Abs_min,Abs_max,tmp,Interval,Sample No + 160022,167590,7568,160026,160022,167590,2057568256,3203711852,1 + 160022,170628,10606,160024,160022,170628,4079222784,3204010824,2 + 160022,169824,9802,160024,160022,170628,1805910016,3203812064,3 + 160022,168832,8810,160030,160022,170628,3827564544,3203792594,4 + 160022,168248,8226,160026,160022,170628,1554251776,3203765920,5 + 160022,167834,7812,160028,160022,170628,3575906304,3203761114,6 + 160022,167442,7420,160024,160022,170628,1302593536,3203769250,7 + 160022,169120,9098,160028,160022,170628,3324248064,3203853340,8 + 160022,170710,10688,160024,160022,170710,1050935296,3203985878,9 + 160022,167952,7930,160024,160022,170710,3072589824,3203733756,10 + 160022,168314,8292,160030,160022,170710,799277056,3203741152,11 + 160022,169672,9650,160024,160022,170710,2820931584,3203739910,12 + 160022,168684,8662,160024,160022,170710,547618816,3203727336,13 + 160022,168246,8224,160024,160022,170710,2569273344,3203739052,14 + 160022,168134,8112,160030,160022,170710,295960576,3203735874,15 + 160022,170230,10208,160024,160022,170710,2317615104,3203996356,16 + 160022,167190,7168,160024,160022,170710,44302336,3203713628,17 + 160022,167304,7282,160024,160022,170710,2065956864,3203717954,18 + 160022,167500,7478,160024,160022,170710,4087611392,3203706674,19 + 160022,167302,7280,160024,160022,170710,1814298624,3203726452,20 + 160022,167266,7244,160024,160022,170710,3835953152,3203702804,21 + 160022,167820,7798,160022,160022,170710,1562640384,3203719138,22 + 160022,168100,8078,160024,160022,170710,3584294912,3203716636,23 + 160022,170408,10386,160024,160022,170710,1310982144,3203946958,24 + 160022,167276,7254,160024,160022,170710,3332636672,3203706236,25 + 160022,167052,7030,160024,160022,170710,1059323904,3203696444,26 + 160022,170322,10300,160024,160022,170710,3080978432,3203747514,27 + 160022,167332,7310,160024,160022,170710,807665664,3203716210,28 + 160022,167426,7404,160026,160022,170710,2829320192,3203700630,29 + 160022,168840,8818,160024,160022,170710,556007424,3203727658,30 + + +Memory Bandwidth +~~~~~~~~~~~~~~~~ + +:: + + $ sudo /home/testuser/mlc --bandwidth_matrix + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --bandwidth_matrix + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + Measuring Memory Bandwidths between nodes within system + Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) + Using all the threads from each core if Hyper-threading is enabled + Using Read-only traffic type + Numa node + Numa node 0 1 + 0 122097.7 51327.9 + 1 51309.2 122005.5 + +:: + + $ sudo /home/testuser/mlc --peak_injection_bandwidth + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --peak_injection_bandwidth + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + + Measuring Peak Injection Memory Bandwidths for the system + Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) + Using all the threads from each core if Hyper-threading is enabled + Using traffic with the following read-write ratios + ALL Reads : 243159.4 + 3:1 Reads-Writes : 219132.5 + 2:1 Reads-Writes : 216603.1 + 1:1 Reads-Writes : 203713.0 + Stream-triad like: 193790.8 + +:: + + $ sudo /home/testuser/mlc --max_bandwidth + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --max_bandwidth + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + + Measuring Maximum Memory Bandwidths for the system + Will take several minutes to complete as multiple injection rates will be tried to get the best bandwidth + Bandwidths are in MB/sec (1 MB/sec = 1,000,000 Bytes/sec) + Using all the threads from each core if Hyper-threading is enabled + Using traffic with the following read-write ratios + ALL Reads : 244114.27 + 3:1 Reads-Writes : 219441.97 + 2:1 Reads-Writes : 216603.72 + 1:1 Reads-Writes : 203679.09 + Stream-triad like: 214902.80 + + +Memory Latency +~~~~~~~~~~~~~~ + +:: + + $ sudo /home/testuser/mlc --latency_matrix + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --latency_matrix + + Using buffer size of 2000.000MiB + Measuring idle latencies (in ns)... + Numa node + Numa node 0 1 + 0 81.2 130.2 + 1 130.2 81.1 + +:: + + $ sudo /home/testuser/mlc --idle_latency + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --idle_latency + + Using buffer size of 2000.000MiB + Each iteration took 186.1 core clocks ( 80.9 ns) + +:: + + $ sudo /home/testuser/mlc --loaded_latency + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --loaded_latency + + Using buffer size of 100.000MiB/thread for reads and an additional 100.000MiB/thread for writes + + Measuring Loaded Latencies for the system + Using all the threads from each core if Hyper-threading is enabled + Using Read-only traffic type + Inject Latency Bandwidth + Delay (ns) MB/sec + ========================== + 00000 233.86 243421.9 + 00002 230.61 243544.1 + 00008 232.56 243394.5 + 00015 229.52 244076.6 + 00050 225.82 244290.6 + 00100 161.65 236744.8 + 00200 100.63 133844.0 + 00300 96.84 90548.2 + 00400 95.71 68504.3 + 00500 95.68 55139.0 + 00700 88.77 39798.4 + 01000 84.74 28200.1 + 01300 83.08 21915.5 + 01700 82.27 16969.3 + 02500 81.66 11810.6 + 03500 81.98 8662.9 + 05000 81.48 6306.8 + 09000 81.17 3857.8 + 20000 80.19 2179.9 + + +L1/L2/LLC Latency +~~~~~~~~~~~~~~~~~ + +:: + + $ sudo /home/testuser/mlc --c2c_latency + Intel(R) Memory Latency Checker - v3.7 + Command line parameters: --c2c_latency + + Measuring cache-to-cache transfer latency (in ns)... + Local Socket L2->L2 HIT latency 55.5 + Local Socket L2->L2 HITM latency 55.6 + Remote Socket L2->L2 HITM latency (data address homed in writer socket) + Reader Numa Node + Writer Numa Node 0 1 + 0 - 115.6 + 1 115.6 - + Remote Socket L2->L2 HITM latency (data address homed in reader socket) + Reader Numa Node + Writer Numa Node 0 1 + 0 - 178.2 + 1 178.4 - + +.. include:: ../introduction/test_environment_sut_meltspec_clx.rst diff --git a/docs/report/introduction/test_environment_sut_meltspec_clx.rst b/docs/report/introduction/test_environment_sut_meltspec_clx.rst new file mode 100644 index 0000000000..9056be839b --- /dev/null +++ b/docs/report/introduction/test_environment_sut_meltspec_clx.rst @@ -0,0 +1,154 @@ +Spectre and Meltdown Checks +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Following section displays the output of a running shell script to tell if +system is vulnerable against the several speculative execution CVEs that were +made public in 2018. Script is available on `Spectre & Meltdown Checker Github +<https://github.com/speed47/spectre-meltdown-checker>`_. + +:: + + Spectre and Meltdown mitigation detection tool v0.42 + + Checking for vulnerabilities on current system + Kernel is Linux 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 + CPU is Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz + + Hardware check + * Hardware support (CPU microcode) for mitigation techniques + * Indirect Branch Restricted Speculation (IBRS) + * SPEC_CTRL MSR is available: YES + * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) + * Indirect Branch Prediction Barrier (IBPB) + * PRED_CMD MSR is available: YES + * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) + * Single Thread Indirect Branch Predictors (STIBP) + * SPEC_CTRL MSR is available: YES + * CPU indicates STIBP capability: YES (Intel STIBP feature bit) + * Speculative Store Bypass Disable (SSBD) + * CPU indicates SSBD capability: YES (Intel SSBD) + * L1 data cache invalidation + * FLUSH_CMD MSR is available: YES + * CPU indicates L1D flush capability: YES (L1D flush feature bit) + * Microarchitecture Data Sampling + * VERW instruction is available: YES (MD_CLEAR feature bit) + * Enhanced IBRS (IBRS_ALL) + * CPU indicates ARCH_CAPABILITIES MSR availability: YES + * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: YES + * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES + * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO + * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES + * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO + * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): YES + * CPU supports Software Guard Extensions (SGX): NO + * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x7 ucode 0x5000021 cpuid 0x50657) + * CPU microcode is the latest known available version: awk: fatal: cannot open file `bash' for reading (No file or directory) + UNKNOWN (latest microcode version for your CPU model is unknown) + * CPU vulnerability to the speculative execution attack variants + * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES + * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES + * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO + * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES + * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES + * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO + * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO + * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO + * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): NO + * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): NO + * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): NO + * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)): NO + + CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass' + * Mitigated according to the /sys interface: YES (Mitigation: usercopy/swapgs barriers and __user pointer saniation) + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has mask_nospec64 (arm64): NO + > STATUS: NOT VULNERABLE (Mitigation: usercopy/swapgs barriers and __user pointer sanitization) + + CVE-2017-5715 aka 'Spectre Variant 2, branch target injection' + * Mitigated according to the /sys interface: YES (Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling) + * Mitigation 1 + * Kernel is compiled with IBRS support: YES + * IBRS enabled and active: YES (Enhanced flavor, performance impact will be greatly reduced) + * Kernel is compiled with IBPB support: YES + * IBPB enabled and active: YES + * Mitigation 2 + * Kernel has branch predictor hardening (arm): NO + * Kernel compiled with retpoline option: YES + * Kernel supports RSB filling: YES + > STATUS: NOT VULNERABLE (Enhanced IBRS + IBPB are mitigating the vulnerability) + + CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports Page Table Isolation (PTI): YES + * PTI enabled and active: NO + * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced) + * Running as a Xen PV DomU: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3640 aka 'Variant 3a, rogue system register read' + * CPU microcode mitigates the vulnerability: YES + > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) + + CVE-2018-3639 aka 'Variant 4, speculative store bypass' + * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + * Kernel supports disabling speculative store bypass (SSB): YES (found in /proc/self/status) + * SSB mitigation is enabled and active: YES (per-thread through prctl) + * SSB mitigation currently active for selected processes: YES ((deleted) systemd-journald systemd-logind systemd-networkd systemd-resolved systemd-timesyncd systemd-udevd) + > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + + CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault' + * CPU microcode mitigates the vulnerability: N/A + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports PTE inversion: YES (found in kernel image) + * PTE inversion enabled and active: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault' + * Information from the /sys interface: Not affected + * This system is a host running a hypervisor: NO + * Mitigation 1 (KVM) + * EPT is disabled: NO + * Mitigation 2 + * L1D flush is supported by kernel: YES (found flush_l1d in /proc/cpuinfo) + * L1D flush enabled: NO + * Hardware-backed L1D flush supported: YES (performance impact of the mitigation will be greatly reduced) + + * Hyper-Threading (SMT) is enabled: YES + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12126 aka 'Fallout, microarchitectural store buffer data sampling (MSBDS)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12130 aka 'ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12127 aka 'RIDL, microarchitectural load port data sampling (MLPDS)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2019-11091 aka 'RIDL, microarchitectural data sampling uncacheable memory (MDSUM)' + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:OK CVE-2018-12130:OK CVE-2018-12127:OK CVE-2019-11091:OK + |