diff options
| author |   Peter Mikus <pmikus@cisco.com> | 2019-08-09 07:48:43 +0000 |
|---|---|---|
| committer |   Tibor Frank <tifrank@cisco.com> | 2019-08-09 08:29:19 +0000 |
| commit | ce1c52b1fd27d3e2b6c4909219fa98418565ba61 (patch) | |
| tree | 80d95c9987fbf21e0d41507b4740034562912812 /docs | |
| parent | d6c832b02ac7d3073dd8651b089a72ec8240b183 (diff) | |
DOC: rls1908 static content
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Ia0778acc543a51fe85b8a75162f12905badaa382
Diffstat (limited to 'docs')
23 files changed, 1054 insertions, 1204 deletions
diff --git a/docs/fdio_csit_dev_plan.txt b/docs/fdio_csit_dev_plan.txt deleted file mode 100644 index acf01657fb..0000000000 --- a/docs/fdio_csit_dev_plan.txt +++ /dev/null @@ -1,84 +0,0 @@ -fdio_csit_dev_plan.txt - DRAFT - -FD.io CSIT High-Level Development Plan -====================================== - -Proposed Work Organisation --------------------------- - -* Each work area is covered by owners. - - Technical Lead (TL) - overall responsibility incl. design, detailed - work plan, DT coordination, managing dependencies. - - Development Team (DT) - doing work following the TL. - - Project Lead (PL) - involved in all work areas at higher level, - focusing on requirements definitions, design and work reviews, - acceptance. - - For work areas with large number of deliverables there could be - multiple TLs e.g. for operations, framework or test refactor. - - FD.io CSIT owners' initials listed with suffix strings: - "[TL;DT1..DTn;PL]", contact details at the end of this note. - -* Actual work breakdown tracked in FD.io CSIT jira: - - Tasks tracked in Jira under CSIT Epics. - -Plan Timeline -------------- - -* Current release cycle: - - FD.io CSIT rls18.07, associated with VPP-18.07. -* Sub-sequent releases: - - FD.io CSIT rls18.10, rls1812, .. - - Work not completed fully in current release cycle marked as backlog - for follow-on release(s). - -Plan Summary ------------- - -* Infrastructure, Framework, Tools - * New Skylake testbed infra to increase FD.io CSIT lab capacity. - [PM;EK,PM;MK] - * Introduce 2-node performance tests for new Skylake testbed infra. - [TF;JG,PM,TF;MK] - * Productize duration aware multi-rate MLR search. [VP;PM,VP;MK] - * Improve continuous performance trending: anomaly detection tunings, - add dpdk. [TF;TF,VP;MK] - * Complete and phase into production continuous per VPP patch - performance tests. [PM;PM,TF,VP;MK] - * Implement proper per-packet latency measurements, reporting and - analytics with TRex HdrHistogram. [TF;PM,TF,VP;MK] - * Evolve presentation and analytics layer (PAL) addressing growing - volumes of test measurement and telemetry data. [TF;EK,PM,TF,VP;MK] - * Start migration from CSIT_VIRL to VPP_Path (make_test) and - VPP_Device integration tests. [JG;EK,JG,TF;MK] - * Enhance CSIT reports, trending pages, PAL backend and trending test - code addressing wider set of data plane workloads and automate - CI/CD trending communication to FD.io community. [TF;PM,TF,VP;MK] - * Automate VPP performance regression search. [TF;TF,VP;MK] - * Other refactor: VAT to PAPI, data driven tests, suite duration, - infra overhead. [VP;JG,PM,TF,VP;MK] - -* Testing, Performance - * New tests: more TCP stack, SRv6, memif; AVF driver (no DPDK). - [MK;JG,PM,TF,VP;MK] - * VPP_Path: migration of P0 VIRL tests to VPP_make_test, followed by - qualification of VIRL P1, P2 tests; adding use case driven - functional tests. - * VPP_Device: new use cases per VPP_Device design note <add link>. - * VPP_Path_Device: continue to add relevant tests. - -* Other - * FD.io Operations. [All] - * ARM, Atom servers. [?] - * API changes across VPP major versions. [?] - * Plugin dependencies. [?] - * DPDK driver dependencies. [?] - -FD.io CSIT Contributors ------------------------ - -* JG - Jan Gelety <jgelety@cisco.com>, irc: jgelety. -* EK - Ed Kern <ejk@cisco.com>, irc: snergster. -* MK - Maciek Konstantynowicz <mkonstan@cisco.com>, irc: mackonstan. -* PM - Peter Mikus <pmikus@cisco.com>, irc: pmikus. -* TF - Tibor Frank <tifrank@cisco.com>, irc: tifrank. -* VP - Vratko Polak <vrpolak@cisco.com>, irc: vrpolak. diff --git a/docs/report/csit_framework_documentation/csit_design.rst b/docs/report/csit_framework_documentation/csit_design.rst index 43dfc343de..4cd29fad6d 100644 --- a/docs/report/csit_framework_documentation/csit_design.rst +++ b/docs/report/csit_framework_documentation/csit_design.rst @@ -107,7 +107,6 @@ A brief bottom-up description is provided here: - VPP; - DPDK-Testpmd; - DPDK-L3Fwd; - - Honeycomb; - Tools: diff --git a/docs/report/dmm_functional_tests/csit_release_notes.rst b/docs/report/dmm_functional_tests/csit_release_notes.rst index f525d1e904..32ee01ace6 100644 --- a/docs/report/dmm_functional_tests/csit_release_notes.rst +++ b/docs/report/dmm_functional_tests/csit_release_notes.rst @@ -4,9 +4,7 @@ Release Notes Changes in |csit-release| ------------------------- -#. DMM FUNCTIONAL TESTS - - - Added DMM lwip integration test case. +No changes Known Issues ------------ diff --git a/docs/report/dpdk_performance_tests/test_environment.rst b/docs/report/dpdk_performance_tests/test_environment.rst index e1eb8fa874..ccedca0795 100644 --- a/docs/report/dpdk_performance_tests/test_environment.rst +++ b/docs/report/dpdk_performance_tests/test_environment.rst @@ -11,8 +11,6 @@ .. include:: ../introduction/test_environment_sut_conf_1.rst -.. include:: ../introduction/test_environment_sut_conf_3.rst - DUT Settings - DPDK ------------------- @@ -27,7 +25,7 @@ DPDK Compile Parameters .. code-block:: bash - make install T=x86_64-native-linuxapp-gcc -j + make install T=<arch>-native-linuxapp-gcc -j Testpmd Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -38,7 +36,7 @@ sending jumbo frames. Startup command template: .. code-block:: bash - testpmd -c $$CORE_MASK -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=$$CORES --max-pkt-len=9000 --txqflags=0 --forward-mode=io --rxq=$$RXQ --txq=$$TXQ --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start + testpmd -c $$CORE_MASK -n 4 -- --numa --nb-ports=2 --portmask=0x3 --nb-cores=$$CORES [--max-pkt-len=9000] --txqflags=0 --forward-mode=io --rxq=$$RXQ --txq=$$TXQ --burst=64 --rxd=1024 --txd=1024 --disable-link-check --auto-start L3FWD Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -49,7 +47,7 @@ jumbo frames. Startup command template: .. code-block:: bash - l3fwd -l $$CORE_LIST -n 4 -- -P -L -p 0x3 --config='${port_config}' --enable-jumbo --max-pkt-len=9000 --eth-dest=0,${adj_mac0} --eth-dest=1,${adj_mac1} --parse-ptype + l3fwd -l $$CORE_LIST -n 4 -- -P -L -p 0x3 --config='${port_config}' [--enable-jumbo --max-pkt-len=9000] --eth-dest=0,${adj_mac0} --eth-dest=1,${adj_mac1} --parse-ptype .. include:: ../introduction/test_environment_tg.rst diff --git a/docs/report/introduction/methodology_kvm_vms_vhost_user.rst b/docs/report/introduction/methodology_kvm_vms_vhost_user.rst index 79f1134881..e6a98596da 100644 --- a/docs/report/introduction/methodology_kvm_vms_vhost_user.rst +++ b/docs/report/introduction/methodology_kvm_vms_vhost_user.rst @@ -9,25 +9,7 @@ to the QEMU binary can be adjusted in `Constants.py`. FD.io CSIT performance lab is testing VPP vhost-user with KVM VMs using following environment settings: -- Tests with varying QEMU virtio queue (a.k.a. vring) sizes: [vr1024] - 1024 descriptors to optimize for packet throughput. -- Tests with varying Linux :abbr:`CFS (Completely Fair Scheduler)` - settings: i) [cfs] default settings, ii) [cfsrr1] CFS RoundRobin(1) - policy applied to all data plane threads handling test packet path - including all VPP worker threads and all QEMU testpmd poll-mode - threads. -- Resulting test cases are all combinations with [vr1024] and - [cfs,cfsrr1] settings. -- Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)` - scheduler policy for data plane threads used in CSIT is documented in - `CSIT Performance Environment Tuning wiki - <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_. - -Testing with different CFS settings enables verifying the impact of -making VPP and VM data plane threads less susceptible to other Linux OS -system tasks hijacking CPU cores running those data plane threads. - -CSIT supports two types of VMs: +CSIT supports two types of VMs: - **Image-VM**: used for all functional, VPP_device, and regular performance tests except NFV density tests. @@ -83,10 +65,10 @@ Example of custom init script for the kernel-VM: mount -t hugetlbfs -o "rw,relatime,pagesize=2M" hugetlbfs /dev/hugepages echo 0000:00:06.0 > /sys/bus/pci/devices/0000:00:06.0/driver/unbind echo 0000:00:07.0 > /sys/bus/pci/devices/0000:00:07.0/driver/unbind - echo uio_pci_generic > /sys/bus/pci/devices/0000:00:06.0/driver_override - echo uio_pci_generic > /sys/bus/pci/devices/0000:00:07.0/driver_override - echo 0000:00:06.0 > /sys/bus/pci/drivers/uio_pci_generic/bind - echo 0000:00:07.0 > /sys/bus/pci/drivers/uio_pci_generic/bind + echo vfio-pci > /sys/bus/pci/devices/0000:00:06.0/driver_override + echo vfio-pci > /sys/bus/pci/devices/0000:00:07.0/driver_override + echo 0000:00:06.0 > /sys/bus/pci/drivers/vfio-pci/bind + echo 0000:00:07.0 > /sys/bus/pci/drivers/vfio-pci/bind $vnf_bin poweroff -f diff --git a/docs/report/introduction/methodology_trex_traffic_generator.rst b/docs/report/introduction/methodology_trex_traffic_generator.rst index 2a25931faa..918a34f73d 100644 --- a/docs/report/introduction/methodology_trex_traffic_generator.rst +++ b/docs/report/introduction/methodology_trex_traffic_generator.rst @@ -22,11 +22,11 @@ is: - TRex is started in the background mode :: - $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /tmp/trex.log 2>&1 &' > /dev/null + $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 > /tmp/trex.log 2>&1 &' > /dev/null - There are traffic streams dynamically prepared for each test, based on traffic profiles. The traffic is sent and the statistics obtained using - :command:`trex_stl_lib.api.STLClient`. + :command:`trex.stl.api.STLClient`. Measuring Packet Loss ~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/report/introduction/methodology_vpp_device_functional.rst b/docs/report/introduction/methodology_vpp_device_functional.rst index 41a8040ef6..0c29624419 100644 --- a/docs/report/introduction/methodology_vpp_device_functional.rst +++ b/docs/report/introduction/methodology_vpp_device_functional.rst @@ -1,13 +1,11 @@ VPP_Device Functional --------------------- -|csit-release| added new VPP_Device test environment for functional VPP +|csit-release| includes VPP_Device test environment for functional VPP device tests integrated into LFN CI/CD infrastructure. VPP_Device tests run on 1-Node testbeds (1n-skx, 1n-arm) and rely on Linux SRIOV Virtual Function (VF), dot1q VLAN tagging and external loopback cables to facilitate packet passing over exernal physical links. Initial focus is -on few baseline tests. Existing CSIT VIRL tests can be moved to -VPP_Device framework by changing L1 and L2 KW(s). RF test definition -code stays unchanged with the exception of requiring adjustments from -3-Node to 2-Node logical topologies. CSIT VIRL to VPP_Device migration -is expected in the next CSIT release. +on few baseline tests. Existing CSIT Performance tests can be moved to +VPP_Device framework. RF test definition code stays unchanged with the +exception of traffic generator related L2 KWs. diff --git a/docs/report/introduction/test_environment_sut_conf_2.rst b/docs/report/introduction/test_environment_sut_conf_2.rst deleted file mode 100644 index 24fcd741e9..0000000000 --- a/docs/report/introduction/test_environment_sut_conf_2.rst +++ /dev/null @@ -1,38 +0,0 @@ - -Linux CFS Tunings -~~~~~~~~~~~~~~~~~ - -Linux CFS scheduler tunings are applied to all QEMU vCPU worker threads -(the ones handling testpmd PMD threads) and VPP data plane worker -threads. List of VPP data plane threads can be obtained by running: - -:: - - $ for psid in $(pgrep vpp) - $ do - $ for tid in $(ps -Lo tid --pid $psid | grep -v TID) - $ do - $ echo $tid - $ done - $ done - -Or: - -:: - - $ cat /proc/`pidof vpp`/task/*/stat | awk '{print $1" "$2" "$39}' - -CFS round-robin scheduling with highest priority is applied using: - -:: - - $ for psid in $(pgrep vpp) - $ do - $ for tid in $(ps -Lo tid --pid $psid | grep -v TID) - $ do - $ chrt -r -p 1 $tid - $ done - $ done - -More information about Linux CFS can be found in `Sched manual pages -<http://man7.org/linux/man-pages/man7/sched.7.html>`_. diff --git a/docs/report/introduction/test_environment_sut_conf_3.rst b/docs/report/introduction/test_environment_sut_conf_3.rst deleted file mode 100644 index 20dc155058..0000000000 --- a/docs/report/introduction/test_environment_sut_conf_3.rst +++ /dev/null @@ -1,9 +0,0 @@ - -Host Writeback Affinity -~~~~~~~~~~~~~~~~~~~~~~~ - -Writebacks are pinned to core 0. The same configuration is applied in host Linux and guest VM. - -:: - - $ echo 1 | sudo tee /sys/bus/workqueue/devices/writeback/cpumask diff --git a/docs/report/introduction/test_environment_sut_meltspec_dnv.rst b/docs/report/introduction/test_environment_sut_meltspec_dnv.rst index 71d1b6808f..a83869ba03 100644 --- a/docs/report/introduction/test_environment_sut_meltspec_dnv.rst +++ b/docs/report/introduction/test_environment_sut_meltspec_dnv.rst @@ -6,121 +6,144 @@ system is vulnerable against the several "speculative execution" CVEs that were made public in 2018. Script is available on `Spectre & Meltdown Checker Github <https://github.com/speed47/spectre-meltdown-checker>`_. -- CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' -- CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' -- CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' -- CVE-2018-3640 [rogue system register read] aka 'Variant 3a' -- CVE-2018-3639 [speculative store bypass] aka 'Variant 4' -- CVE-2018-3615 [L1 terminal fault] aka 'Foreshadow (SGX)' -- CVE-2018-3620 [L1 terminal fault] aka 'Foreshadow-NG (OS)' -- CVE-2018-3646 [L1 terminal fault] aka 'Foreshadow-NG (VMM)' - :: - $ sudo ./spectre-meltdown-checker.sh --no-color - - Spectre and Meltdown mitigation detection tool v0.40 - - Checking for vulnerabilities on current system - Kernel is Linux 4.15.0-36-generic #39~16.04.1-Ubuntu SMP Tue Sep 25 08:59:23 UTC 2018 x86_64 - CPU is Intel(R) Atom(TM) CPU C3858 @ 2.00GHz - - Hardware check - * Hardware support (CPU microcode) for mitigation techniques - * Indirect Branch Restricted Speculation (IBRS) - * SPEC_CTRL MSR is available: YES - * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) - * Indirect Branch Prediction Barrier (IBPB) - * PRED_CMD MSR is available: YES - * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) - * Single Thread Indirect Branch Predictors (STIBP) - * SPEC_CTRL MSR is available: YES - * CPU indicates STIBP capability: YES (Intel STIBP feature bit) - * Speculative Store Bypass Disable (SSBD) - * CPU indicates SSBD capability: YES (Intel SSBD) - * L1 data cache invalidation - * FLUSH_CMD MSR is available: NO - * CPU indicates L1D flush capability: NO - * Enhanced IBRS (IBRS_ALL) - * CPU indicates ARCH_CAPABILITIES MSR availability: YES - * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO - * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO): YES - * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO - * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES - * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO - * CPU supports Software Guard Extensions (SGX): NO - * CPU microcode is known to cause stability problems: NO (model 0x5f family 0x6 stepping 0x1 ucode 0x24 cpuid 0x506f1) - * CPU microcode is the latest known available version: YES (latest version is 0x24 dated 2018/05/11 according to builtin MCExtractor DB v84 - 2018/09/27) - * CPU vulnerability to the speculative execution attack variants - * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES - * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES - * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO - * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES - * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES - * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO - * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): YES - * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): YES - - CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass' - * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) - * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) - * Kernel has the Red Hat/Ubuntu patch: NO - * Kernel has mask_nospec64 (arm64): NO - > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) - - CVE-2017-5715 aka 'Spectre Variant 2, branch target injection' - * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) - * Mitigation 1 - * Kernel is compiled with IBRS support: YES - * IBRS enabled and active: YES (for kernel and firmware code) - * Kernel is compiled with IBPB support: YES - * IBPB enabled and active: YES - * Mitigation 2 - * Kernel has branch predictor hardening (arm): NO - * Kernel compiled with retpoline option: YES - * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) - > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) - - CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load' - * Mitigated according to the /sys interface: YES (Not affected) - * Kernel supports Page Table Isolation (PTI): YES - * PTI enabled and active: NO - * Reduced performance impact of PTI: NO (PCID/INVPCID not supported, performance impact of PTI will be significant) - * Running as a Xen PV DomU: NO - > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) - - CVE-2018-3640 aka 'Variant 3a, rogue system register read' - * CPU microcode mitigates the vulnerability: YES - > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) - - CVE-2018-3639 aka 'Variant 4, speculative store bypass' - * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) - * Kernel supports speculation store bypass: YES (found in /proc/self/status) - > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) - - CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault' - * CPU microcode mitigates the vulnerability: N/A - > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) - - CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault' - * Mitigated according to the /sys interface: YES (Not affected) - * Kernel supports PTE inversion: YES (found in kernel image) - * PTE inversion enabled and active: NO - > STATUS: NOT VULNERABLE (Not affected) - - CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault' - * Information from the /sys interface: - * This system is a host running an hypervisor: NO - * Mitigation 1 (KVM) - * EPT is disabled: NO - * Mitigation 2 - * L1D flush is supported by kernel: YES (found flush_l1d in kernel image) - * L1D flush enabled: UNKNOWN (unrecognized mode) - * Hardware-backed L1D flush supported: NO (flush will be done in software, this is slower) - * Hyper-Threading (SMT) is enabled: NO - > STATUS: NOT VULNERABLE (this system is not running an hypervisor) - - > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK - - Need more detailed information about mitigation options? Use --explain - A false sense of security is worse than no security at all, see --disclaimer + Spectre and Meltdown mitigation detection tool v0.42 + Checking for vulnerabilities on current system + Kernel is Linux 4.15.0-51-generic #55-Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x86_64 + CPU is Intel(R) Atom(TM) CPU C3858 @ 2.00GHz + + Hardware check + * Hardware support (CPU microcode) for mitigation techniques + * Indirect Branch Restricted Speculation (IBRS) + * SPEC_CTRL MSR is available: YES + * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) + * Indirect Branch Prediction Barrier (IBPB) + * PRED_CMD MSR is available: YES + * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) + * Single Thread Indirect Branch Predictors (STIBP) + * SPEC_CTRL MSR is available: YES + * CPU indicates STIBP capability: YES (Intel STIBP feature bit) + * Speculative Store Bypass Disable (SSBD) + * CPU indicates SSBD capability: YES (Intel SSBD) + * L1 data cache invalidation + * FLUSH_CMD MSR is available: NO + * CPU indicates L1D flush capability: NO + * Microarchitecture Data Sampling + * VERW instruction is available: YES (MD_CLEAR feature bit) + * Enhanced IBRS (IBRS_ALL) + * CPU indicates ARCH_CAPABILITIES MSR availability: YES + * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO + * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): YES + * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO + * CPU/Hypervisor indicates L1D flushing is not necessary on this system: YES + * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO + * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): YES + * CPU supports Software Guard Extensions (SGX): NO + * CPU microcode is known to cause stability problems: NO (model 0x5f family 0x6 stepping 0x1 ucode 0x2e cpuid 0x506f1) + * CPU microcode is the latest known available version: awk: fatal: cannot open file `bash for reading (No such file or directory) + UNKNOWN (latest microcode version for your CPU model is unknown) + * CPU vulnerability to the speculative execution attack variants + * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES + * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES + * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): NO + * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES + * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES + * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO + * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): NO + * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): NO + * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): NO + * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): NO + * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): NO + * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)): NO + + CVE-2017-5753 aka Spectre Variant 1, bounds check bypass + * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has mask_nospec64 (arm64): NO + > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) + + CVE-2017-5715 aka Spectre Variant 2, branch target injection + * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling) + * Mitigation 1 + * Kernel is compiled with IBRS support: YES + * IBRS enabled and active: YES (for firmware code only) + * Kernel is compiled with IBPB support: YES + * IBPB enabled and active: YES + * Mitigation 2 + * Kernel has branch predictor hardening (arm): NO + * Kernel compiled with retpoline option: YES + * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) + > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) + + CVE-2017-5754 aka Variant 3, Meltdown, rogue data cache load + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports Page Table Isolation (PTI): YES + * PTI enabled and active: UNKNOWN (dmesg truncated, please reboot and relaunch this script) + * Reduced performance impact of PTI: NO (PCID/INVPCID not supported, performance impact of PTI will be significant) + * Running as a Xen PV DomU: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3640 aka Variant 3a, rogue system register read + * CPU microcode mitigates the vulnerability: YES + > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) + + CVE-2018-3639 aka Variant 4, speculative store bypass + * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + * Kernel supports disabling speculative store bypass (SSB): YES (found in /proc/self/status) + * SSB mitigation is enabled and active: YES (per-thread through prctl) + * SSB mitigation currently active for selected processes: YES (systemd-journald systemd-logind systemd-networkd systemd-resolved systemd-timesyncd systemd-udevd) + > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + + CVE-2018-3615 aka Foreshadow (SGX), L1 terminal fault + * CPU microcode mitigates the vulnerability: N/A + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3620 aka Foreshadow-NG (OS), L1 terminal fault + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports PTE inversion: YES (found in kernel image) + * PTE inversion enabled and active: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3646 aka Foreshadow-NG (VMM), L1 terminal fault + * Information from the /sys interface: Not affected + * This system is a host running a hypervisor: NO + * Mitigation 1 (KVM) + * EPT is disabled: NO + * Mitigation 2 + * L1D flush is supported by kernel: YES (found flush_l1d in kernel image) + * L1D flush enabled: NO + * Hardware-backed L1D flush supported: NO (flush will be done in software, this is slower) + * Hyper-Threading (SMT) is enabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12126 aka Fallout, microarchitectural store buffer data sampling (MSBDS) + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12130 aka ZombieLoad, microarchitectural fill buffer data sampling (MFBDS) + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-12127 aka RIDL, microarchitectural load port data sampling (MLPDS) + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2019-11091 aka RIDL, microarchitectural data sampling uncacheable memory (MDSUM) + * Mitigated according to the /sys interface: YES (Not affected) + * Kernel supports using MD_CLEAR mitigation: YES (md_clear found in /proc/cpuinfo) + * Kernel mitigation is enabled and active: NO + * SMT is either mitigated or disabled: NO + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:OK CVE-2018-12130:OK CVE-2018-12127:OK CVE-2019-11091:OK diff --git a/docs/report/introduction/test_environment_sut_meltspec_hsw.rst b/docs/report/introduction/test_environment_sut_meltspec_hsw.rst index 71787f0691..8634aa4cfa 100644 --- a/docs/report/introduction/test_environment_sut_meltspec_hsw.rst +++ b/docs/report/introduction/test_environment_sut_meltspec_hsw.rst @@ -6,121 +6,133 @@ system is vulnerable against the several "speculative execution" CVEs that were made public in 2018. Script is available on `Spectre & Meltdown Checker Github <https://github.com/speed47/spectre-meltdown-checker>`_. -- CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' -- CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' -- CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' -- CVE-2018-3640 [rogue system register read] aka 'Variant 3a' -- CVE-2018-3639 [speculative store bypass] aka 'Variant 4' -- CVE-2018-3615 [L1 terminal fault] aka 'Foreshadow (SGX)' -- CVE-2018-3620 [L1 terminal fault] aka 'Foreshadow-NG (OS)' -- CVE-2018-3646 [L1 terminal fault] aka 'Foreshadow-NG (VMM)' - :: - $ sudo ./spectre-meltdown-checker.sh --no-color - - Spectre and Meltdown mitigation detection tool v0.40 - - Checking for vulnerabilities on current system - Kernel is Linux 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 - CPU is Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz - - Hardware check - * Hardware support (CPU microcode) for mitigation techniques - * Indirect Branch Restricted Speculation (IBRS) - * SPEC_CTRL MSR is available: YES - * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) - * Indirect Branch Prediction Barrier (IBPB) - * PRED_CMD MSR is available: YES - * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) - * Single Thread Indirect Branch Predictors (STIBP) - * SPEC_CTRL MSR is available: YES - * CPU indicates STIBP capability: YES (Intel STIBP feature bit) - * Speculative Store Bypass Disable (SSBD) - * CPU indicates SSBD capability: YES (Intel SSBD) - * L1 data cache invalidation - * FLUSH_CMD MSR is available: YES - * CPU indicates L1D flush capability: YES (L1D flush feature bit) - * Enhanced IBRS (IBRS_ALL) - * CPU indicates ARCH_CAPABILITIES MSR availability: NO - * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO - * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO): NO - * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO - * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO - * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO - * CPU supports Software Guard Extensions (SGX): NO - * CPU microcode is known to cause stability problems: NO (model 0x3f family 0x6 stepping 0x2 ucode 0x3d cpuid 0x306f2) - * CPU microcode is the latest known available version: YES (latest version is 0x3d dated 2018/04/20 according to builtin MCExtractor DB v84 - 2018/09/27) - * CPU vulnerability to the speculative execution attack variants - * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES - * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES - * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): YES - * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES - * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES - * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO - * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): YES - * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): YES - - CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass' - * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) - * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) - * Kernel has the Red Hat/Ubuntu patch: NO - * Kernel has mask_nospec64 (arm64): NO - > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) - - CVE-2017-5715 aka 'Spectre Variant 2, branch target injection' - * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) - * Mitigation 1 - * Kernel is compiled with IBRS support: YES - * IBRS enabled and active: YES (for kernel and firmware code) - * Kernel is compiled with IBPB support: YES - * IBPB enabled and active: YES - * Mitigation 2 - * Kernel has branch predictor hardening (arm): NO - * Kernel compiled with retpoline option: YES - * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) - > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) - - CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load' - * Mitigated according to the /sys interface: YES (Mitigation: PTI) - * Kernel supports Page Table Isolation (PTI): YES - * PTI enabled and active: YES - * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced) - * Running as a Xen PV DomU: NO - > STATUS: NOT VULNERABLE (Mitigation: PTI) - - CVE-2018-3640 aka 'Variant 3a, rogue system register read' - * CPU microcode mitigates the vulnerability: YES - > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) - - CVE-2018-3639 aka 'Variant 4, speculative store bypass' - * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) - * Kernel supports speculation store bypass: YES (found in /proc/self/status) - > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) - - CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault' - * CPU microcode mitigates the vulnerability: N/A - > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) - - CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault' - * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion) - * Kernel supports PTE inversion: YES (found in kernel image) - * PTE inversion enabled and active: YES - > STATUS: NOT VULNERABLE (Mitigation: PTE Inversion) - - CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault' - * Information from the /sys interface: VMX: conditional cache flushes, SMT disabled - * This system is a host running an hypervisor: NO - * Mitigation 1 (KVM) - * EPT is disabled: NO - * Mitigation 2 - * L1D flush is supported by kernel: YES (found flush_l1d in /proc/cpuinfo) - * L1D flush enabled: YES (conditional flushes) - * Hardware-backed L1D flush supported: YES (performance impact of the mitigation will be greatly reduced) - * Hyper-Threading (SMT) is enabled: NO - > STATUS: NOT VULNERABLE (this system is not running an hypervisor) - - > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK - - Need more detailed information about mitigation options? Use --explain - A false sense of security is worse than no security at all, see --disclaimer + Spectre and Meltdown mitigation detection tool v0.42 + + Checking for vulnerabilities on current system + Kernel is Linux 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 + CPU is Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz + + Hardware check + * Hardware support (CPU microcode) for mitigation techniques + * Indirect Branch Restricted Speculation (IBRS) + * SPEC_CTRL MSR is available: YES + * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) + * Indirect Branch Prediction Barrier (IBPB) + * PRED_CMD MSR is available: YES + * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) + * Single Thread Indirect Branch Predictors (STIBP) + * SPEC_CTRL MSR is available: YES + * CPU indicates STIBP capability: YES (Intel STIBP feature bit) + * Speculative Store Bypass Disable (SSBD) + * CPU indicates SSBD capability: YES (Intel SSBD) + * L1 data cache invalidation + * FLUSH_CMD MSR is available: YES + * CPU indicates L1D flush capability: YES (L1D flush feature bit) + * Microarchitecture Data Sampling + * VERW instruction is available: NO + * Enhanced IBRS (IBRS_ALL) + * CPU indicates ARCH_CAPABILITIES MSR availability: NO + * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO + * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): NO + * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO + * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO + * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO + * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): NO + * CPU supports Software Guard Extensions (SGX): NO + * CPU microcode is known to cause stability problems: NO (model 0x3f family 0x6 stepping 0x2 ucode 0x3d cpuid 0x306f2) + * CPU microcode is the latest known available version: awk: cannot open bash (No such file or directory) + UNKNOWN (latest microcode version for your CPU model is unknown) + * CPU vulnerability to the speculative execution attack variants + * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES + * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES + * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): YES + * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES + * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES + * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO + * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): YES + * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): YES + * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): YES + * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): YES + * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): YES + * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)): YES + + CVE-2017-5753 aka Spectre Variant 1, bounds check bypass + * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has mask_nospec64 (arm64): NO + > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) + + CVE-2017-5715 aka Spectre Variant 2, branch target injection + * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) + * Mitigation 1 + * Kernel is compiled with IBRS support: YES + * IBRS enabled and active: YES (for firmware code only) + * Kernel is compiled with IBPB support: YES + * IBPB enabled and active: YES + * Mitigation 2 + * Kernel has branch predictor hardening (arm): NO + * Kernel compiled with retpoline option: YES + * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) + > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) + + CVE-2017-5754 aka Variant 3, Meltdown, rogue data cache load + * Mitigated according to the /sys interface: YES (Mitigation: PTI) + * Kernel supports Page Table Isolation (PTI): YES + * PTI enabled and active: YES + * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced) + * Running as a Xen PV DomU: NO + > STATUS: NOT VULNERABLE (Mitigation: PTI) + + CVE-2018-3640 aka Variant 3a, rogue system register read + * CPU microcode mitigates the vulnerability: YES + > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) + + CVE-2018-3639 aka Variant 4, speculative store bypass + * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + * Kernel supports disabling speculative store bypass (SSB): YES (found in /proc/self/status) + * SSB mitigation is enabled and active: YES (per-thread through prctl) + * SSB mitigation currently active for selected processes: YES (systemd-journald systemd-logind systemd-networkd systemd-resolved systemd-timesyncd systemd-udevd) + > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + + CVE-2018-3615 aka Foreshadow (SGX), L1 terminal fault + * CPU microcode mitigates the vulnerability: N/A + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3620 aka Foreshadow-NG (OS), L1 terminal fault + * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled) + * Kernel supports PTE inversion: YES (found in kernel image) + * PTE inversion enabled and active: YES + > STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled) + + CVE-2018-3646 aka Foreshadow-NG (VMM), L1 terminal fault + * Information from the /sys interface: Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT disabled + * This system is a host running a hypervisor: NO + * Mitigation 1 (KVM) + * EPT is disabled: NO + * Mitigation 2 + * L1D flush is supported by kernel: YES (found flush_l1d in /proc/cpuinfo) + * L1D flush enabled: YES (conditional flushes) + * Hardware-backed L1D flush supported: YES (performance impact of the mitigation will be greatly reduced) + * Hyper-Threading (SMT) is enabled: NO + > STATUS: NOT VULNERABLE (this system is not running a hypervisor) + + CVE-2018-12126 aka Fallout, microarchitectural store buffer data sampling (MSBDS) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + CVE-2018-12130 aka ZombieLoad, microarchitectural fill buffer data sampling (MFBDS) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + CVE-2018-12127 aka RIDL, microarchitectural load port data sampling (MLPDS) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + CVE-2019-11091 aka RIDL, microarchitectural data sampling uncacheable memory (MDSUM) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:KO CVE-2018-12130:KO CVE-2018-12127:KO CVE-2019-11091:KO diff --git a/docs/report/introduction/test_environment_sut_meltspec_skx.rst b/docs/report/introduction/test_environment_sut_meltspec_skx.rst index 443a7fd484..15b098a9ce 100644 --- a/docs/report/introduction/test_environment_sut_meltspec_skx.rst +++ b/docs/report/introduction/test_environment_sut_meltspec_skx.rst @@ -6,120 +6,134 @@ system is vulnerable against the several "speculative execution" CVEs that were made public in 2018. Script is available on `Spectre & Meltdown Checker Github <https://github.com/speed47/spectre-meltdown-checker>`_. -- CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' -- CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' -- CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' -- CVE-2018-3640 [rogue system register read] aka 'Variant 3a' -- CVE-2018-3639 [speculative store bypass] aka 'Variant 4' -- CVE-2018-3615 [L1 terminal fault] aka 'Foreshadow (SGX)' -- CVE-2018-3620 [L1 terminal fault] aka 'Foreshadow-NG (OS)' -- CVE-2018-3646 [L1 terminal fault] aka 'Foreshadow-NG (VMM)' - :: - $ sudo ./spectre-meltdown-checker.sh --no-color - - Spectre and Meltdown mitigation detection tool v0.40 - - Checking for vulnerabilities on current system - Kernel is Linux 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 - CPU is Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz - - Hardware check - * Hardware support (CPU microcode) for mitigation techniques - * Indirect Branch Restricted Speculation (IBRS) - * SPEC_CTRL MSR is available: YES - * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) - * Indirect Branch Prediction Barrier (IBPB) - * PRED_CMD MSR is available: YES - * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) - * Single Thread Indirect Branch Predictors (STIBP) - * SPEC_CTRL MSR is available: YES - * CPU indicates STIBP capability: YES (Intel STIBP feature bit) - * Speculative Store Bypass Disable (SSBD) - * CPU indicates SSBD capability: NO - * L1 data cache invalidation - * FLUSH_CMD MSR is available: NO - * CPU indicates L1D flush capability: NO - * Enhanced IBRS (IBRS_ALL) - * CPU indicates ARCH_CAPABILITIES MSR availability: NO - * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO - * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO): NO - * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO - * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO - * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO - * CPU supports Software Guard Extensions (SGX): NO - * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x4 ucode 0x2000043 cpuid 0x50654) - * CPU microcode is the latest known available version: NO (latest version is 0x200004d dated 2018/05/15 according to builtin MCExtractor DB v84 - 2018/09/27) - * CPU vulnerability to the speculative execution attack variants - * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES - * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES - * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): YES - * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES - * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES - * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO - * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): YES - * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): YES - - CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass' - * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) - * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) - * Kernel has the Red Hat/Ubuntu patch: NO - * Kernel has mask_nospec64 (arm64): NO - > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) - - CVE-2017-5715 aka 'Spectre Variant 2, branch target injection' - * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) - * Mitigation 1 - * Kernel is compiled with IBRS support: YES - * IBRS enabled and active: YES (for kernel and firmware code) - * Kernel is compiled with IBPB support: YES - * IBPB enabled and active: YES - * Mitigation 2 - * Kernel has branch predictor hardening (arm): NO - * Kernel compiled with retpoline option: YES - * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) - * Kernel supports RSB filling: YES - > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) - - CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load' - * Mitigated according to the /sys interface: YES (Mitigation: PTI) - * Kernel supports Page Table Isolation (PTI): YES - * PTI enabled and active: YES - * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced) - * Running as a Xen PV DomU: NO - > STATUS: NOT VULNERABLE (Mitigation: PTI) - - CVE-2018-3640 aka 'Variant 3a, rogue system register read' - * CPU microcode mitigates the vulnerability: NO - > STATUS: VULNERABLE (an up-to-date CPU microcode is needed to mitigate this vulnerability) - - CVE-2018-3639 aka 'Variant 4, speculative store bypass' - * Mitigated according to the /sys interface: NO (Vulnerable) - * Kernel supports speculation store bypass: YES (found in /proc/self/status) - > STATUS: VULNERABLE (Your CPU doesn't support SSBD) - - CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault' - * CPU microcode mitigates the vulnerability: N/A - > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) - - CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault' - * Kernel supports PTE inversion: NO - * PTE inversion enabled and active: UNKNOWN (sysfs interface not available) - > STATUS: VULNERABLE (Your kernel doesn't support PTE inversion, update it) - - CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault' - * This system is a host running an hypervisor: NO - * Mitigation 1 (KVM) - * EPT is disabled: NO - * Mitigation 2 - * L1D flush is supported by kernel: NO - * L1D flush enabled: UNKNOWN (can't find or read /sys/devices/system/cpu/vulnerabilities/l1tf) - * Hardware-backed L1D flush supported: NO (flush will be done in software, this is slower) - * Hyper-Threading (SMT) is enabled: YES - > STATUS: NOT VULNERABLE (this system is not running an hypervisor) - - > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:KO CVE-2018-3639:KO CVE-2018-3615:OK CVE-2018-3620:KO CVE-2018-3646:OK - - Need more detailed information about mitigation options? Use --explain - A false sense of security is worse than no security at all, see --disclaimer + Spectre and Meltdown mitigation detection tool v0.42 + + Checking for vulnerabilities on current system + Kernel is Linux 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 + CPU is Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz + + Hardware check + * Hardware support (CPU microcode) for mitigation techniques + * Indirect Branch Restricted Speculation (IBRS) + * SPEC_CTRL MSR is available: YES + * CPU indicates IBRS capability: YES (SPEC_CTRL feature bit) + * Indirect Branch Prediction Barrier (IBPB) + * PRED_CMD MSR is available: YES + * CPU indicates IBPB capability: YES (SPEC_CTRL feature bit) + * Single Thread Indirect Branch Predictors (STIBP) + * SPEC_CTRL MSR is available: YES + * CPU indicates STIBP capability: YES (Intel STIBP feature bit) + * Speculative Store Bypass Disable (SSBD) + * CPU indicates SSBD capability: YES (Intel SSBD) + * L1 data cache invalidation + * FLUSH_CMD MSR is available: YES + * CPU indicates L1D flush capability: YES (L1D flush feature bit) + * Microarchitecture Data Sampling + * VERW instruction is available: NO + * Enhanced IBRS (IBRS_ALL) + * CPU indicates ARCH_CAPABILITIES MSR availability: NO + * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability: NO + * CPU explicitly indicates not being vulnerable to Meltdown/L1TF (RDCL_NO): NO + * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO): NO + * CPU/Hypervisor indicates L1D flushing is not necessary on this system: NO + * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA): NO + * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDS_NO): NO + * CPU supports Software Guard Extensions (SGX): NO + * CPU microcode is known to cause stability problems: NO (model 0x55 family 0x6 stepping 0x4 ucode 0x200004d cpuid 0x50654) + * CPU microcode is the latest known available version: awk: cannot open bash (No such file or directory) + UNKNOWN (latest microcode version for your CPU model is unknown) + * CPU vulnerability to the speculative execution attack variants + * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass): YES + * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection): YES + * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load): YES + * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read): YES + * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass): YES + * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault): NO + * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault): YES + * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault): YES + * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)): YES + * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)): YES + * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)): YES + * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)): YES + + CVE-2017-5753 aka Spectre Variant 1, bounds check bypass + * Mitigated according to the /sys interface: YES (Mitigation: __user pointer sanitization) + * Kernel has array_index_mask_nospec: YES (1 occurrence(s) found of x86 64 bits array_index_mask_nospec()) + * Kernel has the Red Hat/Ubuntu patch: NO + * Kernel has mask_nospec64 (arm64): NO + > STATUS: NOT VULNERABLE (Mitigation: __user pointer sanitization) + + CVE-2017-5715 aka Spectre Variant 2, branch target injection + * Mitigated according to the /sys interface: YES (Mitigation: Full generic retpoline, IBPB, IBRS_FW) + * Mitigation 1 + * Kernel is compiled with IBRS support: YES + * IBRS enabled and active: YES (for firmware code only) + * Kernel is compiled with IBPB support: YES + * IBPB enabled and active: YES + * Mitigation 2 + * Kernel has branch predictor hardening (arm): NO + * Kernel compiled with retpoline option: YES + * Kernel compiled with a retpoline-aware compiler: YES (kernel reports full retpoline compilation) + * Kernel supports RSB filling: YES + > STATUS: NOT VULNERABLE (Full retpoline + IBPB are mitigating the vulnerability) + + CVE-2017-5754 aka Variant 3, Meltdown, rogue data cache load + * Mitigated according to the /sys interface: YES (Mitigation: PTI) + * Kernel supports Page Table Isolation (PTI): YES + * PTI enabled and active: YES + * Reduced performance impact of PTI: YES (CPU supports INVPCID, performance impact of PTI will be greatly reduced) + * Running as a Xen PV DomU: NO + > STATUS: NOT VULNERABLE (Mitigation: PTI) + + CVE-2018-3640 aka Variant 3a, rogue system register read + * CPU microcode mitigates the vulnerability: YES + > STATUS: NOT VULNERABLE (your CPU microcode mitigates the vulnerability) + + CVE-2018-3639 aka Variant 4, speculative store bypass + * Mitigated according to the /sys interface: YES (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + * Kernel supports disabling speculative store bypass (SSB): YES (found in /proc/self/status) + * SSB mitigation is enabled and active: YES (per-thread through prctl) + * SSB mitigation currently active for selected processes: YES (systemd-journald systemd-logind systemd-networkd systemd-resolved systemd-timesyncd systemd-udevd) + > STATUS: NOT VULNERABLE (Mitigation: Speculative Store Bypass disabled via prctl and seccomp) + + CVE-2018-3615 aka Foreshadow (SGX), L1 terminal fault + * CPU microcode mitigates the vulnerability: N/A + > STATUS: NOT VULNERABLE (your CPU vendor reported your CPU model as not vulnerable) + + CVE-2018-3620 aka Foreshadow-NG (OS), L1 terminal fault + * Mitigated according to the /sys interface: YES (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable) + * Kernel supports PTE inversion: YES (found in kernel image) + * PTE inversion enabled and active: YES + > STATUS: NOT VULNERABLE (Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable) + + CVE-2018-3646 aka Foreshadow-NG (VMM), L1 terminal fault + * Information from the /sys interface: Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable + * This system is a host running a hypervisor: NO + * Mitigation 1 (KVM) + * EPT is disabled: NO + * Mitigation 2 + * L1D flush is supported by kernel: YES (found flush_l1d in /proc/cpuinfo) + * L1D flush enabled: YES (conditional flushes) + * Hardware-backed L1D flush supported: YES (performance impact of the mitigation will be greatly reduced) + * Hyper-Threading (SMT) is enabled: YES + > STATUS: NOT VULNERABLE (this system is not running a hypervisor) + + CVE-2018-12126 aka Fallout, microarchitectural store buffer data sampling (MSBDS) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + CVE-2018-12130 aka ZombieLoad, microarchitectural fill buffer data sampling (MFBDS) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + CVE-2018-12127 aka RIDL, microarchitectural load port data sampling (MLPDS) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + CVE-2019-11091 aka RIDL, microarchitectural data sampling uncacheable memory (MDSUM) + * Kernel supports using MD_CLEAR mitigation: NO + > STATUS: VULNERABLE (Neither your kernel or your microcode support mitigation, upgrade both to mitigate the vulnerability) + + > SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:KO CVE-2018-12130:KO CVE-2018-12127:KO CVE-2019-11091:KO diff --git a/docs/report/introduction/test_environment_tg.rst b/docs/report/introduction/test_environment_tg.rst index 135c9d478d..60dc81270b 100644 --- a/docs/report/introduction/test_environment_tg.rst +++ b/docs/report/introduction/test_environment_tg.rst @@ -9,7 +9,7 @@ TG Version DPDK Version ~~~~~~~~~~~~ -DPDK v18.08 +DPDK v19.02 TG Build Script Used ~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/report/introduction/test_scenarios_overview.rst b/docs/report/introduction/test_scenarios_overview.rst index ee334a6407..8d66836e9d 100644 --- a/docs/report/introduction/test_scenarios_overview.rst +++ b/docs/report/introduction/test_scenarios_overview.rst @@ -12,18 +12,17 @@ Brief overview of test scenarios covered in this report: #. **VPP Performance**: VPP performance tests are executed in physical FD.io testbeds, focusing on VPP network data plane performance in NIC-to-NIC switching topologies. Tested across Intel Xeon Haswell - and Skylake servers, range of NICs (10GE, 25GE, 40GE) and multi- - thread/multi-core configurations. VPP application runs in bare-metal + and Skylake servers, ARM, Denverton, range of NICs (10GE, 25GE, 40GE) and + multi-thread/multi-core configurations. VPP application runs in bare-metal host user-mode handling NICs. TRex is used as a traffic generator. #. **VPP Vhostuser Performance with KVM VMs**: VPP VM service switching performance tests using vhostuser virtual interface for - interconnecting multiple Testpmd-in-VM instances. VPP vswitch + interconnecting multiple NF-in-VM instances. VPP vswitch instance runs in bare-metal user-mode handling NICs and connecting - over vhost-user interfaces to VM instances each running DPDK - Testpmd with virtio virtual interfaces. Similarly to VPP - Performance, tests are run across a range of configurations. TRex - is used as a traffic generator. + over vhost-user interfaces to VM instances each running VPP with virtio + virtual interfaces. Similarly to VPP Performance, tests are run across a + range of configurations. TRex is used as a traffic generator. #. **VPP Memif Performance with LXC and Docker Containers**: VPP Container service switching performance tests using memif virtual @@ -49,12 +48,11 @@ Brief overview of test scenarios covered in this report: cover vNIC-to-vNIC vNIC-to-nestedVM-to-vNIC forwarding topologies. Scapy is used as a traffic generator. -#. **Honeycomb Functional**: Honeycomb functional tests are executed in - virtual FD.io testbeds, focusing on Honeycomb management and - programming functionality of VPP. Tests cover a range of CRUD - operations executed against VPP. - .. + #. **Honeycomb Functional**: Honeycomb functional tests are executed in + virtual FD.io testbeds, focusing on Honeycomb management and + programming functionality of VPP. Tests cover a range of CRUD + operations executed against VPP. #. **DMM Functional**: DMM functional tests are executed in virtual FD.io testbeds demonstrating a single server (DUT1) and single client (DUT2) scenario using DMM framework and Linux kernel TCP/IP diff --git a/docs/report/vpp_device_tests/csit_release_notes.rst b/docs/report/vpp_device_tests/csit_release_notes.rst index 6f07713413..58ac234f6f 100644 --- a/docs/report/vpp_device_tests/csit_release_notes.rst +++ b/docs/report/vpp_device_tests/csit_release_notes.rst @@ -7,6 +7,14 @@ Changes in |csit-release| #. TEST FRAMEWORK - **Bug fixes**. + - **ARM platform compatibility**. + +#. TEST COVERAGE + + - Increased test coverage: **Dot1q**, **IPsec**, **802.1ad VXLAN**, + **COP whitelist**, **COP blacklist**, **QoS Policer Metering**, + **iACL whitelist**, **AVF driver**, **TAP Interface**. + - Align vpp_device L2 Robot Keywords with performance L2 Robot Keywords. Known Issues ------------ diff --git a/docs/report/vpp_device_tests/overview.rst b/docs/report/vpp_device_tests/overview.rst index a53e3f4971..4a53d619a2 100644 --- a/docs/report/vpp_device_tests/overview.rst +++ b/docs/report/vpp_device_tests/overview.rst @@ -111,6 +111,15 @@ environment: +-----------------------+----------------------------------------------+ | Functionality | Description | +=======================+==============================================+ +| ACL | Ingress Access Control List security for L2 | +| | Bridge-Domain MAC switching, IPv4 routing, | +| | IPv6 routing. | ++-----------------------+----------------------------------------------+ +| COP | COP address white-list and black-list | +| | filtering for IPv4 and IPv6 routing. | ++-----------------------+----------------------------------------------+ +| IPSec | IPSec tunnel and transport modes. | ++-----------------------+----------------------------------------------+ | IPv4 | IPv4 routing, ICMPv4. | +-----------------------+----------------------------------------------+ | IPv6 | IPv4 routing, ICMPv6. | @@ -121,9 +130,19 @@ environment: | L2XC | L2 Cross-Connect switching for untagged | | | Ethernet. | +-----------------------+----------------------------------------------+ +| Memif Interface | Baseline VPP memif interface tests. | ++-----------------------+----------------------------------------------+ +| QoS Policer Metering | Ingress packet rate metering and marking for | +| | IPv4, IPv6. | ++-----------------------+----------------------------------------------+ +| Tap Interface | Baseline Linux tap interface tests. | ++-----------------------+----------------------------------------------+ +| VLAN Tag | L2 VLAN subinterfaces. | ++-----------------------+----------------------------------------------+ | Vhost-user Interface | Baseline VPP vhost-user interface tests. | +-----------------------+----------------------------------------------+ -| Memif Interface | Baseline VPP memif interface tests. | +| VXLAN | VXLAN overlay tunneling for L2-over-IPv4 and | +| | -over-IPv6. | +-----------------------+----------------------------------------------+ Tests Naming diff --git a/docs/report/vpp_device_tests/test_environment.rst b/docs/report/vpp_device_tests/test_environment.rst index 97c296086b..f56f1b913f 100644 --- a/docs/report/vpp_device_tests/test_environment.rst +++ b/docs/report/vpp_device_tests/test_environment.rst @@ -1,2 +1,573 @@ +Integration Tests +================= -.. include:: ../../../../../../docs/vpp-device.rst +Abstract +-------- + +FD.io VPP software data plane technology has become very popular across +a wide range of VPP eco-system use cases, putting higher pressure on +continuous verification of VPP software quality. + +This document describes a proposal for design and implementation of extended +continuous VPP testing by extending existing test environments. +Furthermore it describes and summarizes implementation details of Integration +and System tests platform *1-Node VPP_Device*. It aims to provide a complete +end-to-end view of *1-Node VPP_Device* environment in order to improve +extendability and maintenance, under the guideline of VPP core team. + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", +"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be +interpreted as described in :rfc:`8174`. + +Overview +-------- + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/vpp_device_tests/}} + \includegraphics[width=0.90\textwidth]{vpp_device} + \label{fig:vpp_device} + \end{figure} + +.. only:: html + + .. figure:: vpp_device.svg + :alt: vpp_device + :align: center + +Physical Testbeds +----------------- + +All :abbr:`FD.io (Fast Data Input/Ouput)` :abbr:`CSIT (Continuous System +Integration and Testing)` vpp-device tests are executed on physical testbeds +built with bare-metal servers hosted by :abbr:`LF (Linux Foundation)` FD.io +project. Two 1-node testbed topologies are used: + +- **2-Container Topology**: Consisting of one Docker container acting as SUT + (System Under Test) and one Docker container as TG (Traffic Generator), both + connected in ring topology via physical NIC cross-connecting. + +Current FD.io production testbeds are built with servers based on one +processor generation of Intel Xeons: Skylake (Platinum 8180). Testbeds built +with servers based on Arm processors are in the process of being added to FD.io +production. + +Following section describe existing production 1n-skx testbed. + +1-Node Xeon Skylake (1n-skx) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1n-skx testbed is based on single SuperMicro SYS-7049GP-TRT server equipped with +two Intel Xeon Skylake Platinum 8180 2.5 GHz 28 core processors. Physical +testbed topology is depicted in a figure below. + +.. only:: latex + + .. raw:: latex + + \begin{figure}[H] + \centering + \graphicspath{{../_tmp/src/vpp_device_tests/}} + \includegraphics[width=0.90\textwidth]{vf-2n-nic2nic} + \label{fig:vf-2n-nic2nic} + \end{figure} + +.. only:: html + + .. figure:: vf-2n-nic2nic.svg + :alt: vf-2n-nic2nic + :align: center + +Server is populated with the following NIC models: + +#. NIC-1: x710-da4 4p10GE Intel. +#. NIC-2: x710-da4 4p10GE Intel. + +All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, +doubling the number of logical cores exposed to Linux, with 56 logical +cores and 28 physical cores per processor socket. + +NIC interfaces are shared using Linux vfio_pci and VPP VF drivers: + +- DPDK VF driver, +- Fortville AVF driver. + +Provided Intel x710-da4 4p10GE NICs support 32 VFs per interface, 128 per NIC. + +Complete 1n-skx testbeds specification is available on `CSIT LF Testbeds +<https://wiki.fd.io/view/CSIT/Testbeds:_Xeon_Skx,_Arm,_Atom.>`_ wiki page. + +Total of two 1n-skx testbeds are in operation in FD.io labs. + +1-Node Virtualbox (1n-vbox) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1n-skx testbed can run in single VirtualBox VM machine. This solution replaces +the previously used Vagrant environment based on 3 VMs. + +VirtualBox VM MAY be created by Vagrant and MUST have additional 4 virtio NICs +each pair attached to separate private networks to simulate back-to-back +connections. It SHOULD be 82545EM device model (otherwise can be changed in +boostrap scripts). Example of Vagrant configuration: + +:: + + Vagrant.configure(2) do |c| + c.vm.network "private_network", type: "dhcp", auto_config: false, + virtualbox__intnet: "port1", nic_type: "82545EM" + c.vm.network "private_network", type: "dhcp", auto_config: false, + virtualbox__intnet: "port2", nic_type: "82545EM" + + c.vm.provider :virtualbox do |v| + v.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"] + v.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"] + v.customize ["modifyvm", :id, "--nicpromisc4", "allow-all"] + v.customize ["modifyvm", :id, "--nicpromisc5", "allow-all"] + +Vagrant VM is populated with the following NIC models: + +#. NIC-1: 82545EM Intel. +#. NIC-2: 82545EM Intel. +#. NIC-3: 82545EM Intel. +#. NIC-4: 82545EM Intel. + +Containers +---------- + +It was agreed on :abbr:`TWS (Technical Work Stream)` call to continue with +Ubuntu 18.04 LTS as a baseline system with OPTIONAL extend to Centos 7 and +SuSE per demand [TWSLink]_. + +All :abbr:`DCR (Docker container)` images are REQUIRED to be hosted on Docker +registry available from LF network, publicly available and trackable. For +backup, tracking and contributing purposes all Dockerfiles (including files +needed for building container) MUST be available and stored in [fdiocsitgerrit]_ +repository under appropriate folders. This allows the peer review process to be +done for every change of infrastructure related to scope of this document. +Currently only **csit-shim-dcr** and **csit-sut-dcr** containers will be stored +and maintained under CSIT repository by CSIT contributors. + +At the time of designing solution described in this document the interconnection +between [dockerhub]_ and [fdiocsitgerrit]_ for automated build purposes and +image hosting cannot be established with the trust and respectful to +security of FD.io project. Unless adressed, :abbr:`DCR` images will be placed in +custom registry service [fdioregistry]_. Automated Jenkins jobs will be created +in align of long term solution for container lifecycle and ability to build +new version of docker images. + +In parallel, the effort is started to find the outsourced Docker registry +service. + +Versioning +~~~~~~~~~~ + +As of initial version of vpp-device, we do have only single latest version of +Docker image hosted on [dockerhub]_. This will be addressed as further +improvement with proper semantic versioning. + +jenkins-slave-dcr +~~~~~~~~~~~~~~~~~ + +This :abbr:`DCR` acts as the Jenkins slave (known also as jenkins minion). It +can connect over SSH protocol to TCP port 6022 of **csit-shim-dcr** and executes +non-interactive reservation script. Nomad is responsible for scheduling this +container execution onto specific **1-Node VPP_Device** testbed. It executes +:abbr:`CSIT` environment including :abbr:`CSIT` framework. + +All software dependencies including VPP/DPDK that are not present in +**csit-sut-dcr** container image and/or needs to be compiled prior running on +**csit-sut-dcr** SHOULD be compiled in this container. + +- *Container Image Location*: Docker image at snergster/vpp-ubuntu18. + +- *Container Definition*: Docker file specified at [JenkinsSlaveDcrFile]_. + +- *Initializing*: Container is initialized from within *Consul by HashiCorp* + and *Nomad by HashiCorp*. + +csit-shim-dcr +~~~~~~~~~~~~~ + +This :abbr:`DCR` acts as an intermediate layer running script responsible for +orchestrating topologies under test and reservation. Responsible for managing VF +resources and allocation to :abbr:`DUT (Device Under Test)`, :abbr:`TG +(Traffic Generator)` containers. This MUST to be done on **csit-shim-dcr**. +This image also acts as the generic reservation mechanics arbiter to make sure +that only Y number of simulations are spawned on any given HW node. + +- *Container Image Location*: Docker image at snergster/csit-shim. + +- *Container Definition*: Docker file specified at [CsitShimDcrFile]_. + +- *Initializing*: Container is initialized from within *Consul by HashiCorp* + and *Nomad by HashiCorp*. Required docker parameters, to be able to run + nested containers with VF reservation system are: privileged, net=host, + pid=host. + +- *Connectivity*: Over SSH only, using <host>:6022 format. Currently using + *root* user account as primary. From the jenkins slave it will be able to + connect via env variable, since the jenkins slave doesn't actually know what + host its running on. + + :: + + ssh -p 6022 root@10.30.51.node + +csit-sut-dcr +~~~~~~~~~~~~ + +This :abbr:`DCR` acts as an :abbr:`SUT (System Under Test)`. Any :abbr:`DUT` or +:abbr:`TG` application is installed there. It is RECOMMENDED to install DUT and +all DUT dependencies via commands ``rpm -ihv`` on RedHat based OS or ``dpkg -i`` +on Debian based OS. + +Container is designed to be a very lightweight Docker image that only installs +packages and execute binaries (previously built or downloaded on +**jenkins-slave-dcr**) and contains libraries necessary to run CSIT framework +including those required by DUT/TG. + +- *Container Image Location*: Docker image at snergster/csit-sut. + +- *Container Definition*: Docker file specified at [CsitSutDcrFile]_. + +- *Initializing*: + :: + + docker run + # Run the container in the background and print the new container ID. + --detach=true + # Give extended privileges to this container. A "privileged" container is + # given access to all devices and able to run nested containers. + --privileged + # Publish all exposed ports to random ports on the host interfaces. + --publish-all + # Automatically remove the container when it exits. + --rm + # Size of /dev/shm. + dcr_stc_params+="--shm-size 512M " + # Override access to PCI bus by attaching a filesystem mount to the + # container. + dcr_stc_params+="--mount type=tmpfs,destination=/sys/bus/pci/devices " + # Mount vfio to be able to bind to see binded interfaces. We cannot use + # --device=/dev/vfio as this does not see newly binded interfaces. + dcr_stc_params+="--volume /dev/vfio:/dev/vfio " + # Mount docker.sock to be able to use docker deamon of the host. + dcr_stc_params+="--volume /var/run/docker.sock:/var/run/docker.sock " + # Mount /opt/boot/ where VM kernel and initrd are located. + dcr_stc_params+="--volume /opt/boot/:/opt/boot/ " + # Mount host hugepages for VMs. + dcr_stc_params+="--volume /dev/hugepages/:/dev/hugepages/ " + + Container name is catenated from **csit-** prefix and uuid generated uniquely + for each container instance. + +- *Connectivity*: Over SSH only, using <host>[:<port>] format. Currently using + *root* user account as primary. + :: + + ssh -p <port> root@10.30.51.<node> + +Container required to run as ``--privileged`` due to ability to create nested +containers and have full read/write access to sysfs (for bind/unbind). Docker +automatically pick free network port (``--publish-all``) for ability to connect +over ssh. To be able to limit access to PCI bus, container is creating tmpfs +mount type in PCI bus tree. CSIT reservation script is dynamically linking only +PCI devices (NIC cards) that are reserved for particular container. This +way it is not colliding with other containers. To make vfio work, access to +``/dev/vfio`` must be granted. + +.. todo: Change default user to testuser with non-privileged and install sudo. + +Environment initialization +-------------------------- + +All 1-node servers are to be managed and provisioned via the [ansiblelink]_ set +of playbooks with *vpp-device* role. Full playbooks can be found under +[fdiocsitansible]_ directory. This way we are able to track all configuration +changes of physical servers in gerrit (in structured yaml format) as well as we +are able to extend *vpp-device* to additional servers with less effort or +re-stage servers in case of failure. + +SR-IOV VF initialization is done via ``systemd`` service during host system boot +up. Service with name *csit-initialize-vfs.service* is created under systemd +system context (``/etc/systemd/system/``). By default service is calling +``/usr/local/bin/csit-initialize-vfs.sh`` with single parameter: + +- **start**: Creates maximum number of :abbr:`virtual functions (VFs)` (detected + from ``sriov_totalvfs``) for each whitelisted PCI device. +- **stop**: Removes all :abbr:`VFs` for all whitelisted PCI device. + +Service is considered active even when all of its processes exited successfully. +Stopping service will automatically remove :abbr:`VFs`. + +:: + + [Unit] + Description=CSIT Initialize SR-IOV VFs + After=network.target + + [Service] + Type=one-shot + RemainAfterExit=True + ExecStart=/usr/local/bin/csit-initialize-vfs.sh start + ExecStop=/usr/local/bin/csit-initialize-vfs.sh stop + + [Install] + WantedBy=default.target + +Script is driven by two array variables ``pci_blacklist``/``pci_whitelist``. +They MUST store all PCI addresses in **<domain>:<bus>:<device>.<func>** format, +where: + +- **pci_blacklist**: PCI addresses to be skipped from :abbr:`VFs` + initialization (usefull for e.g. excluding management network interfaces). +- **pci_whitelist**: PCI addresses to be included for :abbr:`VFs` + initialization. + +VF reservation +-------------- + +During topology initialization phase of script, mutex is used to avoid multiple +instances of script to interact with each other during resources allocation. +Mutal exclusion ensure that no two distinct instances of script will get same +resource list. + +Reservation function reads the list of all available virtual function network +devices in system: + +:: + + # Find the first ${device_count} number of available TG Linux network + # VF device names. Only allowed VF PCI IDs are filtered. + for netdev in ${tg_netdev[@]} + do + for netdev_path in $(grep -l "${pci_id}" \ + /sys/class/net/${netdev}*/device/device \ + 2> /dev/null) + do + if [[ ${#TG_NETDEVS[@]} -lt ${device_count} ]]; then + tg_netdev_name=$(dirname ${netdev_path}) + tg_netdev_name=$(dirname ${tg_netdev_name}) + TG_NETDEVS+=($(basename ${tg_netdev_name})) + else + break + fi + done + if [[ ${#TG_NETDEVS[@]} -eq ${device_count} ]]; then + break + fi + done + +Where ``${pci_id}`` is ID of white-listed VF PCI ID. For more information please +see [pciids]_. This act as security constraint to prevent taking other unwanted +interfaces. +The output list of all VF network devices is split into two lists for TG and +SUT side of connection. First two items from each TG or SUT network devices +list are taken to expose directly to namespace of container. This can be done +via commands: + +:: + + $ ip link set ${netdev} netns ${DCR_CPIDS[tg]} + $ ip link set ${netdev} netns ${DCR_CPIDS[dut1]} + +In this stage also symbolic links to PCI devices under sysfs bus directory tree +are created in running containers. Once VF devices are assigned to container +namespace and PCI deivces are linked to running containers and mutex is exited. +Selected VF network device automatically dissapear from parent container +namespace, so another instance of script will not find device under that +namespace. + +Once Docker container exits, network device is returned back into parent +namespace and can be reused. + +Network traffic isolation - Intel i40evf +---------------------------------------- + +In a virtualized environment, on Intel(R) Server Adapters that support SR-IOV, +the virtual function (VF) may be subject to malicious behavior. Software- +generated layer two frames, like IEEE 802.3x (link flow control), IEEE 802.1Qbb +(priority based flow-control), and others of this type, are not expected and +can throttle traffic between the host and the virtual switch, reducing +performance. To resolve this issue, configure all SR-IOV enabled ports for +VLAN tagging. This configuration allows unexpected, and potentially malicious, +frames to be dropped. [inteli40e]_ + +To configure VLAN tagging for the ports on an SR-IOV enabled adapter, +use the following command. The VLAN configuration SHOULD be done +before the VF driver is loaded or the VM is booted. [inteli40e]_ + +:: + + $ ip link set dev <PF netdev id> vf <id> vlan <vlan id> + +For example, the following instructions will configure PF eth0 and +the first VF on VLAN 10. + +:: + + $ ip link set dev eth0 vf 0 vlan 10 + +VLAN Tag Packet Steering allows to send all packets with a specific VLAN tag to +a particular SR-IOV virtual function (VF). Further, this feature allows to +designate a particular VF as trusted, and allows that trusted VF to request +selective promiscuous mode on the Physical Function (PF). [inteli40e]_ + +To set a VF as trusted or untrusted, enter the following command in the +Hypervisor: + +:: + + $ ip link set dev eth0 vf 1 trust [on|off] + +Once the VF is designated as trusted, use the following commands in the VM +to set the VF to promiscuous mode. [inteli40e]_ + +- For promiscuous all: + :: + + $ ip link set eth2 promisc on + +- For promiscuous Multicast: + :: + + $ ip link set eth2 allmulti on + +.. note:: + + By default, the ethtool priv-flag vf-true-promisc-support is set to + *off*, meaning that promiscuous mode for the VF will be limited. To set the + promiscuous mode for the VF to true promiscuous and allow the VF to see + all ingress traffic, use the following command. + $ ethtool set-priv-flags p261p1 vf-true-promisc-support on + The vf-true-promisc-support priv-flag does not enable promiscuous mode; + rather, it designates which type of promiscuous mode (limited or true) + you will get when you enable promiscuous mode using the ip link commands + above. Note that this is a global setting that affects the entire device. + However,the vf-true-promisc-support priv-flag is only exposed to the first + PF of the device. The PF remains in limited promiscuous mode (unless it + is in MFP mode) regardless of the vf-true-promisc-support setting. + [inteli40e]_ + +Service described earlier *csit-initialize-vfs.service* is responsible for +assigning 802.1Q vlan tagging to each vitual function via physical function +from list of white-listed PCI addresses by following (simplified) code. + +:: + + SCRIPT_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))" + source "${SCRIPT_DIR}/csit-initialize-vfs-data.sh" + + # Initilize whitelisted NICs with maximum number of VFs. + pci_idx=0 + for pci_addr in ${PCI_WHITELIST[@]}; do + if ! [[ ${PCI_BLACKLIST[*]} =~ "${pci_addr}" ]]; then + pci_path="/sys/bus/pci/devices/${pci_addr}" + # SR-IOV initialization + case "${1:-start}" in + "start" ) + sriov_totalvfs=$(< "${pci_path}"/sriov_totalvfs) + ;; + "stop" ) + sriov_totalvfs=0 + ;; + esac + echo ${sriov_totalvfs} > "${pci_path}"/sriov_numvfs + # SR-IOV 802.1Q isolation + case "${1:-start}" in + "start" ) + pf=$(basename "${pci_path}"/net/*) + for vf in $(seq "${sriov_totalvfs}"); do + # PCI address index in array (pairing siblings). + if [[ -n ${PF_INDICES[@]} ]] + then + vlan_pf_idx=${PF_INDICES[$pci_addr]} + else + vlan_pf_idx=$((pci_idx % (${#PCI_WHITELIST[@]}/2))) + fi + # 802.1Q base offset. + vlan_bs_off=1100 + # 802.1Q PF PCI address offset. + vlan_pf_off=$(( vlan_pf_idx * 100 + vlan_bs_off )) + # 802.1Q VF PCI address offset. + vlan_vf_off=$(( vlan_pf_off + vf - 1 )) + # VLAN string. + vlan_str="vlan ${vlan_vf_off}" + # MAC string. + mac5="$(printf '%x' ${pci_idx})" + mac6="$(printf '%x' $(( vf - 1 )))" + mac_str="mac ba:dc:0f:fe:${mac5}:${mac6}" + # Set 802.1Q VLAN id and MAC address + ip link set ${pf} vf $(( vf - 1)) ${mac_str} ${vlan_str} + ip link set ${pf} vf $(( vf - 1)) trust on + ip link set ${pf} vf $(( vf - 1)) spoof off + done + pci_idx=$(( pci_idx + 1 )) + ;; + esac + rmmod i40evf + modprobe i40evf + fi + done + +Assignment starts at VLAN 1100 and incrementing by 1 for each VF and by 100 for +each white-listed PCI address up to the middle of the PCI list. Second half of +the lists is assumed to be directly (cable) paired siblings and assigned with +same 802.1Q VLANs as its siblings. + +Open tasks +---------- + +Security +~~~~~~~~ + +.. note:: + + Switch to non-privileged containers: As of now all three container + flavors are using privileged containers to make it working. Explore options + to switch containers to non-privileged with explicit rather implicit + privileges. + +.. note:: + + Switch to testuser account intead of root. + +Maintainability +~~~~~~~~~~~~~~~ + +.. note:: + + Docker image distribution: Create jenkins jobs with full pipiline of + CI/CD for CSIT Docker images. + +Stability +~~~~~~~~~ + +.. note:: + + Implement queueing mechanism: Currently there is no mechanics that + would place starving jobs in queue in case of no resources available. + +.. note:: + + Replace reservation script with Docker network plugin written in + GOLANG/SH/Python - platform independent. + +Links +----- + +.. [TWSLink] `TWS <https://wiki.fd.io/view/CSIT/TWS>`_ +.. [dockerhub] `Docker hub <https://hub.docker.com/>`_ +.. [fdiocsitgerrit] `FD.io/CSIT gerrit <https://gerrit.fd.io/r/CSIT>`_ +.. [fdioregistry] `FD.io registy <registry.fdiopoc.net>`_ +.. [JenkinsSlaveDcrFile] `jenkins-slave-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/ubuntu18/Dockerfile>`_ +.. [CsitShimDcrFile] `csit-shim-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/csit-shim/Dockerfile>`_ +.. [CsitSutDcrFile] `csit-sut-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/csit-sut/Dockerfile>`_ +.. [ansiblelink] `ansible <https://www.ansible.com/>`_ +.. [fdiocsitansible] `Fd.io/CSIT ansible <https://git.fd.io/csit/tree/resources/tools/testbed-setup/ansible>`_ +.. [inteli40e] `Intel i40e <https://downloadmirror.intel.com/26370/eng/readme.txt>`_ +.. [pciids] `pci ids <http://pci-ids.ucw.cz/v2.2/pci.ids>`_ diff --git a/docs/report/vpp_functional_tests/csit_release_notes.rst b/docs/report/vpp_functional_tests/csit_release_notes.rst index 57114cc858..ecb0a63ebc 100644 --- a/docs/report/vpp_functional_tests/csit_release_notes.rst +++ b/docs/report/vpp_functional_tests/csit_release_notes.rst @@ -10,43 +10,17 @@ Changes in |csit-release| #. CSIT TEST MIMGRATION - - **VPP_Path**: Continuing migration of the original FD.io CSIT VIRL - tests to VPP-make_test VPP integration tests for functional - acceptance of VPP feature path(s) driven by use case(s). See P1 - and P2 markup in `CSIT_VIRL migration progress - <https://docs.google.com/spreadsheets/d/1PciV8XN9v1qHbIRUpFJoqyES29_vik7lcFDl73G1usc/edit?usp=sharing>`_ + - **VPP_Device**: Continuing migration of the original FD.io CSIT VIRL + tests to VPP-device tests for functional acceptance of VPP feature path(s) + driven by performance tests. Known Issues ------------ List of known issues in |csit-release| for VPP functional tests in VIRL: -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ -| # | JiraID | Issue Description | -+===+=========================================+=========================================================================================================================+ -| 1 | `CSIT-129 | DHCPv4 client: Client responses to DHCPv4 OFFER sent with different XID. | -| | <https://jira.fd.io/browse/CSIT-129>`_ | Client replies with DHCPv4 REQUEST message when received DHCPv4 OFFER message with different (wrong) XID. | -| | `VPP-99 | | -| | <https://jira.fd.io/browse/VPP-99>`_ | | -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ -| 2 | `CSIT-398 | Softwire - MAP-E: Incorrect calculation of IPv6 destination address when IPv4 prefix is 0. | -| | <https://jira.fd.io/browse/CSIT-398>`_ | IPv6 destination address is wrongly calculated in case that IPv4 prefix is equal to 0 and IPv6 prefix is less than 40. | -| | `VPP-380 | | -| | <https://jira.fd.io/browse/VPP-380>`_ | | -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ -| 3 | `CSIT-399 | Softwire - MAP-E: Map domain is created when incorrect parameters provided. | -| | <https://jira.fd.io/browse/CSIT-399>`_ | Map domain is created in case that the sum of suffix length of IPv4 prefix and PSID length is greater than EA bits | -| | `VPP-435 | length. IPv6 destination address contains bits writen with PSID over the EA-bit length when IPv4 packet is sent. | -| | <https://jira.fd.io/browse/VPP-435>`_ | | -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ -| 4 | `CSIT-409 | IPv6 RA: Incorrect IPv6 destination address in response to ICMPv6 Router Solicitation. | -| | <https://jira.fd.io/browse/CSIT-409>`_ | Wrong IPv6 destination address (ff02::1) is used in ICMPv6 Router Advertisement packet sent as a response to received | -| | `VPP-406 | ICMPv6 Router Solicitation packet. | -| | <https://jira.fd.io/browse/VPP-406>`_ | | -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-565 | Vhost-user: QEMU reconnect does not work. | -| | <https://jira.fd.io/browse/CSIT-565>`_ | QEMU 2.5.0 used in CSIT does not support vhost-user reconnect. Requires upgrading CSIT VIRL environment to QEMU 2.7.0. | -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ -| 6 | `CSIT-1371 | Softwire: Exclude all softwire functional tests until KWs re-worked to PAPI | -| | <https://jira.fd.io/browse/CSIT-1371>`_ | Map commands were remove from VAT by VPP patch https://gerrit.fd.io/r/#/c/16115/. | -+---+-----------------------------------------+-------------------------------------------------------------------------------------------------------------------------+ ++---+--------------------+-----------------------------------------------------+ +| # | JiraID | Issue Description | ++===+====================+=====================================================+ +| | | | ++---+--------------------+-----------------------------------------------------+ diff --git a/docs/report/vpp_functional_tests/overview.rst b/docs/report/vpp_functional_tests/overview.rst index a4635e7f85..510f204bdf 100644 --- a/docs/report/vpp_functional_tests/overview.rst +++ b/docs/report/vpp_functional_tests/overview.rst @@ -106,56 +106,9 @@ environment: +-----------------------+----------------------------------------------+ | Functionality | Description | +=======================+==============================================+ -| ACL | Ingress Access Control List security for L2 | -| | Bridge-Domain MAC switching, IPv4 routing, | -| | IPv6 routing. | -+-----------------------+----------------------------------------------+ -| COP | COP address white-list and black-list | -| | filtering for IPv4 and IPv6 routing. | -+-----------------------+----------------------------------------------+ -| DHCP | Dynamic Host Control Protocol Client and | -| | Proxy for IPv4 and IPv6 routing. | -+-----------------------+----------------------------------------------+ -| GRE | Generic Routing Encapsulation Overlay | -| | Tunnels for IPv4. | -+-----------------------+----------------------------------------------+ -| IPSec | IPSec tunnel and transport modes. | -+-----------------------+----------------------------------------------+ -| IPv4 | IPv4 routing, RPF, ARP, Proxy ARP, ICMPv4. | -+-----------------------+----------------------------------------------+ -| IPv6 | IPv6 routing, NS/ND, RA, ICMPv6. | -+-----------------------+----------------------------------------------+ -| L2BD | L2 Bridge-Domain switching for untagged | -| | Ethernet, dot1q and dot1ad tagged. | -+-----------------------+----------------------------------------------+ -| L2XC | L2 Cross-Connect switching for untagged | -| | Ethernet, dot1q and dot1ad tagged. | -+-----------------------+----------------------------------------------+ | LISP | Locator/ID Separation Protocol overlay | | | tunnels and locator/id mapping control. | +-----------------------+----------------------------------------------+ -| QoS Policer Metering | Ingress packet rate metering and marking for | -| | IPv4, IPv6. | -+-----------------------+----------------------------------------------+ -| Softwire Tunnels | IPv4-in-IPv6 softwire tunnels. | -+-----------------------+----------------------------------------------+ -| Tap Interface | Baseline Linux tap interface tests. | -+-----------------------+----------------------------------------------+ -| IPFIX and SPAN | Telemetry IPFIX netflow statistics and SPAN | -| | port mirroring. | -+-----------------------+----------------------------------------------+ -| uRPF Source Security | Unicast Reverse Path Forwarding security for | -| | IPv4 and IPv6 routing. | -+-----------------------+----------------------------------------------+ -| VLAN Tag Translation | L2 VLAN tag translation 2to2, 2to1, 1to2, | -| | 1to1. | -+-----------------------+----------------------------------------------+ -| VRF Routing | Multi-context VRF IPVPN routing for IPv4 and | -| | IPv6. | -+-----------------------+----------------------------------------------+ -| VXLAN | VXLAN overlay tunneling for L2-over-IPv4 and | -| | -over-IPv6. | -+-----------------------+----------------------------------------------+ Functional Tests Naming ----------------------- diff --git a/docs/report/vpp_performance_tests/csit_release_notes.rst b/docs/report/vpp_performance_tests/csit_release_notes.rst index 74ab68062b..3156ece355 100644 --- a/docs/report/vpp_performance_tests/csit_release_notes.rst +++ b/docs/report/vpp_performance_tests/csit_release_notes.rst @@ -6,35 +6,33 @@ Changes in |csit-release| #. VPP PERFORMANCE TESTS - - **Service density 2n-skx tests**: Added higher NF density tests with two - NFs' data-plane threads sharing a physical core. VPP IPv4 routing is now - used as a VNF payload similar to CNF tests. + - **Service density 2n-skx tests**: Added higher NF density tests with + 802.1q (vlan) and 802.1ad (vxlan) encapsulation from Traffic Generator. - - **Soak Tests**: Optimized performamce soak tests framework - code for extended time duration tests and throughput discovery - at given PLR and at give total test time e.g. minutes, hours, - days, weeks. See updated - :ref:`test_methodology` section for more details. + - **GBP tests**: Added GBP routing test cases with 802.1q (vlan) external + traffic. -#. TEST FRAMEWORK + - **AVF IPv4 scale tests**: Increased coverage of AVF IPv4 base and scale + test cases. + + - **2n-skx tests**: Increased coverage of selected (COP, iACL, Policer) + test cases. + + - **IPsec scale tests**: Added IPsec interface mode scale tests with + 1, 40, 400, 1000, 5000, 10000, 20000, 40000, 60000 tunnels. Removed DPDK + backend dependency. - - **Qemu code refactor**: Complete code refactor of the key components of - QemuUtil.py and QemuManager.py (L1 and L2 KW counterparts). Added - implementation of kernel-image-kvm based VM replacing the previously used - NestedVM images. Added ability to run VPP as a payload in VNF. +#. TEST FRAMEWORK - - **CSIT PAPI Support**: Continued conversion of CSIT VAT L1 keywords to + - **CSIT PAPI Support**: Finished conversion of CSIT VAT L1 keywords to PAPI L1 KWs in CSIT using VPP Python bindings. Redesign of key components - of PAPI Executor and PAPI history. + of PAPI Executor and PAPI history. Currently the only exception is + usage of VAT command for scale configuration. - **General Code Housekeeping**: Ongoing RF keywords optimizations, - removal of redundant RF keywords. - - - **Test suite generator**: Added capability to generate suites for - different NIC models as well as throughput search algorithm types. Uses - base tests suites as source. + removal of redundant RF keywords and aligning of suite/test + setup/teardowns. - - **TOX verification**: Added verifications for test suite generator. #. PRESENTATION AND ANALYTICS LAYER @@ -42,16 +40,6 @@ Changes in |csit-release| for better readibility and maintenance: test grouping, axis labels, descriptions, other informative decoration. -.. - #. MISCELLANEOUS - - - **2n-dnv Tests (3rd Party)**: Published performance tests for 2n- - dnv (2-Node Atom Denverton) from 3rd party testbeds running FD.io - |csit-release| automated testing code. - Only graphs for Packet Throughput and Speedup Multi-core and not - for Packet Latency were published as there are no results for Packet - Latency available. - .. raw:: latex \clearpage @@ -72,21 +60,6 @@ List of known issues in |csit-release| for VPP performance tests: | 2 | `CSIT-1503 | [`TRex-519 <https://trex-tgn.cisco.com/youtrack/issue/trex-519>`_] XL710/XXV710 with FW 6.0.1 will have | | | <https://jira.fd.io/browse/CSIT-1503>`_ | Rx drop rate of 27MPPS. | +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ -| 3 | `CSIT-1501 | Sporadic crypto backend fails loading `VPP-1670 <https://jira.fd.io/browse/VPP-1670>`_ | -| | <https://jira.fd.io/browse/CSIT-1501>`_ | | -+----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ -| 4 | `CSIT-1427 | Sporadic HW aes-128-cbc-sha1 tunnel-interface tests are failing. | -| | <https://jira.fd.io/browse/CSIT-1427>`_ | `VPP-1671 <https://jira.fd.io/browse/VPP-1671>`_ | -+----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ -| 5 | `CSIT-1498 | Memif tests are sporadically failing on initialization of memif connection. | -| | <https://jira.fd.io/browse/CSIT-1498>`_ | | -+----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ -| 6 | `CSIT-1499 | AVF tests are sporadically failing on initialization of AVF interface. | +| 3 | `CSIT-1499 | AVF tests are sporadically failing on initialization of AVF interface. | | | <https://jira.fd.io/browse/CSIT-1499>`_ | | +----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ -| 7 | `VPP-1676 | 9000B ip4 memif errors - ip4-input: ip4 length > l2 length. | -| | <https://jira.fd.io/browse/VPP-1676>`_ | IP4 jumbo frames (9000B) are dropped in case of tests with memif. | -+----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ -| 8 | `VPP-1677 | 9000B ip4 nat44: VPP crash + coredump. | -| | <https://jira.fd.io/browse/VPP-1677>`_ | VPP crashes very often in case that NAT44 is configured and it has to process IP4 jumbo frames (9000B). | -+----+-----------------------------------------+----------------------------------------------------------------------------------------------------------+ diff --git a/docs/report/vpp_performance_tests/overview.rst b/docs/report/vpp_performance_tests/overview.rst index 3c95919c55..a2ead6a0b6 100644 --- a/docs/report/vpp_performance_tests/overview.rst +++ b/docs/report/vpp_performance_tests/overview.rst @@ -291,7 +291,7 @@ performance tested across a range of NIC drivers and NIC models: | IPv6 Scale | IPv6 routing with 20k, 200k and 2M FIB | | | entries. | +-----------------------+----------------------------------------------+ -| IPSecHW | IPSec encryption with AES-GCM, CBC-SHA1 | +| IPSecHW | IPSec encryption with AES-GCM, CBC-SHA-256 | | | ciphers, in combination with IPv4 routing. | | | Intel QAT HW acceleration. | +-----------------------+----------------------------------------------+ @@ -299,7 +299,7 @@ performance tested across a range of NIC drivers and NIC models: | | combination with LISP-GPE overlay tunneling | | | for IPv4-over-IPv4. | +-----------------------+----------------------------------------------+ -| IPSecSW | IPSec encryption with AES-GCM, CBC-SHA1 | +| IPSecSW | IPSec encryption with AES-GCM, CBC-SHA-256 | | | ciphers, in combination with IPv4 routing. | +-----------------------+----------------------------------------------+ | K8s Containers Memif | K8s orchestrated container VPP service chain | @@ -307,7 +307,7 @@ performance tested across a range of NIC drivers and NIC models: | | interface. | +-----------------------+----------------------------------------------+ | KVM VMs vhost-user | Virtual topologies with service | -| | chains of 1 and 2 VMs using vhost-user | +| | chains of 1 VM using vhost-user | | | interfaces, with different VPP forwarding | | | modes incl. L2XC, L2BD, VXLAN with L2BD, | | | IPv4 routing. | diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst index 3c179e1f7a..48045e943e 100644 --- a/docs/report/vpp_performance_tests/test_environment.rst +++ b/docs/report/vpp_performance_tests/test_environment.rst @@ -15,10 +15,6 @@ .. include:: ../introduction/test_environment_sut_conf_1.rst -.. include:: ../introduction/test_environment_sut_conf_2.rst - -.. include:: ../introduction/test_environment_sut_conf_3.rst - DUT Settings - VPP ------------------ @@ -38,7 +34,7 @@ VPP Install Parameters :: - $ dpkg -i --force-all vpp* + $ dpkg -i --force-all *vpp* VPP Startup Configuration ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -99,7 +95,6 @@ below: { num-rx-queues $$NUM_RX_QUEUES } - socket-mem 1024,1024 no-tx-checksum-offload dev $$DEV_1 dev $$DEV_2 diff --git a/docs/vpp-device.rst b/docs/vpp-device.rst deleted file mode 100644 index 4289887c6b..0000000000 --- a/docs/vpp-device.rst +++ /dev/null @@ -1,534 +0,0 @@ -Integration Tests -================= - -Abstract --------- - -FD.io VPP software data plane technology has become very popular across -a wide range of VPP eco-system use cases, putting higher pressure on -continuous verification of VPP software quality. - -This document describes a proposal for design and implementation of extended -continuous VPP testing by extending existing test environments. -Furthermore it describes and summarizes implementation details of Integration -and System tests platform *1-Node VPP_Device*. It aims to provide a complete -end-to-end view of *1-Node VPP_Device* environment in order to improve -extendability and maintenance, under the guideline of VPP core team. - -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", -"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be -interpreted as described in :rfc:`8174`. - -Overview --------- - -.. only:: latex - - .. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_tmp/src/vpp_device_tests/}} - \includegraphics[width=0.90\textwidth]{vpp_device} - \label{fig:vpp_device} - \end{figure} - -.. only:: html - - .. figure:: vpp_device.svg - :alt: vpp_device - :align: center - -Physical Testbeds ------------------ - -All :abbr:`FD.io (Fast Data Input/Ouput)` :abbr:`CSIT (Continuous System -Integration and Testing)` vpp-device tests are executed on physical testbeds -built with bare-metal servers hosted by :abbr:`LF (Linux Foundation)` FD.io -project. Two 1-node testbed topologies are used: - -- **2-Container Topology**: Consisting of one Docker container acting as SUT - (System Under Test) and one Docker container as TG (Traffic Generator), both - connected in ring topology via physical NIC cross-connecting. - -Current FD.io production testbeds are built with servers based on one -processor generation of Intel Xeons: Skylake (Platinum 8180). Testbeds built -with servers based on Arm processors are in the process of being added to FD.io -production. - -Following section describe existing production 1n-skx testbed. - -1-Node Xeon Skylake (1n-skx) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1n-skx testbed is based on single SuperMicro SYS-7049GP-TRT server equipped with -two Intel Xeon Skylake Platinum 8180 2.5 GHz 28 core processors. Physical -testbed topology is depicted in a figure below. - -.. only:: latex - - .. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_tmp/src/vpp_device_tests/}} - \includegraphics[width=0.90\textwidth]{vf-2n-nic2nic} - \label{fig:vf-2n-nic2nic} - \end{figure} - -.. only:: html - - .. figure:: vf-2n-nic2nic.svg - :alt: vf-2n-nic2nic - :align: center - -Server is populated with the following NIC models: - -#. NIC-1: x710-da4 4p10GE Intel. -#. NIC-2: x710-da4 4p10GE Intel. - -All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, -doubling the number of logical cores exposed to Linux, with 56 logical -cores and 28 physical cores per processor socket. - -NIC interfaces are shared using Linux vfio_pci and VPP VF drivers: - -- DPDK VF driver, -- Fortville AVF driver. - -Provided Intel x710-da4 4p10GE NICs support 32 VFs per interface, 128 per NIC. - -Complete 1n-skx testbeds specification is available on `CSIT LF Testbeds -<https://wiki.fd.io/view/CSIT/Testbeds:_Xeon_Skx,_Arm,_Atom.>`_ wiki page. - -Total of two 1n-skx testbeds are in operation in FD.io labs. - -1-Node Virtualbox (1n-vbox) -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1n-skx testbed can run in single VirtualBox VM machine. This solution replaces -the previously used Vagrant environment based on 3 VMs. - -VirtualBox VM MAY be created by Vagrant and MUST have additional 4 virtio NICs -each pair attached to separate private networks to simulate back-to-back -connections. It SHOULD be 82545EM device model (otherwise can be changed in -boostrap scripts). Example of Vagrant configuration: - -:: - - Vagrant.configure(2) do |c| - c.vm.network "private_network", type: "dhcp", auto_config: false, - virtualbox__intnet: "port1", nic_type: "82545EM" - c.vm.network "private_network", type: "dhcp", auto_config: false, - virtualbox__intnet: "port2", nic_type: "82545EM" - - c.vm.provider :virtualbox do |v| - v.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"] - v.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"] - v.customize ["modifyvm", :id, "--nicpromisc4", "allow-all"] - v.customize ["modifyvm", :id, "--nicpromisc5", "allow-all"] - -Vagrant VM is populated with the following NIC models: - -#. NIC-1: 82545EM Intel. -#. NIC-2: 82545EM Intel. -#. NIC-3: 82545EM Intel. -#. NIC-4: 82545EM Intel. - -Containers ----------- - -It was agreed on :abbr:`TWS (Technical Work Stream)` call to continue with -Ubuntu 18.04 LTS as a baseline system with OPTIONAL extend to Centos 7 and -SuSE per demand [TWSLink]_. - -All :abbr:`DCR (Docker container)` images are REQUIRED to be hosted on Docker -registry available from LF network, publicly available and trackable. For -backup, tracking and contributing purposes all Dockerfiles (including files -needed for building container) MUST be available and stored in [fdiocsitgerrit]_ -repository under appropriate folders. This allows the peer review process to be -done for every change of infrastructure related to scope of this document. -Currently only **csit-shim-dcr** and **csit-sut-dcr** containers will be stored -and maintained under CSIT repository by CSIT contributors. - -At the time of designing solution described in this document the interconnection -between [dockerhub]_ and [fdiocsitgerrit]_ for automated build purposes and -image hosting cannot be established with the trust and respectful to -security of FD.io project. Unless adressed, :abbr:`DCR` images will be placed in -custom registry service [fdioregistry]_. Automated Jenkins jobs will be created -in align of long term solution for container lifecycle and ability to build -new version of docker images. - -In parallel, the effort is started to find the outsourced Docker registry -service. - -Versioning -~~~~~~~~~~ - -As of initial version of vpp-device, we do have only single latest version of -Docker image hosted on [dockerhub]_. This will be addressed as further -improvement with proper semantic versioning. - -jenkins-slave-dcr -~~~~~~~~~~~~~~~~~ - -This :abbr:`DCR` acts as the Jenkins slave (known also as jenkins minion). It -can connect over SSH protocol to TCP port 6022 of **csit-shim-dcr** and executes -non-interactive reservation script. Nomad is responsible for scheduling this -container execution onto specific **1-Node VPP_Device** testbed. It executes -:abbr:`CSIT` environment including :abbr:`CSIT` framework. - -All software dependencies including VPP/DPDK that are not present in -**csit-sut-dcr** container image and/or needs to be compiled prior running on -**csit-sut-dcr** SHOULD be compiled in this container. - -- *Container Image Location*: Docker image at snergster/vpp-ubuntu18. - -- *Container Definition*: Docker file specified at [JenkinsSlaveDcrFile]_. - -- *Initializing*: Container is initialized from within *Consul by HashiCorp* - and *Nomad by HashiCorp*. - -csit-shim-dcr -~~~~~~~~~~~~~ - -This :abbr:`DCR` acts as an intermediate layer running script responsible for -orchestrating topologies under test and reservation. Responsible for managing VF -resources and allocation to :abbr:`DUT (Device Under Test)`, :abbr:`TG -(Traffic Generator)` containers. This MUST to be done on **csit-shim-dcr**. -This image also acts as the generic reservation mechanics arbiter to make sure -that only Y number of simulations are spawned on any given HW node. - -- *Container Image Location*: Docker image at snergster/csit-shim. - -- *Container Definition*: Docker file specified at [CsitShimDcrFile]_. - -- *Initializing*: Container is initialized from within *Consul by HashiCorp* - and *Nomad by HashiCorp*. Required docker parameters, to be able to run - nested containers with VF reservation system are: privileged, net=host, - pid=host. - -- *Connectivity*: Over SSH only, using <host>:6022 format. Currently using - *root* user account as primary. From the jenkins slave it will be able to - connect via env variable, since the jenkins slave doesn't actually know what - host its running on. - - :: - - ssh -p 6022 root@10.30.51.node - -csit-sut-dcr -~~~~~~~~~~~~ - -This :abbr:`DCR` acts as an :abbr:`SUT (System Under Test)`. Any :abbr:`DUT` or -:abbr:`TG` application is installed there. It is RECOMMENDED to install DUT and -all DUT dependencies via commands ``rpm -ihv`` on RedHat based OS or ``dpkg -i`` -on Debian based OS. - -Container is designed to be a very lightweight Docker image that only installs -packages and execute binaries (previously built or downloaded on -**jenkins-slave-dcr**) and contains libraries necessary to run CSIT framework -including those required by DUT/TG. - -- *Container Image Location*: Docker image at snergster/csit-sut. - -- *Container Definition*: Docker file specified at [CsitSutDcrFile]_. - -- *Initializing*: - :: - - docker run - # Run the container in the background and print the new container ID. - --detach=true - # Give extended privileges to this container. A "privileged" container is - # given access to all devices and able to run nested containers. - --privileged - # Publish all exposed ports to random ports on the host interfaces. - --publish-all - # Automatically remove the container when it exits. - --rm - # Size of /dev/shm. - --shm-size 512M - # Override access to PCI bus by attaching a filesystem mount to the - # container. - --mount type=tmpfs,destination=/sys/bus/pci/devices - # Mount vfio to be able to bind to see binded interfaces. We cannot use - # --device=/dev/vfio as this does not see newly binded interfaces. - --volume /dev/vfio:/dev/vfio - # Mount nested_vm image to be able to run VM tests. - --volume /var/lib/vm/vhost-nested.img:/var/lib/vm/vhost-nested.img - # Mount docker.sock to be able to use docker deamon of the host. - --volume /var/run/docker.sock:/var/run/docker.sock - # Image of csit-sut-dcr - snergster/csit-vpp-device-test:latest - - Container name is catenated from **csit-** prefix and uuid generated uniquely - for each container instance. - -- *Connectivity*: Over SSH only, using <host>[:<port>] format. Currently using - *root* user account as primary. - :: - - ssh -p <port> root@10.30.51.<node> - -Container required to run as ``--privileged`` due to ability to create nested -containers and have full read/write access to sysfs (for bind/unbind). Docker -automatically pick free network port (``--publish-all``) for ability to connect -over ssh. To be able to limit access to PCI bus, container is creating tmpfs -mount type in PCI bus tree. CSIT reservation script is dynamically linking only -PCI devices (NIC cards) that are reserved for particular container. This -way it is not colliding with other containers. To make vfio work, access to -``/dev/vfio`` must be granted. - -.. todo: Change default user to testuser with non-privileged and install sudo. - -Environment initialization --------------------------- - -All 1-node servers are to be managed and provisioned via the [ansiblelink]_ set -of playbooks with *vpp-device* role. Full playbooks can be found under -[fdiocsitansible]_ directory. This way we are able to track all configuration -changes of physical servers in gerrit (in structured yaml format) as well as we -are able to extend *vpp-device* to additional servers with less effort or -re-stage servers in case of failure. - -SR-IOV VF initialization is done via ``systemd`` service during host system boot -up. Service with name *csit-initialize-vfs.service* is created under systemd -system context (``/etc/systemd/system/``). By default service is calling -``/usr/local/bin/csit-initialize-vfs.sh`` with single parameter: - -- **start**: Creates maximum number of :abbr:`virtual functions (VFs)` (detected - from ``sriov_totalvfs``) for each whitelisted PCI device. -- **stop**: Removes all :abbr:`VFs` for all whitelisted PCI device. - -Service is considered active even when all of its processes exited successfully. -Stopping service will automatically remove :abbr:`VFs`. - -:: - - [Unit] - Description=CSIT Initialize SR-IOV VFs - After=network.target - - [Service] - Type=one-shot - RemainAfterExit=True - ExecStart=/usr/local/bin/csit-initialize-vfs.sh start - ExecStop=/usr/local/bin/csit-initialize-vfs.sh stop - - [Install] - WantedBy=default.target - -Script is driven by two array variables ``pci_blacklist``/``pci_whitelist``. -They MUST store all PCI addresses in **<domain>:<bus>:<device>.<func>** format, -where: - -- **pci_blacklist**: PCI addresses to be skipped from :abbr:`VFs` - initialization (usefull for e.g. excluding management network interfaces). -- **pci_whitelist**: PCI addresses to be included for :abbr:`VFs` - initialization. - -VF reservation --------------- - -During topology initialization phase of script, mutex is used to avoid multiple -instances of script to interact with each other during resources allocation. -Mutal exclusion ensure that no two distinct instances of script will get same -resource list. - -Reservation function reads the list of all available virtual function network -devices in system: - -:: - - net_path="/sys/bus/pci/devices/*/net/*" - - for netdev in \ - $(find ${net_path} -type d -name . -o -prune -exec basename '{}' ';'); - do - if grep -q "${pci_id}" "/sys/class/net/${netdev}/device/device"; then - # found VF - fi - done - -Where ``${pci_id}`` is ID of white-listed VF PCI ID. For more information please -see [pciids]_. This act as security constraint to prevent taking other unwanted -interfaces. -The output list of all VF network devices is split into two lists for TG and -SUT side of connection. First two items from each TG or SUT network devices -list are taken to expose directly to namespace of container. This can be done -via commands: - -:: - - $ ip link set ${netdev} netns ${DCR_CPIDS[tg]} - $ ip link set ${netdev} netns ${DCR_CPIDS[dut1]} - -In this stage also symbolic links to PCI devices under sysfs bus directory tree -are created in running containers. Once VF devices are assigned to container -namespace and PCI deivces are linked to running containers and mutex is exited. -Selected VF network device automatically dissapear from parent container -namespace, so another instance of script will not find device under that -namespace. - -Once Docker container exits, network device is returned back into parent -namespace and can be reused. - -Network traffic isolation - Intel i40evf ----------------------------------------- - -In a virtualized environment, on Intel(R) Server Adapters that support SR-IOV, -the virtual function (VF) may be subject to malicious behavior. Software- -generated layer two frames, like IEEE 802.3x (link flow control), IEEE 802.1Qbb -(priority based flow-control), and others of this type, are not expected and -can throttle traffic between the host and the virtual switch, reducing -performance. To resolve this issue, configure all SR-IOV enabled ports for -VLAN tagging. This configuration allows unexpected, and potentially malicious, -frames to be dropped. [inteli40e]_ - -To configure VLAN tagging for the ports on an SR-IOV enabled adapter, -use the following command. The VLAN configuration SHOULD be done -before the VF driver is loaded or the VM is booted. [inteli40e]_ - -:: - - $ ip link set dev <PF netdev id> vf <id> vlan <vlan id> - -For example, the following instructions will configure PF eth0 and -the first VF on VLAN 10. - -:: - - $ ip link set dev eth0 vf 0 vlan 10 - -VLAN Tag Packet Steering allows to send all packets with a specific VLAN tag to -a particular SR-IOV virtual function (VF). Further, this feature allows to -designate a particular VF as trusted, and allows that trusted VF to request -selective promiscuous mode on the Physical Function (PF). [inteli40e]_ - -To set a VF as trusted or untrusted, enter the following command in the -Hypervisor: - -:: - - $ ip link set dev eth0 vf 1 trust [on|off] - -Once the VF is designated as trusted, use the following commands in the VM -to set the VF to promiscuous mode. [inteli40e]_ - -- For promiscuous all: - :: - - $ ip link set eth2 promisc on - -- For promiscuous Multicast: - :: - - $ ip link set eth2 allmulti on - -.. note:: - - By default, the ethtool priv-flag vf-true-promisc-support is set to - *off*, meaning that promiscuous mode for the VF will be limited. To set the - promiscuous mode for the VF to true promiscuous and allow the VF to see - all ingress traffic, use the following command. - $ ethtool set-priv-flags p261p1 vf-true-promisc-support on - The vf-true-promisc-support priv-flag does not enable promiscuous mode; - rather, it designates which type of promiscuous mode (limited or true) - you will get when you enable promiscuous mode using the ip link commands - above. Note that this is a global setting that affects the entire device. - However,the vf-true-promisc-support priv-flag is only exposed to the first - PF of the device. The PF remains in limited promiscuous mode (unless it - is in MFP mode) regardless of the vf-true-promisc-support setting. - [inteli40e]_ - -Service described earlier *csit-initialize-vfs.service* is responsible for -assigning 802.1Q vlan tagging to each vitual function via physical function -from list of white-listed PCI addresses by following (simplified) code. - -:: - - pci_idx=0 - for pci_addr in ${pci_whitelist[@]}; do - pci_path="/sys/bus/pci/devices/${pci_addr}" - pf=$(basename "${pci_path}"/net/*) - for vf in $(seq "${sriov_totalvfs}"); do - # PCI address index in array (pairing siblings). - vlan_pf_idx=$(( pci_idx % (${#pci_whitelist[@]} / 2) )) - # 802.1Q base offset. - vlan_bs_off=1100 - # 802.1Q PF PCI address offset. - vlan_pf_off=$(( vlan_pf_idx * 100 + vlan_bs_off )) - # 802.1Q VF PCI address offset. - vlan_vf_off=$(( vlan_pf_off + vf - 1 )) - # VLAN string. - vlan_str="vlan ${vlan_vf_off}" - # MAC string. - mac5="$(printf '%x' ${pci_idx})" - mac6="$(printf '%x' $(( vf - 1 )))" - mac_str="mac ba:dc:0f:fe:${mac5}:${mac6}" - # Set 802.1Q VLAN id and MAC address - ip link set ${pf} vf $(( vf - 1 )) ${mac_str} ${vlan_str} - ip link set ${pf} vf $(( vf - 1 )) trust on - ip link set ${pf} vf $(( vf - 1 )) spoof off - done - pci_idx=$(( pci_idx + 1 )) - done - -Assignment starts at VLAN 1100 and incrementing by 1 for each VF and by 100 for -each white-listed PCI address up to the middle of the PCI list. Second half of -the lists is assumed to be directly (cable) paired siblings and assigned with -same 802.1Q VLANs as its siblings. - -Open tasks ----------- - -Security -~~~~~~~~ - -.. note:: - - Switch to non-privileged containers: As of now all three container - flavors are using privileged containers to make it working. Explore options - to switch containers to non-privileged with explicit rather implicit - privileges. - -.. note:: - - Switch to testuser account intead of root. - -Maintainability -~~~~~~~~~~~~~~~ - -.. note:: - - Docker image distribution: Create jenkins jobs with full pipiline of - CI/CD for CSIT Docker images. - -Stability -~~~~~~~~~ - -.. note:: - - Implement queueing mechanism: Currently there is no mechanics that - would place starving jobs in queue in case of no resources available. - -.. note:: - - Replace reservation script with Docker network plugin written in - GOLANG/SH/Python - platform independent. - -Links ------ - -.. [TWSLink] `TWS <https://wiki.fd.io/view/CSIT/TWS>`_ -.. [dockerhub] `Docker hub <https://hub.docker.com/>`_ -.. [fdiocsitgerrit] `FD.io/CSIT gerrit <https://gerrit.fd.io/r/CSIT>`_ -.. [fdioregistry] `FD.io registy <registry.fdiopoc.net>`_ -.. [JenkinsSlaveDcrFile] `jenkins-slave-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/ubuntu18/Dockerfile>`_ -.. [CsitShimDcrFile] `csit-shim-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/csit-shim/Dockerfile>`_ -.. [CsitSutDcrFile] `csit-sut-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/csit-sut/Dockerfile>`_ -.. [ansiblelink] `ansible <https://www.ansible.com/>`_ -.. [fdiocsitansible] `Fd.io/CSIT ansible <https://git.fd.io/csit/tree/resources/tools/testbed-setup/ansible>`_ -.. [inteli40e] `Intel i40e <https://downloadmirror.intel.com/26370/eng/readme.txt>`_ -.. [pciids] `pci ids <http://pci-ids.ucw.cz/v2.2/pci.ids>`_ |