diff options
Diffstat (limited to 'docs/report/vpp_device_tests/test_environment.rst')
| -rw-r--r-- | docs/report/vpp_device_tests/test_environment.rst | 578 |
1 files changed, 0 insertions, 578 deletions
diff --git a/docs/report/vpp_device_tests/test_environment.rst b/docs/report/vpp_device_tests/test_environment.rst deleted file mode 100644 index 33915da83b..0000000000 --- a/docs/report/vpp_device_tests/test_environment.rst +++ /dev/null @@ -1,578 +0,0 @@ -Integration Tests -================= - -Abstract --------- - -FD.io VPP software data plane technology has become very popular across -a wide range of VPP eco-system use cases, putting higher pressure on -continuous verification of VPP software quality. - -This document describes a proposal for design and implementation of extended -continuous VPP testing by extending existing test environments. -Furthermore it describes and summarizes implementation details of Integration -and System tests platform *1-Node VPP_Device*. It aims to provide a complete -end-to-end view of *1-Node VPP_Device* environment in order to improve -extendability and maintenance, under the guideline of VPP core team. - -The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", -"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be -interpreted as described in :rfc:`8174`. - -Overview --------- - -.. only:: latex - - .. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_tmp/src/vpp_device_tests/}} - \includegraphics[width=0.90\textwidth]{vpp_device} - \label{fig:vpp_device} - \end{figure} - -.. only:: html - - .. figure:: vpp_device.svg - :alt: vpp_device - :align: center - -Physical Testbeds ------------------ - -All :abbr:`FD.io (Fast Data Input/Ouput)` :abbr:`CSIT (Continuous System -Integration and Testing)` vpp-device tests are executed on physical testbeds -built with bare-metal servers hosted by :abbr:`LF (Linux Foundation)` FD.io -project. Two 1-node testbed topologies are used: - -- **2-Container Topology**: Consisting of one Docker container acting as SUT - (System Under Test) and one Docker container as TG (Traffic Generator), both - connected in ring topology via physical NIC cross-connecting. - -Current FD.io production testbeds are built with servers based on one -processor generation of Intel Xeons: Skylake (Platinum 8180). Testbeds built -with servers based on Arm processors are in the process of being added to FD.io -production. - -Following section describe existing production 1n-skx testbed. - -1-Node Xeon Skylake (1n-skx) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1n-skx testbed is based on single SuperMicro SYS-7049GP-TRT server equipped -with two Intel Xeon Skylake Platinum 8180 2.5 GHz 28 core processors. Physical -testbed topology is depicted in a figure below. - -.. only:: latex - - .. raw:: latex - - \begin{figure}[H] - \centering - \graphicspath{{../_tmp/src/vpp_device_tests/}} - \includegraphics[width=0.90\textwidth]{vf-2n-nic2nic} - \label{fig:vf-2n-nic2nic} - \end{figure} - -.. only:: html - - .. figure:: vf-2n-nic2nic.svg - :alt: vf-2n-nic2nic - :align: center - -Server is populated with the following NIC models: - -#. NIC-1: x710-da4 4p10GE Intel. -#. NIC-2: E810-2CQDA2 2p100GbE Intel. - -All Intel Xeon Skylake servers run with Intel Hyper-Threading enabled, -doubling the number of logical cores exposed to Linux, with 56 logical -cores and 28 physical cores per processor socket. - -NIC interfaces are shared using Linux vfio_pci and VPP VF drivers: - -- DPDK VF driver, -- Fortville AVF driver. - -Provided Intel x710-da4 4p10GE NICs support 32 VFs per interface, 128 per NIC. - -Total of two 1n-skx testbeds are in operation in FD.io labs. - -1-Node Virtualbox (1n-vbox) -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1n-skx testbed can run in single VirtualBox VM machine. This solution replaces -the previously used Vagrant environment based on 3 VMs. - -VirtualBox VM MAY be created by Vagrant and MUST have additional 4 virtio NICs -each pair attached to separate private networks to simulate back-to-back -connections. It SHOULD be 82545EM device model (otherwise can be changed in -boostrap scripts). Example of Vagrant configuration: - -:: - - Vagrant.configure(2) do |c| - c.vm.network "private_network", type: "dhcp", auto_config: false, - virtualbox__intnet: "port1", nic_type: "82545EM" - c.vm.network "private_network", type: "dhcp", auto_config: false, - virtualbox__intnet: "port2", nic_type: "82545EM" - - c.vm.provider :virtualbox do |v| - v.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"] - v.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"] - v.customize ["modifyvm", :id, "--nicpromisc4", "allow-all"] - v.customize ["modifyvm", :id, "--nicpromisc5", "allow-all"] - -Vagrant VM is populated with the following NIC models: - -#. NIC-1: 82545EM Intel. -#. NIC-2: 82545EM Intel. -#. NIC-3: 82545EM Intel. -#. NIC-4: 82545EM Intel. - -Containers ----------- - -It was agreed on :abbr:`TWS (Technical Work Stream)` call to continue with -Ubuntu 18.04 LTS as a baseline system with OPTIONAL extend to Centos 7 and -SuSE per demand [#TWSLink]_. - -All :abbr:`DCR (Docker container)` images are REQUIRED to be hosted on Docker -registry available from LF network, publicly available and trackable. For -backup, tracking and contributing purposes all Dockerfiles (including files -needed for building container) MUST be available and stored in -[#fdiocsitgerrit]_ repository under appropriate folders. This allows the -peer review process to be done for every change of infrastructure related to -scope of this document. -Currently only **csit-shim-dcr** and **csit-sut-dcr** containers will be stored -and maintained under CSIT repository by CSIT contributors. - -At the time of designing solution described in this document the -interconnection between [#dockerhub]_ and [#fdiocsitgerrit]_ for -automated build purposes and image hosting cannot be established with the trust -and respectful to security of FD.io project. Unless adressed, :abbr:`DCR -(Docker container)` images will be placed in custom registry service -[#fdioregistry]_. -Automated Jenkins jobs will be created in align of long term solution for -container lifecycle and ability to build new version of docker images. - -In parallel, the effort is started to find the outsourced Docker registry -service. - -Versioning -~~~~~~~~~~ - -As of initial version of vpp-device, we do have only single latest version of -Docker image hosted on [#dockerhub]_. This will be addressed as further -improvement with proper semantic versioning. - -jenkins-slave-dcr -~~~~~~~~~~~~~~~~~ - -This :abbr:`DCR (Docker container)` acts as the Jenkins slave (known also as -jenkins minion). It can connect over SSH protocol to TCP port 6022 of -**csit-shim-dcr** and executes non-interactive reservation script. Nomad is -responsible for scheduling this container execution onto specific -**1-Node VPP_Device** testbed. It executes -:abbr:`CSIT (Continuous System Integration and Testing)` environment including -:abbr:`CSIT (Continuous System Integration and Testing)` framework. - -All software dependencies including VPP/DPDK that are not present in -**csit-sut-dcr** container image and/or needs to be compiled prior running on -**csit-sut-dcr** SHOULD be compiled in this container. - -- *Container Image Location*: Docker image at snergster/vpp-ubuntu18. - -- *Container Definition*: Docker file specified at [#JenkinsSlaveDcrFile]_. - -- *Initializing*: Container is initialized from within *Consul by HashiCorp* - and *Nomad by HashiCorp*. - -csit-shim-dcr -~~~~~~~~~~~~~ - -This :abbr:`DCR (Docker container)` acts as an intermediate layer running -script responsible for orchestrating topologies under test and reservation. -Responsible for managing VF resources and allocation to -:abbr:`DUT (Device Under Test)`, :abbr:`TG (Traffic Generator)` containers. -This MUST to be done on **csit-shim-dcr**. -This image also acts as the generic reservation mechanics arbiter to make sure -that only Y number of simulations are spawned on any given HW node. - -- *Container Image Location*: Docker image at snergster/csit-shim. - -- *Container Definition*: Docker file specified at [#CsitShimDcrFile]_. - -- *Initializing*: Container is initialized from within *Consul by HashiCorp* - and *Nomad by HashiCorp*. Required docker parameters, to be able to run - nested containers with VF reservation system are: privileged, net=host, - pid=host. - -- *Connectivity*: Over SSH only, using <host>:6022 format. Currently using - *root* user account as primary. From the jenkins slave it will be able to - connect via env variable, since the jenkins slave doesn't actually know what - host its running on. - - :: - - ssh -p 6022 root@10.30.51.node - -csit-sut-dcr -~~~~~~~~~~~~ - -This :abbr:`DCR (Docker container)` acts as an :abbr:`SUT (System Under Test)`. -Any :abbr:`DUT (Device Under Test)` or :abbr:`TG (Traffic Generator)` -application is installed there. It is RECOMMENDED to install DUT and -all DUT dependencies via commands ``rpm -ihv`` on RedHat based OS or -``dpkg -i`` on Debian based OS. - -Container is designed to be a very lightweight Docker image that only installs -packages and execute binaries (previously built or downloaded on -**jenkins-slave-dcr**) and contains libraries necessary to run CSIT framework -including those required by DUT/TG. - -- *Container Image Location*: Docker image at snergster/csit-sut. - -- *Container Definition*: Docker file specified at [#CsitSutDcrFile]_. - -- *Initializing*: - :: - - docker run - # Run the container in the background and print the new container ID. - --detach=true - # Give extended privileges to this container. A "privileged" container is - # given access to all devices and able to run nested containers. - --privileged - # Publish all exposed ports to random ports on the host interfaces. - --publish-all - # Automatically remove the container when it exits. - --rm - # Size of /dev/shm. - dcr_stc_params+="--shm-size 512M " - # Override access to PCI bus by attaching a filesystem mount to the - # container. - dcr_stc_params+="--mount type=tmpfs,destination=/sys/bus/pci/devices " - # Mount vfio to be able to bind to see bound interfaces. We cannot use - # --device=/dev/vfio as this does not see newly bound interfaces. - dcr_stc_params+="--volume /dev/vfio:/dev/vfio " - # Mount docker.sock to be able to use docker deamon of the host. - dcr_stc_params+="--volume /var/run/docker.sock:/var/run/docker.sock " - # Mount /opt/boot/ where VM kernel and initrd are located. - dcr_stc_params+="--volume /opt/boot/:/opt/boot/ " - # Mount host hugepages for VMs. - dcr_stc_params+="--volume /dev/hugepages/:/dev/hugepages/ " - - Container name is catenated from **csit-** prefix and uuid generated uniquely - for each container instance. - -- *Connectivity*: Over SSH only, using <host>[:<port>] format. Currently using - *root* user account as primary. - :: - - ssh -p <port> root@10.30.51.<node> - -Container required to run as ``--privileged`` due to ability to create nested -containers and have full read/write access to sysfs (for bind/unbind). Docker -automatically pick free network port (``--publish-all``) for ability to connect -over ssh. To be able to limit access to PCI bus, container is creating tmpfs -mount type in PCI bus tree. CSIT reservation script is dynamically linking only -PCI devices (NIC cards) that are reserved for particular container. This -way it is not colliding with other containers. To make vfio work, access to -``/dev/vfio`` must be granted. - -.. todo: Change default user to testuser with non-privileged and install sudo. - -Environment initialization --------------------------- - -All 1-node servers are to be managed and provisioned via the -[#ansiblelink]_ set of playbooks with *vpp-device* role. Full playbooks -can be found under [#fdiocsitansible]_ directory. This way we are able to -track all configuration changes of physical servers in gerrit (in structured -yaml format) as well as we are able to extend *vpp-device* to additional -servers with less effort or re-stage servers in case of failure. - -SR-IOV VF initialization is done via ``systemd`` service during host system boot -up. Service with name *csit-initialize-vfs.service* is created under systemd -system context (``/etc/systemd/system/``). By default service is calling -``/usr/local/bin/csit-initialize-vfs.sh`` with single parameter: - -- **start**: Creates maximum number of :abbr:`virtual functions (VFs)` (detected - from ``sriov_totalvfs``) for each whitelisted PCI device. -- **stop**: Removes all :abbr:`VFs (Virtual Functions)` for all whitelisted PCI - device. - -Service is considered active even when all of its processes exited successfully. -Stopping service will automatically remove :abbr:`VFs (Virtual Functions)`. - -:: - - [Unit] - Description=CSIT Initialize SR-IOV VFs - After=network.target - - [Service] - Type=one-shot - RemainAfterExit=True - ExecStart=/usr/local/bin/csit-initialize-vfs.sh start - ExecStop=/usr/local/bin/csit-initialize-vfs.sh stop - - [Install] - WantedBy=default.target - -Script is driven by two array variables ``pci_blacklist``/``pci_whitelist``. -They MUST store all PCI addresses in **<domain>:<bus>:<device>.<func>** format, -where: - -- **pci_blacklist**: PCI addresses to be skipped from - :abbr:`VFs (Virtual Functions)` initialization (useful for e.g. excluding - management network interfaces). -- **pci_whitelist**: PCI addresses to be included for - :abbr:`VFs (Virtual Functions)` initialization. - -VF reservation --------------- - -During topology initialization phase of script, mutex is used to avoid multiple -instances of script to interact with each other during resources allocation. -Mutal exclusion ensure that no two distinct instances of script will get same -resource list. - -Reservation function reads the list of all available virtual function network -devices in system: - -:: - - # Find the first ${device_count} number of available TG Linux network - # VF device names. Only allowed VF PCI IDs are filtered. - for netdev in ${tg_netdev[@]} - do - for netdev_path in $(grep -l "${pci_id}" \ - /sys/class/net/${netdev}*/device/device \ - 2> /dev/null) - do - if [[ ${#TG_NETDEVS[@]} -lt ${device_count} ]]; then - tg_netdev_name=$(dirname ${netdev_path}) - tg_netdev_name=$(dirname ${tg_netdev_name}) - TG_NETDEVS+=($(basename ${tg_netdev_name})) - else - break - fi - done - if [[ ${#TG_NETDEVS[@]} -eq ${device_count} ]]; then - break - fi - done - -Where ``${pci_id}`` is ID of white-listed VF PCI ID. For more information please -see [#pciids]_. This act as security constraint to prevent taking other -unwanted interfaces. -The output list of all VF network devices is split into two lists for TG and -SUT side of connection. First two items from each TG or SUT network devices -list are taken to expose directly to namespace of container. This can be done -via commands: - -:: - - $ ip link set ${netdev} netns ${DCR_CPIDS[tg]} - $ ip link set ${netdev} netns ${DCR_CPIDS[dut1]} - -In this stage also symbolic links to PCI devices under sysfs bus directory tree -are created in running containers. Once VF devices are assigned to container -namespace and PCI devices are linked to running containers and mutex is exited. -Selected VF network device automatically disappear from parent container -namespace, so another instance of script will not find device under that -namespace. - -Once Docker container exits, network device is returned back into parent -namespace and can be reused. - -Network traffic isolation - Intel i40evf ----------------------------------------- - -In a virtualized environment, on Intel(R) Server Adapters that support SR-IOV, -the virtual function (VF) may be subject to malicious behavior. Software- -generated layer two frames, like IEEE 802.3x (link flow control), IEEE 802.1Qbb -(priority based flow-control), and others of this type, are not expected and -can throttle traffic between the host and the virtual switch, reducing -performance. To resolve this issue, configure all SR-IOV enabled ports for -VLAN tagging. This configuration allows unexpected, and potentially malicious, -frames to be dropped. [#inteli40e]_ - -To configure VLAN tagging for the ports on an SR-IOV enabled adapter, -use the following command. The VLAN configuration SHOULD be done -before the VF driver is loaded or the VM is booted. [#inteli40e]_ - -:: - - $ ip link set dev <PF netdev id> vf <id> vlan <vlan id> - -For example, the following instructions will configure PF eth0 and -the first VF on VLAN 10. - -:: - - $ ip link set dev eth0 vf 0 vlan 10 - -VLAN Tag Packet Steering allows to send all packets with a specific VLAN tag to -a particular SR-IOV virtual function (VF). Further, this feature allows to -designate a particular VF as trusted, and allows that trusted VF to request -selective promiscuous mode on the Physical Function (PF). [#inteli40e]_ - -To set a VF as trusted or untrusted, enter the following command in the -Hypervisor: - -:: - - $ ip link set dev eth0 vf 1 trust [on|off] - -Once the VF is designated as trusted, use the following commands in the VM -to set the VF to promiscuous mode. [#inteli40e]_ - -- For promiscuous all: - :: - - $ ip link set eth2 promisc on - -- For promiscuous Multicast: - :: - - $ ip link set eth2 allmulti on - -.. note:: - - By default, the ethtool priv-flag vf-true-promisc-support is set to - *off*, meaning that promiscuous mode for the VF will be limited. To set the - promiscuous mode for the VF to true promiscuous and allow the VF to see - all ingress traffic, use the following command. - $ ethtool set-priv-flags p261p1 vf-true-promisc-support on - The vf-true-promisc-support priv-flag does not enable promiscuous mode; - rather, it designates which type of promiscuous mode (limited or true) - you will get when you enable promiscuous mode using the ip link commands - above. Note that this is a global setting that affects the entire device. - However,the vf-true-promisc-support priv-flag is only exposed to the first - PF of the device. The PF remains in limited promiscuous mode (unless it - is in MFP mode) regardless of the vf-true-promisc-support setting. - [#inteli40e]_ - -Service described earlier *csit-initialize-vfs.service* is responsible for -assigning 802.1Q vlan tagging to each virtual function via physical function -from list of white-listed PCI addresses by following (simplified) code. - -:: - - SCRIPT_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))" - source "${SCRIPT_DIR}/csit-initialize-vfs-data.sh" - - # Initilize whitelisted NICs with maximum number of VFs. - pci_idx=0 - for pci_addr in ${PCI_WHITELIST[@]}; do - if ! [[ ${PCI_BLACKLIST[*]} =~ "${pci_addr}" ]]; then - pci_path="/sys/bus/pci/devices/${pci_addr}" - # SR-IOV initialization - case "${1:-start}" in - "start" ) - sriov_totalvfs=$(< "${pci_path}"/sriov_totalvfs) - ;; - "stop" ) - sriov_totalvfs=0 - ;; - esac - echo ${sriov_totalvfs} > "${pci_path}"/sriov_numvfs - # SR-IOV 802.1Q isolation - case "${1:-start}" in - "start" ) - pf=$(basename "${pci_path}"/net/*) - for vf in $(seq "${sriov_totalvfs}"); do - # PCI address index in array (pairing siblings). - if [[ -n ${PF_INDICES[@]} ]] - then - vlan_pf_idx=${PF_INDICES[$pci_addr]} - else - vlan_pf_idx=$((pci_idx % (${#PCI_WHITELIST[@]}/2))) - fi - # 802.1Q base offset. - vlan_bs_off=1100 - # 802.1Q PF PCI address offset. - vlan_pf_off=$(( vlan_pf_idx * 100 + vlan_bs_off )) - # 802.1Q VF PCI address offset. - vlan_vf_off=$(( vlan_pf_off + vf - 1 )) - # VLAN string. - vlan_str="vlan ${vlan_vf_off}" - # MAC string. - mac5="$(printf '%x' ${pci_idx})" - mac6="$(printf '%x' $(( vf - 1 )))" - mac_str="mac ba:dc:0f:fe:${mac5}:${mac6}" - # Set 802.1Q VLAN id and MAC address - ip link set ${pf} vf $(( vf - 1)) ${mac_str} ${vlan_str} - ip link set ${pf} vf $(( vf - 1)) trust on - ip link set ${pf} vf $(( vf - 1)) spoof off - done - pci_idx=$(( pci_idx + 1 )) - ;; - esac - rmmod i40evf - modprobe i40evf - fi - done - -Assignment starts at VLAN 1100 and incrementing by 1 for each VF and by 100 for -each white-listed PCI address up to the middle of the PCI list. Second half of -the lists is assumed to be directly (cable) paired siblings and assigned with -same 802.1Q VLANs as its siblings. - -Open tasks ----------- - -Security -~~~~~~~~ - -.. note:: - - Switch to non-privileged containers: As of now all three container - flavors are using privileged containers to make it working. Explore options - to switch containers to non-privileged with explicit rather implicit - privileges. - -.. note:: - - Switch to testuser account instead of root. - -Maintainability -~~~~~~~~~~~~~~~ - -.. note:: - - Docker image distribution: Create jenkins jobs with full pipeline of - CI/CD for CSIT Docker images. - -Stability -~~~~~~~~~ - -.. note:: - - Implement queueing mechanism: Currently there is no mechanics that - would place starving jobs in queue in case of no resources available. - -.. note:: - - Replace reservation script with Docker network plugin written in - GOLANG/SH/Python - platform independent. - -Links ------ - -.. [#TWSLink] `TWS <https://wiki.fd.io/view/CSIT/TWS>`_ -.. [#dockerhub] `Docker hub <https://hub.docker.com/>`_ -.. [#fdiocsitgerrit] `FD.io/CSIT gerrit <https://gerrit.fd.io/r/CSIT>`_ -.. [#fdioregistry] `FD.io registy <registry.fdiopoc.net>`_ -.. [#JenkinsSlaveDcrFile] `jenkins-slave-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/ubuntu18/Dockerfile>`_ -.. [#CsitShimDcrFile] `csit-shim-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/csit-shim/Dockerfile>`_ -.. [#CsitSutDcrFile] `csit-sut-dcr-file <https://github.com/snergfdio/multivppcache/blob/master/csit-sut/Dockerfile>`_ -.. [#ansiblelink] `ansible <https://www.ansible.com/>`_ -.. [#fdiocsitansible] `Fd.io/CSIT ansible <https://git.fd.io/csit/tree/fdio.infra.ansible>`_ -.. [#inteli40e] `Intel i40e <https://downloadmirror.intel.com/26370/eng/readme.txt>`_ -.. [#pciids] `pci ids <http://pci-ids.ucw.cz/v2.2/pci.ids>`_ |