diff options
Diffstat (limited to 'docs/report/introduction/methodology_containers.rst')
| -rw-r--r-- | docs/report/introduction/methodology_containers.rst | 421 |
1 files changed, 0 insertions, 421 deletions
diff --git a/docs/report/introduction/methodology_containers.rst b/docs/report/introduction/methodology_containers.rst deleted file mode 100644 index 14817df64f..0000000000 --- a/docs/report/introduction/methodology_containers.rst +++ /dev/null @@ -1,421 +0,0 @@ - -.. _container_orchestration_in_csit: - -Container Orchestration in CSIT -=============================== - -Overview --------- - -Linux Containers -~~~~~~~~~~~~~~~~ - -Linux Containers is an OS-level virtualization method for running -multiple isolated Linux systems (containers) on a compute host using a -single Linux kernel. Containers rely on Linux kernel cgroups -functionality for controlling usage of shared system resources (i.e. -CPU, memory, block I/O, network) and for namespace isolation. The latter -enables complete isolation of applications' view of operating -environment, including process trees, networking, user IDs and mounted -file systems. - -:abbr:`LXC (Linux Containers)` combine kernel's cgroups and support for isolated -namespaces to provide an isolated environment for applications. Docker -does use LXC as one of its execution drivers, enabling image management -and providing deployment services. More information in [lxc]_, [lxcnamespace]_ -and [stgraber]_. - -Linux containers can be of two kinds: privileged containers and -unprivileged containers. - -Unprivileged Containers -~~~~~~~~~~~~~~~~~~~~~~~ - -Running unprivileged containers is the safest way to run containers in a -production environment. From LXC 1.0 one can start a full system -container entirely as a user, allowing to map a range of UIDs on the -host into a namespace inside of which a user with UID 0 can exist again. -In other words an unprivileged container does mask the userid from the -host, making it impossible to gain a root access on the host even if a -user gets root in a container. With unprivileged containers, non-root -users can create containers and will appear in the container as the -root, but will appear as userid <non-zero> on the host. Unprivileged -containers are also better suited to supporting multi-tenancy operating -environments. More information in [lxcsecurity]_ and [stgraber]_. - -Privileged Containers -~~~~~~~~~~~~~~~~~~~~~ - -Privileged containers do not mask UIDs, and container UID 0 is mapped to -the host UID 0. Security and isolation is controlled by a good -configuration of cgroup access, extensive AppArmor profile preventing -the known attacks as well as container capabilities and SELinux. Here a -list of applicable security control mechanisms: - -- Capabilities - keep (whitelist) or drop (blacklist) Linux capabilities, - [capabilities]_. -- Control groups - cgroups, resource bean counting, resource quotas, access - restrictions, [cgroup1]_, [cgroup2]_. -- AppArmor - apparmor profiles aim to prevent any of the known ways of - escaping a container or cause harm to the host, [apparmor]_. -- SELinux - Security Enhanced Linux is a Linux kernel security module - that provides similar function to AppArmor, supporting access control - security policies including United States Department of Defense-style - mandatory access controls. Mandatory access controls allow an - administrator of a system to define how applications and users can - access different resources such as files, devices, networks and inter- - process communication, [selinux]_. -- Seccomp - secure computing mode, enables filtering of system calls, - [seccomp]_. - -More information in [lxcsecurity]_ and [lxcsecfeatures]_. - -**Linux Containers in CSIT** - -CSIT is using Privileged Containers as the ``sysfs`` is mounted with RW -access. Sysfs is required to be mounted as RW due to VPP accessing -:command:`/sys/bus/pci/drivers/uio_pci_generic/unbind`. This is not the case of -unprivileged containers where ``sysfs`` is mounted as read-only. - - -Orchestrating Container Lifecycle Events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Following Linux container lifecycle events need to be addressed by an -orchestration system: - -1. Acquire - acquiring/downloading existing container images via - :command:`docker pull` or :command:`lxc-create -t download`. - -2. Build - building a container image from scratch or another - container image via :command:`docker build <dockerfile/composefile>` or - customizing LXC templates in - `GitHub <https://github.com/lxc/lxc/tree/master/templates>`_. - -3. (Re-)Create - creating a running instance of a container application - from anew, or re-creating one that failed. A.k.a. (re-)deploy via - :command:`docker run` or :command:`lxc-start` - -4. Execute - execute system operations within the container by attaching to - running container. THis is done by :command:`lxc-attach` or - :command:`docker exec` - -5. Distribute - distributing pre-built container images to the compute - nodes. Currently not implemented in CSIT. - - -Container Orchestration Systems Used in CSIT --------------------------------------------- - -Current CSIT testing framework integrates following Linux container -orchestration mechanisms: - -- LXC/Docker for complete VPP container lifecycle control. - -LXC -~~~ - -LXC is the well-known and heavily tested low-level Linux container -runtime [lxcsource]_, that provides a userspace interface for the Linux kernel -containment features. With a powerful API and simple tools, LXC enables -Linux users to easily create and manage system or application -containers. LXC uses following kernel features to contain processes: - -- Kernel namespaces: ipc, uts, mount, pid, network and user. -- AppArmor and SELinux security profiles. -- Seccomp policies. -- Chroot. -- Cgroups. - -CSIT uses LXC runtime and LXC usertools to test VPP data plane performance in -a range of virtual networking topologies. - -**Known Issues** - -- Current CSIT restriction: only single instance of lxc runtime due to - the cgroup policies used in CSIT. There is plan to add the capability into - code to create cgroups per container instance to address this issue. This sort - of functionality is better supported in LXC 2.1 but can be done is current - version as well. - -- CSIT code is currently using cgroup to control the range of CPU cores the - LXC container runs on. VPP thread pinning is defined vpp startup.conf. - -Docker -~~~~~~ - -Docker builds on top of Linux kernel containment features, and -offers a high-level tool for wrapping the processes, maintaining and -executing them in containers [docker]_. Currently it is using *runc*, -a CLI tool for spawning and running containers according to the -`OCI specification <https://www.opencontainers.org/>`_. - -A Docker container image is a lightweight, stand-alone, executable -package that includes everything needed to run the container: -code, runtime, system tools, system libraries, settings. - -CSIT uses Docker to manage the maintenance and execution of -containerized applications used in CSIT performance tests. - -- Data plane thread pinning to CPU cores - Docker CLI and/or Docker - configuration file controls the range of CPU cores the Docker image - must run on. VPP thread pinning defined vpp startup.conf. - -Implementation --------------- - -CSIT container orchestration is implemented in CSIT Level-1 keyword -Python libraries following the Builder design pattern. Builder design -pattern separates the construction of a complex object from its -representation, so that the same construction process can create -different representations e.g. LXC, Docker, other. - -CSIT Robot Framework keywords are then responsible for higher level -lifecycle control of of the named container groups. One can have -multiple named groups, with 1..N containers in a group performing -different role/functionality e.g. NFs, Switch, Kafka bus, ETCD -datastore, etc. ContainerManager class acts as a Director and uses -ContainerEngine class that encapsulate container control. - -Current CSIT implementation is illustrated using UML Class diagram: - -1. Acquire -2. Build -3. (Re-)Create -4. Execute - -:: - - +-----------------------------------------------------------------------+ - | RF Keywords (high level lifecycle control) | - +-----------------------------------------------------------------------+ - | Construct VNF containers on all DUTs | - | Acquire all '${group}' containers | - | Create all '${group}' containers | - | Install all '${group}' containers | - | Configure all '${group}' containers | - | Stop all '${group}' containers | - | Destroy all '${group}' containers | - +-----------------+-----------------------------------------------------+ - | 1 - | - | 1..N - +-----------------v-----------------+ +--------------------------+ - | ContainerManager | | ContainerEngine | - +-----------------------------------+ +--------------------------+ - | __init()__ | | __init(node)__ | - | construct_container() | | acquire(force) | - | construct_containers() | | create() | - | acquire_all_containers() | | stop() | - | create_all_containers() | 1 1 | destroy() | - | execute_on_container() <>-------| info() | - | execute_on_all_containers() | | execute(command) | - | install_vpp_in_all_containers() | | system_info() | - | configure_vpp_in_all_containers() | | install_supervisor() | - | stop_all_containers() | | install_vpp() | - | destroy_all_containers() | | restart_vpp() | - +-----------------------------------+ | create_vpp_exec_config() | - | create_vpp_startup_config| - | is_container_running() | - | is_container_present() | - | _configure_cgroup() | - +-------------^------------+ - | - | - | - +----------+---------+ - | | - +------+------+ +------+------+ - | LXC | | Docker | - +-------------+ +-------------+ - | (inherited) | | (inherited) | - +------+------+ +------+------+ - | | - +----------+---------+ - | - | constructs - | - +---------v---------+ - | Container | - +-------------------+ - | __getattr__(a) | - | __setattr__(a, v) | - +-------------------+ - -Sequentional diagram that illustrates the creation of a single container. - -:: - - Legend: - e = engine [Docker|LXC] - .. = kwargs (variable number of keyword argument) - - +-------+ +------------------+ +-----------------+ - | RF KW | | ContainerManager | | ContainerEngine | - +---+---+ +--------+---------+ +--------+--------+ - | | | - | 1: new ContainerManager(e) | | - +-+---------------------------->+-+ | - |-| |-| 2: new ContainerEngine | - |-| |-+----------------------->+-+ - |-| |-| |-| - |-| +-+ +-+ - |-| | | - |-| 3: construct_container(..) | | - |-+---------------------------->+-+ | - |-| |-| 4: init() | - |-| |-+----------------------->+-+ - |-| |-| |-| 5: new +-------------+ - |-| |-| |-+-------->| Container A | - |-| |-| |-| +-------------+ - |-| |-|<-----------------------+-| - |-| +-+ +-+ - |-| | | - |-| 6: acquire_all_containers() | | - |-+---------------------------->+-+ | - |-| |-| 7: acquire() | - |-| |-+----------------------->+-+ - |-| |-| |-| - |-| |-| |-+--+ - |-| |-| |-| | 8: is_container_present() - |-| |-| True/False |-|<-+ - |-| |-| |-| - |-| |-| |-| - +---------------------------------------------------------------------------------------------+ - | |-| ALT [isRunning & force] |-| |-|--+ | - | |-| |-| |-| | 8a: destroy() | - | |-| |-| |-<--+ | - +---------------------------------------------------------------------------------------------+ - |-| |-| |-| - |-| +-+ +-+ - |-| | | - |-| 9: create_all_containers() | | - |-+---------------------------->+-+ | - |-| |-| 10: create() | - |-| |-+----------------------->+-+ - |-| |-| |-+--+ - |-| |-| |-| | 11: wait('RUNNING') - |-| |-| |-<--+ - |-| +-+ +-+ - |-| | | - +---------------------------------------------------------------------------------------------+ - | |-| ALT | | | - | |-| (install_vpp, configure_vpp) | | | - | |-| | | | - +---------------------------------------------------------------------------------------------+ - |-| | | - |-| 12: destroy_all_containers() | | - |-+---------------------------->+-+ | - |-| |-| 13: destroy() | - |-| |-+----------------------->+-+ - |-| |-| |-| - |-| +-+ +-+ - |-| | | - +++ | | - | | | - + + + - -Container Data Structure -~~~~~~~~~~~~~~~~~~~~~~~~ - -Container is represented in Python L1 library as a separate Class with instance -variables and no methods except overriden ``__getattr__`` and ``__setattr__``. -Instance variables are assigned to container dynamically during the -``construct_container(**kwargs)`` call and are passed down from the RF keyword. - -There is no parameters check functionality. Passing the correct arguments -is a responsibility of the caller. - -Examples -~~~~~~~~ - -This section contains a high-level example of multiple initialization steps -via ContainerManager; taken from an actual CSIT code, -but with non-code lines (comments, Documentation) removed for brevity. - -: - -.. code-block:: robotframework - - | Start containers for test - | | [Arguments] | ${dut}=${None} | ${nf_chains}=${1} | ${nf_nodes}=${1} - | | ... | ${auto_scale}=${True} | ${pinning}=${True} - | | - | | Set Test Variable | @{container_groups} | @{EMPTY} - | | Set Test Variable | ${container_group} | CNF - | | Set Test Variable | ${nf_nodes} - | | Import Library | resources.libraries.python.ContainerUtils.ContainerManager - | | ... | engine=${container_engine} | WITH NAME | ${container_group} - | | Construct chains of containers - | | ... | dut=${dut} | nf_chains=${nf_chains} | nf_nodes=${nf_nodes} - | | ... | auto_scale=${auto_scale} | pinning=${pinning} - | | Acquire all '${container_group}' containers - | | Create all '${container_group}' containers - | | Configure VPP in all '${container_group}' containers - | | Start VPP in all '${container_group}' containers - | | Append To List | ${container_groups} | ${container_group} - | | Save VPP PIDs - -Kubernetes -~~~~~~~~~~ - -For the future use, Kubernetes [k8sdoc]_ is implemented as separate library -``KubernetesUtils.py``, with a class with the same name. This utility provides -an API for L2 Robot Keywords to control ``kubectl`` installed on each of DUTs. -One time initialization script, ``resources/libraries/bash/k8s_setup.sh`` -does reset/init kubectl, and initializes the ``csit`` namespace. CSIT -namespace is required to not to interfere with existing setups and it -further simplifies apply/get/delete Pod/ConfigMap operations on SUTs. - -Kubernetes utility is based on YAML templates to avoid crafting the huge -YAML configuration files, what would lower the readability of code and -requires complicated algorithms. - -Two types of YAML templates are defined: - -- Static - do not change between deployments, that is infrastructure - containers like Kafka, Calico, ETCD. - -- Dynamic - per test suite/case topology YAML files. - -Making own python wrapper library of ``kubectl`` instead of using the -official Python package allows to control and deploy environment over -the SSH library without the need of using isolated driver running on -each of DUTs. - -Tested Topologies -~~~~~~~~~~~~~~~~~ - -Listed CSIT container networking test topologies are defined with DUT -containerized VPP switch forwarding packets between NF containers. Each -NF container runs their own instance of VPP in L2XC configuration. - -Following container networking topologies are tested in |csit-release|: - -- LXC topologies: - - - eth-l2xcbase-eth-2memif-1lxc. - - eth-l2bdbasemaclrn-eth-2memif-1lxc. - -- Docker topologies: - - - eth-l2xcbase-eth-2memif-1docker. - - eth-l2xcbase-eth-1memif-1docker - -References -~~~~~~~~~~ - -.. [lxc] `Linux Containers <https://linuxcontainers.org/>`_ -.. [lxcnamespace] `Resource management: Linux kernel Namespaces and cgroups <https://www.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lxc-namespace.pdf>`_. -.. [stgraber] `LXC 1.0: Blog post series <https://stgraber.org/2013/12/20/lxc-1-0-blog-post-series/>`_. -.. [lxcsecurity] `Linux Containers Security <https://linuxcontainers.org/lxc/security/>`_. -.. [capabilities] `Linux manual - capabilities - overview of Linux capabilities <http://man7.org/linux/man-pages/man7/capabilities.7.html>`_. -.. [cgroup1] `Linux kernel documentation: cgroups <https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt>`_. -.. [cgroup2] `Linux kernel documentation: Control Group v2 <https://www.kernel.org/doc/Documentation/cgroup-v2.txt>`_. -.. [selinux] `SELinux Project Wiki <http://selinuxproject.org/page/Main_Page>`_. -.. [lxcsecfeatures] `LXC 1.0: Security features <https://stgraber.org/2014/01/01/lxc-1-0-security-features/>`_. -.. [lxcsource] `Linux Containers source <https://github.com/lxc/lxc>`_. -.. [apparmor] `Ubuntu AppArmor <https://wiki.ubuntu.com/AppArmor>`_. -.. [seccomp] `SECure COMPuting with filters <https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt>`_. -.. [docker] `Docker <https://www.docker.com/what-docker>`_. -.. [k8sdoc] `Kubernetes documentation <https://kubernetes.io/docs/home/>`_. |