summaryrefslogtreecommitdiffstats
path: root/src/plugins/ioam/udp-ping/udp_ping.api
blob: 72e578c56748057c32741c421099caa9d6cdb931 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
/* Hey Emacs use -*- mode: C -*- */
/*
 * Copyright (c) 2017 Cisco and/or its affiliates.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at:
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

option version = "2.0.0";

/** \brief UDP-Probe Add/Delete request 
    @param src_ip_address - Source ipv4/v6 address for the udp-ping flow
    @param dst_ip_address - Destination ipv4/v6 address for the udp-ping flow
    @param start_src_port - Starting source port of port range for udp-ping
    @param end_src_port - End source port of port range for udp-ping
    @param start_dst_port - Starting destination port of port range for udp-ping
    @param end_dst_port - End destination port of port range for udp-ping
    @param interval - Time interval in seconds at which udp-probe need to be sent
    @param is_ipv4 - To determine whether IPv4 or IPv6 address is used
    @param dis - TRUE is delete, FALSE if Add
*/
autoreply define udp_ping_add_del {
  u32 client_index;
  u32 context;
  u8 src_ip_address[16];
  u8 dst_ip_address[16];
  u16 start_src_port;
  u16 end_src_port;
  u16 start_dst_port;
  u16 end_dst_port;
  u16 interval;
  u8 is_ipv4;
  u8 dis;
  u8 fault_det;
  u8 reserve[3];
};

/** \brief Udp-probe export add/del request
    @param context - sender context, to match reply w/ request
    @param retval - return value for request
    @param enable - If TRUE then enable export else disable
*/
autoreply define udp_ping_export {
   u32 client_index;
   u32 context;
   u32 enable;
};
href='#n229'>229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421
.. _container_orchestration_in_csit:

Container Orchestration in CSIT
===============================

Overview
--------

Linux Containers
~~~~~~~~~~~~~~~~

Linux Containers is an OS-level virtualization method for running
multiple isolated Linux systems (containers) on a compute host using a
single Linux kernel. Containers rely on Linux kernel cgroups
functionality for controlling usage of shared system resources (i.e.
CPU, memory, block I/O, network) and for namespace isolation. The latter
enables complete isolation of applications' view of operating
environment, including process trees, networking, user IDs and mounted
file systems.

:abbr:`LXC (Linux Containers)` combine kernel's cgroups and support for isolated
namespaces to provide an isolated environment for applications. Docker
does use LXC as one of its execution drivers, enabling image management
and providing deployment services. More information in [lxc]_, [lxcnamespace]_
and [stgraber]_.

Linux containers can be of two kinds: privileged containers and
unprivileged containers.

Unprivileged Containers
~~~~~~~~~~~~~~~~~~~~~~~

Running unprivileged containers is the safest way to run containers in a
production environment. From LXC 1.0 one can start a full system
container entirely as a user, allowing to map a range of UIDs on the
host into a namespace inside of which a user with UID 0 can exist again.
In other words an unprivileged container does mask the userid from the
host, making it impossible to gain a root access on the host even if a
user gets root in a container. With unprivileged containers, non-root
users can create containers and will appear in the container as the
root, but will appear as userid <non-zero> on the host. Unprivileged
containers are also better suited to supporting multi-tenancy operating
environments. More information in [lxcsecurity]_ and [stgraber]_.

Privileged Containers
~~~~~~~~~~~~~~~~~~~~~

Privileged containers do not mask UIDs, and container UID 0 is mapped to
the host UID 0. Security and isolation is controlled by a good
configuration of cgroup access, extensive AppArmor profile preventing
the known attacks as well as container capabilities and SELinux. Here a
list of applicable security control mechanisms:

- Capabilities - keep (whitelist) or drop (blacklist) Linux capabilities,
  [capabilities]_.
- Control groups - cgroups, resource bean counting, resource quotas, access
  restrictions, [cgroup1]_, [cgroup2]_.
- AppArmor - apparmor profiles aim to prevent any of the known ways of
  escaping a container or cause harm to the host, [apparmor]_.
- SELinux - Security Enhanced Linux is a Linux kernel security module
  that provides similar function to AppArmor, supporting access control
  security policies including United States Department of Defense-style
  mandatory access controls. Mandatory access controls allow an
  administrator of a system to define how applications and users can
  access different resources such as files, devices, networks and inter-
  process communication, [selinux]_.
- Seccomp - secure computing mode, enables filtering of system calls,
  [seccomp]_.

More information in [lxcsecurity]_ and [lxcsecfeatures]_.

**Linux Containers in CSIT**

CSIT is using Privileged Containers as the ``sysfs`` is mounted with RW
access. Sysfs is required to be mounted as RW due to VPP accessing
:command:`/sys/bus/pci/drivers/uio_pci_generic/unbind`. This is not the case of
unprivileged containers where ``sysfs`` is mounted as read-only.


Orchestrating Container Lifecycle Events
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Following Linux container lifecycle events need to be addressed by an
orchestration system:

1. Acquire - acquiring/downloading existing container images via
   :command:`docker pull` or :command:`lxc-create -t download`.

2. Build - building a container image from scratch or another
   container image via :command:`docker build <dockerfile/composefile>` or
   customizing LXC templates in
   `GitHub <https://github.com/lxc/lxc/tree/master/templates>`_.

3. (Re-)Create - creating a running instance of a container application
   from anew, or re-creating one that failed. A.k.a. (re-)deploy via
   :command:`docker run` or :command:`lxc-start`

4. Execute - execute system operations within the container by attaching to
   running container. THis is done by :command:`lxc-attach` or
   :command:`docker exec`

5. Distribute - distributing pre-built container images to the compute
   nodes. Currently not implemented in CSIT.


Container Orchestration Systems Used in CSIT
--------------------------------------------

Current CSIT testing framework integrates following Linux container
orchestration mechanisms:

- LXC/Docker for complete VPP container lifecycle control.

LXC
~~~

LXC is the well-known and heavily tested low-level Linux container
runtime [lxcsource]_, that provides a userspace interface for the Linux kernel
containment features. With a powerful API and simple tools, LXC enables
Linux users to easily create and manage system or application
containers. LXC uses following kernel features to contain processes:

- Kernel namespaces: ipc, uts, mount, pid, network and user.
- AppArmor and SELinux security profiles.
- Seccomp policies.
- Chroot.
- Cgroups.

CSIT uses LXC runtime and LXC usertools to test VPP data plane performance in
a range of virtual networking topologies.

**Known Issues**

- Current CSIT restriction: only single instance of lxc runtime due to
  the cgroup policies used in CSIT. There is plan to add the capability into
  code to create cgroups per container instance to address this issue. This sort
  of functionality is better supported in LXC 2.1 but can be done is current
  version as well.

- CSIT code is currently using cgroup to control the range of CPU cores the
  LXC container runs on. VPP thread pinning is defined vpp startup.conf.

Docker
~~~~~~

Docker builds on top of Linux kernel containment features, and
offers a high-level tool for wrapping the processes, maintaining and
executing them in containers [docker]_. Currently it is using *runc*,
a CLI tool for spawning and running containers according to the
`OCI specification <https://www.opencontainers.org/>`_.

A Docker container image is a lightweight, stand-alone, executable
package that includes everything needed to run the container:
code, runtime, system tools, system libraries, settings.

CSIT uses Docker to manage the maintenance and execution of
containerized applications used in CSIT performance tests.

- Data plane thread pinning to CPU cores - Docker CLI and/or Docker
  configuration file controls the range of CPU cores the Docker image
  must run on. VPP thread pinning defined vpp startup.conf.

Implementation
--------------

CSIT container orchestration is implemented in CSIT Level-1 keyword
Python libraries following the Builder design pattern. Builder design
pattern separates the construction of a complex object from its
representation, so that the same construction process can create
different representations e.g. LXC, Docker, other.

CSIT Robot Framework keywords are then responsible for higher level
lifecycle control of of the named container groups. One can have
multiple named groups, with 1..N containers in a group performing
different role/functionality e.g. NFs, Switch, Kafka bus, ETCD
datastore, etc. ContainerManager class acts as a Director and uses
ContainerEngine class that encapsulate container control.

Current CSIT implementation is illustrated using UML Class diagram:

1. Acquire
2. Build
3. (Re-)Create
4. Execute

::

 +-----------------------------------------------------------------------+
 |              RF Keywords (high level lifecycle control)               |
 +-----------------------------------------------------------------------+
 | Construct VNF containers on all DUTs                                  |
 | Acquire all '${group}' containers                                     |
 | Create all '${group}' containers                                      |
 | Install all '${group}' containers                                     |
 | Configure all '${group}' containers                                   |
 | Stop all '${group}' containers                                        |
 | Destroy all '${group}' containers                                     |
 +-----------------+-----------------------------------------------------+
                   |  1
                   |
                   |  1..N
 +-----------------v-----------------+        +--------------------------+
 |          ContainerManager         |        |  ContainerEngine         |
 +-----------------------------------+        +--------------------------+
 | __init()__                        |        | __init(node)__           |
 | construct_container()             |        | acquire(force)           |
 | construct_containers()            |        | create()                 |
 | acquire_all_containers()          |        | stop()                   |
 | create_all_containers()           | 1    1 | destroy()                |
 | execute_on_container()            <>-------| info()                   |
 | execute_on_all_containers()       |        | execute(command)         |
 | install_vpp_in_all_containers()   |        | system_info()            |
 | configure_vpp_in_all_containers() |        | install_supervisor()     |
 | stop_all_containers()             |        | install_vpp()            |
 | destroy_all_containers()          |        | restart_vpp()            |
 +-----------------------------------+        | create_vpp_exec_config() |
                                              | create_vpp_startup_config|
                                              | is_container_running()   |
                                              | is_container_present()   |
                                              | _configure_cgroup()      |
                                              +-------------^------------+
                                                            |
                                                            |
                                                            |
                                                 +----------+---------+
                                                 |                    |
                                          +------+------+      +------+------+
                                          |     LXC     |      |    Docker   |
                                          +-------------+      +-------------+
                                          | (inherited) |      | (inherited) |
                                          +------+------+      +------+------+
                                                 |                    |
                                                 +----------+---------+
                                                            |
                                                            | constructs
                                                            |
                                                  +---------v---------+
                                                  |     Container     |
                                                  +-------------------+
                                                  | __getattr__(a)    |
                                                  | __setattr__(a, v) |
                                                  +-------------------+

Sequentional diagram that illustrates the creation of a single container.

::

 Legend:
    e  = engine [Docker|LXC]
    .. = kwargs (variable number of keyword argument)

 +-------+                  +------------------+       +-----------------+
 | RF KW |                  | ContainerManager |       | ContainerEngine |
 +---+---+                  +--------+---------+       +--------+--------+
     |                               |                          |
     |  1: new ContainerManager(e)   |                          |
    +-+---------------------------->+-+                         |
    |-|                             |-| 2: new ContainerEngine  |
    |-|                             |-+----------------------->+-+
    |-|                             |-|                        |-|
    |-|                             +-+                        +-+
    |-|                              |                          |
    |-| 3: construct_container(..)   |                          |
    |-+---------------------------->+-+                         |
    |-|                             |-| 4: init()               |
    |-|                             |-+----------------------->+-+
    |-|                             |-|                        |-| 5: new  +-------------+
    |-|                             |-|                        |-+-------->| Container A |
    |-|                             |-|                        |-|         +-------------+
    |-|                             |-|<-----------------------+-|
    |-|                             +-+                        +-+
    |-|                              |                          |
    |-| 6: acquire_all_containers()  |                          |
    |-+---------------------------->+-+                         |
    |-|                             |-| 7: acquire()            |
    |-|                             |-+----------------------->+-+
    |-|                             |-|                        |-|
    |-|                             |-|                        |-+--+
    |-|                             |-|                        |-|  | 8: is_container_present()
    |-|                             |-|             True/False |-|<-+
    |-|                             |-|                        |-|
    |-|                             |-|                        |-|
 +---------------------------------------------------------------------------------------------+
 |  |-| ALT [isRunning & force]     |-|                        |-|--+                          |
 |  |-|                             |-|                        |-|  | 8a: destroy()            |
 |  |-|                             |-|                        |-<--+                          |
 +---------------------------------------------------------------------------------------------+
    |-|                             |-|                        |-|
    |-|                             +-+                        +-+
    |-|                              |                          |
    |-| 9: create_all_containers()   |                          |
    |-+---------------------------->+-+                         |
    |-|                             |-| 10: create()            |
    |-|                             |-+----------------------->+-+
    |-|                             |-|                        |-+--+
    |-|                             |-|                        |-|  | 11: wait('RUNNING')
    |-|                             |-|                        |-<--+
    |-|                             +-+                        +-+
    |-|                              |                          |
 +---------------------------------------------------------------------------------------------+
 |  |-| ALT                          |                          |                              |
 |  |-| (install_vpp, configure_vpp) |                          |                              |
 |  |-|                              |                          |                              |
 +---------------------------------------------------------------------------------------------+
    |-|                              |                          |
    |-| 12: destroy_all_containers() |                          |
    |-+---------------------------->+-+                         |
    |-|                             |-| 13: destroy()           |
    |-|                             |-+----------------------->+-+
    |-|                             |-|                        |-|
    |-|                             +-+                        +-+
    |-|                              |                          |
    +++                              |                          |
     |                               |                          |
     +                               +                          +

Container Data Structure
~~~~~~~~~~~~~~~~~~~~~~~~

Container is represented in Python L1 library as a separate Class with instance
variables and no methods except overriden ``__getattr__`` and ``__setattr__``.
Instance variables are assigned to container dynamically during the
``construct_container(**kwargs)`` call and are passed down from the RF keyword.

There is no parameters check functionality. Passing the correct arguments
is a responsibility of the caller.

Examples
~~~~~~~~

This section contains a high-level example of multiple initialization steps
via ContainerManager; taken from an actual CSIT code,
but with non-code lines (comments, Documentation) removed for brevity.

:

.. code-block:: robotframework

  | Start containers for test
  | | [Arguments] | ${dut}=${None} | ${nf_chains}=${1} | ${nf_nodes}=${1}
  | | ... | ${auto_scale}=${True} | ${pinning}=${True}
  | |
  | | Set Test Variable | @{container_groups} | @{EMPTY}
  | | Set Test Variable | ${container_group} | CNF
  | | Set Test Variable | ${nf_nodes}
  | | Import Library | resources.libraries.python.ContainerUtils.ContainerManager
  | | ... | engine=${container_engine} | WITH NAME | ${container_group}
  | | Construct chains of containers
  | | ... | dut=${dut} | nf_chains=${nf_chains} | nf_nodes=${nf_nodes}
  | | ... | auto_scale=${auto_scale} | pinning=${pinning}
  | | Acquire all '${container_group}' containers
  | | Create all '${container_group}' containers
  | | Configure VPP in all '${container_group}' containers
  | | Start VPP in all '${container_group}' containers
  | | Append To List | ${container_groups} | ${container_group}
  | | Save VPP PIDs

Kubernetes
~~~~~~~~~~

For the future use, Kubernetes [k8sdoc]_ is implemented as separate library
``KubernetesUtils.py``, with a class with the same name. This utility provides
an API for L2 Robot Keywords to control ``kubectl`` installed on each of DUTs.
One time initialization script, ``resources/libraries/bash/k8s_setup.sh``
does reset/init kubectl, and initializes the ``csit`` namespace. CSIT
namespace is required to not to interfere with existing setups and it
further simplifies apply/get/delete Pod/ConfigMap operations on SUTs.

Kubernetes utility is based on YAML templates to avoid crafting the huge
YAML configuration files, what would lower the readability of code and
requires complicated algorithms.

Two types of YAML templates are defined:

- Static - do not change between deployments, that is infrastructure
  containers like Kafka, Calico, ETCD.

- Dynamic - per test suite/case topology YAML files.

Making own python wrapper library of ``kubectl`` instead of using the
official Python package allows to control and deploy environment over
the SSH library without the need of using isolated driver running on
each of DUTs.

Tested Topologies
~~~~~~~~~~~~~~~~~

Listed CSIT container networking test topologies are defined with DUT
containerized VPP switch forwarding packets between NF containers. Each
NF container runs their own instance of VPP in L2XC configuration.

Following container networking topologies are tested in |csit-release|:

- LXC topologies:

  - eth-l2xcbase-eth-2memif-1lxc.
  - eth-l2bdbasemaclrn-eth-2memif-1lxc.

- Docker topologies:

  - eth-l2xcbase-eth-2memif-1docker.
  - eth-l2xcbase-eth-1memif-1docker

References
~~~~~~~~~~

.. [lxc] `Linux Containers <https://linuxcontainers.org/>`_
.. [lxcnamespace] `Resource management: Linux kernel Namespaces and cgroups <https://www.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/lxc-namespace.pdf>`_.
.. [stgraber] `LXC 1.0: Blog post series <https://stgraber.org/2013/12/20/lxc-1-0-blog-post-series/>`_.
.. [lxcsecurity] `Linux Containers Security <https://linuxcontainers.org/lxc/security/>`_.
.. [capabilities] `Linux manual - capabilities - overview of Linux capabilities <http://man7.org/linux/man-pages/man7/capabilities.7.html>`_.
.. [cgroup1] `Linux kernel documentation: cgroups <https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt>`_.
.. [cgroup2] `Linux kernel documentation: Control Group v2 <https://www.kernel.org/doc/Documentation/cgroup-v2.txt>`_.
.. [selinux] `SELinux Project Wiki <http://selinuxproject.org/page/Main_Page>`_.
.. [lxcsecfeatures] `LXC 1.0: Security features <https://stgraber.org/2014/01/01/lxc-1-0-security-features/>`_.
.. [lxcsource] `Linux Containers source <https://github.com/lxc/lxc>`_.
.. [apparmor] `Ubuntu AppArmor <https://wiki.ubuntu.com/AppArmor>`_.
.. [seccomp] `SECure COMPuting with filters <https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt>`_.
.. [docker] `Docker <https://www.docker.com/what-docker>`_.
.. [k8sdoc] `Kubernetes documentation <https://kubernetes.io/docs/home/>`_.