ietf draft: nfv service density rev. -00

Change-Id: I0490db919b4da198545f4a332fd722855bac84a8
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

1 files changed, 874 insertions, 0 deletions

diff --git a/docs/ietf/draft-mkonstan-nf-service-density-00.md b/docs/ietf/draft-mkonstan-nf-service-density-00.md
new file mode 100644
index 0000000000..a3216d06dd
--- /dev/null
+++ b/docs/ietf/draft-mkonstan-nf-service-density-00.md
@@ -0,0 +1,874 @@
+---
+title: NFV Service Density Benchmarking
+# abbrev: nf-svc-density
+docname: draft-mkonstan-nf-service-density-00
+date: 2019-03-11
+
+ipr: trust200902
+area: ops
+wg: Benchmarking Working Group
+kw: Internet-Draft
+cat: info
+
+coding: us-ascii
+pi:    # can use array (if all yes) or hash here
+#  - toc
+#  - sortrefs
+#  - symrefs
+  toc: yes
+  sortrefs:   # defaults to yes
+  symrefs: yes
+
+author:
+      -
+        ins: M. Konstantynowicz
+        name: Maciek Konstantynowicz
+        org: Cisco Systems
+        role: editor
+        email: mkonstan@cisco.com
+      -
+        ins: P. Mikus
+        name: Peter Mikus
+        org: Cisco Systems
+        role: editor
+        email: pmikus@cisco.com
+
+normative:
+  RFC2544:
+  RFC8174:
+
+informative:
+  RFC8204:
+  TST009:
+    target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.01.01_60/gs_NFV-TST009v030101p.pdf
+    title: "ETSI GS NFV-TST 009 V3.1.1 (2018-10), Network Functions Virtualisation (NFV) Release 3; Testing; Specification of Networking Benchmarks and Measurement Methods for NFVI"
+    date: 2018-10
+  BSDP:
+    target: https://fd.io/wp-content/uploads/sites/34/2019/03/benchmarking_sw_data_planes_skx_bdx_mar07_2019.pdf
+    title: "Benchmarking Software Data Planes Intel® Xeon® Skylake vs. Broadwell"
+    date: 2019-03
+  draft-vpolak-mkonstan-bmwg-mlrsearch:
+    target: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-00
+    title: "Multiple Loss Ratio Search for Packet Throughput (MLRsearch)"
+    date: 2018-11
+  draft-vpolak-bmwg-plrsearch:
+    target: https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch-00
+    title: "Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)"
+    date: 2018-11
+  LFN-FDio-CSIT:
+    target: https://wiki.fd.io/view/CSIT
+    title: "Fast Data io, Continuous System Integration and Testing Project"
+    date: 2019-03
+  CNCF-CNF-Testbed:
+    target: https://github.com/cncf/cnf-testbed/
+    title: "Cloud native Network Function (CNF) Testbed"
+    date: 2019-03
+  TRex:
+    target: https://github.com/cisco-system-traffic-generator/trex-core
+    title: "TRex Low-Cost, High-Speed Stateful Traffic Generator"
+    date: 2019-03
+  CSIT-1901-testbed-2n-skx:
+    target: https://docs.fd.io/csit/rls1901/report/introduction/physical_testbeds.html#node-xeon-skylake-2n-skx
+    title: "FD.io CSIT Test Bed"
+    date: 2019-03
+  CSIT-1901-test-enviroment:
+    target: https://docs.fd.io/csit/rls1901/report/vpp_performance_tests/test_environment.html
+    title: "FD.io CSIT Test Environment"
+    date: 2019-03
+  CSIT-1901-nfv-density-methodology:
+    target: https://docs.fd.io/csit/rls1901/report/introduction/methodology_nfv_service_density.html
+    title: "FD.io CSIT Test Methodology: NFV Service Density"
+    date: 2019-03
+  CSIT-1901-nfv-density-results:
+    target: https://docs.fd.io/csit/rls1901/report/vpp_performance_tests/nf_service_density/index.html
+    title: "FD.io CSIT Test Results: NFV Service Density"
+    date: 2019-03
+  CNCF-CNF-Testbed-Results:
+    target: https://github.com/cncf/cnf-testbed/blob/master/comparison/doc/cncf-cnfs-results-summary.md
+    title: "CNCF CNF Testbed: NFV Service Density Benchmarking"
+    date: 2018-12
+
+--- abstract
+
+Network Function Virtualization (NFV) system designers and operators
+continuously grapple with the problem of qualifying performance of
+network services realised with software Network Functions (NF) running
+on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges
+is getting repeatable and portable benchmarking results and using them
+to derive deterministic operating range that is production deployment
+worthy.
+
+This document specifies benchmarking methodology for NFV services that
+aims to address this problem space. It defines a way for measuring
+performance of multiple NFV service instances, each composed of multiple
+software NFs, and running them at a varied service “packing” density on
+a single server.
+
+The aim is to discover deterministic usage range of NFV system. In
+addition specified methodology can be used to compare and contrast
+different NFV virtualization technologies.
+
+--- middle
+
+# Terminology
+
+* NFV - Network Function Virtualization, a general industry term
+  describing network functionality implemented in software.
+* NFV service - a software based network service realized by a topology
+  of interconnected constituent software network function applications.
+* NFV service instance - a single instantiation of NFV service.
+* Data-plane optimized software - any software with dedicated threads
+  handling data-plane packet processing e.g. FD.io VPP (Vector Packet
+  Processor), OVS-DPDK.
+
+# Motivation
+
+## Problem Description
+
+Network Function Virtualization (NFV) system designers and operators
+continuously grapple with the problem of qualifying performance of
+network services realised with software Network Functions (NF) running
+on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges
+is getting repeatable and portable benchmarking results and using them
+to derive deterministic operating range that is production deployment
+worthy.
+
+Lack of well defined and standardised NFV centric performance
+methodology and metrics makes it hard to address fundamental questions
+that underpin NFV production deployments:
+
+1. What NFV service and how many instances can run on a single compute
+   node?
+2. How to choose the best compute resource allocation scheme to maximise
+   service yield per node?
+3. How do different NF applications compare from the service density
+   perspective?
+4. How do the virtualisation technologies compare e.g. Virtual Machines,
+   Containers?
+
+Getting answers to these points should allow designers to make a data
+based decision about the NFV technology and service design best suited
+to meet requirements of their use cases. Equally, obtaining the
+benchmarking data underpinning those answers should make it easier for
+operators to work out expected deterministic operating range of chosen
+design.
+
+## Proposed Solution
+
+The primary goal of the proposed benchmarking methodology is to focus on
+NFV technologies used to construct NFV services. More specifically to i)
+measure packet data-plane performance of multiple NFV service instances
+while running them at varied service “packing” densities on a single
+server and ii) quantify the impact of using multiple NFs to construct
+each NFV service instance and introducing multiple packet processing
+hops and links on each packet path.
+
+The overarching aim is to discover a set of deterministic usage ranges
+that are of interest to NFV system designers and operators. In addition,
+specified methodology can be used to compare and contrast different NFV
+virtualisation technologies.
+
+In order to ensure wide applicability of the benchmarking methodology,
+the approach is to separate NFV service packet processing from the
+shared virtualisation infrastructure by decomposing the software
+technology stack into three building blocks:
+
+                  +-------------------------------+
+                  |          NFV Service          |
+                  +-------------------------------+
+                  |   Virtualization Technology   |
+                  +-------------------------------+
+                  |        Host Networking        |
+                  +-------------------------------+
+
+              Figure 1. NFV software technology stack.
+
+Proposed methodology is complementary to existing NFV benchmarking
+industry efforts focusing on vSwitch benchmarking [RFC8204], [TST009]
+and extends the benchmarking scope to NFV services.
+
+This document does not describe a complete benchmarking methodology,
+instead it is focusing on system under test configuration part. Each of
+the compute node configurations identified by (RowIndex, ColumnIndex) is
+to be evaluated for NFV service data-plane performance using existing
+and/or emerging network benchmarking standards. This may include
+methodologies specified in [RFC2544], [TST009],
+[draft-vpolak-mkonstan-bmwg-mlrsearch] and/or
+[draft-vpolak-bmwg-plrsearch].
+
+# NFV Service
+
+It is assumed that each NFV service instance is built of one or more
+constituent NFs and is described by: topology, configuration and
+resulting packet path(s).
+
+Each set of NFs forms an independent NFV service instance, with multiple
+sets present in the host.
+
+## Topology
+
+NFV topology describes the number of network functions per service
+instance, and their inter-connections over packet interfaces. It
+includes all point-to-point virtual packet links within the compute
+node, Layer-2 Ethernet or Layer-3 IP, including the ones to host
+networking data-plane.
+
+Theoretically, a large set of possible NFV topologies can be realised
+using software virtualisation topologies, e.g. ring, partial -/full-
+mesh, star, line, tree, ladder. In practice however, only a few
+topologies are in the actual use as NFV services mostly perform either
+bumps-in-a-wire packet operations (e.g. security filtering/inspection,
+monitoring/telemetry) and/or inter-site forwarding decisions (e.g.
+routing, switching).
+
+Two main NFV topologies have been identified so far for NFV service
+density benchmarking:
+
+1. Chain topology: a set of NFs connect to host data-plane with minimum
+   of two virtual interfaces each, enabling host data-plane to
+   facilitate NF to NF service chain forwarding and provide connectivity
+   with external network.
+
+2. Pipeline topology: a set of NFs connect to each other in a line
+   fashion with edge NFs homed to host data-plane. Host data-plane
+   provides connectivity with external network.
+
+Both topologies are shown in figures below.
+
+NF chain topology:
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | |  S1NF1 |  |  S1NF2 |              |  S1NFn |            |
+    | |        |  |        |     ....     |        | Service1   |
+    | |        |  |        |              |        |            |
+    | +-+----+-+  +-+----+-+    +    +    +-+----+-+            |
+    |   |    |      |    |      |    |      |    |   Virtual    |
+    |   |    |<-CS->|    |<-CS->|    |<-CS->|    |   Interfaces |
+    | +-+----+------+----+------+----+------+----+-+            |
+    | |                                            | CS: Chain  |
+    | |                                            |   Segment  |
+    | |             Host Data-Plane                |            |
+    | +-+--+----------------------------------+--+-+            |
+    |   |  |                                  |  |              |
+    +-----------------------------------------------------------+
+        |  |                                  |  |   Physical
+        |  |                                  |  |   Interfaces
+    +---+--+----------------------------------+--+--------------+
+    |                                                           |
+    |                    Traffic Generator                      |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+      Figure 2. NF chain topology forming a service instance.
+
+NF pipeline topology:
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | |  S1NF1 |  |  S1NF2 |              |  S1NFn |            |
+    | |        +--+        +--+  ....  +--+        | Service1   |
+    | |        |  |        |              |        |            |
+    | +-+------+  +--------+              +------+-+            |
+    |   |                                        |   Virtual    |
+    |   |<-Pipeline Edge          Pipeline Edge->|   Interfaces |
+    | +-+----------------------------------------+-+            |
+    | |                                            |            |
+    | |                                            |            |
+    | |             Host Data-Plane                |            |
+    | +-+--+----------------------------------+--+-+            |
+    |   |  |                                  |  |              |
+    +-----------------------------------------------------------+
+        |  |                                  |  |   Physical
+        |  |                                  |  |   Interfaces
+    +---+--+----------------------------------+--+--------------+
+    |                                                           |
+    |                    Traffic Generator                      |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+      Figure 3. NF pipeline topology forming a service instance.
+
+
+## Configuration
+
+NFV configuration includes all packet processing functions in NFs
+including Layer-2, Layer-3 and/or Layer-4-to-7 processing as appropriate
+to specific NF and NFV service design. L2 sub- interface encapsulations
+(e.g. 802.1q, 802.1ad) and IP overlay encapsulation (e.g. VXLAN, IPSec,
+GRE) may be represented here too as appropriate, although in most cases
+they are used as external encapsulation and handled by host networking
+data-plane.
+
+NFV configuration determines logical network connectivity that is
+Layer-2 and/or IPv4/IPv6 switching/routing modes, as well as NFV service
+specific aspects. In the context of NFV density benchmarking methodology
+the initial focus is on the former.
+
+Building on the two identified NFV topologies, two common NFV
+configurations are considered:
+
+1. Chain configuration:
+   * Relies on chain topology to form NFV service chains.
+   * NF packet forwarding designs:
+     * IPv4/IPv6 routing.
+   * Requirements for host data-plane:
+     * L2 switching with L2 forwarding context per each NF chain
+       segment, or
+     * IPv4/IPv6 routing with IP forwarding context per each NF chain
+       segment or per NF chain.
+
+2. Pipeline configuration:
+   * Relies on pipeline topology to form NFV service pipelines.
+   * Packet forwarding designs:
+     * IPv4/IPv6 routing.
+   * Requirements for host data-plane:
+     * L2 switching with L2 forwarding context per each NF pipeline
+       edge link, or
+     * IPv4/IPv6 routing with IP forwarding context per each NF pipeline
+       edge link or per NF pipeline.
+
+## Packet Path(s)
+
+NFV packet path(s) describe the actual packet forwarding path(s) used
+for benchmarking, resulting from NFV topology and configuration. They
+are aimed to resemble true packet forwarding actions during the NFV
+service lifecycle.
+
+Based on the specified NFV topologies and configurations two NFV packet
+paths are taken for benchmarking:
+
+1. Snake packet path
+   * Requires chain topology and configuration.
+   * Packets enter the NFV chain through one edge NF and progress to the
+     other edge NF of the chain.
+   * Within the chain, packets follow a zigzagging "snake" path entering
+     and leaving host data-plane as they progress through the NF chain.
+   * Host data-plane is involved in packet forwarding operations between
+     NIC interfaces and edge NFs, as well as between NFs in the chain.
+
+2. Pipeline packet path
+   * Requires pipeline topology and configuration.
+   * Packets enter the NFV chain through one edge NF and progress to the
+     other edge NF of the pipeline.
+   * Within the chain, packets follow a straight path entering and
+     leaving subsequent NFs as they progress through the NF pipeline.
+   * Host data-plane is involved in packet forwarding operations between
+     NIC interfaces and edge NFs only.
+
+Both packet paths are shown in figures below.
+
+Snake packet path:
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | |  S1NF1 |  |  S1NF2 |              |  S1NFn |            |
+    | |        |  |        |     ....     |        | Service1   |
+    | |  XXXX  |  |  XXXX  |              |  XXXX  |            |
+    | +-+X--X+-+  +-+X--X+-+    +X  X+    +-+X--X+-+            |
+    |   |X  X|      |X  X|      |X  X|      |X  X|   Virtual    |
+    |   |X  X|      |X  X|      |X  X|      |X  X|   Interfaces |
+    | +-+X--X+------+X--X+------+X--X+------+X--X+-+            |
+    | |  X  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  X  |            |
+    | |  X                                      X  |            |
+    | |  X          Host Data-Plane             X  |            |
+    | +-+X-+----------------------------------+-X+-+            |
+    |   |X |                                  | X|              |
+    +----X--------------------------------------X---------------+
+        |X |                                  | X|   Physical
+        |X |                                  | X|   Interfaces
+    +---+X-+----------------------------------+-X+--------------+
+    |                                                           |
+    |                    Traffic Generator                      |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+        Figure 4. Snake packet path thru NF chain topology.
+
+
+Pipeline packet path:
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | |  S1NF1 |  |  S1NF2 |              |  S1NFn |            |
+    | |        +--+        +--+  ....  +--+        | Service1   |
+    | |  XXXXXXXXXXXXXXXXXXXXXXXX    XXXXXXXXXXXX  |            |
+    | +--X-----+  +--------+              +-----X--+            |
+    |   |X                                      X|   Virtual    |
+    |   |X                                      X|   Interfaces |
+    | +-+X--------------------------------------X+-+            |
+    | |  X                                      X  |            |
+    | |  X                                      X  |            |
+    | |  X          Host Data-Plane             X  |            |
+    | +-+X-+----------------------------------+-X+-+            |
+    |   |X |                                  | X|              |
+    +----X--------------------------------------X---------------+
+        |X |                                  | X|   Physical
+        |X |                                  | X|   Interfaces
+    +---+X-+----------------------------------+-X+--------------+
+    |                                                           |
+    |                    Traffic Generator                      |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+      Figure 5. Pipeline packet path thru NF pipeline topology.
+
+In all cases packets enter NFV system via shared physical NIC interfaces
+controlled by shared host data-plane, are then associated with specific
+NFV service (based on service discriminator) and subsequently are cross-
+connected/switched/routed by host data-plane to and through NF
+topologies per one of above listed schemes.
+
+# Virtualization Technology
+
+NFV services are built of composite isolated NFs, with virtualisation
+technology providing the workload isolation. Following virtualisation
+technology types are considered for NFV service density benchmarking:
+
+1. Virtual Machines (VMs)
+   * Relying on host hypervisor technology e.g. KVM, ESXi, Xen.
+   * NFs running in VMs are referred to as VNFs.
+2. Containers
+   * Relying on Linux container technology e.g. LXC, Docker.
+   * NFs running in Containers are referred to as CNFs.
+
+Different virtual interface types are available to VNFs and CNFs:
+
+1. VNF
+   * virtio-vhostuser: fully user-mode based virtual interface.
+   * virtio-vhostnet: involves kernel-mode based backend.
+2. CNF
+   * memif: fully user-mode based virtual interface.
+   * af_packet: involves kernel-mode based backend.
+   * (add more common ones)
+
+# Host Networking
+
+Host networking data-plane is the central shared resource that underpins
+creation of NFV services. It handles all of the connectivity to external
+physical network devices through physical network connections using
+NICs, through which the benchmarking is done.
+
+Assuming that NIC interface resources are shared, here is the list of
+widely available host data-plane options for providing packet
+connectivity to/from NICs and constructing NFV chain and pipeline
+topologies and configurations:
+
+* Linux Kernel-Mode Networking.
+* Linux User-Mode vSwitch.
+* Virtual Machine vSwitch.
+* Linux Container vSwitch.
+* SRIOV NIC Virtual Function - note: restricted support for chain and
+  pipeline topologies, as it requires hair-pinning through the NIC and
+  oftentimes also through external physical switch.
+
+Analysing properties of each of these options and their Pros/Cons for
+specified NFV topologies and configurations is outside the scope of this
+document.
+
+From all listed options, performance optimised Linux user-mode vswitch
+deserves special attention. Linux user-mode switch decouples NFV service
+from the underlying NIC hardware, offers rich multi-tenant functionality
+and most flexibility for supporting NFV services. But in the same time
+it is consuming compute resources and is harder to benchmark in NFV
+service density scenarios.
+
+Following sections focus on using Linux user-mode vSwitch, focusing on
+its performance benchmarking at increasing levels of NFV service
+density.
+
+# NFV Service Density Matrix
+
+In order to evaluate performance of multiple NFV services running on a
+compute node, NFV service instances are benchmarked at increasing
+density, allowing to construct an NFV Service Density Matrix. Table
+below shows an example of such a matrix, capturing number of NFV service
+instances (row indices), number of NFs per service instance (column
+indices) and resulting total number of NFs (values).
+
+    NFV Service Density - NF Count View
+
+    SVC   001   002   004   006   008   00N
+    001     1     2     4     6     8   1*N
+    002     2     4     8    12    16   2*N
+    004     4     8    16    24    32   4*N
+    006     6    12    24    36    48   6*N
+    008     8    16    32    48    64   8*N
+    00M   M*1   M*2   M*4   M*6   M*8   M*N
+
+    RowIndex:     Number of NFV Service Instances, 1..M.
+    ColumnIndex:  Number of NFs per NFV Service Instance, 1..N.
+    Value:        Total number of NFs running in the system.
+
+In order to deliver good and repeatable network data-plane performance,
+NFs and host data-plane software require direct access to critical
+compute resources. Due to a shared nature of all resources on a compute
+node, a clearly defined resource allocation scheme is defined in the
+next section to address this.
+
+In each tested configuration host data-plane is a gateway between the
+external network and the internal NFV network topologies. Offered packet
+load is generated and received by an external traffic generator per
+usual benchmarking practice.
+
+It is proposed that initial benchmarks are done with the offered packet
+load distributed equally across all configured NFV service instances.
+This could be followed by various per NFV service instance load ratios
+mimicking expected production deployment scenario(s).
+
+Following sections specify compute resource allocation, followed by
+examples of applying NFV service density methodology to VNF and CNF
+benchmarking use cases.
+
+# Compute Resource Allocation
+
+Performance optimized NF and host data-plane software threads require
+timely execution of packet processing instructions and are very
+sensitive to any interruptions (or stalls) to this execution e.g. cpu
+core context switching, or cpu jitter. To that end, NFV service density
+methodology treats controlled mapping ratios of data plane software
+threads to physical processor cores with directly allocated cache
+hierarchies as the first order requirement.
+
+Other compute resources including memory bandwidth and PCIe bandwidth
+have lesser impact and as such are subject for further study. For more
+detail and deep-dive analysis of software data plane performance and
+impact on different shared compute resources is available in [BSDP].
+
+It is assumed that NFs as well as host data-plane (e.g. vswitch) are
+performance optimized, with their tasks executed in two types of
+software threads:
+
+* data-plane - handling data-plane packet processing and forwarding,
+  time critical, requires dedicated cores. To scale data-plane
+  performance, most NF apps use multiple data-plane threads and rely on
+  NIC RSS (Receive Side Scaling), virtual interface multi-queue and/or
+  integrated software hashing to distribute packets across the data
+  threads.
+
+* main-control - handling application management, statistics and
+  control-planes, less time critical, allows for core sharing. For most
+  NF apps this is a single main thread, but often statistics (counters)
+  and various control protocol software are run in separate threads.
+
+Core mapping scheme described below allocates cores for all threads of
+specified type belonging to each NF app instance, and separately lists
+number of threads to a number of logical/physical core mappings for
+processor configurations with enabled/disabled Symmetric Multi-
+Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading).
+
+If NFV service density benchmarking is run on server nodes with
+Symmetric Multi-Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading)
+for higher performance and efficiency, logical cores allocated to data-
+plane threads should be allocated as pairs of sibling logical cores
+corresponding to the hyper-threads running on the same physical core.
+
+Separate core ratios are defined for mapping threads of vSwitch and NFs.
+In order to get consistent benchmarking results, the mapping ratios are
+enforced using Linux core pinning.
+
+| application | thread type | app:core ratio | threads/pcores (SMT disabled)   | threads/lcores map (SMT enabled)   |
+|:-----------:|:-----------:|:--------------:|:-------------------------------:|:----------------------------------:|
+| vSwitch-1c  | data        | 1:1            | 1DT/1PC                         | 2DT/2LC                            |
+|             | main        | 1:S2           | 1MT/S2PC                        | 1MT/1LC                            |
+|             |             |                |                                 |                                    |
+| vSwitch-2c  | data        | 1:2            | 2DT/2PC                         | 4DT/4LC                            |
+|             | main        | 1:S2           | 1MT/S2PC                        | 1MT/1LC                            |
+|             |             |                |                                 |                                    |
+| vSwitch-4c  | data        | 1:4            | 4DT/4PC                         | 8DT/8LC                            |
+|             | main        | 1:S2           | 1MT/S2PC                        | 1MT/1LC                            |
+|             |             |                |                                 |                                    |
+| NF-0.5c     | data        | 1:S2           | 1DT/S2PC                        | 1DT/1LC                            |
+|             | main        | 1:S2           | 1MT/S2PC                        | 1MT/1LC                            |
+|             |             |                |                                 |                                    |
+| NF-1c       | data        | 1:1            | 1DT/1PC                         | 2DT/2LC                            |
+|             | main        | 1:S2           | 1MT/S2PC                        | 1MT/1LC                            |
+|             |             |                |                                 |                                    |
+| NF-2c       | data        | 1:2            | 2DT/2PC                         | 4DT/4LC                            |
+|             | main        | 1:S2           | 1MT/S2PC                        | 1MT/1LC                            |
+
+* Legend to table
+  * Header row
+    * application - network application with optimized data-plane, a
+      vSwitch or Network Function (NF) application.
+    * thread type - either "data", short for data-plane; or "main",
+      short for all main-control threads.
+    * app:core ratio - ratio of per application instance threads of
+      specific thread type to physical cores.
+    * threads/pcores (SMT disabled) - number of threads of specific
+      type (DT for data-plane thread, MT for main thread) running on a
+      number of physical cores, with SMT disabled.
+    * threads/lcores map (SMT enabled) - number of threads of specific
+      type (DT, MT) running on a number of logical cores, with SMT
+      enabled. Two logical cores per one physical core.
+  * Content rows
+    * vSwitch-(1c|2c|4c) - vSwitch with 1 physical core (or 2, or 4)
+      allocated to its data-plane software worker threads.
+    * NF-(0.5c|1c|2c) - NF application with half of a physical core (or
+      1, or 2) allocated to its data-plane software worker threads.
+    * Sn - shared core, sharing ratio of (n).
+    * DT - data-plane thread.
+    * MT - main-control thread.
+    * PC - physical core, with SMT/HT enabled has many (mostly 2 today)
+      logical cores associated with it.
+    * LC - logical core, if more than one lc get allocated in sets of
+      two sibling logical cores running on the same physical core.
+    * SnPC - shared physical core, sharing ratio of (n).
+    * SnLC - shared logical core, sharing ratio of (n).
+
+Maximum benchmarked NFV service densities are limited by a number of
+physical cores on a compute node.
+
+A sample physical core usage view is shown in the matrix below.
+
+    NFV Service Density - Core Usage View
+    vSwitch-1c, NF-1c
+
+    SVC   001   002   004   006   008   010
+    001     2     3     6     9    12    15
+    002     3     6    12    18    24    30
+    004     6    12    24    36    48    60
+    006     9    18    36    54    72    90
+    008    12    24    48    72    96   120
+    010    15    30    60    90   120   150
+
+    RowIndex:     Number of NFV Service Instances, 1..10.
+    ColumnIndex:  Number of NFs per NFV Service Instance, 1..10.
+    Value:        Total number of physical processor cores used for NFs.
+
+# NFV Service Density Benchmarks
+
+To illustrate defined NFV service density applicability, following
+sections describe three sets of NFV service topologies and
+configurations that have been benchmarked in open-source: i) in
+[LFN-FDio-CSIT], a continuous testing and data-plane benchmarking
+project, and ii) as part of CNCF CNF Testbed initiative
+[CNCF-CNF-Testbed].
+
+In both cases each NFV service instance definition is based on the same
+set of NF applications, and varies only by network addressing
+configuration to emulate multi-tenant operating environment.
+
+## Test Methodology - MRR Throughput
+
+Initial NFV density throughput benchmarks have been performed using
+Maximum Receive Rate (MRR) test methodology defined and used in FD.io
+CSIT.
+
+MRR tests measure the packet forwarding rate under the maximum load
+offered by traffic generator over a set trial duration, regardless of
+packet loss. Maximum load for specified Ethernet frame size is set to
+the bi-directional link rate (2x 10GbE in referred results).
+
+Tests were conducted with two traffic profiles: i) continuous stream of
+64B frames, ii) continuous stream of IMIX sequence of (7x 64B, 4x 570B,
+1x 1518B), all sizes are L2 untagged Ethernet.
+
+NFV service topologies tested include: VNF service chains, CNF service
+chains and CNF service pipelines.
+
+## VNF Service Chain
+
+VNF Service Chain (VSC) topology is tested with KVM hypervisor (Ubuntu
+18.04-LTS), with NFV service instances consisting of NFs running in VMs
+(VNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual
+interfaces are virtio-vhostuser. Snake forwarding packet path is tested
+using [TRex] traffic generator, see figure.
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | | S1VNF1 |  | S1VNF2 |              | S1VNFn |            |
+    | |        |  |        |     ....     |        | Service1   |
+    | |  XXXX  |  |  XXXX  |              |  XXXX  |            |
+    | +-+X--X+-+  +-+X--X+-+              +-+X--X+-+            |
+    |   |X  X|      |X  X|                  |X  X|   Virtual    |
+    |   |X  X|      |X  X|      |X  X|      |X  X|   Interfaces |
+    | +-+X--X+------+X--X+------+X--X+------+X--X+-+            |
+    | |  X  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  X  |            |
+    | |  X                                      X  |            |
+    | |  X          FD.io VPP vSwitch           X  |            |
+    | +-+X-+----------------------------------+-X+-+            |
+    |   |X |                                  | X|              |
+    +----X--------------------------------------X---------------+
+        |X |                                  | X|   Physical
+        |X |                                  | X|   Interfaces
+    +---+X-+----------------------------------+-X+--------------+
+    |                                                           |
+    |                  Traffic Generator (TRex)                 |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+            Figure 6. VNF service chain test setup.
+
+
+## CNF Service Chain
+
+CNF Service Chain (CSC) topology is tested with Docker containers
+(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running
+in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch.
+Virtual interfaces are memif. Snake forwarding packet path is tested
+using [TRex] traffic generator, see figure.
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | | S1CNF1 |  | S1CNF2 |              | S1CNFn |            |
+    | |        |  |        |     ....     |        | Service1   |
+    | |  XXXX  |  |  XXXX  |              |  XXXX  |            |
+    | +-+X--X+-+  +-+X--X+-+              +-+X--X+-+            |
+    |   |X  X|      |X  X|                  |X  X|   Virtual    |
+    |   |X  X|      |X  X|      |X  X|      |X  X|   Interfaces |
+    | +-+X--X+------+X--X+------+X--X+------+X--X+-+            |
+    | |  X  XXXXXXXXXX  XXXXXXXXXX  XXXXXXXXXX  X  |            |
+    | |  X                                      X  |            |
+    | |  X          FD.io VPP vSwitch           X  |            |
+    | +-+X-+----------------------------------+-X+-+            |
+    |   |X |                                  | X|              |
+    +----X--------------------------------------X---------------+
+        |X |                                  | X|   Physical
+        |X |                                  | X|   Interfaces
+    +---+X-+----------------------------------+-X+--------------+
+    |                                                           |
+    |                  Traffic Generator (TRex)                 |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+          Figure 7. CNF service chain test setup.
+
+## CNF Service Pipeline
+
+CNF Service Pipeline (CSP) topology is tested with Docker containers
+(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running
+in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch.
+Virtual interfaces are memif. Pipeline forwarding packet path is tested
+using [TRex] traffic generator, see figure.
+
+    +-----------------------------------------------------------+
+    |                     Host Compute Node                     |
+    |                                                           |
+    | +--------+  +--------+              +--------+            |
+    | |  S1NF1 |  |  S1NF2 |              |  S1NFn |            |
+    | |        +--+        +--+  ....  +--+        | Service1   |
+    | |  XXXXXXXXXXXXXXXXXXXXXXXX    XXXXXXXXXXXX  |            |
+    | +--X-----+  +--------+              +-----X--+            |
+    |   |X                                      X|   Virtual    |
+    |   |X                                      X|   Interfaces |
+    | +-+X--------------------------------------X+-+            |
+    | |  X                                      X  |            |
+    | |  X                                      X  |            |
+    | |  X          FD.io VPP vSwitch           X  |            |
+    | +-+X-+----------------------------------+-X+-+            |
+    |   |X |                                  | X|              |
+    +----X--------------------------------------X---------------+
+        |X |                                  | X|   Physical
+        |X |                                  | X|   Interfaces
+    +---+X-+----------------------------------+-X+--------------+
+    |                                                           |
+    |                  Traffic Generator (TRex)                 |
+    |                                                           |
+    +-----------------------------------------------------------+
+
+              Figure 8. CNF service chain test setup.
+
+## Sample Results: FD.io CSIT
+
+FD.io CSIT project introduced NFV density benchmarking in release
+CSIT-1901 and published results for the following NFV service topologies
+and configurations:
+
+1. VNF Service Chains
+   * VNF: DPDK-L3FWD v18.10
+     * IPv4 forwarding
+     * NF-1c
+   * vSwitch: VPP v19.01-release
+     * L2 MAC switching
+     * vSwitch-1c, vSwitch-2c
+   * frame sizes: 64B, IMIX
+2. CNF Service Chains
+   * CNF: VPP v19.01-release
+     * IPv4 routing
+     * NF-1c
+   * vSwitch: VPP v19.01-release
+     * L2 MAC switching
+     * vSwitch-1c, vSwitch-2c
+   * frame sizes: 64B, IMIX
+3. CNF Service Pipelines
+   * CNF: VPP v19.01-release
+     * IPv4 routing
+     * NF-1c
+   * vSwitch: VPP v19.01-release
+     * L2 MAC switching
+     * vSwitch-1c, vSwitch-2c
+   * frame sizes: 64B, IMIX
+
+More information is available in FD.io CSIT-1901 report, with specific
+references listed below:
+
+* Testbed: [CSIT-1901-testbed-2n-skx]
+* Test environment: [CSIT-1901-test-enviroment]
+* Methodology: [CSIT-1901-nfv-density-methodology]
+* Results: [CSIT-1901-nfv-density-results]
+
+## Sample Results: CNCF/CNFs
+
+CNCF CI team introduced a CNF testbed initiative focusing on benchmaring
+NFV density with open-source network applications running as VNFs and
+CNFs. Following NFV service topologies and configurations have been
+tested to date:
+
+1. VNF Service Chains
+   * VNF: VPP v18.10-release
+     * IPv4 routing
+     * NF-1c
+   * vSwitch: VPP v18.10-release
+     * L2 MAC switching
+     * vSwitch-1c, vSwitch-2c
+   * frame sizes: 64B, IMIX
+2. CNF Service Chains
+   * CNF: VPP v18.10-release
+     * IPv4 routing
+     * NF-1c
+   * vSwitch: VPP v18.10-release
+     * L2 MAC switching
+     * vSwitch-1c, vSwitch-2c
+   * frame sizes: 64B, IMIX
+3. CNF Service Pipelines
+   * CNF: VPP v18.10-release
+     * IPv4 routing
+     * NF-1c
+   * vSwitch: VPP v18.10-release
+     * L2 MAC switching
+     * vSwitch-1c, vSwitch-2c
+   * frame sizes: 64B, IMIX
+
+More information is available in CNCF CNF Testbed github, with summary
+test results presented in summary markdown file, references listed
+below:
+
+* Results: [CNCF-CNF-Testbed-Results]
+
+# IANA Considerations
+
+No requests of IANA
+
+# Security Considerations
+
+..
+
+# Acknowledgements
+
+Thanks to Vratko Polak of FD.io CSIT project and Michael Pedersen of the
+CNCF Testbed initiative for their contributions and useful suggestions.
+
+--- back
\ No newline at end of file