From 3be03603203d84ad49fa3fd6f376ecbf0395361b Mon Sep 17 00:00:00 2001 From: Maciek Konstantynowicz Date: Tue, 2 Apr 2019 19:35:34 +0100 Subject: Update draft-mkonstan-nf-service-density-00->01 Change-Id: Ic63aa09bfff98d358b770e378c5571f1114839b8 Signed-off-by: Maciek Konstantynowicz --- docs/ietf/draft-mkonstan-nf-service-density-00.md | 874 ------------------- docs/ietf/draft-mkonstan-nf-service-density-01.md | 983 ++++++++++++++++++++++ 2 files changed, 983 insertions(+), 874 deletions(-) delete mode 100644 docs/ietf/draft-mkonstan-nf-service-density-00.md create mode 100644 docs/ietf/draft-mkonstan-nf-service-density-01.md diff --git a/docs/ietf/draft-mkonstan-nf-service-density-00.md b/docs/ietf/draft-mkonstan-nf-service-density-00.md deleted file mode 100644 index a3216d06dd..0000000000 --- a/docs/ietf/draft-mkonstan-nf-service-density-00.md +++ /dev/null @@ -1,874 +0,0 @@ ---- -title: NFV Service Density Benchmarking -# abbrev: nf-svc-density -docname: draft-mkonstan-nf-service-density-00 -date: 2019-03-11 - -ipr: trust200902 -area: ops -wg: Benchmarking Working Group -kw: Internet-Draft -cat: info - -coding: us-ascii -pi: # can use array (if all yes) or hash here -# - toc -# - sortrefs -# - symrefs - toc: yes - sortrefs: # defaults to yes - symrefs: yes - -author: - - - ins: M. Konstantynowicz - name: Maciek Konstantynowicz - org: Cisco Systems - role: editor - email: mkonstan@cisco.com - - - ins: P. Mikus - name: Peter Mikus - org: Cisco Systems - role: editor - email: pmikus@cisco.com - -normative: - RFC2544: - RFC8174: - -informative: - RFC8204: - TST009: - target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.01.01_60/gs_NFV-TST009v030101p.pdf - title: "ETSI GS NFV-TST 009 V3.1.1 (2018-10), Network Functions Virtualisation (NFV) Release 3; Testing; Specification of Networking Benchmarks and Measurement Methods for NFVI" - date: 2018-10 - BSDP: - target: https://fd.io/wp-content/uploads/sites/34/2019/03/benchmarking_sw_data_planes_skx_bdx_mar07_2019.pdf - title: "Benchmarking Software Data Planes Intel® Xeon® Skylake vs. Broadwell" - date: 2019-03 - draft-vpolak-mkonstan-bmwg-mlrsearch: - target: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch-00 - title: "Multiple Loss Ratio Search for Packet Throughput (MLRsearch)" - date: 2018-11 - draft-vpolak-bmwg-plrsearch: - target: https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch-00 - title: "Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)" - date: 2018-11 - LFN-FDio-CSIT: - target: https://wiki.fd.io/view/CSIT - title: "Fast Data io, Continuous System Integration and Testing Project" - date: 2019-03 - CNCF-CNF-Testbed: - target: https://github.com/cncf/cnf-testbed/ - title: "Cloud native Network Function (CNF) Testbed" - date: 2019-03 - TRex: - target: https://github.com/cisco-system-traffic-generator/trex-core - title: "TRex Low-Cost, High-Speed Stateful Traffic Generator" - date: 2019-03 - CSIT-1901-testbed-2n-skx: - target: https://docs.fd.io/csit/rls1901/report/introduction/physical_testbeds.html#node-xeon-skylake-2n-skx - title: "FD.io CSIT Test Bed" - date: 2019-03 - CSIT-1901-test-enviroment: - target: https://docs.fd.io/csit/rls1901/report/vpp_performance_tests/test_environment.html - title: "FD.io CSIT Test Environment" - date: 2019-03 - CSIT-1901-nfv-density-methodology: - target: https://docs.fd.io/csit/rls1901/report/introduction/methodology_nfv_service_density.html - title: "FD.io CSIT Test Methodology: NFV Service Density" - date: 2019-03 - CSIT-1901-nfv-density-results: - target: https://docs.fd.io/csit/rls1901/report/vpp_performance_tests/nf_service_density/index.html - title: "FD.io CSIT Test Results: NFV Service Density" - date: 2019-03 - CNCF-CNF-Testbed-Results: - target: https://github.com/cncf/cnf-testbed/blob/master/comparison/doc/cncf-cnfs-results-summary.md - title: "CNCF CNF Testbed: NFV Service Density Benchmarking" - date: 2018-12 - ---- abstract - -Network Function Virtualization (NFV) system designers and operators -continuously grapple with the problem of qualifying performance of -network services realised with software Network Functions (NF) running -on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges -is getting repeatable and portable benchmarking results and using them -to derive deterministic operating range that is production deployment -worthy. - -This document specifies benchmarking methodology for NFV services that -aims to address this problem space. It defines a way for measuring -performance of multiple NFV service instances, each composed of multiple -software NFs, and running them at a varied service “packing” density on -a single server. - -The aim is to discover deterministic usage range of NFV system. In -addition specified methodology can be used to compare and contrast -different NFV virtualization technologies. - ---- middle - -# Terminology - -* NFV - Network Function Virtualization, a general industry term - describing network functionality implemented in software. -* NFV service - a software based network service realized by a topology - of interconnected constituent software network function applications. -* NFV service instance - a single instantiation of NFV service. -* Data-plane optimized software - any software with dedicated threads - handling data-plane packet processing e.g. FD.io VPP (Vector Packet - Processor), OVS-DPDK. - -# Motivation - -## Problem Description - -Network Function Virtualization (NFV) system designers and operators -continuously grapple with the problem of qualifying performance of -network services realised with software Network Functions (NF) running -on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges -is getting repeatable and portable benchmarking results and using them -to derive deterministic operating range that is production deployment -worthy. - -Lack of well defined and standardised NFV centric performance -methodology and metrics makes it hard to address fundamental questions -that underpin NFV production deployments: - -1. What NFV service and how many instances can run on a single compute - node? -2. How to choose the best compute resource allocation scheme to maximise - service yield per node? -3. How do different NF applications compare from the service density - perspective? -4. How do the virtualisation technologies compare e.g. Virtual Machines, - Containers? - -Getting answers to these points should allow designers to make a data -based decision about the NFV technology and service design best suited -to meet requirements of their use cases. Equally, obtaining the -benchmarking data underpinning those answers should make it easier for -operators to work out expected deterministic operating range of chosen -design. - -## Proposed Solution - -The primary goal of the proposed benchmarking methodology is to focus on -NFV technologies used to construct NFV services. More specifically to i) -measure packet data-plane performance of multiple NFV service instances -while running them at varied service “packing” densities on a single -server and ii) quantify the impact of using multiple NFs to construct -each NFV service instance and introducing multiple packet processing -hops and links on each packet path. - -The overarching aim is to discover a set of deterministic usage ranges -that are of interest to NFV system designers and operators. In addition, -specified methodology can be used to compare and contrast different NFV -virtualisation technologies. - -In order to ensure wide applicability of the benchmarking methodology, -the approach is to separate NFV service packet processing from the -shared virtualisation infrastructure by decomposing the software -technology stack into three building blocks: - - +-------------------------------+ - | NFV Service | - +-------------------------------+ - | Virtualization Technology | - +-------------------------------+ - | Host Networking | - +-------------------------------+ - - Figure 1. NFV software technology stack. - -Proposed methodology is complementary to existing NFV benchmarking -industry efforts focusing on vSwitch benchmarking [RFC8204], [TST009] -and extends the benchmarking scope to NFV services. - -This document does not describe a complete benchmarking methodology, -instead it is focusing on system under test configuration part. Each of -the compute node configurations identified by (RowIndex, ColumnIndex) is -to be evaluated for NFV service data-plane performance using existing -and/or emerging network benchmarking standards. This may include -methodologies specified in [RFC2544], [TST009], -[draft-vpolak-mkonstan-bmwg-mlrsearch] and/or -[draft-vpolak-bmwg-plrsearch]. - -# NFV Service - -It is assumed that each NFV service instance is built of one or more -constituent NFs and is described by: topology, configuration and -resulting packet path(s). - -Each set of NFs forms an independent NFV service instance, with multiple -sets present in the host. - -## Topology - -NFV topology describes the number of network functions per service -instance, and their inter-connections over packet interfaces. It -includes all point-to-point virtual packet links within the compute -node, Layer-2 Ethernet or Layer-3 IP, including the ones to host -networking data-plane. - -Theoretically, a large set of possible NFV topologies can be realised -using software virtualisation topologies, e.g. ring, partial -/full- -mesh, star, line, tree, ladder. In practice however, only a few -topologies are in the actual use as NFV services mostly perform either -bumps-in-a-wire packet operations (e.g. security filtering/inspection, -monitoring/telemetry) and/or inter-site forwarding decisions (e.g. -routing, switching). - -Two main NFV topologies have been identified so far for NFV service -density benchmarking: - -1. Chain topology: a set of NFs connect to host data-plane with minimum - of two virtual interfaces each, enabling host data-plane to - facilitate NF to NF service chain forwarding and provide connectivity - with external network. - -2. Pipeline topology: a set of NFs connect to each other in a line - fashion with edge NFs homed to host data-plane. Host data-plane - provides connectivity with external network. - -Both topologies are shown in figures below. - -NF chain topology: - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1NF1 | | S1NF2 | | S1NFn | | - | | | | | .... | | Service1 | - | | | | | | | | - | +-+----+-+ +-+----+-+ + + +-+----+-+ | - | | | | | | | | | Virtual | - | | |<-CS->| |<-CS->| |<-CS->| | Interfaces | - | +-+----+------+----+------+----+------+----+-+ | - | | | CS: Chain | - | | | Segment | - | | Host Data-Plane | | - | +-+--+----------------------------------+--+-+ | - | | | | | | - +-----------------------------------------------------------+ - | | | | Physical - | | | | Interfaces - +---+--+----------------------------------+--+--------------+ - | | - | Traffic Generator | - | | - +-----------------------------------------------------------+ - - Figure 2. NF chain topology forming a service instance. - -NF pipeline topology: - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1NF1 | | S1NF2 | | S1NFn | | - | | +--+ +--+ .... +--+ | Service1 | - | | | | | | | | - | +-+------+ +--------+ +------+-+ | - | | | Virtual | - | |<-Pipeline Edge Pipeline Edge->| Interfaces | - | +-+----------------------------------------+-+ | - | | | | - | | | | - | | Host Data-Plane | | - | +-+--+----------------------------------+--+-+ | - | | | | | | - +-----------------------------------------------------------+ - | | | | Physical - | | | | Interfaces - +---+--+----------------------------------+--+--------------+ - | | - | Traffic Generator | - | | - +-----------------------------------------------------------+ - - Figure 3. NF pipeline topology forming a service instance. - - -## Configuration - -NFV configuration includes all packet processing functions in NFs -including Layer-2, Layer-3 and/or Layer-4-to-7 processing as appropriate -to specific NF and NFV service design. L2 sub- interface encapsulations -(e.g. 802.1q, 802.1ad) and IP overlay encapsulation (e.g. VXLAN, IPSec, -GRE) may be represented here too as appropriate, although in most cases -they are used as external encapsulation and handled by host networking -data-plane. - -NFV configuration determines logical network connectivity that is -Layer-2 and/or IPv4/IPv6 switching/routing modes, as well as NFV service -specific aspects. In the context of NFV density benchmarking methodology -the initial focus is on the former. - -Building on the two identified NFV topologies, two common NFV -configurations are considered: - -1. Chain configuration: - * Relies on chain topology to form NFV service chains. - * NF packet forwarding designs: - * IPv4/IPv6 routing. - * Requirements for host data-plane: - * L2 switching with L2 forwarding context per each NF chain - segment, or - * IPv4/IPv6 routing with IP forwarding context per each NF chain - segment or per NF chain. - -2. Pipeline configuration: - * Relies on pipeline topology to form NFV service pipelines. - * Packet forwarding designs: - * IPv4/IPv6 routing. - * Requirements for host data-plane: - * L2 switching with L2 forwarding context per each NF pipeline - edge link, or - * IPv4/IPv6 routing with IP forwarding context per each NF pipeline - edge link or per NF pipeline. - -## Packet Path(s) - -NFV packet path(s) describe the actual packet forwarding path(s) used -for benchmarking, resulting from NFV topology and configuration. They -are aimed to resemble true packet forwarding actions during the NFV -service lifecycle. - -Based on the specified NFV topologies and configurations two NFV packet -paths are taken for benchmarking: - -1. Snake packet path - * Requires chain topology and configuration. - * Packets enter the NFV chain through one edge NF and progress to the - other edge NF of the chain. - * Within the chain, packets follow a zigzagging "snake" path entering - and leaving host data-plane as they progress through the NF chain. - * Host data-plane is involved in packet forwarding operations between - NIC interfaces and edge NFs, as well as between NFs in the chain. - -2. Pipeline packet path - * Requires pipeline topology and configuration. - * Packets enter the NFV chain through one edge NF and progress to the - other edge NF of the pipeline. - * Within the chain, packets follow a straight path entering and - leaving subsequent NFs as they progress through the NF pipeline. - * Host data-plane is involved in packet forwarding operations between - NIC interfaces and edge NFs only. - -Both packet paths are shown in figures below. - -Snake packet path: - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1NF1 | | S1NF2 | | S1NFn | | - | | | | | .... | | Service1 | - | | XXXX | | XXXX | | XXXX | | - | +-+X--X+-+ +-+X--X+-+ +X X+ +-+X--X+-+ | - | |X X| |X X| |X X| |X X| Virtual | - | |X X| |X X| |X X| |X X| Interfaces | - | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | - | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | - | | X X | | - | | X Host Data-Plane X | | - | +-+X-+----------------------------------+-X+-+ | - | |X | | X| | - +----X--------------------------------------X---------------+ - |X | | X| Physical - |X | | X| Interfaces - +---+X-+----------------------------------+-X+--------------+ - | | - | Traffic Generator | - | | - +-----------------------------------------------------------+ - - Figure 4. Snake packet path thru NF chain topology. - - -Pipeline packet path: - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1NF1 | | S1NF2 | | S1NFn | | - | | +--+ +--+ .... +--+ | Service1 | - | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | - | +--X-----+ +--------+ +-----X--+ | - | |X X| Virtual | - | |X X| Interfaces | - | +-+X--------------------------------------X+-+ | - | | X X | | - | | X X | | - | | X Host Data-Plane X | | - | +-+X-+----------------------------------+-X+-+ | - | |X | | X| | - +----X--------------------------------------X---------------+ - |X | | X| Physical - |X | | X| Interfaces - +---+X-+----------------------------------+-X+--------------+ - | | - | Traffic Generator | - | | - +-----------------------------------------------------------+ - - Figure 5. Pipeline packet path thru NF pipeline topology. - -In all cases packets enter NFV system via shared physical NIC interfaces -controlled by shared host data-plane, are then associated with specific -NFV service (based on service discriminator) and subsequently are cross- -connected/switched/routed by host data-plane to and through NF -topologies per one of above listed schemes. - -# Virtualization Technology - -NFV services are built of composite isolated NFs, with virtualisation -technology providing the workload isolation. Following virtualisation -technology types are considered for NFV service density benchmarking: - -1. Virtual Machines (VMs) - * Relying on host hypervisor technology e.g. KVM, ESXi, Xen. - * NFs running in VMs are referred to as VNFs. -2. Containers - * Relying on Linux container technology e.g. LXC, Docker. - * NFs running in Containers are referred to as CNFs. - -Different virtual interface types are available to VNFs and CNFs: - -1. VNF - * virtio-vhostuser: fully user-mode based virtual interface. - * virtio-vhostnet: involves kernel-mode based backend. -2. CNF - * memif: fully user-mode based virtual interface. - * af_packet: involves kernel-mode based backend. - * (add more common ones) - -# Host Networking - -Host networking data-plane is the central shared resource that underpins -creation of NFV services. It handles all of the connectivity to external -physical network devices through physical network connections using -NICs, through which the benchmarking is done. - -Assuming that NIC interface resources are shared, here is the list of -widely available host data-plane options for providing packet -connectivity to/from NICs and constructing NFV chain and pipeline -topologies and configurations: - -* Linux Kernel-Mode Networking. -* Linux User-Mode vSwitch. -* Virtual Machine vSwitch. -* Linux Container vSwitch. -* SRIOV NIC Virtual Function - note: restricted support for chain and - pipeline topologies, as it requires hair-pinning through the NIC and - oftentimes also through external physical switch. - -Analysing properties of each of these options and their Pros/Cons for -specified NFV topologies and configurations is outside the scope of this -document. - -From all listed options, performance optimised Linux user-mode vswitch -deserves special attention. Linux user-mode switch decouples NFV service -from the underlying NIC hardware, offers rich multi-tenant functionality -and most flexibility for supporting NFV services. But in the same time -it is consuming compute resources and is harder to benchmark in NFV -service density scenarios. - -Following sections focus on using Linux user-mode vSwitch, focusing on -its performance benchmarking at increasing levels of NFV service -density. - -# NFV Service Density Matrix - -In order to evaluate performance of multiple NFV services running on a -compute node, NFV service instances are benchmarked at increasing -density, allowing to construct an NFV Service Density Matrix. Table -below shows an example of such a matrix, capturing number of NFV service -instances (row indices), number of NFs per service instance (column -indices) and resulting total number of NFs (values). - - NFV Service Density - NF Count View - - SVC 001 002 004 006 008 00N - 001 1 2 4 6 8 1*N - 002 2 4 8 12 16 2*N - 004 4 8 16 24 32 4*N - 006 6 12 24 36 48 6*N - 008 8 16 32 48 64 8*N - 00M M*1 M*2 M*4 M*6 M*8 M*N - - RowIndex: Number of NFV Service Instances, 1..M. - ColumnIndex: Number of NFs per NFV Service Instance, 1..N. - Value: Total number of NFs running in the system. - -In order to deliver good and repeatable network data-plane performance, -NFs and host data-plane software require direct access to critical -compute resources. Due to a shared nature of all resources on a compute -node, a clearly defined resource allocation scheme is defined in the -next section to address this. - -In each tested configuration host data-plane is a gateway between the -external network and the internal NFV network topologies. Offered packet -load is generated and received by an external traffic generator per -usual benchmarking practice. - -It is proposed that initial benchmarks are done with the offered packet -load distributed equally across all configured NFV service instances. -This could be followed by various per NFV service instance load ratios -mimicking expected production deployment scenario(s). - -Following sections specify compute resource allocation, followed by -examples of applying NFV service density methodology to VNF and CNF -benchmarking use cases. - -# Compute Resource Allocation - -Performance optimized NF and host data-plane software threads require -timely execution of packet processing instructions and are very -sensitive to any interruptions (or stalls) to this execution e.g. cpu -core context switching, or cpu jitter. To that end, NFV service density -methodology treats controlled mapping ratios of data plane software -threads to physical processor cores with directly allocated cache -hierarchies as the first order requirement. - -Other compute resources including memory bandwidth and PCIe bandwidth -have lesser impact and as such are subject for further study. For more -detail and deep-dive analysis of software data plane performance and -impact on different shared compute resources is available in [BSDP]. - -It is assumed that NFs as well as host data-plane (e.g. vswitch) are -performance optimized, with their tasks executed in two types of -software threads: - -* data-plane - handling data-plane packet processing and forwarding, - time critical, requires dedicated cores. To scale data-plane - performance, most NF apps use multiple data-plane threads and rely on - NIC RSS (Receive Side Scaling), virtual interface multi-queue and/or - integrated software hashing to distribute packets across the data - threads. - -* main-control - handling application management, statistics and - control-planes, less time critical, allows for core sharing. For most - NF apps this is a single main thread, but often statistics (counters) - and various control protocol software are run in separate threads. - -Core mapping scheme described below allocates cores for all threads of -specified type belonging to each NF app instance, and separately lists -number of threads to a number of logical/physical core mappings for -processor configurations with enabled/disabled Symmetric Multi- -Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading). - -If NFV service density benchmarking is run on server nodes with -Symmetric Multi-Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading) -for higher performance and efficiency, logical cores allocated to data- -plane threads should be allocated as pairs of sibling logical cores -corresponding to the hyper-threads running on the same physical core. - -Separate core ratios are defined for mapping threads of vSwitch and NFs. -In order to get consistent benchmarking results, the mapping ratios are -enforced using Linux core pinning. - -| application | thread type | app:core ratio | threads/pcores (SMT disabled) | threads/lcores map (SMT enabled) | -|:-----------:|:-----------:|:--------------:|:-------------------------------:|:----------------------------------:| -| vSwitch-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | -| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | -| | | | | | -| vSwitch-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | -| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | -| | | | | | -| vSwitch-4c | data | 1:4 | 4DT/4PC | 8DT/8LC | -| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | -| | | | | | -| NF-0.5c | data | 1:S2 | 1DT/S2PC | 1DT/1LC | -| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | -| | | | | | -| NF-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | -| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | -| | | | | | -| NF-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | -| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | - -* Legend to table - * Header row - * application - network application with optimized data-plane, a - vSwitch or Network Function (NF) application. - * thread type - either "data", short for data-plane; or "main", - short for all main-control threads. - * app:core ratio - ratio of per application instance threads of - specific thread type to physical cores. - * threads/pcores (SMT disabled) - number of threads of specific - type (DT for data-plane thread, MT for main thread) running on a - number of physical cores, with SMT disabled. - * threads/lcores map (SMT enabled) - number of threads of specific - type (DT, MT) running on a number of logical cores, with SMT - enabled. Two logical cores per one physical core. - * Content rows - * vSwitch-(1c|2c|4c) - vSwitch with 1 physical core (or 2, or 4) - allocated to its data-plane software worker threads. - * NF-(0.5c|1c|2c) - NF application with half of a physical core (or - 1, or 2) allocated to its data-plane software worker threads. - * Sn - shared core, sharing ratio of (n). - * DT - data-plane thread. - * MT - main-control thread. - * PC - physical core, with SMT/HT enabled has many (mostly 2 today) - logical cores associated with it. - * LC - logical core, if more than one lc get allocated in sets of - two sibling logical cores running on the same physical core. - * SnPC - shared physical core, sharing ratio of (n). - * SnLC - shared logical core, sharing ratio of (n). - -Maximum benchmarked NFV service densities are limited by a number of -physical cores on a compute node. - -A sample physical core usage view is shown in the matrix below. - - NFV Service Density - Core Usage View - vSwitch-1c, NF-1c - - SVC 001 002 004 006 008 010 - 001 2 3 6 9 12 15 - 002 3 6 12 18 24 30 - 004 6 12 24 36 48 60 - 006 9 18 36 54 72 90 - 008 12 24 48 72 96 120 - 010 15 30 60 90 120 150 - - RowIndex: Number of NFV Service Instances, 1..10. - ColumnIndex: Number of NFs per NFV Service Instance, 1..10. - Value: Total number of physical processor cores used for NFs. - -# NFV Service Density Benchmarks - -To illustrate defined NFV service density applicability, following -sections describe three sets of NFV service topologies and -configurations that have been benchmarked in open-source: i) in -[LFN-FDio-CSIT], a continuous testing and data-plane benchmarking -project, and ii) as part of CNCF CNF Testbed initiative -[CNCF-CNF-Testbed]. - -In both cases each NFV service instance definition is based on the same -set of NF applications, and varies only by network addressing -configuration to emulate multi-tenant operating environment. - -## Test Methodology - MRR Throughput - -Initial NFV density throughput benchmarks have been performed using -Maximum Receive Rate (MRR) test methodology defined and used in FD.io -CSIT. - -MRR tests measure the packet forwarding rate under the maximum load -offered by traffic generator over a set trial duration, regardless of -packet loss. Maximum load for specified Ethernet frame size is set to -the bi-directional link rate (2x 10GbE in referred results). - -Tests were conducted with two traffic profiles: i) continuous stream of -64B frames, ii) continuous stream of IMIX sequence of (7x 64B, 4x 570B, -1x 1518B), all sizes are L2 untagged Ethernet. - -NFV service topologies tested include: VNF service chains, CNF service -chains and CNF service pipelines. - -## VNF Service Chain - -VNF Service Chain (VSC) topology is tested with KVM hypervisor (Ubuntu -18.04-LTS), with NFV service instances consisting of NFs running in VMs -(VNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual -interfaces are virtio-vhostuser. Snake forwarding packet path is tested -using [TRex] traffic generator, see figure. - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1VNF1 | | S1VNF2 | | S1VNFn | | - | | | | | .... | | Service1 | - | | XXXX | | XXXX | | XXXX | | - | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | - | |X X| |X X| |X X| Virtual | - | |X X| |X X| |X X| |X X| Interfaces | - | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | - | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | - | | X X | | - | | X FD.io VPP vSwitch X | | - | +-+X-+----------------------------------+-X+-+ | - | |X | | X| | - +----X--------------------------------------X---------------+ - |X | | X| Physical - |X | | X| Interfaces - +---+X-+----------------------------------+-X+--------------+ - | | - | Traffic Generator (TRex) | - | | - +-----------------------------------------------------------+ - - Figure 6. VNF service chain test setup. - - -## CNF Service Chain - -CNF Service Chain (CSC) topology is tested with Docker containers -(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running -in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch. -Virtual interfaces are memif. Snake forwarding packet path is tested -using [TRex] traffic generator, see figure. - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1CNF1 | | S1CNF2 | | S1CNFn | | - | | | | | .... | | Service1 | - | | XXXX | | XXXX | | XXXX | | - | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | - | |X X| |X X| |X X| Virtual | - | |X X| |X X| |X X| |X X| Interfaces | - | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | - | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | - | | X X | | - | | X FD.io VPP vSwitch X | | - | +-+X-+----------------------------------+-X+-+ | - | |X | | X| | - +----X--------------------------------------X---------------+ - |X | | X| Physical - |X | | X| Interfaces - +---+X-+----------------------------------+-X+--------------+ - | | - | Traffic Generator (TRex) | - | | - +-----------------------------------------------------------+ - - Figure 7. CNF service chain test setup. - -## CNF Service Pipeline - -CNF Service Pipeline (CSP) topology is tested with Docker containers -(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running -in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch. -Virtual interfaces are memif. Pipeline forwarding packet path is tested -using [TRex] traffic generator, see figure. - - +-----------------------------------------------------------+ - | Host Compute Node | - | | - | +--------+ +--------+ +--------+ | - | | S1NF1 | | S1NF2 | | S1NFn | | - | | +--+ +--+ .... +--+ | Service1 | - | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | - | +--X-----+ +--------+ +-----X--+ | - | |X X| Virtual | - | |X X| Interfaces | - | +-+X--------------------------------------X+-+ | - | | X X | | - | | X X | | - | | X FD.io VPP vSwitch X | | - | +-+X-+----------------------------------+-X+-+ | - | |X | | X| | - +----X--------------------------------------X---------------+ - |X | | X| Physical - |X | | X| Interfaces - +---+X-+----------------------------------+-X+--------------+ - | | - | Traffic Generator (TRex) | - | | - +-----------------------------------------------------------+ - - Figure 8. CNF service chain test setup. - -## Sample Results: FD.io CSIT - -FD.io CSIT project introduced NFV density benchmarking in release -CSIT-1901 and published results for the following NFV service topologies -and configurations: - -1. VNF Service Chains - * VNF: DPDK-L3FWD v18.10 - * IPv4 forwarding - * NF-1c - * vSwitch: VPP v19.01-release - * L2 MAC switching - * vSwitch-1c, vSwitch-2c - * frame sizes: 64B, IMIX -2. CNF Service Chains - * CNF: VPP v19.01-release - * IPv4 routing - * NF-1c - * vSwitch: VPP v19.01-release - * L2 MAC switching - * vSwitch-1c, vSwitch-2c - * frame sizes: 64B, IMIX -3. CNF Service Pipelines - * CNF: VPP v19.01-release - * IPv4 routing - * NF-1c - * vSwitch: VPP v19.01-release - * L2 MAC switching - * vSwitch-1c, vSwitch-2c - * frame sizes: 64B, IMIX - -More information is available in FD.io CSIT-1901 report, with specific -references listed below: - -* Testbed: [CSIT-1901-testbed-2n-skx] -* Test environment: [CSIT-1901-test-enviroment] -* Methodology: [CSIT-1901-nfv-density-methodology] -* Results: [CSIT-1901-nfv-density-results] - -## Sample Results: CNCF/CNFs - -CNCF CI team introduced a CNF testbed initiative focusing on benchmaring -NFV density with open-source network applications running as VNFs and -CNFs. Following NFV service topologies and configurations have been -tested to date: - -1. VNF Service Chains - * VNF: VPP v18.10-release - * IPv4 routing - * NF-1c - * vSwitch: VPP v18.10-release - * L2 MAC switching - * vSwitch-1c, vSwitch-2c - * frame sizes: 64B, IMIX -2. CNF Service Chains - * CNF: VPP v18.10-release - * IPv4 routing - * NF-1c - * vSwitch: VPP v18.10-release - * L2 MAC switching - * vSwitch-1c, vSwitch-2c - * frame sizes: 64B, IMIX -3. CNF Service Pipelines - * CNF: VPP v18.10-release - * IPv4 routing - * NF-1c - * vSwitch: VPP v18.10-release - * L2 MAC switching - * vSwitch-1c, vSwitch-2c - * frame sizes: 64B, IMIX - -More information is available in CNCF CNF Testbed github, with summary -test results presented in summary markdown file, references listed -below: - -* Results: [CNCF-CNF-Testbed-Results] - -# IANA Considerations - -No requests of IANA - -# Security Considerations - -.. - -# Acknowledgements - -Thanks to Vratko Polak of FD.io CSIT project and Michael Pedersen of the -CNCF Testbed initiative for their contributions and useful suggestions. - ---- back \ No newline at end of file diff --git a/docs/ietf/draft-mkonstan-nf-service-density-01.md b/docs/ietf/draft-mkonstan-nf-service-density-01.md new file mode 100644 index 0000000000..7ab7876172 --- /dev/null +++ b/docs/ietf/draft-mkonstan-nf-service-density-01.md @@ -0,0 +1,983 @@ +--- +title: NFV Service Density Benchmarking +# abbrev: nf-svc-density +docname: draft-mkonstan-nf-service-density-01 +date: 2019-07-08 + +ipr: trust200902 +area: ops +wg: Benchmarking Working Group +kw: Internet-Draft +cat: info + +coding: us-ascii +pi: # can use array (if all yes) or hash here +# - toc +# - sortrefs +# - symrefs + toc: yes + sortrefs: # defaults to yes + symrefs: yes + +author: + - + ins: M. Konstantynowicz + name: Maciek Konstantynowicz + org: Cisco Systems + role: editor + email: mkonstan@cisco.com + - + ins: P. Mikus + name: Peter Mikus + org: Cisco Systems + role: editor + email: pmikus@cisco.com + +normative: + RFC2544: + RFC8174: + +informative: + RFC8204: + TST009: + target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.01.01_60/gs_NFV-TST009v030101p.pdf + title: "ETSI GS NFV-TST 009 V3.1.1 (2018-10), Network Functions Virtualisation (NFV) Release 3; Testing; Specification of Networking Benchmarks and Measurement Methods for NFVI" + date: 2018-10 + BSDP: + target: https://fd.io/wp-content/uploads/sites/34/2019/03/benchmarking_sw_data_planes_skx_bdx_mar07_2019.pdf + title: "Benchmarking Software Data Planes Intel® Xeon® Skylake vs. Broadwell" + date: 2019-03 + draft-vpolak-mkonstan-bmwg-mlrsearch: + target: https://tools.ietf.org/html/draft-vpolak-mkonstan-bmwg-mlrsearch + title: "Multiple Loss Ratio Search for Packet Throughput (MLRsearch)" + date: 2019-07 + draft-vpolak-bmwg-plrsearch: + target: https://tools.ietf.org/html/draft-vpolak-bmwg-plrsearch + title: "Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)" + date: 2019-07 + LFN-FDio-CSIT: + target: https://wiki.fd.io/view/CSIT + title: "Fast Data io, Continuous System Integration and Testing Project" + date: 2019-07 + CNCF-CNF-Testbed: + target: https://github.com/cncf/cnf-testbed/ + title: "Cloud native Network Function (CNF) Testbed" + date: 2019-07 + TRex: + target: https://github.com/cisco-system-traffic-generator/trex-core + title: "TRex Low-Cost, High-Speed Stateful Traffic Generator" + date: 2019-07 + CSIT-1904-testbed-2n-skx: + target: https://docs.fd.io/csit/rls1904/report/introduction/physical_testbeds.html#node-xeon-skylake-2n-skx + title: "FD.io CSIT Test Bed" + date: 2019-06 + CSIT-1904-test-enviroment: + target: https://docs.fd.io/csit/rls1904/report/vpp_performance_tests/test_environment.html + title: "FD.io CSIT Test Environment" + date: 2019-06 + CSIT-1904-nfv-density-methodology: + target: https://docs.fd.io/csit/rls1904/report/introduction/methodology_nfv_service_density.html + title: "FD.io CSIT Test Methodology: NFV Service Density" + date: 2019-06 + CSIT-1904-nfv-density-results: + target: https://docs.fd.io/csit/rls1904/report/vpp_performance_tests/nf_service_density/index.html + title: "FD.io CSIT Test Results: NFV Service Density" + date: 2019-06 + CNCF-CNF-Testbed-Results: + target: https://github.com/cncf/cnf-testbed/blob/master/comparison/doc/cncf-cnfs-results-summary.md + title: "CNCF CNF Testbed: NFV Service Density Benchmarking" + date: 2018-12 + NFVbench: + target: https://opnfv-nfvbench.readthedocs.io/en/latest/testing/user/userguide/readme.html + title: NFVbench Data Plane Performance Measurement Features + date: 2019-07 + +--- abstract + +Network Function Virtualization (NFV) system designers and operators +continuously grapple with the problem of qualifying performance of +network services realised with software Network Functions (NF) running +on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges +is getting repeatable and portable benchmarking results and using them +to derive deterministic operating range that is production deployment +worthy. + +This document specifies benchmarking methodology for NFV services that +aims to address this problem space. It defines a way for measuring +performance of multiple NFV service instances, each composed of multiple +software NFs, and running them at a varied service “packing” density on +a single server. + +The aim is to discover deterministic usage range of NFV system. In +addition specified methodology can be used to compare and contrast +different NFV virtualization technologies. + +--- middle + +# Terminology + +* NFV: Network Function Virtualization, a general industry term + describing network functionality implemented in software. +* NFV service: a software based network service realized by a topology + of interconnected constituent software network function applications. +* NFV service instance: a single instantiation of NFV service. +* Data-plane optimized software: any software with dedicated threads + handling data-plane packet processing e.g. FD.io VPP (Vector Packet + Processor), OVS-DPDK. +* Packet Loss Ratio (PLR): ratio of packets received relative to packets + transmitted over the test trial duration, calculated using formula: + PLR = ( pkts_transmitted - pkts_received ) / pkts_transmitted. + For bi-directional throughput tests aggregate PLR is calculated based + on the aggregate number of packets transmitted and received. +* Packet Throughput Rate: maximum packet offered load DUT/SUT forwards + within the specified Packet Loss Ratio (PLR). In many cases the rate + depends on the frame size processed by DUT/SUT. Hence packet + throughput rate MUST be quoted with specific frame size as received by + DUT/SUT during the measurement. For bi-directional tests, packet + throughput rate should be reported as aggregate for both directions. + Measured in packets-per-second (pps) or frames-per-second (fps), + equivalent metrics. +* Non Drop Rate (NDR): maximum packet/bandwith throughput rate sustained + by DUT/SUT at PLR equal zero (zero packet loss) specific to tested + frame size(s). MUST be quoted with specific packet size as received by + DUT/SUT during the measurement. Packet NDR measured in + packets-per-second (or fps), bandwidth NDR expressed in + bits-per-second (bps). +* Partial Drop Rate (PDR): maximum packet/bandwith throughput rate + sustained by DUT/SUT at PLR greater than zero (non-zero packet loss) + specific to tested frame size(s). MUST be quoted with specific packet + size as received by DUT/SUT during the measurement. Packet PDR + measured in packets-per-second (or fps), bandwidth PDR expressed in + bits-per-second (bps). +* Maximum Receive Rate (MRR): packet/bandwidth rate regardless of PLR + sustained by DUT/SUT under specified Maximum Transmit Rate (MTR) + packet load offered by traffic generator. MUST be quoted with both + specific packet size and MTR as received by DUT/SUT during the + measurement. Packet MRR measured in packets-per-second (or fps), + bandwidth MRR expressed in bits-per-second (bps). + +# Motivation + +## Problem Description + +Network Function Virtualization (NFV) system designers and operators +continuously grapple with the problem of qualifying performance of +network services realised with software Network Functions (NF) running +on Commercial-Off-The-Shelf (COTS) servers. One of the main challenges +is getting repeatable and portable benchmarking results and using them +to derive deterministic operating range that is production deployment +worthy. + +Lack of well defined and standardised NFV centric performance +methodology and metrics makes it hard to address fundamental questions +that underpin NFV production deployments: + +1. What NFV service and how many instances can run on a single compute + node? +2. How to choose the best compute resource allocation scheme to maximise + service yield per node? +3. How do different NF applications compare from the service density + perspective? +4. How do the virtualisation technologies compare e.g. Virtual Machines, + Containers? + +Getting answers to these points should allow designers to make data +based decisions about the NFV technology and service design best suited +to meet requirements of their use cases. Thereby obtained benchmarking +data would aid in selection of the most appropriate NFV infrastructure +design and platform and enable more accurate capacity planning, an +important element for commercial viability of the NFV service. + +## Proposed Solution + +The primary goal of the proposed benchmarking methodology is to focus on +NFV technologies used to construct NFV services. More specifically to i) +measure packet data-plane performance of multiple NFV service instances +while running them at varied service “packing” densities on a single +server and ii) quantify the impact of using multiple NFs to construct +each NFV service instance and introducing multiple packet processing +hops and links on each packet path. + +The overarching aim is to discover a set of deterministic usage ranges +that are of interest to NFV system designers and operators. In addition, +specified methodology can be used to compare and contrast different NFV +virtualisation technologies. + +In order to ensure wide applicability of the benchmarking methodology, +the approach is to separate NFV service packet processing from the +shared virtualisation infrastructure by decomposing the software +technology stack into three building blocks: + + +-------------------------------+ + | NFV Service | + +-------------------------------+ + | Virtualization Technology | + +-------------------------------+ + | Host Networking | + +-------------------------------+ + + Figure 1. NFV software technology stack. + +Proposed methodology is complementary to existing NFV benchmarking +industry efforts focusing on vSwitch benchmarking [RFC8204], [TST009] +and extends the benchmarking scope to NFV services. + +This document does not describe a complete benchmarking methodology, +instead it is focusing on the system under test configuration. Each of +the compute node configurations identified in this document is +to be evaluated for NFV service data-plane performance using existing +and/or emerging network benchmarking standards. This may include +methodologies specified in [RFC2544], [TST009], +[draft-vpolak-mkonstan-bmwg-mlrsearch] and/or +[draft-vpolak-bmwg-plrsearch]. + +# NFV Service + +It is assumed that each NFV service instance is built of one or more +constituent NFs and is described by: topology, configuration and +resulting packet path(s). + +Each set of NFs forms an independent NFV service instance, with multiple +sets present in the host. + +## Topology + +NFV topology describes the number of network functions per service +instance, and their inter-connections over packet interfaces. It +includes all point-to-point virtual packet links within the compute +node, Layer-2 Ethernet or Layer-3 IP, including the ones to host +networking data-plane. + +Theoretically, a large set of possible NFV topologies can be realised +using software virtualisation topologies, e.g. ring, partial -/full- +mesh, star, line, tree, ladder. In practice however, only a few +topologies are in the actual use as NFV services mostly perform either +bumps-in-a-wire packet operations (e.g. security filtering/inspection, +monitoring/telemetry) and/or inter-site forwarding decisions (e.g. +routing, switching). + +Two main NFV topologies have been identified so far for NFV service +density benchmarking: + +1. Chain topology: a set of NFs connect to host data-plane with minimum + of two virtual interfaces each, enabling host data-plane to + facilitate NF to NF service chain forwarding and provide connectivity + with external network. + +2. Pipeline topology: a set of NFs connect to each other in a line + fashion with edge NFs homed to host data-plane. Host data-plane + provides connectivity with external network. + +In both cases multiple NFV service topologies are running in parallel. +Both topologies are shown in figures 2. and 3. below. + +NF chain topology: + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | SmNF1 SmNF2 SmNFn Service-m | + | ... ... ... ... | + | S2NF1 S2NF2 S2NFn Service-2 | + | +--------+ +--------+ +--------+ | + | | S1NF1 | | S1NF2 | | S1NFn | | + | | | | | .... | | Service-1 | + | | | | | | | | + | +-+----+-+ +-+----+-+ + + +-+----+-+ | + | | | | | | | | | Virtual | + | | |<-CS->| |<-CS->| |<-CS->| | Interfaces | + | +-+----+------+----+------+----+------+----+-+ | + | | | CS: Chain | + | | | Segment | + | | Host Data-Plane | | + | +-+--+----------------------------------+--+-+ | + | | | | | | + +-----------------------------------------------------------+ + | | | | Physical + | | | | Interfaces + +---+--+----------------------------------+--+--------------+ + | | + | Traffic Generator | + | | + +-----------------------------------------------------------+ + + Figure 2. NF chain topology forming a service instance. + +NF pipeline topology: + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | SmNF1 SmNF2 SmNFn Service-m | + | ... ... ... ... | + | S2NF1 S2NF2 S2NFn Service-2 | + | +--------+ +--------+ +--------+ | + | | S1NF1 | | S1NF2 | | S1NFn | | + | | +--+ +--+ .... +--+ | Service1 | + | | | | | | | | + | +-+------+ +--------+ +------+-+ | + | | | Virtual | + | |<-Pipeline Edge Pipeline Edge->| Interfaces | + | +-+----------------------------------------+-+ | + | | | | + | | | | + | | Host Data-Plane | | + | +-+--+----------------------------------+--+-+ | + | | | | | | + +-----------------------------------------------------------+ + | | | | Physical + | | | | Interfaces + +---+--+----------------------------------+--+--------------+ + | | + | Traffic Generator | + | | + +-----------------------------------------------------------+ + + Figure 3. NF pipeline topology forming a service instance. + + +## Configuration + +NFV configuration includes all packet processing functions in NFs +including Layer-2, Layer-3 and/or Layer-4-to-7 processing as appropriate +to specific NF and NFV service design. L2 sub- interface encapsulations +(e.g. 802.1q, 802.1ad) and IP overlay encapsulation (e.g. VXLAN, IPSec, +GRE) may be represented here too as appropriate, although in most cases +they are used as external encapsulation and handled by host networking +data-plane. + +NFV configuration determines logical network connectivity that is +Layer-2 and/or IPv4/IPv6 switching/routing modes, as well as NFV service +specific aspects. In the context of NFV density benchmarking methodology +the initial focus is on logical network connectivity between the NFs, +and no NFV service specific configurations. NF specific functionality is +emulated using IPv4/IPv6 routing. + +Building on the two identified NFV topologies, two common NFV +configurations are considered: + +1. Chain configuration: + * Relies on chain topology to form NFV service chains. + * NF packet forwarding designs: + * IPv4/IPv6 routing. + * Requirements for host data-plane: + * L2 switching with L2 forwarding context per each NF chain + segment, or + * IPv4/IPv6 routing with IP forwarding context per each NF chain + segment or per NF chain. + +2. Pipeline configuration: + * Relies on pipeline topology to form NFV service pipelines. + * Packet forwarding designs: + * IPv4/IPv6 routing. + * Requirements for host data-plane: + * L2 switching with L2 forwarding context per each NF pipeline + edge link, or + * IPv4/IPv6 routing with IP forwarding context per each NF pipeline + edge link or per NF pipeline. + +## Packet Path(s) + +NFV packet path(s) describe the actual packet forwarding path(s) used +for benchmarking, resulting from NFV topology and configuration. They +are aimed to resemble true packet forwarding actions during the NFV +service lifecycle. + +Based on the specified NFV topologies and configurations two NFV packet +paths are taken for benchmarking: + +1. Snake packet path + * Requires chain topology and configuration. + * Packets enter the NFV chain through one edge NF and progress to the + other edge NF of the chain. + * Within the chain, packets follow a zigzagging "snake" path entering + and leaving host data-plane as they progress through the NF chain. + * Host data-plane is involved in packet forwarding operations between + NIC interfaces and edge NFs, as well as between NFs in the chain. + +2. Pipeline packet path + * Requires pipeline topology and configuration. + * Packets enter the NFV chain through one edge NF and progress to the + other edge NF of the pipeline. + * Within the chain, packets follow a straight path entering and + leaving subsequent NFs as they progress through the NF pipeline. + * Host data-plane is involved in packet forwarding operations between + NIC interfaces and edge NFs only. + +Both packet paths are shown in figures below. + +Snake packet path: + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | SmNF1 SmNF2 SmNFn Service-m | + | ... ... ... ... | + | S2NF1 S2NF2 S2NFn Service-2 | + | +--------+ +--------+ +--------+ | + | | S1NF1 | | S1NF2 | | S1NFn | | + | | | | | .... | | Service1 | + | | XXXX | | XXXX | | XXXX | | + | +-+X--X+-+ +-+X--X+-+ +X X+ +-+X--X+-+ | + | |X X| |X X| |X X| |X X| Virtual | + | |X X| |X X| |X X| |X X| Interfaces | + | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | + | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | + | | X X | | + | | X Host Data-Plane X | | + | +-+X-+----------------------------------+-X+-+ | + | |X | | X| | + +----X--------------------------------------X---------------+ + |X | | X| Physical + |X | | X| Interfaces + +---+X-+----------------------------------+-X+--------------+ + | | + | Traffic Generator | + | | + +-----------------------------------------------------------+ + + Figure 4. Snake packet path thru NF chain topology. + + +Pipeline packet path: + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | SmNF1 SmNF2 SmNFn Service-m | + | ... ... ... ... | + | S2NF1 S2NF2 S2NFn Service-2 | + | +--------+ +--------+ +--------+ | + | | S1NF1 | | S1NF2 | | S1NFn | | + | | +--+ +--+ .... +--+ | Service1 | + | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | + | +--X-----+ +--------+ +-----X--+ | + | |X X| Virtual | + | |X X| Interfaces | + | +-+X--------------------------------------X+-+ | + | | X X | | + | | X X | | + | | X Host Data-Plane X | | + | +-+X-+----------------------------------+-X+-+ | + | |X | | X| | + +----X--------------------------------------X---------------+ + |X | | X| Physical + |X | | X| Interfaces + +---+X-+----------------------------------+-X+--------------+ + | | + | Traffic Generator | + | | + +-----------------------------------------------------------+ + + Figure 5. Pipeline packet path thru NF pipeline topology. + +In all cases packets enter NFV system via shared physical NIC interfaces +controlled by shared host data-plane, are then associated with specific +NFV service (based on service discriminator) and subsequently are cross- +connected/switched/routed by host data-plane to and through NF +topologies per one of the above listed schemes. + +# Virtualization Technology + +NFV services are built of composite isolated NFs, with virtualisation +technology providing the workload isolation. Following virtualisation +technology types are considered for NFV service density benchmarking: + +1. Virtual Machines (VMs) + * Relying on host hypervisor technology e.g. KVM, ESXi, Xen. + * NFs running in VMs are referred to as VNFs. +2. Containers + * Relying on Linux container technology e.g. LXC, Docker. + * NFs running in Containers are referred to as CNFs. + +Different virtual interface types are available to VNFs and CNFs: + +1. VNF + * virtio-vhostuser: fully user-mode based virtual interface. + * virtio-vhostnet: involves kernel-mode based backend. +2. CNF + * memif: fully user-mode based virtual interface. + * af_packet: involves kernel-mode based backend. + * (add more common ones) + +# Host Networking + +Host networking data-plane is the central shared resource that underpins +creation of NFV services. It handles all of the connectivity to external +physical network devices through physical network connections using +NICs, through which the benchmarking is done. + +Assuming that NIC interface resources are shared, here is the list of +widely available host data-plane options for providing packet +connectivity to/from NICs and constructing NFV chain and pipeline +topologies and configurations: + +* Linux Kernel-Mode Networking. +* Linux User-Mode vSwitch. +* Virtual Machine vSwitch. +* Linux Container vSwitch. +* SRIOV NIC Virtual Function - note: restricted support for chain and + pipeline topologies, as it requires hair-pinning through the NIC and + oftentimes also through external physical switch. + +Analysing properties of each of these options and their Pros/Cons for +specified NFV topologies and configurations is outside the scope of this +document. + +From all listed options, performance optimised Linux user-mode vswitch +deserves special attention. Linux user-mode switch decouples NFV service +from the underlying NIC hardware, offers rich multi-tenant functionality +and most flexibility for supporting NFV services. But in the same time +it is consuming compute resources and is harder to benchmark in NFV +service density scenarios. + +Following sections focus on using Linux user-mode vSwitch, focusing on +its performance benchmarking at increasing levels of NFV service +density. + +# NFV Service Density Matrix + +In order to evaluate performance of multiple NFV services running on a +compute node, NFV service instances are benchmarked at increasing +density, allowing to construct an NFV Service Density Matrix. Table +below shows an example of such a matrix, capturing number of NFV service +instances (row indices), number of NFs per service instance (column +indices) and resulting total number of NFs (values). + + NFV Service Density - NF Count View + + SVC 001 002 004 006 008 00N + 001 1 2 4 6 8 1*N + 002 2 4 8 12 16 2*N + 004 4 8 16 24 32 4*N + 006 6 12 24 36 48 6*N + 008 8 16 32 48 64 8*N + 00M M*1 M*2 M*4 M*6 M*8 M*N + + RowIndex: Number of NFV Service Instances, 1..M. + ColumnIndex: Number of NFs per NFV Service Instance, 1..N. + Value: Total number of NFs running in the system. + +In order to deliver good and repeatable network data-plane performance, +NFs and host data-plane software require direct access to critical +compute resources. Due to a shared nature of all resources on a compute +node, a clearly defined resource allocation scheme is defined in the +next section to address this. + +In each tested configuration host data-plane is a gateway between the +external network and the internal NFV network topologies. Offered packet +load is generated and received by an external traffic generator per +usual benchmarking practice. + +It is proposed that benchmarks are done with the offered packet load +distributed equally across all configured NFV service instances. +This approach should provide representative benchmarking data for each +tested topology and configuraiton, and a good guesstimate of maximum +performance required for capacity planning. + +Following sections specify compute resource allocation, followed by +examples of applying NFV service density methodology to VNF and CNF +benchmarking use cases. + +# Compute Resource Allocation + +Performance optimized NF and host data-plane software threads require +timely execution of packet processing instructions and are very +sensitive to any interruptions (or stalls) to this execution e.g. cpu +core context switching, or cpu jitter. To that end, NFV service density +methodology treats controlled mapping ratios of data plane software +threads to physical processor cores with directly allocated cache +hierarchies as the first order requirement. + +Other compute resources including memory bandwidth and PCIe bandwidth +have lesser impact and as such are subject for further study. For more +detail and deep-dive analysis of software data plane performance and +impact on different shared compute resources is available in [BSDP]. + +It is assumed that NFs as well as host data-plane (e.g. vswitch) are +performance optimized, with their tasks executed in two types of +software threads: + +* data-plane - handling data-plane packet processing and forwarding, + time critical, requires dedicated cores. To scale data-plane + performance, most NF apps use multiple data-plane threads and rely on + NIC RSS (Receive Side Scaling), virtual interface multi-queue and/or + integrated software hashing to distribute packets across the data + threads. + +* main-control - handling application management, statistics and + control-planes, less time critical, allows for core sharing. For most + NF apps this is a single main thread, but often statistics (counters) + and various control protocol software are run in separate threads. + +Core mapping scheme described below allocates cores for all threads of +specified type belonging to each NF app instance, and separately lists +number of threads to a number of logical/physical core mappings for +processor configurations with enabled/disabled Symmetric Multi- +Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading). + +If NFV service density benchmarking is run on server nodes with +Symmetric Multi-Threading (SMT) (e.g. AMD SMT, Intel Hyper-Threading) +for higher performance and efficiency, logical cores allocated to data- +plane threads should be allocated as pairs of sibling logical cores +corresponding to the hyper-threads running on the same physical core. + +Separate core ratios are defined for mapping threads of vSwitch and NFs. +In order to get consistent benchmarking results, the mapping ratios are +enforced using Linux core pinning. + +| application | thread type | app:core ratio | threads/pcores (SMT disabled) | threads/lcores map (SMT enabled) | +|:-----------:|:-----------:|:--------------:|:-------------------------------:|:----------------------------------:| +| vSwitch-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | +| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | +| | | | | | +| vSwitch-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | +| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | +| | | | | | +| vSwitch-4c | data | 1:4 | 4DT/4PC | 8DT/8LC | +| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | +| | | | | | +| NF-0.5c | data | 1:S2 | 1DT/S2PC | 1DT/1LC | +| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | +| | | | | | +| NF-1c | data | 1:1 | 1DT/1PC | 2DT/2LC | +| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | +| | | | | | +| NF-2c | data | 1:2 | 2DT/2PC | 4DT/4LC | +| | main | 1:S2 | 1MT/S2PC | 1MT/1LC | + +* Legend to table + * Header row + * application - network application with optimized data-plane, a + vSwitch or Network Function (NF) application. + * thread type - either "data", short for data-plane; or "main", + short for all main-control threads. + * app:core ratio - ratio of per application instance threads of + specific thread type to physical cores. + * threads/pcores (SMT disabled) - number of threads of specific + type (DT for data-plane thread, MT for main thread) running on a + number of physical cores, with SMT disabled. + * threads/lcores map (SMT enabled) - number of threads of specific + type (DT, MT) running on a number of logical cores, with SMT + enabled. Two logical cores per one physical core. + * Content rows + * vSwitch-(1c|2c|4c) - vSwitch with 1 physical core (or 2, or 4) + allocated to its data-plane software worker threads. + * NF-(0.5c|1c|2c) - NF application with half of a physical core (or + 1, or 2) allocated to its data-plane software worker threads. + * Sn - shared core, sharing ratio of (n). + * DT - data-plane thread. + * MT - main-control thread. + * PC - physical core, with SMT/HT enabled has many (mostly 2 today) + logical cores associated with it. + * LC - logical core, if more than one lc get allocated in sets of + two sibling logical cores running on the same physical core. + * SnPC - shared physical core, sharing ratio of (n). + * SnLC - shared logical core, sharing ratio of (n). + +Maximum benchmarked NFV service densities are limited by a number of +physical cores on a compute node. + +A sample physical core usage view is shown in the matrix below. + + NFV Service Density - Core Usage View + vSwitch-1c, NF-1c + + SVC 001 002 004 006 008 010 + 001 2 3 6 9 12 15 + 002 3 6 12 18 24 30 + 004 6 12 24 36 48 60 + 006 9 18 36 54 72 90 + 008 12 24 48 72 96 120 + 010 15 30 60 90 120 150 + + RowIndex: Number of NFV Service Instances, 1..10. + ColumnIndex: Number of NFs per NFV Service Instance, 1..10. + Value: Total number of physical processor cores used for NFs. + +# NFV Service Data-Plane Benchmarking + +NF service density scenarios should have their data-plane performance +benchmarked using existing and/or emerging network benchmarking +standards as noted earlier. + +Following metrics should be measured (or calculated) and reported: + +* Packet throughput rate (packets-per-second) + * Specific to tested packet size or packet sequence (e.g. some type of + packet size mix sent in recurrent sequence). + * Applicable types of throughput rate: NDR, PDR, MRR. +* (Calculated) Bandwidth throughput rate (bits-per-second) corresponding + to the measured packet throughput rate. +* Packet one-way latency (seconds) + * Measured at different packet throughput rates load e.g. light, + medium, heavy. + +Listed metrics should be itemized per service instance and per direction +(e.g. forward/reverse) for latency. + +# Sample NFV Service Density Benchmarks + +To illustrate defined NFV service density applicability, following +sections describe three sets of NFV service topologies and +configurations that have been benchmarked in open-source: i) in +[LFN-FDio-CSIT], a continuous testing and data-plane benchmarking +project, ii) as part of CNCF CNF Testbed initiative [CNCF-CNF-Testbed] +and iii) in OPNFV NFVbench project. + +In the first two cases each NFV service instance definition is based on +the same set of NF applications, and varies only by network addressing +configuration to emulate multi-tenant operating environment. + +OPNFV NFVbench project is focusing on benchmarking the actual production +deployments that are aligned with OPNFV specifications. + +## Intrepreting the Sample Results + +TODO How to interpret and avoid misreading included results? And how to +avoid falling into the trap of using these results to draw generilized +conclusions about performance of different virtualization technologies, +e.g. VM and Containers, irrespective of deployment scenarios and what +VNFs and CNFs are in the actual use. + +## Benchmarking MRR Throughput + +Initial NFV density throughput benchmarks have been performed using +Maximum Receive Rate (MRR) test methodology defined and used in FD.io +CSIT. + +MRR tests measure the packet forwarding rate under specified Maximum +Transmit Rate (MTR) packet load offered by traffic generator over a set +trial duration, regardless of packet loss ratio (PLR). MTR for specified +Ethernet frame size was set to the bi-directional link rate, 2x 10GbE in +referred results. + +Tests were conducted with two traffic profiles: i) continuous stream of +64B frames, ii) continuous stream of IMIX sequence of (7x 64B, 4x 570B, +1x 1518B), all sizes are L2 untagged Ethernet. + +NFV service topologies tested include: VNF service chains, CNF service +chains and CNF service pipelines. + +## VNF Service Chain + +VNF Service Chain (VSC) topology is tested with KVM hypervisor (Ubuntu +18.04-LTS), with NFV service instances consisting of NFs running in VMs +(VNFs). Host data-plane is provided by FD.io VPP vswitch. Virtual +interfaces are virtio-vhostuser. Snake forwarding packet path is tested +using [TRex] traffic generator, see figure. + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | +--------+ +--------+ +--------+ | + | | S1VNF1 | | S1VNF2 | | S1VNFn | | + | | | | | .... | | Service1 | + | | XXXX | | XXXX | | XXXX | | + | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | + | |X X| |X X| |X X| Virtual | + | |X X| |X X| |X X| |X X| Interfaces | + | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | + | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | + | | X X | | + | | X FD.io VPP vSwitch X | | + | +-+X-+----------------------------------+-X+-+ | + | |X | | X| | + +----X--------------------------------------X---------------+ + |X | | X| Physical + |X | | X| Interfaces + +---+X-+----------------------------------+-X+--------------+ + | | + | Traffic Generator (TRex) | + | | + +-----------------------------------------------------------+ + + Figure 6. VNF service chain test setup. + + +## CNF Service Chain + +CNF Service Chain (CSC) topology is tested with Docker containers +(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running +in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch. +Virtual interfaces are memif. Snake forwarding packet path is tested +using [TRex] traffic generator, see figure. + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | +--------+ +--------+ +--------+ | + | | S1CNF1 | | S1CNF2 | | S1CNFn | | + | | | | | .... | | Service1 | + | | XXXX | | XXXX | | XXXX | | + | +-+X--X+-+ +-+X--X+-+ +-+X--X+-+ | + | |X X| |X X| |X X| Virtual | + | |X X| |X X| |X X| |X X| Interfaces | + | +-+X--X+------+X--X+------+X--X+------+X--X+-+ | + | | X XXXXXXXXXX XXXXXXXXXX XXXXXXXXXX X | | + | | X X | | + | | X FD.io VPP vSwitch X | | + | +-+X-+----------------------------------+-X+-+ | + | |X | | X| | + +----X--------------------------------------X---------------+ + |X | | X| Physical + |X | | X| Interfaces + +---+X-+----------------------------------+-X+--------------+ + | | + | Traffic Generator (TRex) | + | | + +-----------------------------------------------------------+ + + Figure 7. CNF service chain test setup. + +## CNF Service Pipeline + +CNF Service Pipeline (CSP) topology is tested with Docker containers +(Ubuntu 18.04-LTS), with NFV service instances consisting of NFs running +in Containers (CNFs). Host data-plane is provided by FD.io VPP vswitch. +Virtual interfaces are memif. Pipeline forwarding packet path is tested +using [TRex] traffic generator, see figure. + + +-----------------------------------------------------------+ + | Host Compute Node | + | | + | +--------+ +--------+ +--------+ | + | | S1NF1 | | S1NF2 | | S1NFn | | + | | +--+ +--+ .... +--+ | Service1 | + | | XXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXX | | + | +--X-----+ +--------+ +-----X--+ | + | |X X| Virtual | + | |X X| Interfaces | + | +-+X--------------------------------------X+-+ | + | | X X | | + | | X X | | + | | X FD.io VPP vSwitch X | | + | +-+X-+----------------------------------+-X+-+ | + | |X | | X| | + +----X--------------------------------------X---------------+ + |X | | X| Physical + |X | | X| Interfaces + +---+X-+----------------------------------+-X+--------------+ + | | + | Traffic Generator (TRex) | + | | + +-----------------------------------------------------------+ + + Figure 8. CNF service chain test setup. + +## Sample Results: FD.io CSIT + +FD.io CSIT project introduced NFV density benchmarking in release +CSIT-1904 and published results for the following NFV service topologies +and configurations: + +1. VNF Service Chains + * VNF: DPDK-L3FWD v19.02 + * IPv4 forwarding + * NF-1c + * vSwitch: VPP v19.04-release + * L2 MAC switching + * vSwitch-1c, vSwitch-2c + * frame sizes: 64B, IMIX +2. CNF Service Chains + * CNF: VPP v19.04-release + * IPv4 routing + * NF-1c + * vSwitch: VPP v19.04-release + * L2 MAC switching + * vSwitch-1c, vSwitch-2c + * frame sizes: 64B, IMIX +3. CNF Service Pipelines + * CNF: VPP v19.04-release + * IPv4 routing + * NF-1c + * vSwitch: VPP v19.04-release + * L2 MAC switching + * vSwitch-1c, vSwitch-2c + * frame sizes: 64B, IMIX + +More information is available in FD.io CSIT-1904 report, with specific +references listed below: + +* Testbed: [CSIT-1904-testbed-2n-skx] +* Test environment: [CSIT-1904-test-enviroment] +* Methodology: [CSIT-1904-nfv-density-methodology] +* Results: [CSIT-1904-nfv-density-results] + +## Sample Results: CNCF/CNFs + +CNCF CI team introduced a CNF testbed initiative focusing on benchmaring +NFV density with open-source network applications running as VNFs and +CNFs. Following NFV service topologies and configurations have been +tested to date: + +1. VNF Service Chains + * VNF: VPP v18.10-release + * IPv4 routing + * NF-1c + * vSwitch: VPP v18.10-release + * L2 MAC switching + * vSwitch-1c, vSwitch-2c + * frame sizes: 64B, IMIX +2. CNF Service Chains + * CNF: VPP v18.10-release + * IPv4 routing + * NF-1c + * vSwitch: VPP v18.10-release + * L2 MAC switching + * vSwitch-1c, vSwitch-2c + * frame sizes: 64B, IMIX +3. CNF Service Pipelines + * CNF: VPP v18.10-release + * IPv4 routing + * NF-1c + * vSwitch: VPP v18.10-release + * L2 MAC switching + * vSwitch-1c, vSwitch-2c + * frame sizes: 64B, IMIX + +More information is available in CNCF CNF Testbed github, with summary +test results presented in summary markdown file, references listed +below: + +* Results: [CNCF-CNF-Testbed-Results] + +## Sample Results: OPNFV NFVbench + +TODO Add short NFVbench based test description, and NFVbench sweep chart +with single VM per service instance: Y-axis packet throughput rate or +bandwidth throughput rate, X-axis number of concurrent service +instances. + +# IANA Considerations + +No requests of IANA. + +# Security Considerations + +Benchmarking activities as described in this memo are limited to +technology characterization of a DUT/SUT using controlled stimuli in a +laboratory environment, with dedicated address space and the constraints +specified in the sections above. + +The benchmarking network topology will be an independent test setup and +MUST NOT be connected to devices that may forward the test traffic into +a production network or misroute traffic to the test management network. + +Further, benchmarking is performed on a "black-box" basis, relying +solely on measurements observable external to the DUT/SUT. + +Special capabilities SHOULD NOT exist in the DUT/SUT specifically for +benchmarking purposes. Any implications for network security arising +from the DUT/SUT SHOULD be identical in the lab and in production +networks. + +# Acknowledgements + +Thanks to Vratko Polak of FD.io CSIT project and Michael Pedersen of the +CNCF Testbed initiative for their contributions and useful suggestions. +Extended thanks to Alec Hothan of OPNFV NFVbench project for numerous +comments, suggestions and references to his/team work in the +OPNFV/NVFbench project. + +--- back \ No newline at end of file -- cgit