diff options
Diffstat (limited to 'doc/guides')
200 files changed, 18406 insertions, 8423 deletions
diff --git a/doc/guides/bbdevs/null.rst b/doc/guides/bbdevs/null.rst index 9baf2a99..0b885d17 100644 --- a/doc/guides/bbdevs/null.rst +++ b/doc/guides/bbdevs/null.rst @@ -4,7 +4,7 @@ BBDEV null Poll Mode Driver ============================ -The (**bbdev_null**) is a bbdev poll mode driver which provides a minimal +The (**baseband_null**) is a bbdev poll mode driver which provides a minimal implementation of a software bbdev device. As a null device it does not modify the data in the mbuf on which the bbdev operation is to operate and it only works for operation type ``RTE_BBDEV_OP_NONE``. @@ -30,9 +30,9 @@ Initialization To use the PMD in an application, user must: -- Call ``rte_vdev_init("bbdev_null")`` within the application. +- Call ``rte_vdev_init("baseband_null")`` within the application. -- Use ``--vdev="bbdev_null"`` in the EAL options, which will call ``rte_vdev_init()`` internally. +- Use ``--vdev="baseband_null"`` in the EAL options, which will call ``rte_vdev_init()`` internally. The following parameters (all optional) can be provided in the previous two calls: @@ -46,4 +46,4 @@ Example: .. code-block:: console - ./test-bbdev.py -e="--vdev=bbdev_null,socket_id=0,max_nb_queues=8" + ./test-bbdev.py -e="--vdev=baseband_null,socket_id=0,max_nb_queues=8" diff --git a/doc/guides/bbdevs/turbo_sw.rst b/doc/guides/bbdevs/turbo_sw.rst index b3fed163..0b96fbb1 100644 --- a/doc/guides/bbdevs/turbo_sw.rst +++ b/doc/guides/bbdevs/turbo_sw.rst @@ -4,7 +4,7 @@ SW Turbo Poll Mode Driver ========================= -The SW Turbo PMD (**turbo_sw**) provides a poll mode bbdev driver that utilizes +The SW Turbo PMD (**baseband_turbo_sw**) provides a poll mode bbdev driver that utilizes Intel optimized libraries for LTE Layer 1 workloads acceleration. This PMD supports the functions: Turbo FEC, Rate Matching and CRC functions. @@ -26,6 +26,8 @@ For the decode operation: * ``RTE_BBDEV_TURBO_CRC_TYPE_24B`` * ``RTE_BBDEV_TURBO_POS_LLR_1_BIT_IN`` * ``RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN`` +* ``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` +* ``RTE_BBDEV_TURBO_EARLY_TERMINATION`` Limitations @@ -39,25 +41,39 @@ Installation FlexRAN SDK Download ~~~~~~~~~~~~~~~~~~~~ -To build DPDK with the *turbo_sw* PMD the user is required to download -the export controlled ``FlexRAN SDK`` Libraries. An account at Intel Resource -Design Center needs to be registered from -`<https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_. +To build DPDK with the *baseband_turbo_sw* PMD the user is required to download +the export controlled ``FlexRAN SDK`` Libraries. An account at `Intel Resource +Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_ +needs to be registered. Once registered, the user needs to log in, and look for -*Intel SWA_SW_FlexRAN_Release_Package R1_3_0* and click for download. Or use -this direct download link `<https://cdrd.intel.com/v1/dl/getContent/575367>`_. +*Intel FlexRAN Software Release Package -1-6-0* to download or directly through +this `link <https://cdrdv2.intel.com/v1/dl/getContent/600609>`_. After download is complete, the user needs to unpack and compile on their system before building DPDK. +The following table maps DPDK versions with past FlexRAN SDK releases: + +.. _table_flexran_releases: + +.. table:: DPDK and FlexRAN SDK releases compliance + + ===================== ============================ + DPDK version FlexRAN SDK release + ===================== ============================ + 18.02 1.3.0 + 18.05 1.4.0 + 18.08 1.6.0 + ===================== ============================ + FlexRAN SDK Installation ~~~~~~~~~~~~~~~~~~~~~~~~ The following are pre-requisites for building FlexRAN SDK Libraries: (a) An AVX2 supporting machine - (b) Windriver TS 2 or CentOS 7 operating systems - (c) Intel ICC compiler installed + (b) CentOS Linux release 7.2.1511 (Core) operating system + (c) Intel ICC 18.0.1 20171018 compiler installed The following instructions should be followed in this exact order: @@ -67,31 +83,25 @@ The following instructions should be followed in this exact order: source <path-to-icc-compiler-install-folder>/linux/bin/compilervars.sh intel64 -platform linux - -#. Extract the ``FlexRAN-1.3.0.tar.gz.zip`` package, then run the SDK extractor - script and accept the license: +#. Extract the ``flexran-1-6-0-tar.gz.zip`` package: .. code-block:: console - cd <path-to-workspace>/FlexRAN-1.3.0/ - ./SDK-R1.3.0.sh + unzip flexran-1-6-0-tar.gz.zip + tar xvzf flexran-1-6-0-tar.gz -C FlexRAN-1.6.0/ -#. To allow ``FlexRAN SDK R1.3.0`` to work with bbdev properly, the following - hotfix is required. Change the return of function ``rate_matching_turbo_lte_avx2()`` - located in file - ``<path-to-workspace>/FlexRAN-1.3.0/SDK-R1.3.0/sdk/source/phy/lib_rate_matching/phy_rate_match_avx2.cpp`` - to return 0 instead of 1. +#. Run the SDK extractor script and accept the license: - .. code-block:: c + .. code-block:: console - - return 1; - + return 0; + cd <path-to-workspace>/FlexRAN-1.6.0/ + ./SDK-R1.6.0.sh #. Generate makefiles based on system configuration: .. code-block:: console - cd <path-to-workspace>/FlexRAN-1.3.0/SDK-R1.3.0/sdk/ + cd <path-to-workspace>/FlexRAN-1.6.0/SDK-R1.6.0/sdk/ ./create-makefiles-linux.sh #. A build folder is generated in this form ``build-<ISA>-<CC>``, enter that @@ -100,7 +110,7 @@ The following instructions should be followed in this exact order: .. code-block:: console cd build-avx2-icc/ - make install + make && make install Initialization @@ -118,8 +128,8 @@ Example: .. code-block:: console - export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-1.3.0/SDK-R1.3.0/sdk/build-avx2-icc/install - export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-1.3.0/SDK-R1.3.0/sdk/ + export FLEXRAN_SDK=<path-to-workspace>/FlexRAN-1.6.0/SDK-R1.6.0/sdk/build-avx2-icc/install + export DIR_WIRELESS_SDK=<path-to-workspace>/FlexRAN-1.6.0/SDK-R1.6.0/sdk/ * Set ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` in DPDK common configuration @@ -127,9 +137,9 @@ Example: To use the PMD in an application, user must: -- Call ``rte_vdev_init("turbo_sw")`` within the application. +- Call ``rte_vdev_init("baseband_turbo_sw")`` within the application. -- Use ``--vdev="turbo_sw"`` in the EAL options, which will call ``rte_vdev_init()`` internally. +- Use ``--vdev="baseband_turbo_sw"`` in the EAL options, which will call ``rte_vdev_init()`` internally. The following parameters (all optional) can be provided in the previous two calls: @@ -143,5 +153,5 @@ Example: .. code-block:: console - ./test-bbdev.py -e="--vdev=turbo_sw,socket_id=0,max_nb_queues=8" \ - -c validation -v ./test_vectors/bbdev_vector_t?_default.data + ./test-bbdev.py -e="--vdev=baseband_turbo_sw,socket_id=0,max_nb_queues=8" \ + -c validation -v ./turbo_*_default.data diff --git a/doc/guides/compressdevs/features/default.ini b/doc/guides/compressdevs/features/default.ini new file mode 100644 index 00000000..829e4df6 --- /dev/null +++ b/doc/guides/compressdevs/features/default.ini @@ -0,0 +1,26 @@ +; +; Features of a default compression driver. +; +; This file defines the features that are valid for inclusion in +; the other driver files and also the order that they appear in +; the features table in the documentation. +; +[Features] +HW Accelerated = +CPU SSE = +CPU AVX = +CPU AVX2 = +CPU AVX512 = +CPU NEON = +Stateful = +Pass-through = +OOP SGL In SGL Out = +OOP SGL In LB Out = +OOP LB In SGL Out = +Deflate = +LZS = +Adler32 = +Crc32 = +Adler32&Crc32 = +Fixed = +Dynamic = diff --git a/doc/guides/compressdevs/features/isal.ini b/doc/guides/compressdevs/features/isal.ini new file mode 100644 index 00000000..919cf705 --- /dev/null +++ b/doc/guides/compressdevs/features/isal.ini @@ -0,0 +1,16 @@ +; +; Refer to default.ini for the full list of available PMD features. +; +; Supported features of 'ISA-L' compression driver. +; +[Features] +CPU SSE = Y +CPU AVX = Y +CPU AVX2 = Y +CPU AVX512 = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +Deflate = Y +Fixed = Y +Dynamic = Y diff --git a/doc/guides/compressdevs/features/octeontx.ini b/doc/guides/compressdevs/features/octeontx.ini new file mode 100644 index 00000000..884a8b07 --- /dev/null +++ b/doc/guides/compressdevs/features/octeontx.ini @@ -0,0 +1,10 @@ +; +; Refer to default.ini for the full list of available PMD features. +; +; Supported features of 'OCTEONTX ZIP' compression driver. +; +[Features] +HW Accelerated = Y +Deflate = Y +Fixed = Y +Dynamic = Y diff --git a/doc/guides/compressdevs/features/qat.ini b/doc/guides/compressdevs/features/qat.ini new file mode 100644 index 00000000..5cd4524b --- /dev/null +++ b/doc/guides/compressdevs/features/qat.ini @@ -0,0 +1,15 @@ +; +; Refer to default.ini for the full list of available PMD features. +; +; Supported features of 'QAT' compression driver. +; +[Features] +HW Accelerated = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +Deflate = Y +Adler32 = Y +Crc32 = Y +Adler32&Crc32 = Y +Fixed = Y diff --git a/doc/guides/compressdevs/features/zlib.ini b/doc/guides/compressdevs/features/zlib.ini new file mode 100644 index 00000000..58a4ee3a --- /dev/null +++ b/doc/guides/compressdevs/features/zlib.ini @@ -0,0 +1,10 @@ +; +; Refer to default.ini for the full list of available PMD features. +; +; Supported features of 'ZLIB' compression driver. +; +[Features] +Pass-through = Y +Deflate = Y +Fixed = Y +Dynamic = Y diff --git a/doc/guides/compressdevs/index.rst b/doc/guides/compressdevs/index.rst new file mode 100644 index 00000000..1f37e260 --- /dev/null +++ b/doc/guides/compressdevs/index.rst @@ -0,0 +1,16 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Compression Device Drivers +========================== + + +.. toctree:: + :maxdepth: 2 + :numbered: + + overview + isal + octeontx + qat_comp + zlib diff --git a/doc/guides/compressdevs/isal.rst b/doc/guides/compressdevs/isal.rst new file mode 100644 index 00000000..3bc30229 --- /dev/null +++ b/doc/guides/compressdevs/isal.rst @@ -0,0 +1,123 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +ISA-L Compression Poll Mode Driver +================================== + +The ISA-L PMD (**librte_pmd_isal_comp**) provides poll mode compression & +decompression driver support for utilizing Intel ISA-L library, +which implements the deflate algorithm for both Deflate(compression) and Inflate(decompression). + + +Features +-------- + +ISA-L PMD has support for: + +Compression/Decompression algorithm: + + * DEFLATE + +Huffman code type: + + * FIXED + * DYNAMIC + +Window size support: + + * 32K + +Level guide: + +The ISA-L levels have been mapped to somewhat correspond to the same ZLIB level, +i.e. ZLIB L1 gives a compression ratio similar to ISA-L L1. +Compressdev level 0 enables "No Compression", which passes the uncompressed +data to the output buffer, plus deflate headers. +The ISA-L library does not support this, therefore compressdev level 0 is not supported. + +The compressdev API has 10 levels, 0-9. ISA-L has 4 levels of compression, 0-3. +As a result the level mappings from the API to the PMD are shown below. + +.. _table_ISA-L_compression_levels: + +.. table:: Level mapping from Compressdev to ISA-L PMD. + + +-------------+----------------------------------------------+-----------------------------------------------+ + | Compressdev | PMD Functionality | Internal ISA-L | + | API Level | | Level | + +=============+==============================================+===============================================+ + | 0 | No compression, Not Supported | --- | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 1 | Dynamic (Fast compression) | 1 | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 2 | Dynamic | 2 | + | | (Higher compression ratio) | | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 3 | Dynamic | 3 | + | | (Best compression ratio) | (Level 2 if | + | | | no AVX512/AVX2) | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 4 | Dynamic (Best compression ratio) | Same as above | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 5 | Dynamic (Best compression ratio) | Same as above | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 6 | Dynamic (Best compression ratio) | Same as above | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 7 | Dynamic (Best compression ratio) | Same as above | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 8 | Dynamic (Best compression ratio) | Same as above | + +-------------+----------------------------------------------+-----------------------------------------------+ + | 9 | Dynamic (Best compression ratio) | Same as above | + +-------------+----------------------------------------------+-----------------------------------------------+ + +.. Note:: + + The above table only shows mapping when API calls for dynamic compression. + For fixed compression, regardless of API level, internally ISA-L level 0 is always used. + +Limitations +----------- + +* Compressdev level 0, no compression, is not supported. + +* Checksums will not be supported until future release. + +Installation +------------ + +* To build DPDK with Intel's ISA-L library, the user is required to download the library from `<https://github.com/01org/isa-l>`_. + +* Once downloaded, the user needs to build the library, the ISA-L autotools are usually sufficient:: + + ./autogen.sh + ./configure + +* make can be used to install the library on their system, before building DPDK:: + + make + sudo make install + +* To build with meson, the **libisal.pc** file, must be copied into "pkgconfig", + e.g. /usr/lib/pkgconfig or /usr/lib64/pkgconfig depending on your system, + for meson to find the ISA-L library. The **libisal.pc** is located in library sources:: + + cp isal/libisal.pc /usr/lib/pkgconfig/ + + +Initialization +-------------- + +In order to enable this virtual compression PMD, user must: + +* Set ``CONFIG_RTE_LIBRTE_PMD_ISAL=y`` in config/common_base. + +To use the PMD in an application, user must: + +* Call ``rte_vdev_init("compress_isal")`` within the application. + +* Use ``--vdev="compress_isal"`` in the EAL options, which will call ``rte_vdev_init()`` internally. + +The following parameter (optional) can be provided in the previous two calls: + +* ``socket_id:`` Specify the socket where the memory for the device is going to be allocated + (by default, socket_id will be the socket where the core that is creating the PMD is running on). diff --git a/doc/guides/compressdevs/octeontx.rst b/doc/guides/compressdevs/octeontx.rst new file mode 100644 index 00000000..5a32d5d1 --- /dev/null +++ b/doc/guides/compressdevs/octeontx.rst @@ -0,0 +1,105 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Cavium Networks. + +Octeontx ZIP Compression Poll Mode Driver +========================================= + +The Octeontx ZIP PMD (**librte_pmd_octeontx_zip**) provides poll mode +compression & decompression driver for ZIP HW offload device, found in +**Cavium OCTEONTX** SoC family. + +More information can be found at `Cavium, Inc Official Website +<http://www.cavium.com/OCTEON-TX_ARM_Processors.html>`_. + +Features +-------- + +Octeontx ZIP PMD has support for: + +Compression/Decompression algorithm: + +* DEFLATE + +Huffman code type: + +* FIXED +* DYNAMIC + +Window size support: + +* 2 to 2^14 + +Limitations +----------- + +* Chained mbufs are not supported. + +Supported OCTEONTX SoCs +----------------------- + +- CN83xx + +Steps To Setup Platform +----------------------- + + Octeontx SDK includes kernel image which provides Octeontx ZIP PF + driver to manage configuration of ZIPVF device + Required version of SDK is "OCTEONTX-SDK-6.2.0-build35" or above. + + SDK can be install by using below command. + #rpm -ivh CTEONTX-SDK-6.2.0-build35.x86_64.rpm --force --nodeps + It will install OCTEONTX-SDK at following default location + /usr/local/Cavium_Networks/OCTEONTX-SDK/ + + For more information on building and booting linux kernel on OCTEONTX + please refer /usr/local/Cavium_Networks/OCTEONTX-SDK/docs/OcteonTX-SDK-UG_6.2.0.pdf. + + SDK and related information can be obtained from: `Cavium support site <https://support.cavium.com/>`_. + +Installation +------------ + +Driver Compilation +~~~~~~~~~~~~~~~~~~ + +To compile the OCTEONTX ZIP PMD for Linux arm64 gcc target, run the +following ``make`` command: + + .. code-block:: console + + cd <DPDK-source-directory> + make config T=arm64-thunderx-linuxapp-gcc install + + +Initialization +-------------- + +The octeontx zip is exposed as pci device which consists of a set of +PCIe VF devices. On EAL initialization, ZIP PCIe VF devices will be +probed. To use the PMD in an application, user must: + +* run dev_bind script to bind eight ZIP PCIe VFs to the ``vfio-pci`` driver: + + .. code-block:: console + + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.1 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.2 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.3 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.4 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.5 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.6 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:00.7 + ./usertools/dpdk-devbind.py -b vfio-pci 0001:04:01.0 + +* The unit test cases can be tested as below: + + .. code-block:: console + + reserve enough huge pages + cd to the top-level DPDK directory + export RTE_TARGET=arm64-thunderx-linuxapp-gcc + export RTE_SDK=`pwd` + cd to test/test + type the command "make" to compile + run the tests with "./test" + type the command "compressdev_autotest" to test diff --git a/doc/guides/compressdevs/overview.rst b/doc/guides/compressdevs/overview.rst new file mode 100644 index 00000000..70bbe82b --- /dev/null +++ b/doc/guides/compressdevs/overview.rst @@ -0,0 +1,32 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Compression Device Supported Functionality Matrices +=================================================== + +Supported Feature Flags +----------------------- + +.. _table_compression_pmd_features: + +.. include:: overview_feature_table.txt + +.. Note:: + + - "Pass-through" feature flag refers to the ability of the PMD + to let input buffers pass-through it, copying the input to the output, + without making any modifications to it (no compression done). + + - "OOP SGL In SGL Out" feature flag stands for + "Out-of-place Scatter-gather list Input, Scatter-gater list Output", + which means PMD supports different scatter-gather styled input and output buffers + (i.e. both can consists of multiple segments). + + - "OOP SGL In LB Out" feature flag stands for + "Out-of-place Scatter-gather list Input, Linear Buffers Output", + which means PMD supports input from scatter-gathered styled buffers, outputting linear buffers + (i.e. single segment). + + - "OOP LB In SGL Out" feature flag stands for + "Out-of-place Linear Buffers Input, Scatter-gather list Output", + which means PMD supports input from linear buffer, outputting scatter-gathered styled buffers. diff --git a/doc/guides/compressdevs/qat_comp.rst b/doc/guides/compressdevs/qat_comp.rst new file mode 100644 index 00000000..8b1270b7 --- /dev/null +++ b/doc/guides/compressdevs/qat_comp.rst @@ -0,0 +1,47 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Intel(R) QuickAssist (QAT) Compression Poll Mode Driver +======================================================= + +The QAT compression PMD provides poll mode compression & decompression driver +support for the following hardware accelerator devices: + +* ``Intel QuickAssist Technology C62x`` +* ``Intel QuickAssist Technology C3xxx`` + + +Features +-------- + +QAT compression PMD has support for: + +Compression/Decompression algorithm: + + * DEFLATE + +Huffman code type: + + * FIXED + +Window size support: + + * 32K + +Checksum generation: + + * CRC32, Adler and combined checksum + +Limitations +----------- + +* Compressdev level 0, no compression, is not supported. + +* Dynamic Huffman encoding is not yet supported. + +Installation +------------ + +The QAT compression PMD is built by default with a standard DPDK build. + +It depends on a QAT kernel driver, see :ref:`qat_kernel_installation`. diff --git a/doc/guides/compressdevs/zlib.rst b/doc/guides/compressdevs/zlib.rst new file mode 100644 index 00000000..986c59d4 --- /dev/null +++ b/doc/guides/compressdevs/zlib.rst @@ -0,0 +1,69 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Cavium Networks. + +ZLIB Compression Poll Mode Driver +================================== + +The ZLIB PMD (**librte_pmd_zlib**) provides poll mode compression & +decompression driver based on SW zlib library, + +Features +-------- + +ZLIB PMD has support for: + +Compression/Decompression algorithm: + +* DEFLATE + +Huffman code type: + +* FIXED +* DYNAMIC + +Window size support: + +* Min - 256 bytes +* Max - 32K + +Limitations +----------- + +* Scatter-Gather and Stateful not supported. + +Installation +------------ + +* To build DPDK with ZLIB library, the user is required to download the ``libz`` library. +* Use following command for installation. + +* For Fedora users:: + sudo yum install zlib-devel +* For Ubuntu users:: + sudo apt-get install zlib1g-dev + +* Once downloaded, the user needs to build the library. + +* To build from sources + download zlib sources from http://zlib.net/ and do following before building DPDK:: + + make + sudo make install + +Initialization +-------------- + +In order to enable this virtual compression PMD, user must: + +* Set ``CONFIG_RTE_LIBRTE_PMD_ZLIB=y`` in config/common_base. + +To use the PMD in an application, user must: + +* Call ``rte_vdev_init("compress_zlib")`` within the application. + +* Use ``--vdev="compress_zlib"`` in the EAL options, which will call ``rte_vdev_init()`` internally. + +The following parameter (optional) can be provided in the previous two calls: + +* ``socket_id:`` Specify the socket where the memory for the device is going to be allocated + (by default, socket_id will be the socket where the core that is creating the PMD is running on). diff --git a/doc/guides/conf.py b/doc/guides/conf.py index cf06f257..c883306d 100644 --- a/doc/guides/conf.py +++ b/doc/guides/conf.py @@ -190,18 +190,23 @@ def generate_overview_table(output_filename, table_id, section, table_name, titl ini_files.sort() # Build up a list of the table header names from the ini filenames. - header_names = [] + pmd_names = [] for ini_filename in ini_files: name = ini_filename[:-4] name = name.replace('_vf', 'vf') + pmd_names.append(name) - # Pad the table header names to match the existing format. + # Pad the table header names. + max_header_len = len(max(pmd_names, key=len)) + header_names = [] + for name in pmd_names: if '_vec' in name: pmd, vec = name.split('_') - name = '{0:{fill}{align}7}vec'.format(pmd, fill='.', align='<') + name = '{0:{fill}{align}{width}}vec'.format(pmd, + fill='.', align='<', width=max_header_len-3) else: - name = '{0:{fill}{align}10}'.format(name, fill=' ', align='<') - + name = '{0:{fill}{align}{width}}'.format(name, + fill=' ', align='<', width=max_header_len) header_names.append(name) # Create a dict of the defined features for each driver from the ini files. @@ -253,7 +258,7 @@ def print_table_header(outfile, num_cols, header_names, title): print_table_row(outfile, title, line) - for i in range(1, 10): + for i in range(1, len(header_names[0])): line = '' for name in header_names: line += ' ' + name[i] @@ -310,7 +315,7 @@ def print_table_css(outfile, table_id): text-align: center; } table#idx th { - font-size: 80%; + font-size: 72%; white-space: pre-wrap; vertical-align: top; padding: 0.5em 0; @@ -383,6 +388,11 @@ def setup(app): 'AEAD', 'AEAD algorithms in crypto drivers', 'AEAD algorithm') + table_file = dirname(__file__) + '/compressdevs/overview_feature_table.txt' + generate_overview_table(table_file, 1, + 'Features', + 'Features availability in compression drivers', + 'Feature') if LooseVersion(sphinx_version) < LooseVersion('1.3.1'): print('Upgrade sphinx to version >= 1.3.1 for ' diff --git a/doc/guides/contributing/cheatsheet.rst b/doc/guides/contributing/cheatsheet.rst index 7bc07715..0debd118 100644 --- a/doc/guides/contributing/cheatsheet.rst +++ b/doc/guides/contributing/cheatsheet.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + Patch Cheatsheet ================ diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst index b0f0adb8..b1bf0d15 100644 --- a/doc/guides/contributing/coding_style.rst +++ b/doc/guides/contributing/coding_style.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + .. _coding_style: DPDK Coding Style @@ -614,8 +617,8 @@ In the DPDK environment, use the logging interface provided: * is DEBUG) */ rte_log_set_level(my_logtype2, RTE_LOG_NOTICE); - /* enable all PMD logs (whose identifier string starts with "pmd") */ - rte_log_set_level_regexp("pmd.*", RTE_LOG_DEBUG); + /* enable all PMD logs (whose identifier string starts with "pmd.") */ + rte_log_set_level_pattern("pmd.*", RTE_LOG_DEBUG); /* log in debug level */ rte_log_set_global_level(RTE_LOG_DEBUG); diff --git a/doc/guides/contributing/design.rst b/doc/guides/contributing/design.rst index 88d3a433..651fd224 100644 --- a/doc/guides/contributing/design.rst +++ b/doc/guides/contributing/design.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + Design ====== diff --git a/doc/guides/contributing/documentation.rst b/doc/guides/contributing/documentation.rst index 82f2e1bb..6a075553 100644 --- a/doc/guides/contributing/documentation.rst +++ b/doc/guides/contributing/documentation.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + .. _doc_guidelines: DPDK Documentation Guidelines diff --git a/doc/guides/contributing/index.rst b/doc/guides/contributing/index.rst index 329b678a..f90df451 100644 --- a/doc/guides/contributing/index.rst +++ b/doc/guides/contributing/index.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + Contributor's Guidelines ======================== diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst index 2287835f..a3d78802 100644 --- a/doc/guides/contributing/patches.rst +++ b/doc/guides/contributing/patches.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + .. submitting_patches: Contributing Code to DPDK @@ -256,6 +259,66 @@ In addition to the ``Signed-off-by:`` name the commit messages can also have tags for who reported, suggested, tested and reviewed the patch being posted. Please refer to the `Tested, Acked and Reviewed by`_ section. +Patch Fix Related Issues +~~~~~~~~~~~~~~~~~~~~~~~~ + +`Coverity <https://scan.coverity.com/projects/dpdk-data-plane-development-kit>`_ +is a tool for static code analysis. +It is used as a cloud-based service used to scan the DPDK source code, +and alert developers of any potential defects in the source code. +When fixing an issue found by Coverity, the patch must contain a Coverity issue ID +in the body of the commit message. For example:: + + + doc: fix some parameter description + + Update the docs, fixing description of some parameter. + + Coverity issue: 12345 + Fixes: abcdefgh1234 ("doc: add some parameter") + Cc: author@example.com + + Signed-off-by: Alex Smith <alex.smith@example.com> + + +`Bugzilla <https://dpdk.org/tracker>`_ +is a bug- or issue-tracking system. +Bug-tracking systems allow individual or groups of developers +effectively to keep track of outstanding problems with their product. +When fixing an issue raised in Bugzilla, the patch must contain +a Bugzilla issue ID in the body of the commit message. +For example:: + + doc: fix some parameter description + + Update the docs, fixing description of some parameter. + + Bugzilla ID: 12345 + Fixes: abcdefgh1234 ("doc: add some parameter") + Cc: author@example.com + + Signed-off-by: Alex Smith <alex.smith@example.com> + +Patch for Stable Releases +~~~~~~~~~~~~~~~~~~~~~~~~~ + +All fix patches to the master branch that are candidates for backporting +should also be CCed to the `stable@dpdk.org <http://dpdk.org/ml/listinfo/stable>`_ +mailing list. +In the commit message body the Cc: stable@dpdk.org should be inserted as follows:: + + doc: fix some parameter description + + Update the docs, fixing description of some parameter. + + Fixes: abcdefgh1234 ("doc: add some parameter") + Cc: stable@dpdk.org + + Signed-off-by: Alex Smith <alex.smith@example.com> + +For further information on stable contribution you can go to +:doc:`Stable Contribution Guide <stable>`. + Creating Patches ---------------- @@ -450,6 +513,20 @@ Experienced committers may send patches directly with ``git send-email`` without The options ``--annotate`` and ``confirm = always`` are recommended for checking patches before sending. +Backporting patches for Stable Releases +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sometimes a maintainer or contributor wishes, or can be asked, to send a patch +for a stable release rather than mainline. +In this case the patch(es) should be sent to ``stable@dpdk.org``, +not to ``dev@dpdk.org``. + +Given that there are multiple stable releases being maintained at the same time, +please specify exactly which branch(es) the patch is for +using ``git send-email --subject-prefix='PATCH 16.11' ...`` +and also optionally in the cover letter or in the annotation. + + The Review Process ------------------ diff --git a/doc/guides/contributing/stable.rst b/doc/guides/contributing/stable.rst index 0f2f1f37..1746c046 100644 --- a/doc/guides/contributing/stable.rst +++ b/doc/guides/contributing/stable.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + .. stable_lts_releases: DPDK Stable Releases and Long Term Support @@ -57,7 +60,25 @@ that a tagged release has been tested. What changes should be backported --------------------------------- -Backporting should be limited to bug fixes. +Backporting should be limited to bug fixes. All patches accepted on the master +branch with a Fixes: tag should be backported to the relevant stable/LTS +branches, unless the submitter indicates otherwise. If there are exceptions, +they will be discussed on the mailing lists. + +Fixes suitable for backport should have a ``Cc: stable@dpdk.org`` tag in the +commit message body as follows:: + + doc: fix some parameter description + + Update the docs, fixing description of some parameter. + + Fixes: abcdefgh1234 ("doc: add some parameter") + Cc: stable@dpdk.org + + Signed-off-by: Alex Smith <alex.smith@example.com> + + +Fixes not suitable for backport should not include the ``Cc: stable@dpdk.org`` tag. Features should not be backported to stable releases. It may be acceptable, in limited cases, to back port features for the LTS release where: diff --git a/doc/guides/contributing/versioning.rst b/doc/guides/contributing/versioning.rst index c495294d..01b36247 100644 --- a/doc/guides/contributing/versioning.rst +++ b/doc/guides/contributing/versioning.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + Managing ABI updates ==================== @@ -43,29 +46,6 @@ ABI versions are set at the time of major release labeling, and the ABI may change multiple times, without warning, between the last release label and the HEAD label of the git tree. -APIs marked as ``experimental`` are not considered part of the ABI and may -change without warning at any time. Since changes to APIs are most likely -immediately after their introduction, as users begin to take advantage of -those new APIs and start finding issues with them, new DPDK APIs will be -automatically marked as ``experimental`` to allow for a period of stabilization -before they become part of a tracked ABI. - -Note that marking an API as experimental is a multi step process. -To mark an API as experimental, the symbols which are desired to be exported -must be placed in an EXPERIMENTAL version block in the corresponding libraries' -version map script. -Secondly, the corresponding definitions of those exported functions, and -their forward declarations (in the development header files), must be marked -with the ``__rte_experimental`` tag (see ``rte_compat.h``). -The DPDK build makefiles perform a check to ensure that the map file and the -C code reflect the same list of symbols. -This check can be circumvented by defining ``ALLOW_EXPERIMENTAL_API`` -during compilation in the corresponding library Makefile. - -In addition to tagging the code with ``__rte_experimental``, -the doxygen markup must also contain the EXPERIMENTAL string, -and the MAINTAINERS file should note the EXPERIMENTAL libraries. - ABI versions, once released, are available until such time as their deprecation has been noted in the Release Notes for at least one major release cycle. For example consider the case where the ABI for DPDK 2.0 has been @@ -119,6 +99,37 @@ readability purposes should be avoided. follow the relevant deprecation policy procedures as above: 3 acks and announcement at least one release in advance. +Experimental APIs +~~~~~~~~~~~~~~~~~ + +APIs marked as ``experimental`` are not considered part of the ABI and may +change without warning at any time. Since changes to APIs are most likely +immediately after their introduction, as users begin to take advantage of +those new APIs and start finding issues with them, new DPDK APIs will be +automatically marked as ``experimental`` to allow for a period of stabilization +before they become part of a tracked ABI. + +Note that marking an API as experimental is a multi step process. +To mark an API as experimental, the symbols which are desired to be exported +must be placed in an EXPERIMENTAL version block in the corresponding libraries' +version map script. +Secondly, the corresponding definitions of those exported functions, and +their forward declarations (in the development header files), must be marked +with the ``__rte_experimental`` tag (see ``rte_compat.h``). +The DPDK build makefiles perform a check to ensure that the map file and the +C code reflect the same list of symbols. +This check can be circumvented by defining ``ALLOW_EXPERIMENTAL_API`` +during compilation in the corresponding library Makefile. + +In addition to tagging the code with ``__rte_experimental``, +the doxygen markup must also contain the EXPERIMENTAL string, +and the MAINTAINERS file should note the EXPERIMENTAL libraries. + +For removing the experimental tag associated with an API, deprecation notice +is not required. Though, an API should remain in experimental state for at least +one release. Thereafter, normal process of posting patch for review to mailing +list can be followed. + Examples of Deprecation Notices ------------------------------- diff --git a/doc/guides/cryptodevs/aesni_gcm.rst b/doc/guides/cryptodevs/aesni_gcm.rst index ffd6ba90..e0346080 100644 --- a/doc/guides/cryptodevs/aesni_gcm.rst +++ b/doc/guides/cryptodevs/aesni_gcm.rst @@ -36,12 +36,13 @@ Installation To build DPDK with the AESNI_GCM_PMD the user is required to download the multi-buffer library from `here <https://github.com/01org/intel-ipsec-mb>`_ and compile it on their user system before building DPDK. -The latest version of the library supported by this PMD is v0.48, which -can be downloaded in `<https://github.com/01org/intel-ipsec-mb/archive/v0.48.zip>`_. +The latest version of the library supported by this PMD is v0.50, which +can be downloaded in `<https://github.com/01org/intel-ipsec-mb/archive/v0.50.zip>`_. .. code-block:: console - make + make + make install As a reference, the following table shows a mapping between the past DPDK versions and the external crypto libraries supported by them: @@ -55,7 +56,8 @@ and the external crypto libraries supported by them: ============= ================================ 16.04 - 16.11 Multi-buffer library 0.43 - 0.44 17.02 - 17.05 ISA-L Crypto v2.18 - 17.08+ Multi-buffer library 0.46+ + 17.08 - 18.02 Multi-buffer library 0.46 - 0.48 + 18.05+ Multi-buffer library 0.49+ ============= ================================ @@ -64,9 +66,6 @@ Initialization In order to enable this virtual crypto PMD, user must: -* Export the environmental variable AESNI_MULTI_BUFFER_LIB_PATH with the path where - the library was extracted. - * Build the multi buffer library (explained in Installation section). * Set CONFIG_RTE_LIBRTE_PMD_AESNI_GCM=y in config/common_base. diff --git a/doc/guides/cryptodevs/aesni_mb.rst b/doc/guides/cryptodevs/aesni_mb.rst index 3950daae..c2929500 100644 --- a/doc/guides/cryptodevs/aesni_mb.rst +++ b/doc/guides/cryptodevs/aesni_mb.rst @@ -27,6 +27,7 @@ Cipher algorithms: * RTE_CRYPTO_CIPHER_AES256_CTR * RTE_CRYPTO_CIPHER_AES_DOCSISBPI * RTE_CRYPTO_CIPHER_DES_CBC +* RTE_CRYPTO_CIPHER_3DES_CBC * RTE_CRYPTO_CIPHER_DES_DOCSISBPI Hash algorithms: @@ -38,6 +39,7 @@ Hash algorithms: * RTE_CRYPTO_HASH_SHA384_HMAC * RTE_CRYPTO_HASH_SHA512_HMAC * RTE_CRYPTO_HASH_AES_XCBC_HMAC +* RTE_CRYPTO_HASH_AES_CMAC AEAD algorithms: @@ -56,12 +58,13 @@ Installation To build DPDK with the AESNI_MB_PMD the user is required to download the multi-buffer library from `here <https://github.com/01org/intel-ipsec-mb>`_ and compile it on their user system before building DPDK. -The latest version of the library supported by this PMD is v0.48, which -can be downloaded from `<https://github.com/01org/intel-ipsec-mb/archive/v0.48.zip>`_. +The latest version of the library supported by this PMD is v0.50, which +can be downloaded from `<https://github.com/01org/intel-ipsec-mb/archive/v0.50.zip>`_. .. code-block:: console - make + make + make install As a reference, the following table shows a mapping between the past DPDK versions and the Multi-Buffer library version supported by them: @@ -77,7 +80,8 @@ and the Multi-Buffer library version supported by them: 17.02 0.44 17.05 - 17.08 0.45 - 0.48 17.11 0.47 - 0.48 - 18.02+ 0.48 + 18.02 0.48 + 18.05+ 0.49+ ============== ============================ @@ -86,9 +90,6 @@ Initialization In order to enable this virtual crypto PMD, user must: -* Export the environmental variable AESNI_MULTI_BUFFER_LIB_PATH with the path where - the library was extracted. - * Build the multi buffer library (explained in Installation section). * Set CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y in config/common_base. diff --git a/doc/guides/cryptodevs/ccp.rst b/doc/guides/cryptodevs/ccp.rst new file mode 100644 index 00000000..034d2036 --- /dev/null +++ b/doc/guides/cryptodevs/ccp.rst @@ -0,0 +1,140 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Advanced Micro Devices, Inc. All rights reserved. + +AMD CCP Poll Mode Driver +======================== + +This code provides the initial implementation of the ccp poll mode driver. +The CCP poll mode driver library (librte_pmd_ccp) implements support for +AMD’s cryptographic co-processor (CCP). The CCP PMD is a virtual crypto +poll mode driver which schedules crypto operations to one or more available +CCP hardware engines on the platform. The CCP PMD provides poll mode crypto +driver support for the following hardware accelerator devices:: + + AMD Cryptographic Co-processor (0x1456) + AMD Cryptographic Co-processor (0x1468) + +Features +-------- + +CCP crypto PMD has support for: + +Cipher algorithms: + +* ``RTE_CRYPTO_CIPHER_AES_CBC`` +* ``RTE_CRYPTO_CIPHER_AES_ECB`` +* ``RTE_CRYPTO_CIPHER_AES_CTR`` +* ``RTE_CRYPTO_CIPHER_3DES_CBC`` + +Hash algorithms: + +* ``RTE_CRYPTO_AUTH_SHA1`` +* ``RTE_CRYPTO_AUTH_SHA1_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA224`` +* ``RTE_CRYPTO_AUTH_SHA224_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA256`` +* ``RTE_CRYPTO_AUTH_SHA256_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA384`` +* ``RTE_CRYPTO_AUTH_SHA384_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA512`` +* ``RTE_CRYPTO_AUTH_SHA512_HMAC`` +* ``RTE_CRYPTO_AUTH_MD5_HMAC`` +* ``RTE_CRYPTO_AUTH_AES_CMAC`` +* ``RTE_CRYPTO_AUTH_SHA3_224`` +* ``RTE_CRYPTO_AUTH_SHA3_224_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA3_256`` +* ``RTE_CRYPTO_AUTH_SHA3_256_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA3_384`` +* ``RTE_CRYPTO_AUTH_SHA3_384_HMAC`` +* ``RTE_CRYPTO_AUTH_SHA3_512`` +* ``RTE_CRYPTO_AUTH_SHA3_512_HMAC`` + +AEAD algorithms: + +* ``RTE_CRYPTO_AEAD_AES_GCM`` + +Installation +------------ + +To compile ccp PMD, it has to be enabled in the config/common_base file and openssl +packages have to be installed in the build environment. + +* ``CONFIG_RTE_LIBRTE_PMD_CCP=y`` + +For Ubuntu 16.04 LTS use below to install openssl in the build system: + +.. code-block:: console + + sudo apt-get install openssl + +This code was verified on Ubuntu 16.04. + +Initialization +-------------- + +Bind the CCP devices to DPDK UIO driver module before running the CCP PMD stack. +e.g. for the 0x1456 device:: + + cd to the top-level DPDK directory + modprobe uio + insmod ./build/kmod/igb_uio.ko + echo "1022 1456" > /sys/bus/pci/drivers/igb_uio/new_id + +Another way to bind the CCP devices to DPDK UIO driver is by using the ``dpdk-devbind.py`` script. +The following command assumes ``BFD`` as ``0000:09:00.2``:: + + cd to the top-level DPDK directory + ./usertools/dpdk-devbind.py -b igb_uio 0000:09:00.2 + +In order to enable the ccp crypto PMD, user must set CONFIG_RTE_LIBRTE_PMD_CCP=y in config/common_base. + +To use the PMD in an application, user must: + +* Call rte_vdev_init("crypto_ccp") within the application. + +* Use --vdev="crypto_ccp" in the EAL options, which will call rte_vdev_init() internally. + +The following parameters (all optional) can be provided in the previous two calls: + +* socket_id: Specify the socket where the memory for the device is going to be allocated. + (by default, socket_id will be the socket where the core that is creating the PMD is running on). + +* max_nb_queue_pairs: Specify the maximum number of queue pairs in the device. + +* max_nb_sessions: Specify the maximum number of sessions that can be created (2048 by default). + +* ccp_auth_opt: Specify authentication operations to perform on CPU using openssl APIs. + +To validate ccp pmd, l2fwd-crypto example can be used with following command: + +.. code-block:: console + + sudo ./build/l2fwd-crypto -l 1 -n 4 --vdev "crypto_ccp" -- -p 0x1 + --chain CIPHER_HASH --cipher_op ENCRYPT --cipher_algo AES_CBC + --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f + --iv 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:ff + --auth_op GENERATE --auth_algo SHA1_HMAC + --auth_key 11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11 + :11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11 + :11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11 + +The CCP PMD also supports computing authentication over CPU with cipher offloaded to CCP. +To enable this feature, pass an additional argument as ccp_auth_opt=1 to --vdev parameters as +following: + +.. code-block:: console + + sudo ./build/l2fwd-crypto -l 1 -n 4 --vdev "crypto_ccp,ccp_auth_opt=1" -- -p 0x1 + --chain CIPHER_HASH --cipher_op ENCRYPT --cipher_algo AES_CBC + --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f + --iv 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:ff + --auth_op GENERATE --auth_algo SHA1_HMAC + --auth_key 11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11 + :11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11 + :11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11:11 + +Limitations +----------- + +* Chained mbufs are not supported. +* MD5_HMAC is supported only for CPU based authentication. diff --git a/doc/guides/cryptodevs/dpaa2_sec.rst b/doc/guides/cryptodevs/dpaa2_sec.rst index 5460a92d..9191704e 100644 --- a/doc/guides/cryptodevs/dpaa2_sec.rst +++ b/doc/guides/cryptodevs/dpaa2_sec.rst @@ -134,10 +134,20 @@ Supported DPAA2 SoCs * LS2088A/LS2048A * LS1088A/LS1048A +Whitelisting & Blacklisting +--------------------------- + +For blacklisting a DPAA2 SEC device, following commands can be used. + + .. code-block:: console + + <dpdk app> <EAL args> -b "fslmc:dpseci.x" -- ... + +Where x is the device object id as configured in resource container. + Limitations ----------- -* Chained mbufs are not supported. * Hash followed by Cipher mode is not supported * Only supports the session-oriented API implementation (session-less APIs are not supported). @@ -189,20 +199,6 @@ Please note that enabling debugging options may affect system performance. By default it is only enabled in defconfig_arm64-dpaa2-* config. Toggle compilation of the ``librte_pmd_dpaa2_sec`` driver. -* ``CONFIG_RTE_LIBRTE_DPAA2_SEC_DEBUG_INIT`` (default ``n``) - Toggle display of initialization related driver messages - -* ``CONFIG_RTE_LIBRTE_DPAA2_SEC_DEBUG_DRIVER`` (default ``n``) - Toggle display of driver runtime messages - -* ``CONFIG_RTE_LIBRTE_DPAA2_SEC_DEBUG_RX`` (default ``n``) - Toggle display of receive fast path run-time message - -* ``CONFIG_RTE_DPAA2_SEC_PMD_MAX_NB_SESSIONS`` - By default it is set as 2048 in defconfig_arm64-dpaa2-* config. - It indicates Number of sessions to create in the session memory pool - on a single DPAA2 SEC device. - Installations ------------- To compile the DPAA2_SEC PMD for Linux arm64 gcc target, run the @@ -212,3 +208,15 @@ following ``make`` command: cd <DPDK-source-directory> make config T=arm64-dpaa2-linuxapp-gcc install + +Enabling logs +------------- + +For enabling logs, use the following EAL parameter: + +.. code-block:: console + + ./your_crypto_application <EAL args> --log-level=pmd.crypto.dpaa2:<level> + +Using ``crypto.dpaa2`` as log matching criteria, all Crypto PMD logs can be +enabled which are lower than logging ``level``. diff --git a/doc/guides/cryptodevs/dpaa_sec.rst b/doc/guides/cryptodevs/dpaa_sec.rst index b98f7864..dd683894 100644 --- a/doc/guides/cryptodevs/dpaa_sec.rst +++ b/doc/guides/cryptodevs/dpaa_sec.rst @@ -78,10 +78,22 @@ Supported DPAA SoCs * LS1046A/LS1026A * LS1043A/LS1023A +Whitelisting & Blacklisting +--------------------------- + +For blacklisting a DPAA device, following commands can be used. + + .. code-block:: console + + <dpdk app> <EAL args> -b "dpaa_bus:dpaa-secX" -- ... + e.g. "dpaa_bus:dpaa-sec0" + + or to disable all 4 SEC devices + -b "dpaa_sec:dpaa-sec0" -b "dpaa_sec:dpaa-sec1" -b "dpaa_sec:dpaa-sec2" -b "dpaa_sec:dpaa-sec3" + Limitations ----------- -* Chained mbufs are not supported. * Hash followed by Cipher mode is not supported * Only supports the session-oriented API implementation (session-less APIs are not supported). @@ -132,20 +144,6 @@ Please note that enabling debugging options may affect system performance. By default it is only enabled in defconfig_arm64-dpaa-* config. Toggle compilation of the ``librte_pmd_dpaa_sec`` driver. -* ``CONFIG_RTE_LIBRTE_DPAA_SEC_DEBUG_INIT`` (default ``n``) - Toggle display of initialization related driver messages - -* ``CONFIG_RTE_LIBRTE_DPAA_SEC_DEBUG_DRIVER`` (default ``n``) - Toggle display of driver runtime messages - -* ``CONFIG_RTE_LIBRTE_DPAA_SEC_DEBUG_RX`` (default ``n``) - Toggle display of receive fast path run-time message - -* ``CONFIG_RTE_DPAA_SEC_PMD_MAX_NB_SESSIONS`` - By default it is set as 2048 in defconfig_arm64-dpaa-* config. - It indicates Number of sessions to create in the session memory pool - on a single DPAA SEC device. - Installations ------------- To compile the DPAA_SEC PMD for Linux arm64 gcc target, run the @@ -155,3 +153,15 @@ following ``make`` command: cd <DPDK-source-directory> make config T=arm64-dpaa-linuxapp-gcc install + +Enabling logs +------------- + +For enabling logs, use the following EAL parameter: + +.. code-block:: console + + ./your_crypto_application <EAL args> --log-level=pmd.crypto.dpaa:<level> + +Using ``pmd.crypto.dpaa`` as log matching criteria, all Crypto PMD logs can be +enabled which are lower than logging ``level``. diff --git a/doc/guides/cryptodevs/features/aesni_gcm.ini b/doc/guides/cryptodevs/features/aesni_gcm.ini index 920b6b6a..b9e9c906 100644 --- a/doc/guides/cryptodevs/features/aesni_gcm.ini +++ b/doc/guides/cryptodevs/features/aesni_gcm.ini @@ -10,7 +10,8 @@ CPU AESNI = Y CPU SSE = Y CPU AVX = Y CPU AVX2 = Y -Mbuf scatter gather = Y +OOP SGL In LB Out = Y +OOP LB In LB Out = Y ; ; Supported crypto algorithms of the 'aesni_gcm' crypto driver. ; diff --git a/doc/guides/cryptodevs/features/aesni_mb.ini b/doc/guides/cryptodevs/features/aesni_mb.ini index a5a45a6d..f7295745 100644 --- a/doc/guides/cryptodevs/features/aesni_mb.ini +++ b/doc/guides/cryptodevs/features/aesni_mb.ini @@ -24,6 +24,7 @@ AES CTR (192) = Y AES CTR (256) = Y AES DOCSIS BPI = Y DES CBC = Y +3DES CBC = Y DES DOCSIS BPI = Y ; @@ -37,6 +38,7 @@ SHA256 HMAC = Y SHA384 HMAC = Y SHA512 HMAC = Y AES XCBC MAC = Y +AES CMAC (128) = Y ; ; Supported AEAD algorithms of the 'aesni_mb' crypto driver. diff --git a/doc/guides/cryptodevs/features/ccp.ini b/doc/guides/cryptodevs/features/ccp.ini new file mode 100644 index 00000000..4722e135 --- /dev/null +++ b/doc/guides/cryptodevs/features/ccp.ini @@ -0,0 +1,59 @@ +; +; Supported features of the 'ccp' crypto poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Symmetric crypto = Y +Sym operation chaining = Y +HW Accelerated = Y + +; +; Supported crypto algorithms of the 'ccp' crypto driver. +; +[Cipher] +AES CBC (128) = Y +AES CBC (192) = Y +AES CBC (256) = Y +AES ECB (128) = Y +AES ECB (192) = Y +AES ECB (256) = Y +AES CTR (128) = Y +AES CTR (192) = Y +AES CTR (256) = Y +3DES CBC = Y + +; +; Supported authentication algorithms of the 'ccp' crypto driver. +; +[Auth] +MD5 HMAC = Y +SHA1 = Y +SHA1 HMAC = Y +SHA224 = Y +SHA224 HMAC = Y +SHA256 = Y +SHA256 HMAC = Y +SHA384 = Y +SHA384 HMAC = Y +SHA512 = Y +SHA512 HMAC = Y +AES CMAC (128) = Y +AES CMAC (192) = Y +AES CMAC (256) = Y +SHA3_224 = Y +SHA3_224 HMAC = Y +SHA3_256 = Y +SHA3_256 HMAC = Y +SHA3_384 = Y +SHA3_384 HMAC = Y +SHA3_512 = Y +SHA3_512 HMAC = Y + +; +; Supported AEAD algorithms of the 'ccp' crypto driver. +; +[AEAD] +AES GCM (128) = Y +AES GCM (192) = Y +AES GCM (256) = Y diff --git a/doc/guides/cryptodevs/features/default.ini b/doc/guides/cryptodevs/features/default.ini index 728ce3b7..92a7ccf3 100644 --- a/doc/guides/cryptodevs/features/default.ini +++ b/doc/guides/cryptodevs/features/default.ini @@ -18,7 +18,11 @@ CPU AVX512 = CPU AESNI = CPU NEON = CPU ARM CE = -Mbuf scatter gather = +In Place SGL = +OOP SGL In SGL Out = +OOP SGL In LB Out = +OOP LB In SGL Out = +OOP LB In LB Out = ; ; Supported crypto algorithms of a default crypto driver. @@ -28,6 +32,9 @@ NULL = AES CBC (128) = AES CBC (192) = AES CBC (256) = +AES ECB (128) = +AES ECB (192) = +AES ECB (256) = AES CTR (128) = AES CTR (192) = AES CTR (256) = @@ -62,6 +69,17 @@ AES GMAC = SNOW3G UIA2 = KASUMI F9 = ZUC EIA3 = +AES CMAC (128) = +AES CMAC (192) = +AES CMAC (256) = +SHA3_224 = +SHA3_224 HMAC = +SHA3_256 = +SHA3_256 HMAC = +SHA3_384 = +SHA3_384 HMAC = +SHA3_512 = +SHA3_512 HMAC = ; ; Supported AEAD algorithms of a default crypto driver. diff --git a/doc/guides/cryptodevs/features/dpaa2_sec.ini b/doc/guides/cryptodevs/features/dpaa2_sec.ini index 68c9960d..69700df4 100644 --- a/doc/guides/cryptodevs/features/dpaa2_sec.ini +++ b/doc/guides/cryptodevs/features/dpaa2_sec.ini @@ -8,7 +8,11 @@ Symmetric crypto = Y Sym operation chaining = Y HW Accelerated = Y Protocol offload = Y -Mbuf scatter gather = Y +In Place SGL = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +OOP LB In LB Out = Y ; ; Supported crypto algorithms of the 'dpaa2_sec' crypto driver. diff --git a/doc/guides/cryptodevs/features/dpaa_sec.ini b/doc/guides/cryptodevs/features/dpaa_sec.ini index 260fae72..937b621c 100644 --- a/doc/guides/cryptodevs/features/dpaa_sec.ini +++ b/doc/guides/cryptodevs/features/dpaa_sec.ini @@ -8,7 +8,11 @@ Symmetric crypto = Y Sym operation chaining = Y HW Accelerated = Y Protocol offload = Y -Mbuf scatter gather = Y +In Place SGL = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +OOP LB In LB Out = Y ; ; Supported crypto algorithms of the 'dpaa_sec' crypto driver. diff --git a/doc/guides/cryptodevs/features/mrvl.ini b/doc/guides/cryptodevs/features/mvsam.ini index 6d2fe6aa..b7c105af 100644 --- a/doc/guides/cryptodevs/features/mrvl.ini +++ b/doc/guides/cryptodevs/features/mvsam.ini @@ -1,4 +1,4 @@ -; Supported features of the 'mrvl' crypto driver. +; Supported features of the 'mvsam' crypto driver. ; ; Refer to default.ini for the full list of available PMD features. ; diff --git a/doc/guides/cryptodevs/features/null.ini b/doc/guides/cryptodevs/features/null.ini index a9e172da..ecf5779a 100644 --- a/doc/guides/cryptodevs/features/null.ini +++ b/doc/guides/cryptodevs/features/null.ini @@ -6,7 +6,7 @@ [Features] Symmetric crypto = Y Sym operation chaining = Y -Mbuf scatter gather = Y +In Place SGL = Y ; ; Supported crypto algorithms of the 'null' crypto driver. diff --git a/doc/guides/cryptodevs/features/openssl.ini b/doc/guides/cryptodevs/features/openssl.ini index 69156586..b9c0bdcc 100644 --- a/doc/guides/cryptodevs/features/openssl.ini +++ b/doc/guides/cryptodevs/features/openssl.ini @@ -6,7 +6,9 @@ [Features] Symmetric crypto = Y Sym operation chaining = Y -Mbuf scatter gather = Y +OOP SGL In LB Out = Y +OOP LB In LB Out = Y +Asymmetric crypto = Y ; ; Supported crypto algorithms of the 'openssl' crypto driver. @@ -49,3 +51,13 @@ AES GCM (256) = Y AES CCM (128) = Y AES CCM (192) = Y AES CCM (256) = Y + +; +; Supported Asymmetric algorithms of the 'openssl' crypto driver. +; +[Asymmetric] +RSA = Y +DSA = Y +Modular Exponentiation = Y +Modular Inversion = Y +Diffie-hellman = Y diff --git a/doc/guides/cryptodevs/features/qat.ini b/doc/guides/cryptodevs/features/qat.ini index 51ed5967..29d865e0 100644 --- a/doc/guides/cryptodevs/features/qat.ini +++ b/doc/guides/cryptodevs/features/qat.ini @@ -7,7 +7,11 @@ Symmetric crypto = Y Sym operation chaining = Y HW Accelerated = Y -Mbuf scatter gather = Y +In Place SGL = Y +OOP SGL In SGL Out = Y +OOP SGL In LB Out = Y +OOP LB In SGL Out = Y +OOP LB In LB Out = Y ; ; Supported crypto algorithms of the 'qat' crypto driver. diff --git a/doc/guides/cryptodevs/features/virtio.ini b/doc/guides/cryptodevs/features/virtio.ini new file mode 100644 index 00000000..168fc174 --- /dev/null +++ b/doc/guides/cryptodevs/features/virtio.ini @@ -0,0 +1,26 @@ +; Supported features of the 'virtio' crypto driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Symmetric crypto = Y +Sym operation chaining = Y + +; +; Supported crypto algorithms of the 'virtio' crypto driver. +; +[Cipher] +AES CBC (128) = Y +AES CBC (192) = Y +AES CBC (256) = Y + +; +; Supported authentication algorithms of the 'virtio' crypto driver. +; +[Auth] +SHA1 HMAC = Y + +; +; Supported AEAD algorithms of the 'virtio' crypto driver. +; +[AEAD] diff --git a/doc/guides/cryptodevs/index.rst b/doc/guides/cryptodevs/index.rst index 558c9267..e9928a4e 100644 --- a/doc/guides/cryptodevs/index.rst +++ b/doc/guides/cryptodevs/index.rst @@ -13,13 +13,15 @@ Crypto Device Drivers aesni_mb aesni_gcm armv8 + ccp dpaa2_sec dpaa_sec kasumi openssl - mrvl + mvsam null scheduler snow3g qat + virtio zuc diff --git a/doc/guides/cryptodevs/kasumi.rst b/doc/guides/cryptodevs/kasumi.rst index f56b5475..2265eee4 100644 --- a/doc/guides/cryptodevs/kasumi.rst +++ b/doc/guides/cryptodevs/kasumi.rst @@ -34,11 +34,11 @@ Installation ------------ To build DPDK with the KASUMI_PMD the user is required to download -the export controlled ``libsso_kasumi`` library, by requesting it from -`<https://networkbuilders.intel.com/network-technologies/dpdk>`_. -Once approval has been granted, the user needs to log in -`<https://networkbuilders.intel.com/dpdklogin>`_ -and click on "Kasumi Bit Stream crypto library" link, to download the library. +the export controlled ``libsso_kasumi`` library, by registering in +`Intel Resource & Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_. +Once approval has been granted, the user needs to search for +*Kasumi F8 F9 3GPP cryptographic algorithms Software Library* to download the +library or directly through this `link <https://cdrdv2.intel.com/v1/dl/getContent/575866>`_. After downloading the library, the user needs to unpack and compile it on their system before building DPDK:: diff --git a/doc/guides/cryptodevs/mrvl.rst b/doc/guides/cryptodevs/mvsam.rst index 6a0b08c5..fd418c26 100644 --- a/doc/guides/cryptodevs/mrvl.rst +++ b/doc/guides/cryptodevs/mvsam.rst @@ -29,10 +29,10 @@ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -MRVL Crypto Poll Mode Driver -============================ +MVSAM Crypto Poll Mode Driver +============================= -The MRVL CRYPTO PMD (**librte_crypto_mrvl_pmd**) provides poll mode crypto driver +The MVSAM CRYPTO PMD (**librte_crypto_mvsam_pmd**) provides poll mode crypto driver support by utilizing MUSDK library, which provides cryptographic operations acceleration by using Security Acceleration Engine (EIP197) directly from user-space with minimum overhead and high performance. @@ -40,7 +40,7 @@ user-space with minimum overhead and high performance. Features -------- -MRVL CRYPTO PMD has support for: +MVSAM CRYPTO PMD has support for: * Symmetric crypto * Sym operation chaining @@ -73,22 +73,22 @@ Limitations Installation ------------ -MRVL CRYPTO PMD driver compilation is disabled by default due to external dependencies. +MVSAM CRYPTO PMD driver compilation is disabled by default due to external dependencies. Currently there are two driver specific compilation options in ``config/common_base`` available: -- ``CONFIG_RTE_LIBRTE_MRVL_CRYPTO`` (default ``n``) +- ``CONFIG_RTE_LIBRTE_MVSAM_CRYPTO`` (default ``n``) - Toggle compilation of the librte_pmd_mrvl driver. + Toggle compilation of the librte_pmd_mvsam driver. -- ``CONFIG_RTE_LIBRTE_MRVL_CRYPTO_DEBUG`` (default ``n``) +- ``CONFIG_RTE_LIBRTE_MVSAM_CRYPTO_DEBUG`` (default ``n``) Toggle display of debugging messages. For a list of prerequisites please refer to `Prerequisites` section in -:ref:`MRVL Poll Mode Driver <mrvl_poll_mode_driver>` guide. +:ref:`MVPP2 Poll Mode Driver <mvpp2_poll_mode_driver>` guide. -MRVL CRYPTO PMD requires MUSDK built with EIP197 support thus following +MVSAM CRYPTO PMD requires MUSDK built with EIP197 support thus following extra option must be passed to the library configuration script: .. code-block:: console @@ -101,7 +101,7 @@ to `doc/musdk_get_started.txt`. Initialization -------------- -After successfully building MRVL CRYPTO PMD, the following modules need to be +After successfully building MVSAM CRYPTO PMD, the following modules need to be loaded: .. code-block:: console @@ -118,12 +118,12 @@ The following parameters (all optional) are exported by the driver: * max_nb_sessions: maximum number of sessions that can be created (2048 by default). * socket_id: socket on which to allocate the device resources on. -l2fwd-crypto example application can be used to verify MRVL CRYPTO PMD +l2fwd-crypto example application can be used to verify MVSAM CRYPTO PMD operation: .. code-block:: console - ./l2fwd-crypto --vdev=net_mrvl,iface=eth0 --vdev=crypto_mrvl -- \ + ./l2fwd-crypto --vdev=eth_mvpp2,iface=eth0 --vdev=crypto_mvsam -- \ --cipher_op ENCRYPT --cipher_algo aes-cbc \ --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f \ --auth_op GENERATE --auth_algo sha1-hmac \ diff --git a/doc/guides/cryptodevs/openssl.rst b/doc/guides/cryptodevs/openssl.rst index 427fc807..bdc30f66 100644 --- a/doc/guides/cryptodevs/openssl.rst +++ b/doc/guides/cryptodevs/openssl.rst @@ -80,6 +80,7 @@ crypto processing. Test name is cryptodev_openssl_autotest. For performance test cryptodev_openssl_perftest can be used. +For asymmetric crypto operations testing, run cryptodev_openssl_asym_autotest. To verify real traffic l2fwd-crypto example can be used with this command: diff --git a/doc/guides/cryptodevs/overview.rst b/doc/guides/cryptodevs/overview.rst index b3cb6cae..3f776f07 100644 --- a/doc/guides/cryptodevs/overview.rst +++ b/doc/guides/cryptodevs/overview.rst @@ -11,6 +11,33 @@ Supported Feature Flags .. include:: overview_feature_table.txt +.. Note:: + + - "In Place SGL" feature flag stands for "In place Scatter-gather list", + which means that an input buffer can consist of multiple segments, + being the operation in-place (input address = output address). + + - "OOP SGL In SGL Out" feature flag stands for + "Out-of-place Scatter-gather list Input, Scatter-gater list Output", + which means pmd supports different scatter-gather styled input and output buffers + (i.e. both can consists of multiple segments). + + - "OOP SGL In LB Out" feature flag stands for + "Out-of-place Scatter-gather list Input, Linear Buffers Output", + which means PMD supports input from scatter-gathered styled buffers, + outputting linear buffers (i.e. single segment). + + - "OOP LB In SGL Out" feature flag stands for + "Out-of-place Linear Buffers Input, Scatter-gather list Output", + which means PMD supports input from linear buffer, outputting + scatter-gathered styled buffers. + + - "OOP LB In LB Out" feature flag stands for + "Out-of-place Linear Buffers Input, Scatter-gather list Output", + which means that Out-of-place operation is supported, + with linear input and output buffers. + + Supported Cipher Algorithms --------------------------- diff --git a/doc/guides/cryptodevs/qat.rst b/doc/guides/cryptodevs/qat.rst index 8c8fefaa..bdc58eb2 100644 --- a/doc/guides/cryptodevs/qat.rst +++ b/doc/guides/cryptodevs/qat.rst @@ -68,12 +68,32 @@ Limitations * Queue pairs are not thread-safe (that is, within a single queue pair, RX and TX from different lcores is not supported). -Installation ------------- +Extra notes on KASUMI F9 +------------------------ + +When using KASUMI F9 authentication algorithm, the input buffer must be +constructed according to the 3GPP KASUMI specifications (section 4.4, page 13): +`<http://cryptome.org/3gpp/35201-900.pdf>`_. +Input buffer has to have COUNT (4 bytes), FRESH (4 bytes), MESSAGE and DIRECTION (1 bit) +concatenated. After the DIRECTION bit, a single '1' bit is appended, followed by +between 0 and 7 '0' bits, so that the total length of the buffer is multiple of 8 bits. +Note that the actual message can be any length, specified in bits. + +Once this buffer is passed this way, when creating the crypto operation, +length of data to authenticate (op.sym.auth.data.length) must be the length +of all the items described above, including the padding at the end. +Also, offset of data to authenticate (op.sym.auth.data.offset) +must be such that points at the start of the COUNT bytes. + + +Building the DPDK QAT cryptodev PMD +----------------------------------- + -To enable QAT in DPDK, follow the instructions for modifying the compile-time +To enable QAT crypto in DPDK, follow the instructions for modifying the compile-time configuration file as described `here <http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html>`_. + Quick instructions are as follows: .. code-block:: console @@ -81,29 +101,95 @@ Quick instructions are as follows: cd to the top-level DPDK directory make config T=x86_64-native-linuxapp-gcc sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_QAT\)=n,\1=y,' build/.config + sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_QAT_SYM\)=n,\1=y,' build/.config make -To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The VF -devices exposed by this driver will be used by the QAT PMD. The devices and -available kernel drivers and device ids are : + +.. _qat_kernel_installation: + +Dependency on the QAT kernel driver +----------------------------------- + +To use the QAT PMD an SRIOV-enabled QAT kernel driver is required. The VF +devices created and initialised by this driver will be used by the QAT PMD. + +Instructions for installation are below, but first an explanation of the +relationships between the PF/VF devices and the PMDs visible to +DPDK applications. + + +Acceleration services - cryptography and compression - are provided to DPDK +applications via PMDs which register to implement the corresponding +cryptodev and compressdev APIs. + +Each QuickAssist VF device can expose one cryptodev PMD and/or one compressdev PMD. +These QAT PMDs share the same underlying device and pci-mgmt code, but are +enumerated independently on their respective APIs and appear as independent +devices to applications. + +.. Note:: + + Each VF can only be used by one DPDK process. It is not possible to share + the same VF across multiple processes, even if these processes are using + different acceleration services. + + Conversely one DPDK process can use one or more QAT VFs and can expose both + cryptodev and compressdev instances on each of those VFs. + + + +Device and driver naming +------------------------ + +* The qat cryptodev driver name is "crypto_qat". + The rte_cryptodev_devices_get() returns the devices exposed by this driver. + +* Each qat crypto device has a unique name, in format + <pci bdf>_<service>, e.g. "0000:41:01.0_qat_sym". + This name can be passed to rte_cryptodev_get_dev_id() to get the device_id. + +.. Note:: + + The qat crypto driver name is passed to the dpdk-test-crypto-perf tool in the -devtype parameter. + + The qat crypto device name is in the format of the slave parameter passed to the crypto scheduler. + +* The qat compressdev driver name is "comp_qat". + The rte_compressdev_devices_get() returns the devices exposed by this driver. + +* Each qat compression device has a unique name, in format + <pci bdf>_<service>, e.g. "0000:41:01.0_qat_comp". + This name can be passed to rte_compressdev_get_dev_id() to get the device_id. + + +Available kernel drivers +------------------------ + +Kernel drivers for each device are listed in the following table. Scroll right +to check that the driver and device supports the servic you require. + .. _table_qat_pmds_drivers: .. table:: QAT device generations, devices and drivers - +-----+----------+--------+---------------+------------+--------+------+--------+--------+ - | Gen | Device | Driver | Kernel Module | Pci Driver | PF Did | #PFs | Vf Did | VFs/PF | - +=====+==========+========+===============+============+========+======+========+========+ - | 1 | DH895xCC | 01.org | icp_qa_al | n/a | 435 | 1 | 443 | 32 | - +-----+----------+--------+---------------+------------+--------+------+--------+--------+ - | 1 | DH895xCC | 4.4+ | qat_dh895xcc | dh895xcc | 435 | 1 | 443 | 32 | - +-----+----------+--------+---------------+------------+--------+------+--------+--------+ - | 2 | C62x | 4.5+ | qat_c62x | c6xx | 37c8 | 3 | 37c9 | 16 | - +-----+----------+--------+---------------+------------+--------+------+--------+--------+ - | 2 | C3xxx | 4.5+ | qat_c3xxx | c3xxx | 19e2 | 1 | 19e3 | 16 | - +-----+----------+--------+---------------+------------+--------+------+--------+--------+ - | 2 | D15xx | p | qat_d15xx | d15xx | 6f54 | 1 | 6f55 | 16 | - +-----+----------+--------+---------------+------------+--------+------+--------+--------+ + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | Gen | Device | Driver/ver | Kernel Module | Pci Driver | PF Did | #PFs | VF Did | VFs/PF | cryptodev | compressdev | + +=====+==========+===============+===============+============+========+======+========+========+===========+=============+ + | 1 | DH895xCC | linux/4.4+ | qat_dh895xcc | dh895xcc | 435 | 1 | 443 | 32 | Yes | No | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | " | " | 01.org/4.2.0+ | " | " | " | " | " | " | Yes | No | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | 2 | C62x | linux/4.5+ | qat_c62x | c6xx | 37c8 | 3 | 37c9 | 16 | Yes | No | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | " | " | 01.org/4.2.0+ | " | " | " | " | " | " | Yes | Yes | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | 2 | C3xxx | linux/4.5+ | qat_c3xxx | c3xxx | 19e2 | 1 | 19e3 | 16 | Yes | No | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | " | " | 01.org/4.2.0+ | " | " | " | " | " | " | Yes | Yes | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ + | 2 | D15xx | p | qat_d15xx | d15xx | 6f54 | 1 | 6f55 | 16 | Yes | No | + +-----+----------+---------------+---------------+------------+--------+------+--------+--------+-----------+-------------+ The ``Driver`` column indicates either the Linux kernel version in which @@ -196,9 +282,9 @@ Consult the *Getting Started Guide* at the same URL for further information. The steps below assume you are: -* Building on a platform with one ``DH895xCC`` device. -* Using package ``qatmux.l.2.3.0-34.tgz``. -* On Fedora21 kernel ``3.17.4-301.fc21.x86_64``. +* Building on a platform with one ``C62x`` device. +* Using package ``qat1.7.l.4.2.0-000xx.tar.gz``. +* On Fedora26 kernel ``4.11.11-300.fc26.x86_64``. In the BIOS ensure that SRIOV is enabled and VT-d is disabled. @@ -206,21 +292,30 @@ Uninstall any existing QAT driver, for example by running: * ``./installer.sh uninstall`` in the directory where originally installed. -* or ``rmmod qat_dh895xcc; rmmod intel_qat``. Build and install the SRIOV-enabled QAT driver:: mkdir /QAT cd /QAT - # Copy qatmux.l.2.3.0-34.tgz to this location - tar zxof qatmux.l.2.3.0-34.tgz + # Copy the package to this location and unpack + tar zxof qat1.7.l.4.2.0-000xx.tar.gz + + ./configure --enable-icp-sriov=host + make install + +You can use ``cat /sys/kernel/debug/qat<your device type and bdf>/version/fw`` to confirm the driver is correctly installed and is using firmware version 4.2.0. +You can use ``lspci -d:37c9`` to confirm the presence of the 16 VF devices available per ``C62x`` PF. + +Confirm the driver is correctly installed and is using firmware version 4.2.0:: + + cat /sys/kernel/debug/qat<your device type and bdf>/version/fw - export ICP_WITHOUT_IOMMU=1 - ./installer.sh install QAT1.6 host -You can use ``cat /proc/icp_dh895xcc_dev0/version`` to confirm the driver is correctly installed. -You can use ``lspci -d:443`` to confirm the of the 32 VF devices available per ``DH895xCC`` device. +Confirm the presence of 48 VF devices - 16 per PF:: + + lspci -d:37c9 + To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. @@ -261,6 +356,7 @@ To complete the installation - follow instructions in `Binding the available VFs sudo yum install zlib-devel sudo yum install openssl-devel + sudo yum install libudev-devel .. Note:: @@ -343,19 +439,28 @@ Another way to bind the VFs to the DPDK UIO driver is by using the ./usertools/dpdk-devbind.py -b igb_uio 0000:03:01.1 -Extra notes on KASUMI F9 ------------------------- +Debugging +---------------------------------------- -When using KASUMI F9 authentication algorithm, the input buffer must be -constructed according to the 3GPP KASUMI specifications (section 4.4, page 13): -`<http://cryptome.org/3gpp/35201-900.pdf>`_. -Input buffer has to have COUNT (4 bytes), FRESH (4 bytes), MESSAGE and DIRECTION (1 bit) -concatenated. After the DIRECTION bit, a single '1' bit is appended, followed by -between 0 and 7 '0' bits, so that the total length of the buffer is multiple of 8 bits. -Note that the actual message can be any length, specified in bits. +There are 2 sets of trace available via the dynamic logging feature: -Once this buffer is passed this way, when creating the crypto operation, -length of data to authenticate (op.sym.auth.data.length) must be the length -of all the items described above, including the padding at the end. -Also, offset of data to authenticate (op.sym.auth.data.offset) -must be such that points at the start of the COUNT bytes. +* pmd.qat_dp exposes trace on the data-path. +* pmd.qat_general exposes all other trace. + +pmd.qat exposes both sets of traces. +They can be enabled using the log-level option (where 8=maximum log level) on +the process cmdline, e.g. using any of the following:: + + --log-level="pmd.qat_general,8" + --log-level="pmd.qat_dp,8" + --log-level="pmd.qat,8" + +.. Note:: + + The global RTE_LOG_DP_LEVEL overrides data-path trace so must be set to + RTE_LOG_DEBUG to see all the trace. This variable is in config/rte_config.h + for meson build and config/common_base for gnu make. + Also the dynamic global log level overrides both sets of trace, so e.g. no + QAT trace would display in this case:: + + --log-level="7" --log-level="pmd.qat_general,8" diff --git a/doc/guides/cryptodevs/scheduler.rst b/doc/guides/cryptodevs/scheduler.rst index d67894d5..a754a27e 100644 --- a/doc/guides/cryptodevs/scheduler.rst +++ b/doc/guides/cryptodevs/scheduler.rst @@ -71,6 +71,11 @@ two calls: mode parameter values are specified in the "Cryptodev Scheduler Modes Overview" section. +* mode_param: Specify the mode-specific parameter. Some scheduling modes + may be initialized with specific parameters other than the default ones, + such as the **threshold** packet size of **packet-size-distr** mode. This + parameter fulfills the purpose. + * ordering: Specify the status of the crypto operations ordering feature. The value of this parameter can be "enable" or "disable". This feature is disabled by default. @@ -132,7 +137,12 @@ operation: **option_type** must be **CDEV_SCHED_OPTION_THRESHOLD** and **option** should point to a rte_cryptodev_scheduler_threshold_option structure filled with appropriate threshold value. Please NOTE this threshold has be a power-of-2 - unsigned integer. + unsigned integer. It is possible to use **mode_param** initialization + parameter to achieve the same purpose. For example: + + ... --vdev "crypto_scheduler,mode=packet-size-distr,mode_param=threshold:512" ... + + The above parameter will overwrite the threshold value to 512. * **CDEV_SCHED_MODE_FAILOVER:** diff --git a/doc/guides/cryptodevs/snow3g.rst b/doc/guides/cryptodevs/snow3g.rst index 24b4f661..7cba712c 100644 --- a/doc/guides/cryptodevs/snow3g.rst +++ b/doc/guides/cryptodevs/snow3g.rst @@ -33,11 +33,11 @@ Installation ------------ To build DPDK with the SNOW3G_PMD the user is required to download -the export controlled ``libsso_snow3g`` library, by requesting it from -`<https://networkbuilders.intel.com/network-technologies/dpdk>`_. -Once approval has been granted, the user needs to log in -`<https://networkbuilders.intel.com/dpdklogin>`_ -and click on "Snow3G Bit Stream crypto library" link, to download the library. +the export controlled ``libsso_snow3g`` library, by registering in +`Intel Resource & Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_. +Once approval has been granted, the user needs to search for +*Snow3G F8 F9 3GPP cryptographic algorithms Software Library* to download the +library or directly through this `link <https://cdrdv2.intel.com/v1/dl/getContent/575867>`_. After downloading the library, the user needs to unpack and compile it on their system before building DPDK:: diff --git a/doc/guides/cryptodevs/virtio.rst b/doc/guides/cryptodevs/virtio.rst new file mode 100644 index 00000000..f3aa7c65 --- /dev/null +++ b/doc/guides/cryptodevs/virtio.rst @@ -0,0 +1,117 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 HUAWEI TECHNOLOGIES CO., LTD. + +Virtio Crypto Poll Mode Driver +============================== + +The virtio crypto PMD provides poll mode driver support for the virtio crypto +device. + +Features +-------- + +The virtio crypto PMD has support for: + +Cipher algorithms: + +* ``RTE_CRYPTO_CIPHER_AES_CBC`` + +Hash algorithms: + +* ``RTE_CRYPTO_AUTH_SHA1_HMAC`` + +Limitations +----------- + +* Only supports the session-oriented API implementation (session-less APIs are + not supported). +* Only supports modern mode since virtio crypto conforms to virtio-1.0. +* Only has two types of queues: data queue and control queue. These two queues + only support indirect buffers to communication with the virtio backend. +* Only supports AES_CBC cipher only algorithm and AES_CBC with HMAC_SHA1 + chaining algorithm since the vhost crypto backend only these algorithms + are supported. +* Does not support Link State interrupt. +* Does not support runtime configuration. + +Virtio crypto PMD Rx/Tx Callbacks +--------------------------------- + +Rx callbacks: + +* ``virtio_crypto_pkt_rx_burst`` + +Tx callbacks: + +* ``virtio_crypto_pkt_tx_burst`` + +Installation +------------ + +Quick instructions are as follows: + +Firstly run DPDK vhost crypto sample as a server side and build QEMU with +vhost crypto enabled. +QEMU can then be started using the following parameters: + +.. code-block:: console + + qemu-system-x86_64 \ + [...] \ + -chardev socket,id=charcrypto0,path=/path/to/your/socket \ + -object cryptodev-vhost-user,id=cryptodev0,chardev=charcrypto0 \ + -device virtio-crypto-pci,id=crypto0,cryptodev=cryptodev0 + [...] + +Secondly bind the uio_generic driver for the virtio-crypto device. +For example, 0000:00:04.0 is the domain, bus, device and function +number of the virtio-crypto device: + +.. code-block:: console + + modprobe uio_pci_generic + echo -n 0000:00:04.0 > /sys/bus/pci/drivers/virtio-pci/unbind + echo "1af4 1054" > /sys/bus/pci/drivers/uio_pci_generic/new_id + +Finally the front-end virtio crypto PMD driver can be installed: + +.. code-block:: console + + cd to the top-level DPDK directory + sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_VIRTIO_CRYPTO\)=n,\1=y,' config/common_base + make config T=x86_64-native-linuxapp-gcc + make install T=x86_64-native-linuxapp-gcc + +Tests +----- + +The unit test cases can be tested as below: + +.. code-block:: console + + reserve enough huge pages + cd to the top-level DPDK directory + export RTE_TARGET=x86_64-native-linuxapp-gcc + export RTE_SDK=`pwd` + cd to test/test + type the command "make" to compile + run the tests with "./test" + type the command "cryptodev_virtio_autotest" to test + +The performance can be tested as below: + +.. code-block:: console + + reserve enough huge pages + cd to the top-level DPDK directory + export RTE_TARGET=x86_64-native-linuxapp-gcc + export RTE_SDK=`pwd` + cd to app/test-crypto-perf + type the command "make" to compile + run the tests with the following command: + + ./dpdk-test-crypto-perf -l 0,1 -- --devtype crypto_virtio \ + --ptest throughput --optype cipher-then-auth --cipher-algo aes-cbc \ + --cipher-op encrypt --cipher-key-sz 16 --auth-algo sha1-hmac \ + --auth-op generate --auth-key-sz 64 --digest-sz 12 \ + --total-ops 100000000 --burst-sz 64 --buffer-sz 2048 diff --git a/doc/guides/cryptodevs/zuc.rst b/doc/guides/cryptodevs/zuc.rst index e226ef9d..e3898996 100644 --- a/doc/guides/cryptodevs/zuc.rst +++ b/doc/guides/cryptodevs/zuc.rst @@ -35,11 +35,11 @@ Installation ------------ To build DPDK with the ZUC_PMD the user is required to download -the export controlled ``libsso_zuc`` library, by requesting it from -`<https://networkbuilders.intel.com/network-technologies/dpdk>`_. -Once approval has been granted, the user needs to log in -`<https://networkbuilders.intel.com/dpdklogin>`_ -and click on "ZUC Library" link, to download the library. +the export controlled ``libsso_zuc`` library, by registering in +`Intel Resource & Design Center <https://www.intel.com/content/www/us/en/design/resource-design-center.html>`_. +Once approval has been granted, the user needs to search for +*ZUC 128-EAA3 and 128-EIA3 3GPP cryptographic algorithms Software Library* to download the +library or directly through this `link <https://cdrdv2.intel.com/v1/dl/getContent/575868>`_. After downloading the library, the user needs to unpack and compile it on their system before building DPDK:: diff --git a/doc/guides/eventdevs/dpaa2.rst b/doc/guides/eventdevs/dpaa2.rst index 5b8da95d..ad94f24b 100644 --- a/doc/guides/eventdevs/dpaa2.rst +++ b/doc/guides/eventdevs/dpaa2.rst @@ -129,7 +129,19 @@ Example: .. code-block:: console - ./your_eventdev_application --vdev="event_dpaa2" + ./your_eventdev_application --vdev="event_dpaa2" + +Enabling logs +------------- + +For enabling logs, use the following EAL parameter: + +.. code-block:: console + + ./your_eventdev_application <EAL args> --log-level=pmd.event.dpaa2,<level> + +Using ``eventdev.dpaa2`` as log matching criteria, all Event PMD logs can be +enabled which are lower than logging ``level``. Limitations ----------- diff --git a/doc/guides/eventdevs/octeontx.rst b/doc/guides/eventdevs/octeontx.rst index 4fabe54f..18cfc7a9 100644 --- a/doc/guides/eventdevs/octeontx.rst +++ b/doc/guides/eventdevs/octeontx.rst @@ -28,6 +28,9 @@ Features of the OCTEONTX SSOVF PMD are: - Open system with configurable amount of outstanding events - HW accelerated dequeue timeout support to enable power management - SR-IOV VF +- HW managed event timers support through TIMVF, with high precision and + time granularity of 1us. +- Up to 64 event timer adapters. Supported OCTEONTX SoCs ----------------------- @@ -36,7 +39,7 @@ Supported OCTEONTX SoCs Prerequisites ------------- -See :doc: `../platform/octeontx` for setup information. +See :doc:`../platform/octeontx` for setup information. Pre-Installation Configuration ------------------------------ @@ -93,7 +96,17 @@ The tests are run once the vdev creation is successfully complete. .. code-block:: console - --vdev="event_octeontx,self_test=1" + --vdev="event_octeontx,selftest=1" + + +Enable TIMvf stats +------------------ +TIMvf stats can be enabled by using this option, by default the stats are +disabled. + +.. code-block:: console + + --vdev="event_octeontx,timvf_stats=1" Limitations @@ -110,3 +123,19 @@ Rx adapter support When eth_octeontx is used as Rx adapter event schedule type ``RTE_SCHED_TYPE_PARALLEL`` is not supported. + +Event timer adapter support +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When timvf is used as Event timer adapter the clock source mapping is as +follows: + +.. code-block:: console + + RTE_EVENT_TIMER_ADAPTER_CPU_CLK = TIM_CLK_SRC_SCLK + RTE_EVENT_TIMER_ADAPTER_EXT_CLK0 = TIM_CLK_SRC_GPIO + RTE_EVENT_TIMER_ADAPTER_EXT_CLK1 = TIM_CLK_SRC_GTI + RTE_EVENT_TIMER_ADAPTER_EXT_CLK2 = TIM_CLK_SRC_PTP + +When timvf is used as Event timer adapter event schedule type +``RTE_SCHED_TYPE_PARALLEL`` is not supported. diff --git a/doc/guides/faq/faq.rst b/doc/guides/faq/faq.rst index c1132fb4..f19c1389 100644 --- a/doc/guides/faq/faq.rst +++ b/doc/guides/faq/faq.rst @@ -62,19 +62,16 @@ the wrong socket, the application simply will not start. On application startup, there is a lot of EAL information printed. Is there any way to reduce this? --------------------------------------------------------------------------------------------------- -Yes, the option ``--log-level=`` accepts one of these numbers: - -.. code-block:: c - - #define RTE_LOG_EMERG 1U /* System is unusable. */ - #define RTE_LOG_ALERT 2U /* Action must be taken immediately. */ - #define RTE_LOG_CRIT 3U /* Critical conditions. */ - #define RTE_LOG_ERR 4U /* Error conditions. */ - #define RTE_LOG_WARNING 5U /* Warning conditions. */ - #define RTE_LOG_NOTICE 6U /* Normal but significant condition. */ - #define RTE_LOG_INFO 7U /* Informational. */ - #define RTE_LOG_DEBUG 8U /* Debug-level messages. */ - +Yes, the option ``--log-level=`` accepts either symbolic names (or numbers): + +1. emergency +2. alert +3. critical +4. error +5. warning +6. notice +7. info +8. debug How can I tune my network application to achieve lower latency? --------------------------------------------------------------- diff --git a/doc/guides/freebsd_gsg/build_sample_apps.rst b/doc/guides/freebsd_gsg/build_sample_apps.rst index 90f05023..a085b618 100644 --- a/doc/guides/freebsd_gsg/build_sample_apps.rst +++ b/doc/guides/freebsd_gsg/build_sample_apps.rst @@ -128,6 +128,9 @@ The EAL options for FreeBSD are as follows: * ``--proc-type``: The type of process instance. +* ``-m MB``: + Memory to allocate from hugepages, regardless of processor socket. + Other options, specific to Linux and are not supported under FreeBSD are as follows: * ``socket-mem``: @@ -142,10 +145,6 @@ Other options, specific to Linux and are not supported under FreeBSD are as foll * ``--file-prefix``: The prefix text used for hugepage filenames. -* ``-m MB``: - Memory to allocate from hugepages, regardless of processor socket. - It is recommended that ``--socket-mem`` be used instead of this option. - The ``-c`` or ``-l`` option is mandatory; the others are optional. Copy the DPDK application binary to your target, then run the application diff --git a/doc/guides/howto/rte_flow.rst b/doc/guides/howto/rte_flow.rst index 4a27c995..caa4e1af 100644 --- a/doc/guides/howto/rte_flow.rst +++ b/doc/guides/howto/rte_flow.rst @@ -1,33 +1,5 @@ -.. BSD LICENSE - Copyright(c) 2017 Mellanox Corporation. All rights reserved. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Mellanox Corporation nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 Mellanox Technologies, Ltd Generic flow API - examples =========================== @@ -276,7 +248,7 @@ Code /* end the pattern array */ pattern[2].type = RTE_FLOW_ITEM)TYPE_END; - /* create the drop action */ + /* create the queue action */ actions[0].type = RTE_FLOW_ACTION_TYPE_QUEUE; actions[0].conf = &queue actions[1].type = RTE_FLOW_ACTION_TYPE_END; diff --git a/doc/guides/howto/virtio_user_for_container_networking.rst b/doc/guides/howto/virtio_user_for_container_networking.rst index aa68b531..476ce3a6 100644 --- a/doc/guides/howto/virtio_user_for_container_networking.rst +++ b/doc/guides/howto/virtio_user_for_container_networking.rst @@ -109,7 +109,8 @@ We have below limitations in this solution: * Cannot work with --no-huge option. Currently, DPDK uses anonymous mapping under this option which cannot be reopened to share with vhost backend. * Cannot work when there are more than VHOST_MEMORY_MAX_NREGIONS(8) hugepages. - In another word, do not use 2MB hugepage so far. + If you have more regions (especially when 2MB hugepages are used), the option, + --single-file-segments, can help to reduce the number of shared files. * Applications should not use file name like HUGEFILE_FMT ("%smap_%d"). That will bring confusion when sharing hugepage files with backend by name. * Root privilege is a must. DPDK resolves physical addresses of hugepages diff --git a/doc/guides/index.rst b/doc/guides/index.rst index d60529dd..8a9ed65c 100644 --- a/doc/guides/index.rst +++ b/doc/guides/index.rst @@ -17,7 +17,9 @@ DPDK documentation nics/index bbdevs/index cryptodevs/index + compressdevs/index eventdevs/index + rawdevs/index mempool/index platform/index contributing/index diff --git a/doc/guides/linux_gsg/build_sample_apps.rst b/doc/guides/linux_gsg/build_sample_apps.rst index 39a47b71..332424e0 100644 --- a/doc/guides/linux_gsg/build_sample_apps.rst +++ b/doc/guides/linux_gsg/build_sample_apps.rst @@ -110,7 +110,13 @@ The EAL options are as follows: ``[domain:]bus:devid.func`` values. Cannot be used with ``-b`` option. * ``--socket-mem``: - Memory to allocate from hugepages on specific sockets. + Memory to allocate from hugepages on specific sockets. In dynamic memory mode, + this memory will also be pinned (i.e. not released back to the system until + application closes). + +* ``--socket-limit``: + Limit maximum memory available for allocation on each socket. Does not support + legacy memory mode. * ``-d``: Add a driver or driver directory to be loaded. @@ -148,6 +154,14 @@ The EAL options are as follows: * ``--vfio-intr``: Specify interrupt type to be used by VFIO (has no effect if VFIO is not used). +* ``--legacy-mem``: + Run DPDK in legacy memory mode (disable memory reserve/unreserve at runtime, + but provide more IOVA-contiguous memory). + +* ``--single-file-segments``: + Store memory segments in fewer files (dynamic memory mode only - does not + affect legacy memory mode). + The ``-c`` or ``-l`` and option is mandatory; the others are optional. Copy the DPDK application binary to your target, then run the application as follows diff --git a/doc/guides/linux_gsg/cross_build_dpdk_for_arm64.rst b/doc/guides/linux_gsg/cross_build_dpdk_for_arm64.rst new file mode 100644 index 00000000..9d1f0fa0 --- /dev/null +++ b/doc/guides/linux_gsg/cross_build_dpdk_for_arm64.rst @@ -0,0 +1,132 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 ARM Corporation. + +Cross compile DPDK for ARM64 +============================ +This chapter describes how to cross compile DPDK for ARM64 from x86 build hosts. + +.. note:: + + Whilst it is recommended to natively build DPDK on ARM64 (just + like with x86), it is also possible to cross-build DPDK for ARM64. An + ARM64 cross compile GNU toolchain is used for this. + +Obtain the cross tool chain +--------------------------- +The latest cross compile tool chain can be downloaded from: +https://releases.linaro.org/components/toolchain/binaries/latest/aarch64-linux-gnu/. + +Following is the step to get the version 7.2.1, latest one at the time of this writing. + +.. code-block:: console + + wget https://releases.linaro.org/components/toolchain/binaries/latest/aarch64-linux-gnu/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu.tar.xz + +Unzip and add into the PATH +--------------------------- + +.. code-block:: console + + tar -xvf gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu.tar.xz + export PATH=$PATH:<cross_install_dir>/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin + +.. note:: + + For the host requirements and other info, refer to the release note section: https://releases.linaro.org/components/toolchain/binaries/latest/ + +Getting the prerequisite library +-------------------------------- + +NUMA is required by most modern machines, not needed for non-NUMA architectures. + +.. note:: + + For compiling the NUMA lib, run libtool --version to ensure the libtool version >= 2.2, + otherwise the compilation will fail with errors. + +.. code-block:: console + + git clone https://github.com/numactl/numactl.git + cd numactl + git checkout v2.0.11 -b v2.0.11 + ./autogen.sh + autoconf -i + ./configure --host=aarch64-linux-gnu CC=aarch64-linux-gnu-gcc --prefix=<numa install dir> + make install + +The numa header files and lib file is generated in the include and lib folder respectively under <numa install dir>. + +.. _augment_the_cross_toolchain_with_numa_support: + +Augment the cross toolchain with NUMA support +--------------------------------------------- + +.. note:: + + This way is optional, an alternative is to use extra CFLAGS and LDFLAGS, depicted in :ref:`configure_and_cross_compile_dpdk_build` below. + +Copy the NUMA header files and lib to the cross compiler's directories: + +.. code-block:: console + + cp <numa_install_dir>/include/numa*.h <cross_install_dir>/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/bin/../aarch64-linux-gnu/libc/usr/include/ + cp <numa_install_dir>/lib/libnuma.a <cross_install_dir>/gcc-linaro-7.2.1-2017.11-x86_64_aarch64-linux-gnu/lib/gcc/aarch64-linux-gnu/7.2.1/ + +.. _configure_and_cross_compile_dpdk_build: + +Configure and cross compile DPDK Build +-------------------------------------- +To configure a build, choose one of the target configurations, like arm64-dpaa2-linuxapp-gcc and arm64-thunderx-linuxapp-gcc. + +.. code-block:: console + + make config T=arm64-armv8a-linuxapp-gcc + +To cross-compile, without compiling the kernel modules, use the following command: + +.. code-block:: console + + make -j CROSS=aarch64-linux-gnu- CONFIG_RTE_KNI_KMOD=n CONFIG_RTE_EAL_IGB_UIO=n + +To cross-compile, including the kernel modules, the kernel source tree needs to be specified by setting +RTE_KERNELDIR: + +.. code-block:: console + + make -j CROSS=aarch64-linux-gnu- RTE_KERNELDIR=<kernel_src_rootdir> CROSS_COMPILE=aarch64-linux-gnu- + +To compile for non-NUMA targets, without compiling the kernel modules, use the following command: + +.. code-block:: console + + make -j CROSS=aarch64-linux-gnu- CONFIG_RTE_KNI_KMOD=n CONFIG_RTE_EAL_IGB_UIO=n CONFIG_RTE_LIBRTE_VHOST_NUMA=n CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n + +.. note:: + + 1. EXTRA_CFLAGS and EXTRA_LDFLAGS should be added to include the NUMA headers and link the library respectively, + if the above step :ref:`augment_the_cross_toolchain_with_numa_support` was skipped therefore the toolchain was not + augmented with NUMA support. + + 2. "-isystem <numa_install_dir>/include" should be add to EXTRA_CFLAGS, otherwise the numa.h file will get a lot of compiling + errors of Werror=cast-qual, Werror=strict-prototypes and Werror=old-style-definition. + + An example is given below: + + .. code-block:: console + + make -j CROSS=aarch64-linux-gnu- CONFIG_RTE_KNI_KMOD=n CONFIG_RTE_EAL_IGB_UIO=n EXTRA_CFLAGS="-isystem <numa_install_dir>/include" EXTRA_LDFLAGS="-L<numa_install_dir>/lib -lnuma" + +Meson Cross Compiling DPDK +-------------------------- + +To cross-compile DPDK on a desired target machine we can use the following +command:: + + meson cross-build --cross-file <target_machine_configuration> + ninja -C cross-build + +For example if the target machine is arm64 we can use the following +command:: + + meson arm64-build --cross-file config/arm/arm64_armv8_linuxapp_gcc + ninja -C arm64-build diff --git a/doc/guides/linux_gsg/index.rst b/doc/guides/linux_gsg/index.rst index 2a7bdfe9..077f9302 100644 --- a/doc/guides/linux_gsg/index.rst +++ b/doc/guides/linux_gsg/index.rst @@ -13,6 +13,7 @@ Getting Started Guide for Linux intro sys_reqs build_dpdk + cross_build_dpdk_for_arm64 linux_drivers build_sample_apps enable_func diff --git a/doc/guides/linux_gsg/linux_drivers.rst b/doc/guides/linux_gsg/linux_drivers.rst index 14381eed..371a817f 100644 --- a/doc/guides/linux_gsg/linux_drivers.rst +++ b/doc/guides/linux_gsg/linux_drivers.rst @@ -1,6 +1,6 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2010-2015 Intel Corporation. - Copyright(c) 2017 Mellanox Corporation. + Copyright 2017 Mellanox Technologies, Ltd All rights reserved. .. _linux_gsg_linux_drivers: diff --git a/doc/guides/nics/axgbe.rst b/doc/guides/nics/axgbe.rst new file mode 100644 index 00000000..e30f4944 --- /dev/null +++ b/doc/guides/nics/axgbe.rst @@ -0,0 +1,89 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright (c) 2018 Advanced Micro Devices, Inc. All rights reserved. + +AXGBE Poll Mode Driver +====================== + +The AXGBE poll mode driver library (**librte_pmd_axgbe**) implements support +for AMD 10 Gbps family of adapters. It is compiled and tested in standard linux distro like Ubuntu. + +Detailed information about SoCs that use these devices can be found here: + +- `AMD EPYC™ EMBEDDED 3000 family <https://www.amd.com/en/products/embedded-epyc-3000-series>`_. + + +Supported Features +------------------ + +AXGBE PMD has support for: + +- Base L2 features +- TSS (Transmit Side Scaling) +- Promiscuous mode +- Port statistics +- Multicast mode +- RSS (Receive Side Scaling) +- Checksum offload +- Jumbo Frame upto 9K + + +Configuration Information +------------------------- + +The following options can be modified in the ``.config`` file. Please note that +enabling debugging options may affect system performance. + +- ``CONFIG_RTE_LIBRTE_AXGBE_PMD`` (default **y**) + + Toggle compilation of axgbe PMD. + +- ``CONFIG_RTE_LIBRTE_AXGBE_PMD_DEBUG`` (default **n**) + + Toggle display for PMD debug related messages. + + +Building DPDK +------------- + +See the :ref:`DPDK Getting Started Guide for Linux <linux_gsg>` for +instructions on how to build DPDK. + +By default the AXGBE PMD library will be built into the DPDK library. + +For configuring and using UIO frameworks, please also refer :ref:`the +documentation that comes with DPDK suite <linux_gsg>`. + + +Prerequisites and Pre-conditions +-------------------------------- +- Prepare the system as recommended by DPDK suite. + +- Bind the intended AMD device to ``igb_uio`` or ``vfio-pci`` module. + +Now system is ready to run DPDK application. + + +Usage Example +------------- + +Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` +for details. + +Example output: + +.. code-block:: console + + [...] + EAL: PCI device 0000:02:00.4 on NUMA socket 0 + EAL: probe driver: 1022:1458 net_axgbe + Interactive-mode selected + USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=171456, size=2176, socket=0 + USER1: create a new mbuf pool <mbuf_pool_socket_1>: n=171456, size=2176, socket=1 + USER1: create a new mbuf pool <mbuf_pool_socket_2>: n=171456, size=2176, socket=2 + USER1: create a new mbuf pool <mbuf_pool_socket_3>: n=171456, size=2176, socket=3 + Configuring Port 0 (socket 0) + Port 0: 00:00:1A:1C:6A:17 + Checking link statuses... + Port 0 Link Up - speed 10000 Mbps - full-duplex + Done + testpmd> diff --git a/doc/guides/nics/bnx2x.rst b/doc/guides/nics/bnx2x.rst index 31f146a0..cecbfc2e 100644 --- a/doc/guides/nics/bnx2x.rst +++ b/doc/guides/nics/bnx2x.rst @@ -194,6 +194,7 @@ This section provides instructions to configure SR-IOV with Linux OS. using the instructions outlined in the Application notes below. #. Running testpmd: + (Supply ``--log-level="pmd.net.bnx2x.driver",7`` to view informational messages): Follow instructions available in the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` diff --git a/doc/guides/nics/bnxt.rst b/doc/guides/nics/bnxt.rst index 9826b350..697b97e6 100644 --- a/doc/guides/nics/bnxt.rst +++ b/doc/guides/nics/bnxt.rst @@ -1,46 +1,20 @@ -.. BSD LICENSE - Copyright 2016 Broadcom Limited - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Broadcom Limited nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2016-2018 Broadcom BNXT Poll Mode Driver ===================== The bnxt poll mode library (**librte_pmd_bnxt**) implements support for: - * **Broadcom NetXtreme-C®/NetXtreme-E® BCM5730X and BCM574XX family of - Ethernet Network Controllers** + * **Broadcom NetXtreme-C®/NetXtreme-E®/NetXtreme-S® + BCM5730X / BCM574XX / BCM58000 family of Ethernet Network Controllers** These adapters support Standards compliant 10/25/50/100Gbps 30MPPS full-duplex throughput. Information about the NetXtreme family of adapters can be found in the `NetXtreme® Brand section - <https://www.broadcom.com/products/ethernet-communication-and-switching?technology%5B%5D=88>`_ + <https://www.broadcom.com/products/ethernet-connectivity/controllers/>`_ of the `Broadcom website <http://www.broadcom.com/>`_. * **Broadcom StrataGX® BCM5871X Series of Communucations Processors** diff --git a/doc/guides/nics/cxgbe.rst b/doc/guides/nics/cxgbe.rst index 8651a7be..58d88eef 100644 --- a/doc/guides/nics/cxgbe.rst +++ b/doc/guides/nics/cxgbe.rst @@ -1,32 +1,6 @@ -.. BSD LICENSE - Copyright 2015-2017 Chelsio Communications. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Chelsio Communications nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2014-2018 Chelsio Communications. + All rights reserved. CXGBE Poll Mode Driver ====================== @@ -35,22 +9,28 @@ The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support for **Chelsio Terminator** 10/25/40/100 Gbps family of adapters. CXGBE PMD has support for the latest Linux and FreeBSD operating systems. +CXGBEVF PMD provides poll mode driver support for SR-IOV Virtual functions +and has support for the latest Linux operating systems. + More information can be found at `Chelsio Communications Official Website <http://www.chelsio.com>`_. Features -------- -CXGBE PMD has support for: +CXGBE and CXGBEVF PMD has support for: - Multiple queues for TX and RX - Receiver Side Steering (RSS) + Receiver Side Steering (RSS) on IPv4, IPv6, IPv4-TCP/UDP, IPv6-TCP/UDP. + For 4-tuple, enabling 'RSS on TCP' and 'RSS on TCP + UDP' is supported. - VLAN filtering - Checksum offload - Promiscuous mode - All multicast mode - Port hardware statistics - Jumbo frames +- Flow API - Support for both Wildcard (LE-TCAM) and Exact (HASH) match filters. Limitations ----------- @@ -63,6 +43,8 @@ port. For this reason, one cannot whitelist/blacklist a single port without whitelisting/blacklisting the other ports on the same device. +.. _t5-nics: + Supported Chelsio T5 NICs ------------------------- @@ -71,16 +53,24 @@ Supported Chelsio T5 NICs - 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR - Other T5 NICs: T522-CR +.. _t6-nics: + Supported Chelsio T6 NICs ------------------------- - 25G NICs: T6425-CR, T6225-CR, T6225-LL-CR, T6225-SO-CR - 100G NICs: T62100-CR, T62100-LP-CR, T62100-SO-CR +Supported SR-IOV Chelsio NICs +----------------------------- + +SR-IOV virtual functions are supported on all the Chelsio NICs listed +in :ref:`t5-nics` and :ref:`t6-nics`. + Prerequisites ------------- -- Requires firmware version **1.16.43.0** and higher. Visit +- Requires firmware version **1.17.14.0** and higher. Visit `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware bundled with the latest Chelsio Unified Wire package. @@ -110,6 +100,10 @@ enabling debugging options may affect system performance. Toggle compilation of librte_pmd_cxgbe driver. + .. note:: + + This controls compilation of both CXGBE and CXGBEVF PMD. + - ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**) Toggle display of generic debugging messages. @@ -134,6 +128,28 @@ enabling debugging options may affect system performance. Toggle behaviour to prefer Throughput or Latency. +Runtime Options +~~~~~~~~~~~~~~~ + +The following ``devargs`` options can be enabled at runtime. They must +be passed as part of EAL arguments. For example, + +.. code-block:: console + + testpmd -w 02:00.4,keep_ovlan=1 -- -i + +- ``keep_ovlan`` (default **0**) + + Toggle behaviour to keep/strip outer VLAN in Q-in-Q packets. If + enabled, the outer VLAN tag is preserved in Q-in-Q packets. Otherwise, + the outer VLAN tag is stripped in Q-in-Q packets. + +- ``force_link_up`` (default **0**) + + When set to 1, CXGBEVF PMD always forces link as up for all VFs on + underlying Chelsio NICs. This enables multiple VFs on the same NIC + to send traffic to each other even when the physical link is down. + .. _driver-compilation: Driver compilation and testing @@ -208,7 +224,7 @@ Unified Wire package for Linux operating system are as follows: .. code-block:: console - firmware-version: 1.16.43.0, TP 0.1.4.9 + firmware-version: 1.17.14.0, TP 0.1.4.9 Running testpmd ~~~~~~~~~~~~~~~ @@ -266,7 +282,7 @@ devices managed by librte_pmd_cxgbe in Linux operating system. EAL: PCI memory mapped at 0x7fd7c0200000 EAL: PCI memory mapped at 0x7fd77cdfd000 EAL: PCI memory mapped at 0x7fd7c10b7000 - PMD: rte_cxgbe_pmd: fw: 1.16.43.0, TP: 0.1.4.9 + PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter Interactive-mode selected Configuring Port 0 (socket 0) @@ -286,6 +302,114 @@ devices managed by librte_pmd_cxgbe in Linux operating system. Flow control pause TX/RX is disabled by default and can be enabled via testpmd. Refer section :ref:`flow-control` for more details. +Configuring SR-IOV Virtual Functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section demonstrates how to enable SR-IOV virtual functions +on Chelsio NICs and demonstrates how to run testpmd with SR-IOV +virtual functions. + +#. Load the kernel module: + + .. code-block:: console + + modprobe cxgb4 + +#. Get the PCI bus addresses of the interfaces bound to cxgb4 driver: + + .. code-block:: console + + dmesg | tail -2 + + Example output: + + .. code-block:: console + + cxgb4 0000:02:00.4 p1p1: renamed from eth0 + cxgb4 0000:02:00.4 p1p2: renamed from eth1 + + .. note:: + + Both the interfaces of a Chelsio 2-port adapter are bound to the + same PCI bus address. + +#. Use ifconfig to get the interface name assigned to Chelsio card: + + .. code-block:: console + + ifconfig -a | grep "00:07:43" + + Example output: + + .. code-block:: console + + p1p1 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C0 + p1p2 Link encap:Ethernet HWaddr 00:07:43:2D:EA:C8 + +#. Bring up the interfaces: + + .. code-block:: console + + ifconfig p1p1 up + ifconfig p1p2 up + +#. Instantiate SR-IOV Virtual Functions. PF0..3 can be used for + SR-IOV VFs. Multiple VFs can be instantiated on each of PF0..3. + To instantiate one SR-IOV VF on each PF0 and PF1: + + .. code-block:: console + + echo 1 > /sys/bus/pci/devices/0000\:02\:00.0/sriov_numvfs + echo 1 > /sys/bus/pci/devices/0000\:02\:00.1/sriov_numvfs + +#. Get the PCI bus addresses of the virtual functions: + + .. code-block:: console + + lspci | grep -i "Chelsio" | grep -i "VF" + + Example output: + + .. code-block:: console + + 02:01.0 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF] + 02:01.1 Ethernet controller: Chelsio Communications Inc T540-CR Unified Wire Ethernet Controller [VF] + +#. Running testpmd + + Follow instructions available in the document + :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` + to bind virtual functions and run testpmd. + + Example output: + + .. code-block:: console + + [...] + EAL: PCI device 0000:02:01.0 on NUMA socket 0 + EAL: probe driver: 1425:5803 net_cxgbevf + PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0 + PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9 + PMD: rte_cxgbe_pmd: Chelsio rev 0 + PMD: rte_cxgbe_pmd: No bootstrap loaded + PMD: rte_cxgbe_pmd: No Expansion ROM loaded + PMD: rte_cxgbe_pmd: 0000:02:01.0 Chelsio rev 0 1G/10GBASE-SFP + EAL: PCI device 0000:02:01.1 on NUMA socket 0 + EAL: probe driver: 1425:5803 net_cxgbevf + PMD: rte_cxgbe_pmd: Firmware version: 1.17.14.0 + PMD: rte_cxgbe_pmd: TP Microcode version: 0.1.4.9 + PMD: rte_cxgbe_pmd: Chelsio rev 0 + PMD: rte_cxgbe_pmd: No bootstrap loaded + PMD: rte_cxgbe_pmd: No Expansion ROM loaded + PMD: rte_cxgbe_pmd: 0000:02:01.1 Chelsio rev 0 1G/10GBASE-SFP + Configuring Port 0 (socket 0) + Port 0: 06:44:29:44:40:00 + Configuring Port 1 (socket 0) + Port 1: 06:44:29:44:40:10 + Checking link statuses... + Done + testpmd> + FreeBSD ------- @@ -350,7 +474,7 @@ Unified Wire package for FreeBSD operating system are as follows: .. code-block:: console - dev.t5nex.0.firmware_version: 1.16.43.0 + dev.t5nex.0.firmware_version: 1.17.14.0 Running testpmd ~~~~~~~~~~~~~~~ @@ -468,7 +592,7 @@ devices managed by librte_pmd_cxgbe in FreeBSD operating system. EAL: PCI memory mapped at 0x8007ec000 EAL: PCI memory mapped at 0x842800000 EAL: PCI memory mapped at 0x80086c000 - PMD: rte_cxgbe_pmd: fw: 1.16.43.0, TP: 0.1.4.9 + PMD: rte_cxgbe_pmd: fw: 1.17.14.0, TP: 0.1.4.9 PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter Interactive-mode selected Configuring Port 0 (socket 0) diff --git a/doc/guides/nics/dpaa.rst b/doc/guides/nics/dpaa.rst index 0a13996c..620c045d 100644 --- a/doc/guides/nics/dpaa.rst +++ b/doc/guides/nics/dpaa.rst @@ -162,6 +162,16 @@ Manager. this pool. +Whitelisting & Blacklisting +--------------------------- + +For blacklisting a DPAA device, following commands can be used. + + .. code-block:: console + + <dpdk app> <EAL args> -b "dpaa_bus:fmX-macY" -- ... + e.g. "dpaa_bus:fm1-mac4" + Supported DPAA SoCs ------------------- diff --git a/doc/guides/nics/dpaa2.rst b/doc/guides/nics/dpaa2.rst index 9c66edd4..66c03e10 100644 --- a/doc/guides/nics/dpaa2.rst +++ b/doc/guides/nics/dpaa2.rst @@ -494,28 +494,12 @@ Please note that enabling debugging options may affect system performance. - ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_DRIVER`` (default ``n``) - Toggle display of generic debugging messages + Toggle display of debugging messages/logic - ``CONFIG_RTE_LIBRTE_DPAA2_USE_PHYS_IOVA`` (default ``y``) Toggle to use physical address vs virtual address for hardware accelerators. -- ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_INIT`` (default ``n``) - - Toggle display of initialization related messages. - -- ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_RX`` (default ``n``) - - Toggle display of receive fast path run-time message - -- ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX`` (default ``n``) - - Toggle display of transmit fast path run-time message - -- ``CONFIG_RTE_LIBRTE_DPAA2_DEBUG_TX_FREE`` (default ``n``) - - Toggle display of transmit fast path buffer free run-time message - Driver compilation and testing ------------------------------ @@ -532,8 +516,7 @@ for details. .. code-block:: console - ./arm64-dpaa2-linuxapp-gcc/testpmd -c 0xff -n 1 \ - -- -i --portmask=0x3 --nb-cores=1 --no-flush-rx + ./testpmd -c 0xff -n 1 -- -i --portmask=0x3 --nb-cores=1 --no-flush-rx ..... EAL: Registered [pci] bus. @@ -557,6 +540,38 @@ for details. Done testpmd> +Enabling logs +------------- + +For enabling logging for DPAA2 PMD, following log-level prefix can be used: + + .. code-block:: console + + <dpdk app> <EAL args> --log-level=bus.fslmc:<level> -- ... + +Using ``bus.fslmc`` as log matching criteria, all FSLMC bus logs can be enabled +which are lower than logging ``level``. + + Or + + .. code-block:: console + + <dpdk app> <EAL args> --log-level=pmd.net.dpaa2:<level> -- ... + +Using ``pmd.dpaa2`` as log matching criteria, all PMD logs can be enabled +which are lower than logging ``level``. + +Whitelisting & Blacklisting +--------------------------- + +For blacklisting a DPAA2 device, following commands can be used. + + .. code-block:: console + + <dpdk app> <EAL args> -b "fslmc:dpni.x" -- ... + +Where x is the device object id as configured in resource container. + Limitations ----------- diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst index 4dffce1a..438a83d5 100644 --- a/doc/guides/nics/enic.rst +++ b/doc/guides/nics/enic.rst @@ -1,32 +1,7 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright (c) 2017, Cisco Systems, Inc. All rights reserved. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - 1. Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - 2. Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN - ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. - ENIC Poll Mode Driver ===================== @@ -114,11 +89,24 @@ Configuration information - **Interrupts** - Only one interrupt per vNIC interface should be configured in the UCS + At least one interrupt per vNIC interface should be configured in the UCS manager regardless of the number receive/transmit queues. The ENIC PMD uses this interrupt to get information about link status and errors in the fast path. + In addition to the interrupt for link status and errors, when using Rx queue + interrupts, increase the number of configured interrupts so that there is at + least one interrupt for each Rx queue. For example, if the app uses 3 Rx + queues and wants to use per-queue interrupts, configure 4 (3 + 1) interrupts. + + - **Receive Side Scaling** + + In order to fully utilize RSS in DPDK, enable all RSS related settings in + CIMC or UCSM. These include the following items listed under + Receive Side Scaling: + TCP, IPv4, TCP-IPv4, IPv6, TCP-IPv6, IPv6 Extension, TCP-IPv6 Extension. + + .. _enic-flow-director: Flow director support @@ -140,20 +128,21 @@ perfect filtering of the 5-tuple with no masking of fields supported. SR-IOV mode utilization ----------------------- -UCS blade servers configured with dynamic vNIC connection policies in UCS -manager are capable of supporting assigned devices on virtual machines (VMs) -through a KVM hypervisor. Assigned devices, also known as 'passthrough' -devices, are SR-IOV virtual functions (VFs) on the host which are exposed -to VM instances. +UCS blade servers configured with dynamic vNIC connection policies in UCSM +are capable of supporting SR-IOV. SR-IOV virtual functions (VFs) are +specialized vNICs, distinct from regular Ethernet vNICs. These VFs can be +directly assigned to virtual machines (VMs) as 'passthrough' devices. -The Cisco Virtual Machine Fabric Extender (VM-FEX) gives the VM a dedicated +In UCS, SR-IOV VFs require the use of the Cisco Virtual Machine Fabric Extender +(VM-FEX), which gives the VM a dedicated interface on the Fabric Interconnect (FI). Layer 2 switching is done at the FI. This may eliminate the requirement for software switching on the host to route intra-host VM traffic. Please refer to `Creating a Dynamic vNIC Connection Policy <http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/vm_fex/vmware/gui/config_guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide/b_GUI_VMware_VM-FEX_UCSM_Configuration_Guide_chapter_010.html#task_433E01651F69464783A68E66DA8A47A5>`_ -for information on configuring SR-IOV adapter policies using UCS manager. +for information on configuring SR-IOV adapter policies and port profiles +using UCSM. Once the policies are in place and the host OS is rebooted, VFs should be visible on the host, E.g.: @@ -170,30 +159,37 @@ visible on the host, E.g.: 0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) 0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) -Enable Intel IOMMU on the host and install KVM and libvirt. A VM instance should -be created with an assigned device. When using libvirt, this configuration can -be done within the domain (i.e. VM) config file. For example this entry maps -host VF 0d:00:01 into the VM. +Enable Intel IOMMU on the host and install KVM and libvirt, and reboot again as +required. Then, using libvirt, create a VM instance with an assigned device. +Below is an example ``interface`` block (part of the domain configuration XML) +that adds the host VF 0d:00:01 to the VM. ``profileid='pp-vlan-25'`` indicates +the port profile that has been configured in UCSM. .. code-block:: console <interface type='hostdev' managed='yes'> <mac address='52:54:00:ac:ff:b6'/> + <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/> </source> + <virtualport type='802.1Qbh'> + <parameters profileid='pp-vlan-25'/> + </virtualport> + </interface> + Alternatively, the configuration can be done in a separate file using the ``network`` keyword. These methods are described in the libvirt documentation for `Network XML format <https://libvirt.org/formatnetwork.html>`_. -When the VM instance is started, the ENIC KVM driver will bind the host VF to +When the VM instance is started, libvirt will bind the host VF to vfio, complete provisioning on the FI and bring up the link. .. note:: It is not possible to use a VF directly from the host because it is not - fully provisioned until the hypervisor brings up the VM that it is assigned + fully provisioned until libvirt brings up the VM that it is assigned to. In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is @@ -207,9 +203,27 @@ seen on the VM instance and should be available for binding to a DPDK. Follow the normal DPDK install procedure, binding the VF to either ``igb_uio`` or ``vfio`` in non-IOMMU mode. +In the VM, the kernel enic driver may be automatically bound to the VF during +boot. Unbinding it currently hangs due to a known issue with the driver. To +work around the issue, blacklist the enic module as follows. Please see :ref:`Limitations <enic_limitations>` for limitations in the use of SR-IOV. +.. code-block:: console + + # cat /etc/modprobe.d/enic.conf + blacklist enic + + # dracut --force + +.. note:: + + Passthrough does not require SR-IOV. If VM-FEX is not desired, the user + may create as many regular vNICs as necessary and assign them to VMs as + passthrough devices. Since these vNICs are not SR-IOV VFs, using them as + passthrough devices do not require libvirt, port profiles, and VM-FEX. + + .. _enic-genic-flow-api: Generic Flow API support @@ -227,7 +241,7 @@ Generic Flow API is supported. The baseline support is: - Actions: queue and void - Selectors: 'is' -- **1300 series VICS with advanced filters disabled** +- **1300 and later series VICS with advanced filters disabled** With advanced filters disabled, an IPv4 or IPv6 item must be specified in the pattern. @@ -238,17 +252,99 @@ Generic Flow API is supported. The baseline support is: - Selectors: 'is', 'spec' and 'mask'. 'last' is not supported - In total, up to 64 bytes of mask is allowed across all headers -- **1300 series VICS with advanced filters enabled** +- **1300 and later series VICS with advanced filters enabled** - Attributes: ingress - Items: eth, ipv4, ipv6, udp, tcp, vxlan, inner eth, ipv4, ipv6, udp, tcp - - Actions: queue, mark, flag and void + - Actions: queue, mark, drop, flag and void - Selectors: 'is', 'spec' and 'mask'. 'last' is not supported - In total, up to 64 bytes of mask is allowed across all headers More features may be added in future firmware and new versions of the VIC. Please refer to the release notes. +.. _overlay_offload: + +Overlay Offload +--------------- + +Recent hardware models support overlay offload. When enabled, the NIC performs +the following operations for VXLAN, NVGRE, and GENEVE packets. In all cases, +inner and outer packets can be IPv4 or IPv6. + +- TSO for VXLAN and GENEVE packets. + + Hardware supports NVGRE TSO, but DPDK currently has no NVGRE offload flags. + +- Tx checksum offloads. + + The NIC fills in IPv4/UDP/TCP checksums for both inner and outer packets. + +- Rx checksum offloads. + + The NIC validates IPv4/UDP/TCP checksums of both inner and outer packets. + Good checksum flags (e.g. ``PKT_RX_L4_CKSUM_GOOD``) indicate that the inner + packet has the correct checksum, and if applicable, the outer packet also + has the correct checksum. Bad checksum flags (e.g. ``PKT_RX_L4_CKSUM_BAD``) + indicate that the inner and/or outer packets have invalid checksum values. + +- Inner Rx packet type classification + + PMD sets inner L3/L4 packet types (e.g. ``RTE_PTYPE_INNER_L4_TCP``), and + ``RTE_PTYPE_TUNNEL_GRENAT`` to indicate that the packet is tunneled. + PMD does not set L3/L4 packet types for outer packets. + +- Inner RSS + + RSS hash calculation, therefore queue selection, is done on inner packets. + +In order to enable overlay offload, the 'Enable VXLAN' box should be checked +via CIMC or UCSM followed by a reboot of the server. When PMD successfully +enables overlay offload, it prints the following message on the console. + +.. code-block:: console + + Overlay offload is enabled + +By default, PMD enables overlay offload if hardware supports it. To disable +it, set ``devargs`` parameter ``disable-overlay=1``. For example:: + + -w 12:00.0,disable-overlay=1 + +By default, the NIC uses 4789 as the VXLAN port. The user may change +it through ``rte_eth_dev_udp_tunnel_port_{add,delete}``. However, as +the current NIC has a single VXLAN port number, the user cannot +configure multiple port numbers. + +Ingress VLAN Rewrite +-------------------- + +VIC adapters can tag, untag, or modify the VLAN headers of ingress +packets. The ingress VLAN rewrite mode controls this behavior. By +default, it is set to pass-through, where the NIC does not modify the +VLAN header in any way so that the application can see the original +header. This mode is sufficient for many applications, but may not be +suitable for others. Such applications may change the mode by setting +``devargs`` parameter ``ig-vlan-rewrite`` to one of the following. + +- ``pass``: Pass-through mode. The NIC does not modify the VLAN + header. This is the default mode. + +- ``priority``: Priority-tag default VLAN mode. If the ingress packet + is tagged with the default VLAN, the NIC replaces its VLAN header + with the priority tag (VLAN ID 0). + +- ``trunk``: Default trunk mode. The NIC tags untagged ingress packets + with the default VLAN. Tagged ingress packets are not modified. To + the application, every packet appears as tagged. + +- ``untag``: Untag default VLAN mode. If the ingress packet is tagged + with the default VLAN, the NIC removes or untags its VLAN header so + that the application sees an untagged packet. As a result, the + default VLAN becomes `untagged`. This mode can be useful for + applications such as OVS-DPDK performance benchmarks that utilize + only the default VLAN and want to see only untagged packets. + .. _enic_limitations: Limitations @@ -264,9 +360,10 @@ Limitations In test setups where an Ethernet port of a Cisco adapter in TRUNK mode is connected point-to-point to another adapter port or connected though a router instead of a switch, all ingress packets will be VLAN tagged. Programs such - as l3fwd which do not account for VLAN tags in packets will misbehave. The - solution is to enable VLAN stripping on ingress. The following code fragment is - an example of how to accomplish this: + as l3fwd may not account for VLAN tags in packets and may misbehave. One + solution is to enable VLAN stripping on ingress so the VLAN tag is removed + from the packet and put into the mbuf->vlan_tci field. Here is an example + of how to accomplish this: .. code-block:: console @@ -274,6 +371,14 @@ Limitations vlan_offload |= ETH_VLAN_STRIP_OFFLOAD; rte_eth_dev_set_vlan_offload(port, vlan_offload); +Another alternative is modify the adapter's ingress VLAN rewrite mode so that +packets with the default VLAN tag are stripped by the adapter and presented to +DPDK as untagged packets. In this case mbuf->vlan_tci and the PKT_RX_VLAN and +PKT_RX_VLAN_STRIPPED mbuf flags would not be set. This mode is enabled with the +``devargs`` parameter ``ig-vlan-rewrite=untag``. For example:: + + -w 12:00.0,ig-vlan-rewrite=untag + - Limited flow director support on 1200 series and 1300 series Cisco VIC adapters with old firmware. Please see :ref:`enic-flow-director`. @@ -305,6 +410,24 @@ Limitations were added. Since there currently is no grouping or priority support, 'catch-all' filters should be added last. +- **Statistics** + + - ``rx_good_bytes`` (ibytes) always includes VLAN header (4B) and CRC bytes (4B). + This behavior applies to 1300 and older series VIC adapters. + 1400 series VICs do not count CRC bytes, and count VLAN header only when VLAN + stripping is disabled. + - When the NIC drops a packet because the Rx queue has no free buffers, + ``rx_good_bytes`` still increments by 4B if the packet is not VLAN tagged or + VLAN stripping is disabled, or by 8B if the packet is VLAN tagged and stripping + is enabled. + This behavior applies to 1300 and older series VIC adapters. 1400 series VICs + do not increment this byte counter when packets are dropped. + +- **RSS Hashing** + + - Hardware enables and disables UDP and TCP RSS hashing together. The driver + cannot control UDP and TCP hashing individually. + How to build the suite ---------------------- @@ -322,17 +445,9 @@ Supported Cisco VIC adapters ENIC PMD supports all recent generations of Cisco VIC adapters including: -- VIC 1280 -- VIC 1240 -- VIC 1225 -- VIC 1285 -- VIC 1225T -- VIC 1227 -- VIC 1227T -- VIC 1380 -- VIC 1340 -- VIC 1385 -- VIC 1387 +- VIC 1200 series +- VIC 1300 series +- VIC 1400 series Supported Operating Systems --------------------------- @@ -356,10 +471,16 @@ Supported features - VLAN filtering (supported via UCSM/CIMC only) - Execution of application by unprivileged system users - IPV4, IPV6 and TCP RSS hashing +- UDP RSS hashing (1400 series and later adapters) - Scattered Rx - MTU update - SR-IOV on UCS managed servers connected to Fabric Interconnects - Flow API +- Overlay offload + + - Rx/Tx checksum offloads for VXLAN, NVGRE, GENEVE + - TSO for VXLAN and GENEVE packets + - Inner RSS Known bugs and unsupported features in this release --------------------------------------------------- @@ -369,8 +490,8 @@ Known bugs and unsupported features in this release - VLAN based flow direction - Non-IPV4 flow direction - Setting of extended VLAN -- UDP RSS hashing - MTU update only works if Scattered Rx mode is disabled +- Maximum receive packet length is ignored if Scattered Rx mode is used Prerequisites ------------- @@ -427,4 +548,4 @@ Any questions or bugs should be reported to DPDK community and to the ENIC PMD maintainers: - John Daley <johndale@cisco.com> -- Nelson Escobar <neescoba@cisco.com> +- Hyong Youb Kim <hyonkim@cisco.com> diff --git a/doc/guides/nics/fail_safe.rst b/doc/guides/nics/fail_safe.rst index 3f72b593..6c02d7ef 100644 --- a/doc/guides/nics/fail_safe.rst +++ b/doc/guides/nics/fail_safe.rst @@ -1,32 +1,6 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2017 6WIND S.A. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of 6WIND S.A. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - Fail-safe poll mode driver library ================================== diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst index 1b4fb979..cddc877d 100644 --- a/doc/guides/nics/features.rst +++ b/doc/guides/nics/features.rst @@ -278,6 +278,17 @@ Supports RSS hashing on RX. * **[provides] mbuf**: ``mbuf.ol_flags:PKT_RX_RSS_HASH``, ``mbuf.rss``. +.. _nic_features_inner_rss: + +Inner RSS +--------- + +Supports RX RSS hashing on Inner headers. + +* **[users] rte_flow_action_rss**: ``level``. +* **[provides] mbuf**: ``mbuf.ol_flags:PKT_RX_RSS_HASH``, ``mbuf.rss``. + + .. _nic_features_rss_key_update: RSS key update @@ -503,7 +514,7 @@ CRC offload Supports CRC stripping by hardware. -* **[uses] rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_CRC_STRIP``. +* **[uses] rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_CRC_STRIP,DEV_RX_OFFLOAD_KEEP_CRC``. .. _nic_features_vlan_offload: @@ -566,7 +577,6 @@ Supports L4 checksum offload. * **[uses] rte_eth_rxconf,rte_eth_rxmode**: ``offloads:DEV_RX_OFFLOAD_UDP_CKSUM,DEV_RX_OFFLOAD_TCP_CKSUM``. * **[uses] rte_eth_txconf,rte_eth_txmode**: ``offloads:DEV_TX_OFFLOAD_UDP_CKSUM,DEV_TX_OFFLOAD_TCP_CKSUM,DEV_TX_OFFLOAD_SCTP_CKSUM``. -* **[uses] user config**: ``dev_conf.rxmode.hw_ip_checksum``. * **[uses] mbuf**: ``mbuf.ol_flags:PKT_TX_IPV4`` | ``PKT_TX_IPV6``, ``mbuf.ol_flags:PKT_TX_L4_NO_CKSUM`` | ``PKT_TX_TCP_CKSUM`` | ``PKT_TX_SCTP_CKSUM`` | ``PKT_TX_UDP_CKSUM``. @@ -749,6 +759,17 @@ Supports getting/setting device eeprom data. ``rte_eth_dev_set_eeprom()``. +.. _nic_features_module_eeprom_dump: + +Module EEPROM dump +------------------ + +Supports getting information and data of plugin module eeprom. + +* **[implements] eth_dev_ops**: ``get_module_info``, ``get_module_eeprom``. +* **[related] API**: ``rte_eth_dev_get_module_info()``, ``rte_eth_dev_get_module_eeprom()``. + + .. _nic_features_register_dump: Registers dump @@ -892,7 +913,25 @@ Documentation describes performance values. See ``dpdk.org/doc/perf/*``. +.. _nic_features_runtime_rx_queue_setup: + +Runtime Rx queue setup +---------------------- + +Supports Rx queue setup after device started. + +* **[provides] rte_eth_dev_info**: ``dev_capa:RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP``. +* **[related] API**: ``rte_eth_dev_info_get()``. +.. _nic_features_runtime_tx_queue_setup: + +Runtime Tx queue setup +---------------------- + +Supports Tx queue setup after device started. + +* **[provides] rte_eth_dev_info**: ``dev_capa:RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP``. +* **[related] API**: ``rte_eth_dev_info_get()``. .. _nic_features_other: diff --git a/doc/guides/nics/features/avf.ini b/doc/guides/nics/features/avf.ini index ccb9edde..35ceada2 100644 --- a/doc/guides/nics/features/avf.ini +++ b/doc/guides/nics/features/avf.ini @@ -6,7 +6,6 @@ [Features] Speed capabilities = Y Link status = Y -Link status event = Y Rx interrupt = Y Queue start/stop = Y MTU update = Y diff --git a/doc/guides/nics/features/avf_vec.ini b/doc/guides/nics/features/avf_vec.ini index 89249948..3050bc4a 100644 --- a/doc/guides/nics/features/avf_vec.ini +++ b/doc/guides/nics/features/avf_vec.ini @@ -6,7 +6,6 @@ [Features] Speed capabilities = Y Link status = Y -Link status event = Y Rx interrupt = Y Queue start/stop = Y MTU update = Y diff --git a/doc/guides/nics/features/axgbe.ini b/doc/guides/nics/features/axgbe.ini new file mode 100644 index 00000000..ab4da559 --- /dev/null +++ b/doc/guides/nics/features/axgbe.ini @@ -0,0 +1,19 @@ +; +; Supported features of the 'axgbe' network poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Speed capabilities = Y +Link status = Y +Jumbo frame = Y +Promiscuous mode = Y +Allmulticast mode = Y +RSS hash = Y +CRC offload = Y +L3 checksum offload = Y +L4 checksum offload = Y +Basic stats = Y +Linux UIO = Y +x86-32 = Y +x86-64 = Y diff --git a/doc/guides/nics/features/cxgbe.ini b/doc/guides/nics/features/cxgbe.ini index 3d0fde2f..88f2f92b 100644 --- a/doc/guides/nics/features/cxgbe.ini +++ b/doc/guides/nics/features/cxgbe.ini @@ -14,7 +14,9 @@ TSO = Y Promiscuous mode = Y Allmulticast mode = Y RSS hash = Y +RSS key update = Y Flow control = Y +Flow API = Y CRC offload = Y VLAN offload = Y L3 checksum offload = Y @@ -24,6 +26,7 @@ Basic stats = Y Stats per queue = Y EEPROM dump = Y Registers dump = Y +Multiprocess aware = Y BSD nic_uio = Y Linux UIO = Y Linux VFIO = Y diff --git a/doc/guides/nics/features/cxgbevf.ini b/doc/guides/nics/features/cxgbevf.ini new file mode 100644 index 00000000..b41fc365 --- /dev/null +++ b/doc/guides/nics/features/cxgbevf.ini @@ -0,0 +1,29 @@ +; +; Supported features of the 'cxgbevf' network poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Speed capabilities = Y +Link status = Y +Queue start/stop = Y +MTU update = Y +Jumbo frame = Y +Scattered Rx = Y +TSO = Y +Promiscuous mode = Y +Allmulticast mode = Y +RSS hash = Y +CRC offload = Y +VLAN offload = Y +L3 checksum offload = Y +L4 checksum offload = Y +Packet type parsing = Y +Basic stats = Y +Stats per queue = Y +Multiprocess aware = Y +Linux UIO = Y +Linux VFIO = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini index dae2ad77..f1a39d0f 100644 --- a/doc/guides/nics/features/default.ini +++ b/doc/guides/nics/features/default.ini @@ -17,6 +17,8 @@ Lock-free Tx queue = Fast mbuf free = Free Tx mbuf on demand = Queue start/stop = +Runtime Rx queue setup = +Runtime Tx queue setup = MTU update = Jumbo frame = Scattered Rx = @@ -29,6 +31,7 @@ Multicast MAC filter = RSS hash = RSS key update = RSS reta update = +Inner RSS = VMDq = SR-IOV = DCB = @@ -63,6 +66,7 @@ Extended stats = Stats per queue = FW version = EEPROM dump = +Module EEPROM dump = Registers dump = LED = Multiprocess aware = diff --git a/doc/guides/nics/features/enic.ini b/doc/guides/nics/features/enic.ini index 498341f0..8a4bad29 100644 --- a/doc/guides/nics/features/enic.ini +++ b/doc/guides/nics/features/enic.ini @@ -6,23 +6,29 @@ [Features] Link status = Y Link status event = Y +Rx interrupt = Y Queue start/stop = Y MTU update = Y Jumbo frame = Y Scattered Rx = Y TSO = Y Promiscuous mode = Y +Allmulticast mode = Y Unicast MAC filter = Y -Multicast MAC filter = Y +Multicast MAC filter = RSS hash = Y +RSS key update = Y +RSS reta update = Y +Inner RSS = Y SR-IOV = Y -VLAN filter = Y CRC offload = Y VLAN offload = Y Flow director = Y Flow API = Y L3 checksum offload = Y L4 checksum offload = Y +Inner L3 checksum = Y +Inner L4 checksum = Y Packet type parsing = Y Basic stats = Y Multiprocess aware = Y diff --git a/doc/guides/nics/features/fm10k.ini b/doc/guides/nics/features/fm10k.ini index f0f61a7d..0acdf0d3 100644 --- a/doc/guides/nics/features/fm10k.ini +++ b/doc/guides/nics/features/fm10k.ini @@ -5,6 +5,8 @@ ; [Features] Speed capabilities = P +Link status = Y +Link status event = Y Rx interrupt = Y Queue start/stop = Y Jumbo frame = Y @@ -24,6 +26,8 @@ VLAN offload = Y L3 checksum offload = Y L4 checksum offload = Y Packet type parsing = Y +Rx descriptor status = Y +Tx descriptor status = Y Basic stats = Y Extended stats = Y Stats per queue = Y diff --git a/doc/guides/nics/features/fm10k_vf.ini b/doc/guides/nics/features/fm10k_vf.ini index 32b93df4..44b50faa 100644 --- a/doc/guides/nics/features/fm10k_vf.ini +++ b/doc/guides/nics/features/fm10k_vf.ini @@ -5,6 +5,8 @@ ; [Features] Speed capabilities = P +Link status = Y +Link status event = Y Rx interrupt = Y Queue start/stop = Y Jumbo frame = Y diff --git a/doc/guides/nics/features/i40e.ini b/doc/guides/nics/features/i40e.ini index e862712c..16eab7f4 100644 --- a/doc/guides/nics/features/i40e.ini +++ b/doc/guides/nics/features/i40e.ini @@ -9,6 +9,8 @@ Link status = Y Link status event = Y Rx interrupt = Y Queue start/stop = Y +Runtime Rx queue setup = Y +Runtime Tx queue setup = Y Jumbo frame = Y Scattered Rx = Y TSO = Y @@ -44,6 +46,7 @@ Tx descriptor status = Y Basic stats = Y Extended stats = Y FW version = Y +Module EEPROM dump = Y Multiprocess aware = Y BSD nic_uio = Y Linux UIO = Y diff --git a/doc/guides/nics/features/i40e_vec.ini b/doc/guides/nics/features/i40e_vec.ini index 7d7b3a92..c65e8b03 100644 --- a/doc/guides/nics/features/i40e_vec.ini +++ b/doc/guides/nics/features/i40e_vec.ini @@ -34,6 +34,7 @@ Rx descriptor status = Y Tx descriptor status = Y Basic stats = Y Extended stats = Y +Module EEPROM dump = Y Multiprocess aware = Y BSD nic_uio = Y Linux UIO = Y diff --git a/doc/guides/nics/features/i40e_vf.ini b/doc/guides/nics/features/i40e_vf.ini index 46e0d9fc..ba2d8cbe 100644 --- a/doc/guides/nics/features/i40e_vf.ini +++ b/doc/guides/nics/features/i40e_vf.ini @@ -5,6 +5,7 @@ ; [Features] Rx interrupt = Y +Link status = Y Queue start/stop = Y Jumbo frame = Y Scattered Rx = Y diff --git a/doc/guides/nics/features/i40e_vf_vec.ini b/doc/guides/nics/features/i40e_vf_vec.ini index c2c6c19f..421ed919 100644 --- a/doc/guides/nics/features/i40e_vf_vec.ini +++ b/doc/guides/nics/features/i40e_vf_vec.ini @@ -5,6 +5,7 @@ ; [Features] Rx interrupt = Y +Link status = Y Queue start/stop = Y Jumbo frame = Y Scattered Rx = Y diff --git a/doc/guides/nics/features/ifcvf.ini b/doc/guides/nics/features/ifcvf.ini new file mode 100644 index 00000000..ef1fc471 --- /dev/null +++ b/doc/guides/nics/features/ifcvf.ini @@ -0,0 +1,8 @@ +; +; Supported features of the 'ifcvf' vDPA driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +x86-32 = Y +x86-64 = Y diff --git a/doc/guides/nics/features/igb.ini b/doc/guides/nics/features/igb.ini index 33d64d99..c53fd075 100644 --- a/doc/guides/nics/features/igb.ini +++ b/doc/guides/nics/features/igb.ini @@ -41,6 +41,7 @@ Basic stats = Y Extended stats = Y FW version = Y EEPROM dump = Y +Module EEPROM dump = Y Registers dump = Y BSD nic_uio = Y Linux UIO = Y diff --git a/doc/guides/nics/features/igb_vf.ini b/doc/guides/nics/features/igb_vf.ini index e641a2c9..d9653234 100644 --- a/doc/guides/nics/features/igb_vf.ini +++ b/doc/guides/nics/features/igb_vf.ini @@ -4,6 +4,7 @@ ; Refer to default.ini for the full list of available PMD features. ; [Features] +Link status = Y Rx interrupt = Y Scattered Rx = Y TSO = Y diff --git a/doc/guides/nics/features/ixgbe.ini b/doc/guides/nics/features/ixgbe.ini index 1d68ee8e..41431117 100644 --- a/doc/guides/nics/features/ixgbe.ini +++ b/doc/guides/nics/features/ixgbe.ini @@ -51,6 +51,7 @@ Extended stats = Y Stats per queue = Y FW version = Y EEPROM dump = Y +Module EEPROM dump = Y Registers dump = Y Multiprocess aware = Y BSD nic_uio = Y diff --git a/doc/guides/nics/features/ixgbe_vec.ini b/doc/guides/nics/features/ixgbe_vec.ini index 28bc0547..ef3ee688 100644 --- a/doc/guides/nics/features/ixgbe_vec.ini +++ b/doc/guides/nics/features/ixgbe_vec.ini @@ -40,6 +40,7 @@ Basic stats = Y Extended stats = Y Stats per queue = Y EEPROM dump = Y +Module EEPROM dump = Y Registers dump = Y Multiprocess aware = Y BSD nic_uio = Y diff --git a/doc/guides/nics/features/mlx4.ini b/doc/guides/nics/features/mlx4.ini index f6efd21d..98a3f611 100644 --- a/doc/guides/nics/features/mlx4.ini +++ b/doc/guides/nics/features/mlx4.ini @@ -13,6 +13,7 @@ Queue start/stop = Y MTU update = Y Jumbo frame = Y Scattered Rx = Y +TSO = Y Promiscuous mode = Y Allmulticast mode = Y Unicast MAC filter = Y diff --git a/doc/guides/nics/features/mlx5.ini b/doc/guides/nics/features/mlx5.ini index c3636391..b28b43e5 100644 --- a/doc/guides/nics/features/mlx5.ini +++ b/doc/guides/nics/features/mlx5.ini @@ -21,6 +21,7 @@ Multicast MAC filter = Y RSS hash = Y RSS key update = Y RSS reta update = Y +Inner RSS = Y SR-IOV = Y VLAN filter = Y Flow director = Y @@ -29,6 +30,9 @@ CRC offload = Y VLAN offload = Y L3 checksum offload = Y L4 checksum offload = Y +Timestamp offload = Y +Inner L3 checksum = Y +Inner L4 checksum = Y Packet type parsing = Y Rx descriptor status = Y Tx descriptor status = Y @@ -39,5 +43,6 @@ Multiprocess aware = Y Other kdrv = Y ARMv8 = Y Power8 = Y +x86-32 = Y x86-64 = Y Usage doc = Y diff --git a/doc/guides/nics/features/mrvl.ini b/doc/guides/nics/features/mvpp2.ini index 00d96218..ef47546d 100644 --- a/doc/guides/nics/features/mrvl.ini +++ b/doc/guides/nics/features/mvpp2.ini @@ -1,5 +1,5 @@ ; -; Supported features of the 'mrvl' network poll mode driver. +; Supported features of the 'mvpp2' network poll mode driver. ; ; Refer to default.ini for the full list of available PMD features. ; @@ -13,11 +13,13 @@ Allmulticast mode = Y Unicast MAC filter = Y Multicast MAC filter = Y RSS hash = Y +Flow control = Y VLAN filter = Y CRC offload = Y L3 checksum offload = Y L4 checksum offload = Y Packet type parsing = Y Basic stats = Y +Extended stats = Y ARMv8 = Y Usage doc = Y diff --git a/doc/guides/nics/features/netvsc.ini b/doc/guides/nics/features/netvsc.ini new file mode 100644 index 00000000..2ff6042b --- /dev/null +++ b/doc/guides/nics/features/netvsc.ini @@ -0,0 +1,23 @@ +; +; Supported features of the 'netvsc' network poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +Speed capabilities = P +Link status = Y +Queue start/stop = Y +Scattered Rx = Y +Promiscuous mode = Y +Allmulticast mode = Y +Basic stats = Y +Stats per queue = Y +Extended stats = Y +Multiprocess aware = Y +Other kdrv = Y +ARMv7 = Y +ARMv8 = Y +x86-32 = Y +x86-64 = Y +Usage doc = Y +MTU update = Y diff --git a/doc/guides/nics/features/qede.ini b/doc/guides/nics/features/qede.ini index cbadc194..0d081002 100644 --- a/doc/guides/nics/features/qede.ini +++ b/doc/guides/nics/features/qede.ini @@ -6,10 +6,11 @@ [Features] Speed capabilities = Y Link status = Y -Link status event = Y MTU update = Y Jumbo frame = Y Scattered Rx = Y +LRO = Y +TSO = Y Promiscuous mode = Y Allmulticast mode = Y Unicast MAC filter = Y @@ -18,12 +19,14 @@ RSS hash = Y RSS key update = Y RSS reta update = Y VLAN filter = Y +N-tuple filter = Y +Tunnel filter = Y +Flow director = Y Flow control = Y CRC offload = Y VLAN offload = Y L3 checksum offload = Y L4 checksum offload = Y -Tunnel filter = Y Inner L3 checksum = Y Inner L4 checksum = Y Packet type parsing = Y @@ -32,11 +35,8 @@ Extended stats = Y Stats per queue = Y Multiprocess aware = Y Linux UIO = Y +Linux VFIO = Y ARMv8 = Y x86-32 = Y x86-64 = Y Usage doc = Y -N-tuple filter = Y -Flow director = Y -LRO = Y -TSO = Y diff --git a/doc/guides/nics/features/qede_vf.ini b/doc/guides/nics/features/qede_vf.ini index 18857b6e..e796b313 100644 --- a/doc/guides/nics/features/qede_vf.ini +++ b/doc/guides/nics/features/qede_vf.ini @@ -6,7 +6,6 @@ [Features] Speed capabilities = Y Link status = Y -Link status event = Y MTU update = Y Jumbo frame = Y Scattered Rx = Y @@ -30,6 +29,7 @@ Extended stats = Y Stats per queue = Y Multiprocess aware = Y Linux UIO = Y +Linux VFIO = Y ARMv8 = Y x86-32 = Y x86-64 = Y diff --git a/doc/guides/nics/features/softnic.ini b/doc/guides/nics/features/softnic.ini new file mode 100644 index 00000000..0583381c --- /dev/null +++ b/doc/guides/nics/features/softnic.ini @@ -0,0 +1,9 @@ +; +; Supported features of the 'softnic' poll mode driver. +; +; Refer to default.ini for the full list of available PMD features. +; +[Features] +x86-32 = Y +x86-64 = Y +Usage doc = Y diff --git a/doc/guides/nics/features/vhost.ini b/doc/guides/nics/features/vhost.ini index dffd1f49..ef81abb4 100644 --- a/doc/guides/nics/features/vhost.ini +++ b/doc/guides/nics/features/vhost.ini @@ -5,7 +5,6 @@ ; [Features] Link status = Y -Link status event = Y Free Tx mbuf on demand = Y Queue status event = Y Basic stats = Y diff --git a/doc/guides/nics/features/virtio.ini b/doc/guides/nics/features/virtio.ini index 16e577df..a16b8172 100644 --- a/doc/guides/nics/features/virtio.ini +++ b/doc/guides/nics/features/virtio.ini @@ -6,6 +6,7 @@ [Features] Speed capabilities = P Link status = Y +Link status event = Y Rx interrupt = Y Queue start/stop = Y Scattered Rx = Y diff --git a/doc/guides/nics/features/virtio_vec.ini b/doc/guides/nics/features/virtio_vec.ini index c06c860d..e60fe36a 100644 --- a/doc/guides/nics/features/virtio_vec.ini +++ b/doc/guides/nics/features/virtio_vec.ini @@ -6,6 +6,7 @@ [Features] Speed capabilities = P Link status = Y +Link status event = Y Rx interrupt = Y Queue start/stop = Y Promiscuous mode = Y diff --git a/doc/guides/nics/fm10k.rst b/doc/guides/nics/fm10k.rst index c44e226e..d1391e99 100644 --- a/doc/guides/nics/fm10k.rst +++ b/doc/guides/nics/fm10k.rst @@ -79,14 +79,14 @@ Other features are supported using optional MACRO configuration. They include: To enable via ``RX_OLFLAGS`` use ``RTE_LIBRTE_FM10K_RX_OLFLAGS_ENABLE=y``. -To guarantee the constraint, the following configuration flags in ``dev_conf.rxmode`` +To guarantee the constraint, the following capabilities in ``dev_conf.rxmode.offloads`` will be checked: -* ``hw_vlan_extend`` +* ``DEV_RX_OFFLOAD_VLAN_EXTEND`` -* ``hw_ip_checksum`` +* ``DEV_RX_OFFLOAD_CHECKSUM`` -* ``header_split`` +* ``DEV_RX_OFFLOAD_HEADER_SPLIT`` * ``fdir_conf->mode`` @@ -106,19 +106,9 @@ TX Constraint Features not Supported by TX Vector PMD ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -TX vPMD only works when ``txq_flags`` is set to ``FM10K_SIMPLE_TX_FLAG``. -This means that it does not support TX multi-segment, VLAN offload or TX csum -offload. The following MACROs are used for these three features: +TX vPMD only works when offloads is set to 0 -* ``ETH_TXQ_FLAGS_NOMULTSEGS`` - -* ``ETH_TXQ_FLAGS_NOVLANOFFL`` - -* ``ETH_TXQ_FLAGS_NOXSUMSCTP`` - -* ``ETH_TXQ_FLAGS_NOXSUMUDP`` - -* ``ETH_TXQ_FLAGS_NOXSUMTCP`` +This means that it does not support any TX offload. Limitations ----------- @@ -149,9 +139,8 @@ CRC striping ~~~~~~~~~~~~ The FM10000 family of NICs strip the CRC for every packets coming into the -host interface. So, CRC will be stripped even when the -``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``. - +host interface. So, CRC will be stripped even when ``DEV_RX_OFFLOAD_CRC_STRIP`` +in ``rxmode.offloads`` is NOT set in ``struct rte_eth_conf``. Maximum packet length ~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/nics/i40e.rst b/doc/guides/nics/i40e.rst index e1b8083c..65d87f86 100644 --- a/doc/guides/nics/i40e.rst +++ b/doc/guides/nics/i40e.rst @@ -4,14 +4,16 @@ I40E Poll Mode Driver ====================== -The I40E PMD (librte_pmd_i40e) provides poll mode driver support -for the Intel X710/XL710/X722 10/40 Gbps family of adapters. +The i40e PMD (librte_pmd_i40e) provides poll mode driver support for +10/25/40 Gbps Intel® Ethernet 700 Series Network Adapters based on +the Intel Ethernet Controller X710/XL710/XXV710 and Intel Ethernet +Connection X722 (only support part of features). Features -------- -Features of the I40E PMD are: +Features of the i40e PMD are: - Multiple queues for TX and RX - Receiver Side Scaling (RSS) @@ -40,6 +42,7 @@ Features of the I40E PMD are: - VF Daemon (VFD) - EXPERIMENTAL - Dynamic Device Personalization (DDP) - Queue region configuration +- Virtual Function Port Representors Prerequisites ------------- @@ -53,7 +56,37 @@ Prerequisites section of the :ref:`Getting Started Guide for Linux <linux_gsg>`. - Upgrade the NVM/FW version following the `Intel® Ethernet NVM Update Tool Quick Usage Guide for Linux - <https://www-ssl.intel.com/content/www/us/en/embedded/products/networking/nvm-update-tool-quick-linux-usage-guide.html>`_ if needed. + <https://www-ssl.intel.com/content/www/us/en/embedded/products/networking/nvm-update-tool-quick-linux-usage-guide.html>`_ and `Intel® Ethernet NVM Update Tool: Quick Usage Guide for EFI <https://www.intel.com/content/www/us/en/embedded/products/networking/nvm-update-tool-quick-efi-usage-guide.html>`_ if needed. + +Recommended Matching List +------------------------- + +It is highly recommended to upgrade the i40e kernel driver and firmware to +avoid the compatibility issues with i40e PMD. Here is the suggested matching +list which has been tested and verified. The detailed information can refer +to chapter Tested Platforms/Tested NICs in release notes. + + +--------------+-----------------------+------------------+ + | DPDK version | Kernel driver version | Firmware version | + +==============+=======================+==================+ + | 18.05 | 2.4.6 | 6.01 | + +--------------+-----------------------+------------------+ + | 18.02 | 2.4.3 | 6.01 | + +--------------+-----------------------+------------------+ + | 17.11 | 2.1.26 | 6.01 | + +--------------+-----------------------+------------------+ + | 17.08 | 2.0.19 | 6.01 | + +--------------+-----------------------+------------------+ + | 17.05 | 1.5.23 | 5.05 | + +--------------+-----------------------+------------------+ + | 17.02 | 1.5.23 | 5.05 | + +--------------+-----------------------+------------------+ + | 16.11 | 1.5.23 | 5.05 | + +--------------+-----------------------+------------------+ + | 16.07 | 1.4.25 | 5.04 | + +--------------+-----------------------+------------------+ + | 16.04 | 1.4.25 | 5.02 | + +--------------+-----------------------+------------------+ Pre-Installation Configuration ------------------------------ @@ -93,11 +126,6 @@ Please note that enabling debugging options may affect system performance. Number of queues reserved for each VMDQ Pool. -- ``CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL`` (default ``-1``) - - Interrupt Throttling interval. - - Runtime Config Options ~~~~~~~~~~~~~~~~~~~~~~ @@ -121,6 +149,20 @@ Runtime Config Options will switch PF interrupt from IntN to Int0 to avoid interrupt conflict between DPDK and Linux Kernel. +- ``Support VF Port Representor`` (default ``not enabled``) + + The i40e PF PMD supports the creation of VF port representors for the control + and monitoring of i40e virtual function devices. Each port representor + corresponds to a single virtual function of that device. Using the ``devargs`` + option ``representor`` the user can specify which virtual functions to create + port representors for on initialization of the PF PMD by passing the VF IDs of + the VFs which are required.:: + + -w DBDF,representor=[0,1,4] + + Currently hot-plugging of representor ports is not supported so all required + representors must be specified on the creation of the PF. + Driver compilation and testing ------------------------------ @@ -324,7 +366,7 @@ Delete all flow director rules on a port: Floating VEB ~~~~~~~~~~~~~ -The Intel® Ethernet Controller X710 and XL710 Family support a feature called +The Intel® Ethernet 700 Series support a feature called "Floating VEB". A Virtual Ethernet Bridge (VEB) is an IEEE Edge Virtual Bridging (EVB) term @@ -370,21 +412,22 @@ or greater. Dynamic Device Personalization (DDP) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The Intel® Ethernet Controller X*710 support a feature called "Dynamic Device -Personalization (DDP)", which is used to configure hardware by downloading -a profile to support protocols/filters which are not supported by default. -The DDP functionality requires a NIC firmware version of 6.0 or greater. +The Intel® Ethernet 700 Series except for the Intel Ethernet Connection +X722 support a feature called "Dynamic Device Personalization (DDP)", +which is used to configure hardware by downloading a profile to support +protocols/filters which are not supported by default. The DDP +functionality requires a NIC firmware version of 6.0 or greater. -Current implementation supports MPLSoUDP/MPLSoGRE/GTP-C/GTP-U/PPPoE/PPPoL2TP, +Current implementation supports GTP-C/GTP-U/PPPoE/PPPoL2TP, steering can be used with rte_flow API. -Load a profile which supports MPLSoUDP/MPLSoGRE and store backup profile: +Load a profile which supports GTP and store backup profile: .. code-block:: console - testpmd> ddp add 0 ./mpls.pkgo,./backup.pkgo + testpmd> ddp add 0 ./gtp.pkgo,./backup.pkgo -Delete a MPLS profile and restore backup profile: +Delete a GTP profile and restore backup profile: .. code-block:: console @@ -396,11 +439,11 @@ Get loaded DDP package info list: testpmd> ddp get list 0 -Display information about a MPLS profile: +Display information about a GTP profile: .. code-block:: console - testpmd> ddp get info ./mpls.pkgo + testpmd> ddp get info ./gtp.pkgo Input set configuration ~~~~~~~~~~~~~~~~~~~~~~~ @@ -416,7 +459,7 @@ For example, to use only 48bit prefix for IPv6 src address for IPv6 TCP RSS: Queue region configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The Ethernet Controller X710/XL710 supports a feature of queue regions +The Intel® Ethernet 700 Series supports a feature of queue regions configuration for RSS in the PF, so that different traffic classes or different packet classification types can be separated to different queues in different queue regions. There is an API for configuration @@ -440,8 +483,8 @@ details please refer to :doc:`../testpmd_app_ug/index`. Limitations or Known issues --------------------------- -MPLS packet classification on X710/XL710 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +MPLS packet classification +~~~~~~~~~~~~~~~~~~~~~~~~~~ For firmware versions prior to 5.0, MPLS packets are not recognized by the NIC. The L2 Payload flow type in flow director can be used to classify MPLS packet @@ -489,14 +532,14 @@ Incorrect Rx statistics when packet is oversize ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When a packet is over maximum frame size, the packet is dropped. -However the Rx statistics, when calling `rte_eth_stats_get` incorrectly +However, the Rx statistics, when calling `rte_eth_stats_get` incorrectly shows it as received. VF & TC max bandwidth setting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The per VF max bandwidth and per TC max bandwidth cannot be enabled in parallel. -The dehavior is different when handling per VF and per TC max bandwidth setting. +The behavior is different when handling per VF and per TC max bandwidth setting. When enabling per VF max bandwidth, SW will check if per TC max bandwidth is enabled. If so, return failure. When enabling per TC max bandwidth, SW will check if per VF max bandwidth @@ -517,11 +560,11 @@ VF performance is impacted by PCI extended tag setting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To reach maximum NIC performance in the VF the PCI extended tag must be -enabled. The DPDK I40E PF driver will set this feature during initialization, +enabled. The DPDK i40e PF driver will set this feature during initialization, but the kernel PF driver does not. So when running traffic on a VF which is managed by the kernel PF driver, a significant NIC performance downgrade has -been observed (for 64 byte packets, there is about 25% linerate downgrade for -a 25G device and about 35% for a 40G device). +been observed (for 64 byte packets, there is about 25% line-rate downgrade for +a 25GbE device and about 35% for a 40GbE device). For kernel version >= 4.11, the kernel's PCI driver will enable the extended tag if it detects that the device supports it. So by default, this is not an @@ -562,12 +605,12 @@ with DPDK, then the configuration will also impact port B in the NIC with kernel driver, which don't want to use the TPID. So PMD reports warning to clarify what is changed by writing global register. -High Performance of Small Packets on 40G NIC --------------------------------------------- +High Performance of Small Packets on 40GbE NIC +---------------------------------------------- As there might be firmware fixes for performance enhancement in latest version of firmware image, the firmware update might be needed for getting high performance. -Check with the local Intel's Network Division application engineers for firmware updates. +Check the Intel support website for the latest firmware updates. Users should consult the release notes specific to a DPDK release to identify the validated firmware version for a NIC using the i40e driver. @@ -577,23 +620,13 @@ Use 16 Bytes RX Descriptor Size As i40e PMD supports both 16 and 32 bytes RX descriptor sizes, and 16 bytes size can provide helps to high performance of small packets. Configuration of ``CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC`` in config files can be changed to use 16 bytes size RX descriptors. -High Performance and per Packet Latency Tradeoff -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Due to the hardware design, the interrupt signal inside NIC is needed for per -packet descriptor write-back. The minimum interval of interrupts could be set -at compile time by ``CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL`` in configuration files. -Though there is a default configuration, the interval could be tuned by the -users with that configuration item depends on what the user cares about more, -performance or per packet latency. - Example of getting best performance with l3fwd example ------------------------------------------------------ -The following is an example of running the DPDK ``l3fwd`` sample application to get high performance with an -Intel server platform and Intel XL710 NICs. +The following is an example of running the DPDK ``l3fwd`` sample application to get high performance with a +server with Intel Xeon processors and Intel Ethernet CNA XL710. -The example scenario is to get best performance with two Intel XL710 40GbE ports. +The example scenario is to get best performance with two Intel Ethernet CNA XL710 40GbE ports. See :numref:`figure_intel_perf_test_setup` for the performance test setup. .. _figure_intel_perf_test_setup: @@ -603,9 +636,9 @@ See :numref:`figure_intel_perf_test_setup` for the performance test setup. Performance Test Setup -1. Add two Intel XL710 NICs to the platform, and use one port per card to get best performance. - The reason for using two NICs is to overcome a PCIe Gen3's limitation since it cannot provide 80G bandwidth - for two 40G ports, but two different PCIe Gen3 x8 slot can. +1. Add two Intel Ethernet CNA XL710 to the platform, and use one port per card to get best performance. + The reason for using two NICs is to overcome a PCIe v3.0 limitation since it cannot provide 80GbE bandwidth + for two 40GbE ports, but two different PCIe v3.0 x8 slot can. Refer to the sample NICs output above, then we can select ``82:00.0`` and ``85:00.0`` as test ports:: 82:00.0 Ethernet [0200]: Intel XL710 for 40GbE QSFP+ [8086:1583] @@ -621,7 +654,7 @@ See :numref:`figure_intel_perf_test_setup` for the performance test setup. 4. Bind these two ports to igb_uio. -5. As to XL710 40G port, we need at least two queue pairs to achieve best performance, then two queues per port +5. As to Intel Ethernet CNA XL710 40GbE port, we need at least two queue pairs to achieve best performance, then two queues per port will be required, and each queue pair will need a dedicated CPU core for receiving/transmitting packets. 6. The DPDK sample application ``l3fwd`` will be used for performance testing, with using two ports for bi-directional forwarding. diff --git a/doc/guides/nics/ifc.rst b/doc/guides/nics/ifc.rst new file mode 100644 index 00000000..48f9adf1 --- /dev/null +++ b/doc/guides/nics/ifc.rst @@ -0,0 +1,96 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +IFCVF vDPA driver +================= + +The IFCVF vDPA (vhost data path acceleration) driver provides support for the +Intel FPGA 100G VF (IFCVF). IFCVF's datapath is virtio ring compatible, it +works as a HW vhost backend which can send/receive packets to/from virtio +directly by DMA. Besides, it supports dirty page logging and device state +report/restore, this driver enables its vDPA functionality. + + +Pre-Installation Configuration +------------------------------ + +Config File Options +~~~~~~~~~~~~~~~~~~~ + +The following option can be modified in the ``config`` file. + +- ``CONFIG_RTE_LIBRTE_IFCVF_VDPA_PMD`` (default ``y`` for linux) + + Toggle compilation of the ``librte_ifcvf_vdpa`` driver. + + +IFCVF vDPA Implementation +------------------------- + +IFCVF's vendor ID and device ID are same as that of virtio net pci device, +with its specific subsystem vendor ID and device ID. To let the device be +probed by IFCVF driver, adding "vdpa=1" parameter helps to specify that this +device is to be used in vDPA mode, rather than polling mode, virtio pmd will +skip when it detects this message. + +Different VF devices serve different virtio frontends which are in different +VMs, so each VF needs to have its own DMA address translation service. During +the driver probe a new container is created for this device, with this +container vDPA driver can program DMA remapping table with the VM's memory +region information. + +Key IFCVF vDPA driver ops +~~~~~~~~~~~~~~~~~~~~~~~~~ + +- ifcvf_dev_config: + Enable VF data path with virtio information provided by vhost lib, including + IOMMU programming to enable VF DMA to VM's memory, VFIO interrupt setup to + route HW interrupt to virtio driver, create notify relay thread to translate + virtio driver's kick to a MMIO write onto HW, HW queues configuration. + + This function gets called to set up HW data path backend when virtio driver + in VM gets ready. + +- ifcvf_dev_close: + Revoke all the setup in ifcvf_dev_config. + + This function gets called when virtio driver stops device in VM. + +To create a vhost port with IFC VF +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Create a vhost socket and assign a VF's device ID to this socket via + vhost API. When QEMU vhost connection gets ready, the assigned VF will + get configured automatically. + + +Features +-------- + +Features of the IFCVF driver are: + +- Compatibility with virtio 0.95 and 1.0. + + +Prerequisites +------------- + +- Platform with IOMMU feature. IFC VF needs address translation service to + Rx/Tx directly with virtio driver in VM. + + +Limitations +----------- + +Dependency on vfio-pci +~~~~~~~~~~~~~~~~~~~~~~ + +vDPA driver needs to setup VF MSIX interrupts, each queue's interrupt vector +is mapped to a callfd associated with a virtio ring. Currently only vfio-pci +allows multiple interrupts, so the IFCVF driver is dependent on vfio-pci. + +Live Migration with VIRTIO_NET_F_GUEST_ANNOUNCE +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +IFC VF doesn't support RARP packet generation, virtio frontend supporting +VIRTIO_NET_F_GUEST_ANNOUNCE feature can help to do that. diff --git a/doc/guides/nics/img/szedata2_nfb200g_architecture.svg b/doc/guides/nics/img/szedata2_nfb200g_architecture.svg new file mode 100644 index 00000000..e152e4a8 --- /dev/null +++ b/doc/guides/nics/img/szedata2_nfb200g_architecture.svg @@ -0,0 +1,214 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + id="svg2" + stroke-miterlimit="10" + stroke-linecap="square" + stroke="none" + fill="none" + viewBox="0.0 0.0 568.7322834645669 352.3937007874016" + version="1.1"> + <metadata + id="metadata65"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <defs + id="defs63" /> + <clipPath + id="p.0"> + <path + id="path5" + clip-rule="nonzero" + d="m0 0l568.7323 0l0 352.3937l-568.7323 0l0 -352.3937z" /> + </clipPath> + <g + id="g7" + clip-path="url(#p.0)"> + <path + id="path9" + fill-rule="evenodd" + d="m0 0l568.7323 0l0 352.3937l-568.7323 0z" + fill-opacity="0.0" + fill="#000000" /> + <path + id="path11" + d="m 40.564137,14.365075 254.362203,0 0,131.842535 -254.362203,0 z" + style="fill:#47c3d3;fill-rule:evenodd" /> + <path + id="path15" + d="m 54.075948,146.2076 227.338592,0 0,32.94488 -227.338592,0 z" + style="fill:#c2c2c2;fill-rule:evenodd" /> + <path + id="path19" + d="m 321.90535,146.2076 227.33856,0 0,32.94488 -227.33856,0 z" + style="fill:#c2c2c2;fill-rule:evenodd" /> + <path + id="path23" + d="m 440.30217,146.24338 -11.82364,-20.50632 6.86313,0 0,-44.550399 -120.12924,0 0,6.938519 -20.28345,-11.953539 20.28345,-11.953547 0,6.93852 130.0503,0 0,54.580446 6.8631,0 z" + style="fill:#9a9a9a;fill-rule:evenodd" /> + <path + id="path25" + d="m 112.39353,263.09765 0,0 c 0,-8.08875 6.55722,-14.64597 14.64597,-14.64597 l 58.58208,0 0,0 c 3.88435,0 7.60962,1.54305 10.35626,4.28971 2.74666,2.74664 4.28971,6.47189 4.28971,10.35626 l 0,58.58209 c 0,8.08875 -6.55722,14.64597 -14.64597,14.64597 l -58.58208,0 c -8.08875,0 -14.64597,-6.55722 -14.64597,-14.64597 z" + style="fill:#c2c2c2;fill-rule:evenodd" /> + <path + id="path29" + d="m 391.63763,263.09765 0,0 c 0,-8.08875 6.55722,-14.64597 14.64597,-14.64597 l 58.58209,0 0,0 c 3.88437,0 7.60962,1.54305 10.35626,4.28971 2.74664,2.74664 4.2897,6.47189 4.2897,10.35626 l 0,58.58209 c 0,8.08875 -6.55722,14.64597 -14.64596,14.64597 l -58.58209,0 c -8.08875,0 -14.64597,-6.55722 -14.64597,-14.64597 z" + style="fill:#c2c2c2;fill-rule:evenodd" /> + <path + id="path33" + d="m 135.20981,199.01075 19.85826,-19.85826 19.85828,19.85826 -9.92914,0 0,29.5748 9.92914,0 -19.85828,19.85827 -19.85826,-19.85827 9.92914,0 0,-29.5748 z" + style="fill:#9a9a9a;fill-rule:evenodd" /> + <path + id="path35" + d="m 415.71635,199.01064 19.85828,-19.85826 19.85827,19.85826 -9.92914,0 0,29.57481 9.92914,0 -19.85827,19.85826 -19.85828,-19.85826 9.92914,0 0,-29.57481 z" + style="fill:#9a9a9a;fill-rule:evenodd" /> + <path + id="path37" + d="m 15.205,31.273212 74.362206,0 0,32.944885 -74.362206,0 z" + style="fill:#ff8434;fill-rule:evenodd" /> + <path + id="path41" + d="m 16.05531,80.231216 74.3622,0 0,32.944884 -74.3622,0 z" + style="fill:#ff8434;fill-rule:evenodd" /> + <path + id="path45" + d="m 275.44377,174.07111 0,111.55905 -37.16536,0 0,-111.55905 z" + style="fill:#ff8434;fill-rule:evenodd" /> + <path + id="path49" + d="m 97.923493,174.07111 0,111.55905 -37.16535,0 0,-111.55905 z" + style="fill:#ff8434;fill-rule:evenodd" /> + <path + id="path53" + d="m 366.27543,174.07111 0,111.55905 -37.16537,0 0,-111.55905 z" + style="fill:#ff8434;fill-rule:evenodd" /> + <path + id="path57" + d="m 542.0392,174.07111 0,111.55905 -37.16534,0 0,-111.55905 z" + style="fill:#ff8434;fill-rule:evenodd" /> + <text + id="text4480" + y="54.570911" + x="24.425898" + style="font-style:normal;font-weight:normal;font-size:18.75px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + y="54.570911" + x="24.425898" + id="tspan4482">ETH 0</tspan></text> + <text + id="text4480-3" + y="103.53807" + x="25.51882" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502" + y="103.53807" + x="25.51882">ETH 1</tspan></text> + <text + id="text4480-7" + y="86.200645" + x="103.15979" + style="font-style:normal;font-weight:normal;font-size:18.75px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + id="tspan4524" + y="86.200645" + x="103.15979">NFB-200G2QL card</tspan></text> + <text + id="text4480-7-3" + y="169.2041" + x="92.195312" + style="font-style:normal;font-weight:normal;font-size:18.75px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4546" + y="169.2041" + x="92.195312">PCI-E master slot</tspan></text> + <text + id="text4480-7-3-6" + y="169.20409" + x="367.98856" + style="font-style:normal;font-weight:normal;font-size:18.75px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4546-2" + y="169.20409" + x="367.98856">PCI-E slave slot</tspan></text> + <text + transform="matrix(0,1,-1,0,0,0)" + id="text4480-3-9" + y="-73.591309" + x="182.29367" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502-1" + y="-73.591309" + x="182.29367">QUEUE 0</tspan></text> + <text + transform="matrix(0,1.0000002,-0.99999976,0,0,0)" + id="text4480-3-9-2" + y="-251.11163" + x="182.29283" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502-1-7" + y="-251.11163" + x="182.29283">QUEUE 15</tspan></text> + <text + transform="matrix(0,1,-1,0,0,0)" + id="text4480-3-9-2-0" + y="-341.94324" + x="182.29311" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502-1-7-9" + y="-341.94324" + x="182.29311">QUEUE 16</tspan></text> + <text + transform="matrix(0,1,-1,0,0,0)" + id="text4480-3-9-2-3" + y="-517.70703" + x="182.29356" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502-1-7-6" + y="-517.70703" + x="182.29356">QUEUE 31</tspan></text> + <text + id="text4480-3-0" + y="299.21396" + x="128.3978" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502-6" + y="299.21396" + x="128.3978">CPU 0</tspan></text> + <text + id="text4480-3-0-2" + y="299.21396" + x="407.88452" + style="font-style:normal;font-weight:normal;font-size:20px;line-height:125%;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + xml:space="preserve"><tspan + style="font-size:18.75px" + id="tspan4502-6-6" + y="299.21396" + x="407.88452">CPU 1</tspan></text> + </g> +</svg> diff --git a/doc/guides/nics/index.rst b/doc/guides/nics/index.rst index 59419f43..59f6063d 100644 --- a/doc/guides/nics/index.rst +++ b/doc/guides/nics/index.rst @@ -13,6 +13,7 @@ Network Interface Controller Drivers build_and_test ark avp + axgbe bnx2x bnxt cxgbe @@ -23,6 +24,7 @@ Network Interface Controller Drivers enic fm10k i40e + ifc igb ixgbe intel_vf @@ -30,11 +32,13 @@ Network Interface Controller Drivers liquidio mlx4 mlx5 - mrvl + mvpp2 + netvsc nfp octeontx qede sfc_efx + softnic szedata2 tap thunderx diff --git a/doc/guides/nics/ixgbe.rst b/doc/guides/nics/ixgbe.rst index 0c660f29..16d63902 100644 --- a/doc/guides/nics/ixgbe.rst +++ b/doc/guides/nics/ixgbe.rst @@ -68,15 +68,15 @@ Other features are supported using optional MACRO configuration. They include: * HW extend dual VLAN -To guarantee the constraint, configuration flags in dev_conf.rxmode will be checked: +To guarantee the constraint, capabilities in dev_conf.rxmode.offloads will be checked: -* hw_vlan_strip +* DEV_RX_OFFLOAD_VLAN_STRIP -* hw_vlan_extend +* DEV_RX_OFFLOAD_VLAN_EXTEND -* hw_ip_checksum +* DEV_RX_OFFLOAD_CHECKSUM -* header_split +* DEV_RX_OFFLOAD_HEADER_SPLIT * dev_conf @@ -102,20 +102,9 @@ Consequently, by default the tx_rs_thresh value is in the range 32 to 64. Feature not Supported by TX Vector PMD ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -TX vPMD only works when txq_flags is set to IXGBE_SIMPLE_FLAGS. +TX vPMD only works when offloads is set to 0 -This means that it does not support TX multi-segment, VLAN offload and TX csum offload. -The following MACROs are used for these three features: - -* ETH_TXQ_FLAGS_NOMULTSEGS - -* ETH_TXQ_FLAGS_NOVLANOFFL - -* ETH_TXQ_FLAGS_NOXSUMSCTP - -* ETH_TXQ_FLAGS_NOXSUMUDP - -* ETH_TXQ_FLAGS_NOXSUMTCP +This means that it does not support any TX offload. Application Programming Interface --------------------------------- @@ -130,13 +119,13 @@ l3fwd ~~~~~ When running l3fwd with vPMD, there is one thing to note. -In the configuration, ensure that port_conf.rxmode.hw_ip_checksum=0. +In the configuration, ensure that DEV_RX_OFFLOAD_CHECKSUM in port_conf.rxmode.offloads is NOT set. Otherwise, by default, RX vPMD is disabled. load_balancer ~~~~~~~~~~~~~ -As in the case of l3fwd, set configure port_conf.rxmode.hw_ip_checksum=0 to enable vPMD. +As in the case of l3fwd, to enable vPMD, do NOT set DEV_RX_OFFLOAD_CHECKSUM in port_conf.rxmode.offloads. In addition, for improved performance, use -bsz "(32,32),(64,64),(32,32)" in load_balancer to avoid using the default burst size of 144. @@ -228,6 +217,20 @@ For more details see the IPsec Security Gateway Sample Application and Security library documentation. +Virtual Function Port Representors +---------------------------------- +The IXGBE PF PMD supports the creation of VF port representors for the control +and monitoring of IXGBE virtual function devices. Each port representor +corresponds to a single virtual function of that device. Using the ``devargs`` +option ``representor`` the user can specify which virtual functions to create +port representors for on initialization of the PF PMD by passing the VF IDs of +the VFs which are required.:: + + -w DBDF,representor=[0,1,4] + +Currently hot-plugging of representor ports is not supported so all required +representors must be specified on the creation of the PF. + Supported Chipsets and NICs --------------------------- diff --git a/doc/guides/nics/liquidio.rst b/doc/guides/nics/liquidio.rst index 61485ade..87b42cdc 100644 --- a/doc/guides/nics/liquidio.rst +++ b/doc/guides/nics/liquidio.rst @@ -201,6 +201,4 @@ Number of descriptors for Rx/Tx ring should be in the range 128 to 512. CRC striping ~~~~~~~~~~~~ -LiquidIO adapters strip ethernet FCS of every packet coming to the host -interface. So, CRC will be stripped even when the ``rxmode.hw_strip_crc`` -member is set to 0 in ``struct rte_eth_conf``. +LiquidIO adapters strip ethernet FCS of every packet coming to the host interface. diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst index 98b97166..4a57c7a6 100644 --- a/doc/guides/nics/mlx4.rst +++ b/doc/guides/nics/mlx4.rst @@ -1,32 +1,6 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2012 6WIND S.A. - Copyright 2015 Mellanox - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of 6WIND S.A. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Copyright 2015 Mellanox Technologies, Ltd MLX4 poll mode driver library ============================= @@ -98,9 +72,10 @@ These options can be modified in the ``.config`` file. missing with ``ldd(1)``. It works by moving these dependencies to a purpose-built rdma-core "glue" - plug-in, which must either be installed in ``CONFIG_RTE_EAL_PMD_PATH`` if - set, or in a standard location for the dynamic linker (e.g. ``/lib``) if - left to the default empty string (``""``). + plug-in which must either be installed in a directory whose name is based + on ``CONFIG_RTE_EAL_PMD_PATH`` suffixed with ``-glue`` if set, or in a + standard location for the dynamic linker (e.g. ``/lib``) if left to the + default empty string (``""``). This option has no performance impact. @@ -110,14 +85,6 @@ These options can be modified in the ``.config`` file. adds additional run-time checks and debugging messages at the cost of lower performance. -- ``CONFIG_RTE_LIBRTE_MLX4_TX_MP_CACHE`` (default **8**) - - Maximum number of cached memory pools (MPs) per TX queue. Each MP from - which buffers are to be transmitted must be associated to memory regions - (MRs). This is a slow operation that must be cached. - - This value is always 1 for RX queues since they use a single MP. - Environment variables ~~~~~~~~~~~~~~~~~~~~~ @@ -168,6 +135,16 @@ below. following limitation: VLAN filtering is not supported with this mode. This is the recommended mode in case VLAN filter is not needed. +Limitations +----------- + +- CRC stripping is supported by default and always reported as "true". + The ability to enable/disable CRC stripping requires OFED version + 4.3-1.5.0.0 and above or rdma-core version v18 and above. + +- TSO (Transmit Segmentation Offload) is supported in OFED version + 4.4 and above. + Prerequisites ------------- @@ -236,7 +213,7 @@ Current RDMA core package and Linux kernel (recommended) Mellanox OFED as a fallback ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- `Mellanox OFED`_ version: **4.2, 4.3**. +- `Mellanox OFED`_ version: **4.3, 4.4**. - firmware version: **2.42.5000** and above. .. _`Mellanox OFED`: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers @@ -396,6 +373,12 @@ Performance tuning The XXX can be different on different systems. Make sure to configure according to the setpci output. +6. To minimize overhead of searching Memory Regions: + + - '--socket-mem' is recommended to pin memory by predictable amount. + - Configure per-lcore cache when creating Mempools for packet buffer. + - Refrain from dynamically allocating/freeing memory in run-time. + Usage example ------------- diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index 0e6e525c..52e1213c 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1,40 +1,14 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2015 6WIND S.A. - Copyright 2015 Mellanox - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of 6WIND S.A. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Copyright 2015 Mellanox Technologies, Ltd MLX5 poll mode driver ===================== The MLX5 poll mode driver library (**librte_pmd_mlx5**) provides support -for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** and **Mellanox -ConnectX-5** families of 10/25/40/50/100 Gb/s adapters as well as their -virtual functions (VF) in SR-IOV context. +for **Mellanox ConnectX-4**, **Mellanox ConnectX-4 Lx** , **Mellanox +ConnectX-5** and **Mellanox Bluefield** families of 10/25/40/50/100 Gb/s +adapters as well as their virtual functions (VF) in SR-IOV context. Information and documentation about these adapters can be found on the `Mellanox website <http://www.mellanox.com>`__. Help is also provided by the @@ -75,7 +49,7 @@ libibverbs. Features -------- -- Multi arch support: x86_64, POWER8, ARMv8. +- Multi arch support: x86_64, POWER8, ARMv8, i686. - Multiple TX and RX queues. - Support for scattered TX and RX frames. - IPv4, IPv6, TCPv4, TCPv6, UDPv4 and UDPv6 RSS on any number of queues. @@ -95,17 +69,17 @@ Features - Multiple process. - KVM and VMware ESX SR-IOV modes are supported. - RSS hash result is supported. -- Hardware TSO. -- Hardware checksum TX offload for VXLAN and GRE. +- Hardware TSO for generic IP or UDP tunnel, including VXLAN and GRE. +- Hardware checksum Tx offload for generic IP or UDP tunnel, including VXLAN and GRE. - RX interrupts. - Statistics query including Basic, Extended and per queue. - Rx HW timestamp. +- Tunnel types: VXLAN, L3 VXLAN, VXLAN-GPE, GRE, MPLSoGRE, MPLSoUDP. +- Tunnel HW offloads: packet type, inner/outer RSS, IP and UDP checksum verification. Limitations ----------- -- Inner RSS for VXLAN frames is not supported yet. -- Hardware checksum RX offloads for VXLAN inner header are not supported yet. - For secondary process: - Forked secondary process not supported. @@ -131,11 +105,37 @@ Limitations - A multi segment packet must have less than 6 segments in case the Tx burst function is set to multi-packet send or Enhanced multi-packet send. Otherwise it must have less than 50 segments. + - Count action for RTE flow is **only supported in Mellanox OFED**. + - Flows with a VXLAN Network Identifier equal (or ends to be equal) to 0 are not supported. + - VXLAN TSO and checksum offloads are not supported on VM. +- L3 VXLAN and VXLAN-GPE tunnels cannot be supported together with MPLSoGRE and MPLSoUDP. + +- VF: flow rules created on VF devices can only match traffic targeted at the + configured MAC addresses (see ``rte_eth_dev_mac_addr_add()``). + +.. note:: + + MAC addresses not already present in the bridge table of the associated + kernel network device will be added and cleaned up by the PMD when closing + the device. In case of ungraceful program termination, some entries may + remain present and should be removed manually by other means. + +- When Multi-Packet Rx queue is configured (``mprq_en``), a Rx packet can be + externally attached to a user-provided mbuf with having EXT_ATTACHED_MBUF in + ol_flags. As the mempool for the external buffer is managed by PMD, all the + Rx mbufs must be freed before the device is closed. Otherwise, the mempool of + the external buffers will be freed by PMD and the application which still + holds the external buffers may be corrupted. + +- If Multi-Packet Rx queue is configured (``mprq_en``) and Rx CQE compression is + enabled (``rxq_cqe_comp_en``) at the same time, RSS hash result is not fully + supported. Some Rx packets may not have PKT_RX_RSS_HASH. + Statistics ---------- @@ -171,9 +171,10 @@ These options can be modified in the ``.config`` file. missing with ``ldd(1)``. It works by moving these dependencies to a purpose-built rdma-core "glue" - plug-in, which must either be installed in ``CONFIG_RTE_EAL_PMD_PATH`` if - set, or in a standard location for the dynamic linker (e.g. ``/lib``) if - left to the default empty string (``""``). + plug-in which must either be installed in a directory whose name is based + on ``CONFIG_RTE_EAL_PMD_PATH`` suffixed with ``-glue`` if set, or in a + standard location for the dynamic linker (e.g. ``/lib``) if left to the + default empty string (``""``). This option has no performance impact. @@ -183,14 +184,6 @@ These options can be modified in the ``.config`` file. adds additional run-time checks and debugging messages at the cost of lower performance. -- ``CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE`` (default **8**) - - Maximum number of cached memory pools (MPs) per TX queue. Each MP from - which buffers are to be transmitted must be associated to memory regions - (MRs). This is a slow operation that must be cached. - - This value is always 1 for RX queues since they use a single MP. - Environment variables ~~~~~~~~~~~~~~~~~~~~~ @@ -250,8 +243,55 @@ Run-time configuration Supported on: - - x86_64 with ConnectX-4, ConnectX-4 LX and ConnectX-5. - - POWER8 and ARMv8 with ConnectX-4 LX and ConnectX-5. + - x86_64 with ConnectX-4, ConnectX-4 LX, ConnectX-5 and Bluefield. + - POWER8 and ARMv8 with ConnectX-4 LX, ConnectX-5 and Bluefield. + +- ``mprq_en`` parameter [int] + + A nonzero value enables configuring Multi-Packet Rx queues. Rx queue is + configured as Multi-Packet RQ if the total number of Rx queues is + ``rxqs_min_mprq`` or more and Rx scatter isn't configured. Disabled by + default. + + Multi-Packet Rx Queue (MPRQ a.k.a Striding RQ) can further save PCIe bandwidth + by posting a single large buffer for multiple packets. Instead of posting a + buffers per a packet, one large buffer is posted in order to receive multiple + packets on the buffer. A MPRQ buffer consists of multiple fixed-size strides + and each stride receives one packet. MPRQ can improve throughput for + small-packet tarffic. + + When MPRQ is enabled, max_rx_pkt_len can be larger than the size of + user-provided mbuf even if DEV_RX_OFFLOAD_SCATTER isn't enabled. PMD will + configure large stride size enough to accommodate max_rx_pkt_len as long as + device allows. Note that this can waste system memory compared to enabling Rx + scatter and multi-segment packet. + +- ``mprq_log_stride_num`` parameter [int] + + Log 2 of the number of strides for Multi-Packet Rx queue. Configuring more + strides can reduce PCIe tarffic further. If configured value is not in the + range of device capability, the default value will be set with a warning + message. The default value is 4 which is 16 strides per a buffer, valid only + if ``mprq_en`` is set. + + The size of Rx queue should be bigger than the number of strides. + +- ``mprq_max_memcpy_len`` parameter [int] + + The maximum length of packet to memcpy in case of Multi-Packet Rx queue. Rx + packet is mem-copied to a user-provided mbuf if the size of Rx packet is less + than or equal to this parameter. Otherwise, PMD will attach the Rx packet to + the mbuf by external buffer attachment - ``rte_pktmbuf_attach_extbuf()``. + A mempool for external buffers will be allocated and managed by PMD. If Rx + packet is externally attached, ol_flags field of the mbuf will have + EXT_ATTACHED_MBUF and this flag must be preserved. ``RTE_MBUF_HAS_EXTBUF()`` + checks the flag. The default value is 128, valid only if ``mprq_en`` is set. + +- ``rxqs_min_mprq`` parameter [int] + + Configure Rx queues as Multi-Packet RQ if the total number of Rx queues is + greater or equal to this value. The default value is 12, valid only if + ``mprq_en`` is set. - ``txq_inline`` parameter [int] @@ -270,34 +310,35 @@ Run-time configuration This option should be used in combination with ``txq_inline`` above. - On ConnectX-4, ConnectX-4 LX and ConnectX-5 without Enhanced MPW: + On ConnectX-4, ConnectX-4 LX, ConnectX-5 and Bluefield without + Enhanced MPW: - Disabled by default. - In case ``txq_inline`` is set recommendation is 4. - On ConnectX-5 with Enhanced MPW: + On ConnectX-5 and Bluefield with Enhanced MPW: - Set to 8 by default. - ``txq_mpw_en`` parameter [int] A nonzero value enables multi-packet send (MPS) for ConnectX-4 Lx and - enhanced multi-packet send (Enhanced MPS) for ConnectX-5. MPS allows the - TX burst function to pack up multiple packets in a single descriptor - session in order to save PCI bandwidth and improve performance at the - cost of a slightly higher CPU usage. When ``txq_inline`` is set along - with ``txq_mpw_en``, TX burst function tries to copy entire packet data - on to TX descriptor instead of including pointer of packet only if there - is enough room remained in the descriptor. ``txq_inline`` sets - per-descriptor space for either pointers or inlined packets. In addition, - Enhanced MPS supports hybrid mode - mixing inlined packets and pointers - in the same descriptor. + enhanced multi-packet send (Enhanced MPS) for ConnectX-5 and Bluefield. + MPS allows the TX burst function to pack up multiple packets in a + single descriptor session in order to save PCI bandwidth and improve + performance at the cost of a slightly higher CPU usage. When + ``txq_inline`` is set along with ``txq_mpw_en``, TX burst function tries + to copy entire packet data on to TX descriptor instead of including + pointer of packet only if there is enough room remained in the + descriptor. ``txq_inline`` sets per-descriptor space for either pointers + or inlined packets. In addition, Enhanced MPS supports hybrid mode - + mixing inlined packets and pointers in the same descriptor. This option cannot be used with certain offloads such as ``DEV_TX_OFFLOAD_TCP_TSO, DEV_TX_OFFLOAD_VXLAN_TNL_TSO, DEV_TX_OFFLOAD_GRE_TNL_TSO, DEV_TX_OFFLOAD_VLAN_INSERT``. When those offloads are requested the MPS send function will not be used. - It is currently only supported on the ConnectX-4 Lx and ConnectX-5 + It is currently only supported on the ConnectX-4 Lx, ConnectX-5 and Bluefield families of adapters. Enabled by default. - ``txq_mpw_hdr_dseg_en`` parameter [int] @@ -318,14 +359,14 @@ Run-time configuration - ``tx_vec_en`` parameter [int] - A nonzero value enables Tx vector on ConnectX-5 only NIC if the number of + A nonzero value enables Tx vector on ConnectX-5 and Bluefield NICs if the number of global Tx queues on the port is lesser than MLX5_VPMD_MIN_TXQS. This option cannot be used with certain offloads such as ``DEV_TX_OFFLOAD_TCP_TSO, DEV_TX_OFFLOAD_VXLAN_TNL_TSO, DEV_TX_OFFLOAD_GRE_TNL_TSO, DEV_TX_OFFLOAD_VLAN_INSERT``. When those offloads are requested the MPS send function will not be used. - Enabled by default on ConnectX-5. + Enabled by default on ConnectX-5 and Bluefield. - ``rx_vec_en`` parameter [int] @@ -334,6 +375,53 @@ Run-time configuration Enabled by default. +- ``vf_nl_en`` parameter [int] + + A nonzero value enables Netlink requests from the VF to add/remove MAC + addresses or/and enable/disable promiscuous/all multicast on the Netdevice. + Otherwise the relevant configuration must be run with Linux iproute2 tools. + This is a prerequisite to receive this kind of traffic. + + Enabled by default, valid only on VF devices ignored otherwise. + +- ``l3_vxlan_en`` parameter [int] + + A nonzero value allows L3 VXLAN and VXLAN-GPE flow creation. To enable + L3 VXLAN or VXLAN-GPE, users has to configure firmware and enable this + parameter. This is a prerequisite to receive this kind of traffic. + + Disabled by default. + +- ``representor`` parameter [list] + + This parameter can be used to instantiate DPDK Ethernet devices from + existing port (or VF) representors configured on the device. + + It is a standard parameter whose format is described in + :ref:`ethernet_device_standard_device_arguments`. + + For instance, to probe port representors 0 through 2:: + + representor=[0-2] + +Firmware configuration +~~~~~~~~~~~~~~~~~~~~~~ + +- L3 VXLAN and VXLAN-GPE destination UDP port + + .. code-block:: console + + mlxconfig -d <mst device> set IP_OVER_VXLAN_EN=1 + mlxconfig -d <mst device> set IP_OVER_VXLAN_PORT=<udp dport> + + Verify configurations are set: + + .. code-block:: console + + mlxconfig -d <mst device> query | grep IP_OVER_VXLAN + IP_OVER_VXLAN_EN True(1) + IP_OVER_VXLAN_PORT <udp dport> + Prerequisites ------------- @@ -353,12 +441,19 @@ DPDK and must be installed separately: - **libmlx5** - Low-level user space driver library for Mellanox ConnectX-4/ConnectX-5 - devices, it is automatically loaded by libibverbs. + Low-level user space driver library for Mellanox + ConnectX-4/ConnectX-5/Bluefield devices, it is automatically loaded + by libibverbs. This library basically implements send/receive calls to the hardware queues. +- **libmnl** + + Minimalistic Netlink library mainly relied on to manage E-Switch flow + rules (i.e. those with the "transfer" attribute and typically involving + port representors). + - **Kernel modules** They provide the kernel-side Verbs API and low level device drivers that @@ -368,15 +463,16 @@ DPDK and must be installed separately: Unlike most other PMDs, these modules must remain loaded and bound to their devices: - - mlx5_core: hardware driver managing Mellanox ConnectX-4/ConnectX-5 - devices and related Ethernet kernel network devices. + - mlx5_core: hardware driver managing Mellanox + ConnectX-4/ConnectX-5/Bluefield devices and related Ethernet kernel + network devices. - mlx5_ib: InifiniBand device driver. - ib_uverbs: user space driver for Verbs (entry point for libibverbs). - **Firmware update** - Mellanox OFED releases include firmware updates for ConnectX-4/ConnectX-5 - adapters. + Mellanox OFED releases include firmware updates for + ConnectX-4/ConnectX-5/Bluefield adapters. Because each release provides new features, these updates must be applied to match the kernel modules and libraries they come with. @@ -399,6 +495,10 @@ RMDA Core with Linux Kernel - Minimal kernel version : v4.14 or the most recent 4.14-rc (see `Linux installation documentation`_) - Minimal rdma-core version: v15+ commit 0c5f5765213a ("Merge pull request #227 from yishaih/tm") (see `RDMA Core installation documentation`_) +- When building for i686 use: + + - rdma-core version 18.0 or above built with 32bit support. + - Kernel version 4.14.41 or above. .. _`Linux installation documentation`: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/plain/Documentation/admin-guide/README.rst .. _`RDMA Core installation documentation`: https://raw.githubusercontent.com/linux-rdma/rdma-core/master/README.md @@ -406,13 +506,14 @@ RMDA Core with Linux Kernel Mellanox OFED ^^^^^^^^^^^^^ -- Mellanox OFED version: **4.2, 4.3**. +- Mellanox OFED version: **4.3, 4.4**. - firmware version: - ConnectX-4: **12.21.1000** and above. - ConnectX-4 Lx: **14.21.1000** and above. - ConnectX-5: **16.21.1000** and above. - ConnectX-5 Ex: **16.21.1000** and above. + - Bluefield: **18.99.3950** and above. While these libraries and kernel modules are available on OpenFabrics Alliance's `website <https://www.openfabrics.org/>`__ and provided by package @@ -431,6 +532,19 @@ required from that distribution. this DPDK release was developed and tested against is strongly recommended. Please check the `prerequisites`_. +Libmnl +^^^^^^ + +Minimal version for libmnl is **1.0.3**. + +As a dependency of the **iproute2** suite, this library is often installed +by default. It is otherwise readily available through standard system +packages. + +Its development headers must be installed in order to compile this PMD. +These packages are usually named **libmnl-dev** or **libmnl-devel** +depending on the Linux distribution. + Supported NICs -------------- @@ -603,6 +717,12 @@ Performance tuning The XXX can be different on different systems. Make sure to configure according to the setpci output. +7. To minimize overhead of searching Memory Regions: + + - '--socket-mem' is recommended to pin memory by predictable amount. + - Configure per-lcore cache when creating Mempools for packet buffer. + - Refrain from dynamically allocating/freeing memory in run-time. + Notes for testpmd ----------------- @@ -624,7 +744,7 @@ Usage example ------------- This section demonstrates how to launch **testpmd** with Mellanox -ConnectX-4/ConnectX-5 devices managed by librte_pmd_mlx5. +ConnectX-4/ConnectX-5/Bluefield devices managed by librte_pmd_mlx5. #. Load the kernel modules: diff --git a/doc/guides/nics/mrvl.rst b/doc/guides/nics/mrvl.rst deleted file mode 100644 index b7f32921..00000000 --- a/doc/guides/nics/mrvl.rst +++ /dev/null @@ -1,275 +0,0 @@ -.. BSD LICENSE - Copyright(c) 2017 Marvell International Ltd. - Copyright(c) 2017 Semihalf. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of the copyright holder nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - -.. _mrvl_poll_mode_driver: - -MRVL Poll Mode Driver -====================== - -The MRVL PMD (librte_pmd_mrvl) provides poll mode driver support -for the Marvell PPv2 (Packet Processor v2) 1/10 Gbps adapter. - -Detailed information about SoCs that use PPv2 can be obtained here: - -* https://www.marvell.com/embedded-processors/armada-70xx/ -* https://www.marvell.com/embedded-processors/armada-80xx/ - -.. Note:: - - Due to external dependencies, this driver is disabled by default. It must - be enabled manually by setting relevant configuration option manually. - Please refer to `Config File Options`_ section for further details. - - -Features --------- - -Features of the MRVL PMD are: - -- Speed capabilities -- Link status -- Queue start/stop -- MTU update -- Jumbo frame -- Promiscuous mode -- Allmulticast mode -- Unicast MAC filter -- Multicast MAC filter -- RSS hash -- VLAN filter -- CRC offload -- L3 checksum offload -- L4 checksum offload -- Packet type parsing -- Basic stats -- QoS - - -Limitations ------------ - -- Number of lcores is limited to 9 by MUSDK internal design. If more lcores - need to be allocated, locking will have to be considered. Number of available - lcores can be changed via ``MRVL_MUSDK_HIFS_RESERVED`` define in - ``mrvl_ethdev.c`` source file. - -- Flushing vlans added for filtering is not possible due to MUSDK missing - functionality. Current workaround is to reset board so that PPv2 has a - chance to start in a sane state. - - -Prerequisites -------------- - -- Custom Linux Kernel sources - - .. code-block:: console - - git clone https://github.com/MarvellEmbeddedProcessors/linux-marvell.git -b linux-4.4.52-armada-17.10 - -- Out of tree `mvpp2x_sysfs` kernel module sources - - .. code-block:: console - - git clone https://github.com/MarvellEmbeddedProcessors/mvpp2x-marvell.git -b mvpp2x-armada-17.10 - -- MUSDK (Marvell User-Space SDK) sources - - .. code-block:: console - - git clone https://github.com/MarvellEmbeddedProcessors/musdk-marvell.git -b musdk-armada-17.10 - - MUSDK is a light-weight library that provides direct access to Marvell's - PPv2 (Packet Processor v2). Alternatively prebuilt MUSDK library can be - requested from `Marvell Extranet <https://extranet.marvell.com>`_. Once - approval has been granted, library can be found by typing ``musdk`` in - the search box. - - MUSDK must be configured with the following features: - - .. code-block:: console - - --enable-bpool-dma=64 - -- DPDK environment - - Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup - DPDK environment. - - -Config File Options -------------------- - -The following options can be modified in the ``config`` file. - -- ``CONFIG_RTE_LIBRTE_MRVL_PMD`` (default ``n``) - - Toggle compilation of the librte_pmd_mrvl driver. - - -QoS Configuration ------------------ - -QoS configuration is done through external configuration file. Path to the -file must be given as `cfg` in driver's vdev parameter list. - -Configuration syntax -~~~~~~~~~~~~~~~~~~~~ - -.. code-block:: console - - [port <portnum> default] - default_tc = <default_tc> - mapping_priority = <mapping_priority> - - [port <portnum> tc <traffic_class>] - rxq = <rx_queue_list> - pcp = <pcp_list> - dscp = <dscp_list> - - [port <portnum> tc <traffic_class>] - rxq = <rx_queue_list> - pcp = <pcp_list> - dscp = <dscp_list> - -Where: - -- ``<portnum>``: DPDK Port number (0..n). - -- ``<default_tc>``: Default traffic class (e.g. 0) - -- ``<mapping_priority>``: QoS priority for mapping (`ip`, `vlan`, `ip/vlan` or `vlan/ip`). - -- ``<traffic_class>``: Traffic Class to be configured. - -- ``<rx_queue_list>``: List of DPDK RX queues (e.g. 0 1 3-4) - -- ``<pcp_list>``: List of PCP values to handle in particular TC (e.g. 0 1 3-4 7). - -- ``<dscp_list>``: List of DSCP values to handle in particular TC (e.g. 0-12 32-48 63). - -Setting PCP/DSCP values for the default TC is not required. All PCP/DSCP -values not assigned explicitly to particular TC will be handled by the -default TC. - -Configuration file example -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. code-block:: console - - [port 0 default] - default_tc = 0 - qos_mode = ip - - [port 0 tc 0] - rxq = 0 1 - - [port 0 tc 1] - rxq = 2 - pcp = 5 6 7 - dscp = 26-38 - - [port 1 default] - default_tc = 0 - qos_mode = vlan/ip - - [port 1 tc 0] - rxq = 0 - - [port 1 tc 1] - rxq = 1 2 - pcp = 5 6 7 - dscp = 26-38 - -Usage example -^^^^^^^^^^^^^ - -.. code-block:: console - - ./testpmd --vdev=eth_mrvl,iface=eth0,iface=eth2,cfg=/home/user/mrvl.conf \ - -c 7 -- -i -a --rxq=2 - - -Building DPDK -------------- - -Driver needs precompiled MUSDK library during compilation. - -.. code-block:: console - - export CROSS_COMPILE=<toolchain>/bin/aarch64-linux-gnu- - ./bootstrap - ./configure --host=aarch64-linux-gnu --enable-bpool-dma=64 - make install - -MUSDK will be installed to `usr/local` under current directory. -For the detailed build instructions please consult ``doc/musdk_get_started.txt``. - -Before the DPDK build process the environmental variable ``LIBMUSDK_PATH`` with -the path to the MUSDK installation directory needs to be exported. - -.. code-block:: console - - export LIBMUSDK_PATH=<musdk>/usr/local - export CROSS=aarch64-linux-gnu- - make config T=arm64-armv8a-linuxapp-gcc - sed -ri 's,(MRVL_PMD=)n,\1y,' build/.config - make - -Usage Example -------------- - -MRVL PMD requires extra out of tree kernel modules to function properly. -`musdk_uio` and `mv_pp_uio` sources are part of the MUSDK. Please consult -``doc/musdk_get_started.txt`` for the detailed build instructions. -For `mvpp2x_sysfs` please consult ``Documentation/pp22_sysfs.txt`` for the -detailed build instructions. - -.. code-block:: console - - insmod musdk_uio.ko - insmod mv_pp_uio.ko - insmod mvpp2x_sysfs.ko - -Additionally interfaces used by DPDK application need to be put up: - -.. code-block:: console - - ip link set eth0 up - ip link set eth2 up - -In order to run testpmd example application following command can be used: - -.. code-block:: console - - ./testpmd --vdev=eth_mrvl,iface=eth0,iface=eth2 -c 7 -- \ - --burst=128 --txd=2048 --rxd=1024 --rxq=2 --txq=2 --nb-cores=2 \ - -i -a --rss-udp diff --git a/doc/guides/nics/mvpp2.rst b/doc/guides/nics/mvpp2.rst new file mode 100644 index 00000000..0408752c --- /dev/null +++ b/doc/guides/nics/mvpp2.rst @@ -0,0 +1,520 @@ +.. BSD LICENSE + Copyright(c) 2017 Marvell International Ltd. + Copyright(c) 2017 Semihalf. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +.. _mvpp2_poll_mode_driver: + +MVPP2 Poll Mode Driver +====================== + +The MVPP2 PMD (librte_pmd_mvpp2) provides poll mode driver support +for the Marvell PPv2 (Packet Processor v2) 1/10 Gbps adapter. + +Detailed information about SoCs that use PPv2 can be obtained here: + +* https://www.marvell.com/embedded-processors/armada-70xx/ +* https://www.marvell.com/embedded-processors/armada-80xx/ + +.. Note:: + + Due to external dependencies, this driver is disabled by default. It must + be enabled manually by setting relevant configuration option manually. + Please refer to `Config File Options`_ section for further details. + + +Features +-------- + +Features of the MVPP2 PMD are: + +- Speed capabilities +- Link status +- Queue start/stop +- MTU update +- Jumbo frame +- Promiscuous mode +- Allmulticast mode +- Unicast MAC filter +- Multicast MAC filter +- RSS hash +- VLAN filter +- CRC offload +- L3 checksum offload +- L4 checksum offload +- Packet type parsing +- Basic stats +- Extended stats +- QoS +- RX flow control +- TX queue start/stop + + +Limitations +----------- + +- Number of lcores is limited to 9 by MUSDK internal design. If more lcores + need to be allocated, locking will have to be considered. Number of available + lcores can be changed via ``MRVL_MUSDK_HIFS_RESERVED`` define in + ``mrvl_ethdev.c`` source file. + +- Flushing vlans added for filtering is not possible due to MUSDK missing + functionality. Current workaround is to reset board so that PPv2 has a + chance to start in a sane state. + + +Prerequisites +------------- + +- Custom Linux Kernel sources + + .. code-block:: console + + git clone https://github.com/MarvellEmbeddedProcessors/linux-marvell.git -b linux-4.4.52-armada-17.10 + +- Out of tree `mvpp2x_sysfs` kernel module sources + + .. code-block:: console + + git clone https://github.com/MarvellEmbeddedProcessors/mvpp2x-marvell.git -b mvpp2x-armada-17.10 + +- MUSDK (Marvell User-Space SDK) sources + + .. code-block:: console + + git clone https://github.com/MarvellEmbeddedProcessors/musdk-marvell.git -b musdk-armada-17.10 + + MUSDK is a light-weight library that provides direct access to Marvell's + PPv2 (Packet Processor v2). Alternatively prebuilt MUSDK library can be + requested from `Marvell Extranet <https://extranet.marvell.com>`_. Once + approval has been granted, library can be found by typing ``musdk`` in + the search box. + + To get better understanding of the library one can consult documentation + available in the ``doc`` top level directory of the MUSDK sources. + + MUSDK must be configured with the following features: + + .. code-block:: console + + --enable-bpool-dma=64 + +- DPDK environment + + Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup + DPDK environment. + + +Config File Options +------------------- + +The following options can be modified in the ``config`` file. + +- ``CONFIG_RTE_LIBRTE_MVPP2_PMD`` (default ``n``) + + Toggle compilation of the librte mvpp2 driver. + + +QoS Configuration +----------------- + +QoS configuration is done through external configuration file. Path to the +file must be given as `cfg` in driver's vdev parameter list. + +Configuration syntax +~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + [port <portnum> default] + default_tc = <default_tc> + mapping_priority = <mapping_priority> + policer_enable = <policer_enable> + token_unit = <token_unit> + color = <color_mode> + cir = <cir> + ebs = <ebs> + cbs = <cbs> + + rate_limit_enable = <rate_limit_enable> + rate_limit = <rate_limit> + burst_size = <burst_size> + + [port <portnum> tc <traffic_class>] + rxq = <rx_queue_list> + pcp = <pcp_list> + dscp = <dscp_list> + default_color = <default_color> + + [port <portnum> tc <traffic_class>] + rxq = <rx_queue_list> + pcp = <pcp_list> + dscp = <dscp_list> + + [port <portnum> txq <txqnum>] + sched_mode = <sched_mode> + wrr_weight = <wrr_weight> + + rate_limit_enable = <rate_limit_enable> + rate_limit = <rate_limit> + burst_size = <burst_size> + +Where: + +- ``<portnum>``: DPDK Port number (0..n). + +- ``<default_tc>``: Default traffic class (e.g. 0) + +- ``<mapping_priority>``: QoS priority for mapping (`ip`, `vlan`, `ip/vlan` or `vlan/ip`). + +- ``<traffic_class>``: Traffic Class to be configured. + +- ``<rx_queue_list>``: List of DPDK RX queues (e.g. 0 1 3-4) + +- ``<pcp_list>``: List of PCP values to handle in particular TC (e.g. 0 1 3-4 7). + +- ``<dscp_list>``: List of DSCP values to handle in particular TC (e.g. 0-12 32-48 63). + +- ``<policer_enable>``: Enable ingress policer. + +- ``<token_unit>``: Policer token unit (`bytes` or `packets`). + +- ``<color_mode>``: Policer color mode (`aware` or `blind`). + +- ``<cir>``: Committed information rate in unit of kilo bits per second (data rate) or packets per second. + +- ``<cbs>``: Committed burst size in unit of kilo bytes or number of packets. + +- ``<ebs>``: Excess burst size in unit of kilo bytes or number of packets. + +- ``<default_color>``: Default color for specific tc. + +- ``<rate_limit_enable>``: Enables per port or per txq rate limiting. + +- ``<rate_limit>``: Committed information rate, in kilo bits per second. + +- ``<burst_size>``: Committed burst size, in kilo bytes. + +- ``<sched_mode>``: Egress scheduler mode (`wrr` or `sp`). + +- ``<wrr_weight>``: Txq weight. + +Setting PCP/DSCP values for the default TC is not required. All PCP/DSCP +values not assigned explicitly to particular TC will be handled by the +default TC. + +Configuration file example +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: console + + [port 0 default] + default_tc = 0 + mapping_priority = ip + + rate_limit_enable = 1 + rate_limit = 1000 + burst_size = 2000 + + [port 0 tc 0] + rxq = 0 1 + + [port 0 txq 0] + sched_mode = wrr + wrr_weight = 10 + + [port 0 txq 1] + sched_mode = wrr + wrr_weight = 100 + + [port 0 txq 2] + sched_mode = sp + + [port 0 tc 1] + rxq = 2 + pcp = 5 6 7 + dscp = 26-38 + + [port 1 default] + default_tc = 0 + mapping_priority = vlan/ip + + policer_enable = 1 + token_unit = bytes + color = blind + cir = 100000 + ebs = 64 + cbs = 64 + + [port 1 tc 0] + rxq = 0 + dscp = 10 + + [port 1 tc 1] + rxq = 1 + dscp = 11-20 + + [port 1 tc 2] + rxq = 2 + dscp = 30 + + [port 1 txq 0] + rate_limit_enable = 1 + rate_limit = 10000 + burst_size = 2000 + +Usage example +^^^^^^^^^^^^^ + +.. code-block:: console + + ./testpmd --vdev=eth_mvpp2,iface=eth0,iface=eth2,cfg=/home/user/mrvl.conf \ + -c 7 -- -i -a --disable-hw-vlan-strip --rxq=3 --txq=3 + + +Building DPDK +------------- + +Driver needs precompiled MUSDK library during compilation. + +.. code-block:: console + + export CROSS_COMPILE=<toolchain>/bin/aarch64-linux-gnu- + ./bootstrap + ./configure --host=aarch64-linux-gnu --enable-bpool-dma=64 + make install + +MUSDK will be installed to `usr/local` under current directory. +For the detailed build instructions please consult ``doc/musdk_get_started.txt``. + +Before the DPDK build process the environmental variable ``LIBMUSDK_PATH`` with +the path to the MUSDK installation directory needs to be exported. + +.. code-block:: console + + export LIBMUSDK_PATH=<musdk>/usr/local + export CROSS=aarch64-linux-gnu- + make config T=arm64-armv8a-linuxapp-gcc + sed -ri 's,(MVPP2_PMD=)n,\1y,' build/.config + make + +Flow API +-------- + +PPv2 offers packet classification capabilities via classifier engine which +can be configured via generic flow API offered by DPDK. + +Supported flow actions +~~~~~~~~~~~~~~~~~~~~~~ + +Following flow action items are supported by the driver: + +* DROP +* QUEUE + +Supported flow items +~~~~~~~~~~~~~~~~~~~~ + +Following flow items and their respective fields are supported by the driver: + +* ETH + + * source MAC + * destination MAC + * ethertype + +* VLAN + + * PCP + * VID + +* IPV4 + + * DSCP + * protocol + * source address + * destination address + +* IPV6 + + * flow label + * next header + * source address + * destination address + +* UDP + + * source port + * destination port + +* TCP + + * source port + * destination port + +Classifier match engine +~~~~~~~~~~~~~~~~~~~~~~~ + +Classifier has an internal match engine which can be configured to +operate in either exact or maskable mode. + +Mode is selected upon creation of the first unique flow rule as follows: + +* maskable, if key size is up to 8 bytes. +* exact, otherwise, i.e for keys bigger than 8 bytes. + +Where the key size equals the number of bytes of all fields specified +in the flow items. + +.. table:: Examples of key size calculation + + +----------------------------------------------------------------------------+-------------------+-------------+ + | Flow pattern | Key size in bytes | Used engine | + +============================================================================+===================+=============+ + | ETH (destination MAC) / VLAN (VID) | 6 + 2 = 8 | Maskable | + +----------------------------------------------------------------------------+-------------------+-------------+ + | VLAN (VID) / IPV4 (source address) | 2 + 4 = 6 | Maskable | + +----------------------------------------------------------------------------+-------------------+-------------+ + | TCP (source port, destination port) | 2 + 2 = 4 | Maskable | + +----------------------------------------------------------------------------+-------------------+-------------+ + | VLAN (priority) / IPV4 (source address) | 1 + 4 = 5 | Maskable | + +----------------------------------------------------------------------------+-------------------+-------------+ + | IPV4 (destination address) / UDP (source port, destination port) | 6 + 2 + 2 = 10 | Exact | + +----------------------------------------------------------------------------+-------------------+-------------+ + | VLAN (VID) / IPV6 (flow label, destination address) | 2 + 3 + 16 = 21 | Exact | + +----------------------------------------------------------------------------+-------------------+-------------+ + | IPV4 (DSCP, source address, destination address) | 1 + 4 + 4 = 9 | Exact | + +----------------------------------------------------------------------------+-------------------+-------------+ + | IPV6 (flow label, source address, destination address) | 3 + 16 + 16 = 35 | Exact | + +----------------------------------------------------------------------------+-------------------+-------------+ + +From the user perspective maskable mode means that masks specified +via flow rules are respected. In case of exact match mode, masks +which do not provide exact matching (all bits masked) are ignored. + +If the flow matches more than one classifier rule the first +(with the lowest index) matched takes precedence. + +Flow rules usage example +~~~~~~~~~~~~~~~~~~~~~~~~ + +Before proceeding run testpmd user application: + +.. code-block:: console + + ./testpmd --vdev=eth_mvpp2,iface=eth0,iface=eth2 -c 3 -- -i --p 3 -a --disable-hw-vlan-strip + +Example #1 +^^^^^^^^^^ + +.. code-block:: console + + testpmd> flow create 0 ingress pattern eth src is 10:11:12:13:14:15 / end actions drop / end + +In this case key size is 6 bytes thus maskable type is selected. Testpmd +will set mask to ff:ff:ff:ff:ff:ff i.e traffic explicitly matching +above rule will be dropped. + +Example #2 +^^^^^^^^^^ + +.. code-block:: console + + testpmd> flow create 0 ingress pattern ipv4 src spec 10.10.10.0 src mask 255.255.255.0 / tcp src spec 0x10 src mask 0x10 / end action drop / end + +In this case key size is 8 bytes thus maskable type is selected. +Flows which have IPv4 source addresses ranging from 10.10.10.0 to 10.10.10.255 +and tcp source port set to 16 will be dropped. + +Example #3 +^^^^^^^^^^ + +.. code-block:: console + + testpmd> flow create 0 ingress pattern vlan vid spec 0x10 vid mask 0x10 / ipv4 src spec 10.10.1.1 src mask 255.255.0.0 dst spec 11.11.11.1 dst mask 255.255.255.0 / end actions drop / end + +In this case key size is 10 bytes thus exact type is selected. +Even though each item has partial mask set, masks will be ignored. +As a result only flows with VID set to 16 and IPv4 source and destination +addresses set to 10.10.1.1 and 11.11.11.1 respectively will be dropped. + +Limitations +~~~~~~~~~~~ + +Following limitations need to be taken into account while creating flow rules: + +* For IPv4 exact match type the key size must be up to 12 bytes. +* For IPv6 exact match type the key size must be up to 36 bytes. +* Following fields cannot be partially masked (all masks are treated as + if they were exact): + + * ETH: ethertype + * VLAN: PCP, VID + * IPv4: protocol + * IPv6: next header + * TCP/UDP: source port, destination port + +* Only one classifier table can be created thus all rules in the table + have to match table format. Table format is set during creation of + the first unique flow rule. +* Up to 5 fields can be specified per flow rule. +* Up to 20 flow rules can be added. + +For additional information about classifier please consult +``doc/musdk_cls_user_guide.txt``. + +Usage Example +------------- + +MVPP2 PMD requires extra out of tree kernel modules to function properly. +`musdk_uio` and `mv_pp_uio` sources are part of the MUSDK. Please consult +``doc/musdk_get_started.txt`` for the detailed build instructions. +For `mvpp2x_sysfs` please consult ``Documentation/pp22_sysfs.txt`` for the +detailed build instructions. + +.. code-block:: console + + insmod musdk_uio.ko + insmod mv_pp_uio.ko + insmod mvpp2x_sysfs.ko + +Additionally interfaces used by DPDK application need to be put up: + +.. code-block:: console + + ip link set eth0 up + ip link set eth2 up + +In order to run testpmd example application following command can be used: + +.. code-block:: console + + ./testpmd --vdev=eth_mvpp2,iface=eth0,iface=eth2 -c 7 -- \ + --burst=128 --txd=2048 --rxd=1024 --rxq=2 --txq=2 --nb-cores=2 \ + -i -a --rss-udp diff --git a/doc/guides/nics/netvsc.rst b/doc/guides/nics/netvsc.rst new file mode 100644 index 00000000..345f393c --- /dev/null +++ b/doc/guides/nics/netvsc.rst @@ -0,0 +1,105 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) Microsoft Corporation. + +Netvsc poll mode driver +======================= + +The Netvsc Poll Mode driver (PMD) provides support for the paravirtualized +network device for Microsoft Hyper-V. It can be used with +Window Server 2008/2012/2016, Windows 10. +The device offers multi-queue support (if kernel and host support it), +checksum and segmentation offloads. + + +Features and Limitations of Hyper-V PMD +--------------------------------------- + +In this release, the hyper PMD driver provides the basic functionality of packet reception and transmission. + +* It supports merge-able buffers per packet when receiving packets and scattered buffer per packet + when transmitting packets. The packet size supported is from 64 to 65536. + +* The PMD supports multicast packets and promiscuous mode subject to restrictions on the host. + In order to this to work, the guest network configuration on Hyper-V must be configured to allow MAC address + spoofing. + +* The device has only a single MAC address. + Hyper-V driver does not support MAC or VLAN filtering because the Hyper-V host does not support it. + +* VLAN tags are always stripped and presented in mbuf tci field. + +* The Hyper-V driver does not use or support Link State or Rx interrupt. + +* The maximum number of queues is limited by the host (currently 64). + When used with 4.16 kernel only a single queue is available. + +.. note:: + This driver is intended for use with **Hyper-V only** and is + not recommended for use on Azure because accelerated Networking + (SR-IOV) is not supported. + + On Azure, use the :doc:`vdev_netvsc` which + automatically configures the necessary TAP and failsave drivers. + + +Installation +------------ + +The Netvsc PMD is a standalone driver, similar to virtio and vmxnet3. +Using Netvsc PMD requires that the associated VMBUS device be bound to the userspace +I/O device driver for Hyper-V (uio_hv_generic). By default, all netvsc devices +will be bound to the Linux kernel driver; in order to use netvsc PMD the +device must first be overridden. + +The first step is to identify the network device to override. +VMBUS uses Universal Unique Identifiers +(`UUID`_) to identify devices on the bus similar to how PCI uses Domain:Bus:Function. +The UUID associated with a Linux kernel network device can be determined +by looking at the sysfs information. To find the UUID for eth1 and +store it in a shell variable: + + .. code-block:: console + + DEV_UUID=$(basename $(readlink /sys/class/net/eth1/device)) + + +.. _`UUID`: https://en.wikipedia.org/wiki/Universally_unique_identifier + +There are several possible ways to assign the uio device driver for a device. +The easiest way (but only on 4.18 or later) +is to use the `driverctl Device Driver control utility`_ to override +the normal kernel device. + + .. code-block:: console + + driverctl -b vmbus set-override $DEV_UUID uio_hv_generic + +.. _`driverctl Device Driver control utility`: https://gitlab.com/driverctl/driverctl + +Any settings done with driverctl are by default persistent and will be reapplied +on reboot. + +On older kernels, the same effect can be had by manual sysfs bind and unbind +operations: + + .. code-block:: console + + NET_UUID="f8615163-df3e-46c5-913f-f2d2f965ed0e" + modprobe uio_hv_generic + echo $NET_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/new_id + echo $DEV_UUID > /sys/bus/vmbus/drivers/hv_netvsc/unbind + echo $DEV_UUID > /sys/bus/vmbus/drivers/uio_hv_generic/bind + +.. Note:: + + The dpkd-devbind.py script can not be used since it only handles PCI devices. + + +Prerequisites +------------- + +The following prerequisites apply: + +* Linux kernel support for UIO on vmbus is done with the uio_hv_generic driver. + Full support of multiple queues requires the 4.17 kernel. It is possible + to use the netvsc PMD with 4.16 kernel but it is limited to a single queue. diff --git a/doc/guides/nics/nfp.rst b/doc/guides/nics/nfp.rst index 99a3b76e..927c03c6 100644 --- a/doc/guides/nics/nfp.rst +++ b/doc/guides/nics/nfp.rst @@ -34,14 +34,14 @@ NFP poll mode driver library Netronome's sixth generation of flow processors pack 216 programmable cores and over 100 hardware accelerators that uniquely combine packet, flow, security and content processing in a single device that scales -up to 400 Gbps. +up to 400-Gb/s. This document explains how to use DPDK with the Netronome Poll Mode Driver (PMD) supporting Netronome's Network Flow Processor 6xxx (NFP-6xxx) and Netronome's Flow Processor 4xxx (NFP-4xxx). NFP is a SRIOV capable device and the PMD driver supports the physical -function (PF) and virtual functions (VFs). +function (PF) and the virtual functions (VFs). Dependencies ------------ @@ -49,17 +49,18 @@ Dependencies Before using the Netronome's DPDK PMD some NFP configuration, which is not related to DPDK, is required. The system requires installation of **Netronome's BSP (Board Support Package)** along -with some specific NFP firmware application. Netronome's NSP ABI +with a specific NFP firmware application. Netronome's NSP ABI version should be 0.20 or higher. If you have a NFP device you should already have the code and -documentation for doing all this configuration. Contact +documentation for this configuration. Contact **support@netronome.com** to obtain the latest available firmware. -The NFP Linux netdev kernel driver for VFs is part of vanilla kernel -since kernel version 4.5, and support for the PF since kernel version -4.11. Support for older kernels can be obtained on Github at -**https://github.com/Netronome/nfp-drv-kmods** along with build +The NFP Linux netdev kernel driver for VFs has been a part of the +vanilla kernel since kernel version 4.5, and support for the PF +since kernel version 4.11. Support for older kernels can be obtained +on Github at +**https://github.com/Netronome/nfp-drv-kmods** along with the build instructions. NFP PMD needs to be used along with UIO ``igb_uio`` or VFIO (``vfio-pci``) @@ -70,15 +71,15 @@ Building the software Netronome's PMD code is provided in the **drivers/net/nfp** directory. Although NFP PMD has Netronome´s BSP dependencies, it is possible to -compile it along with other DPDK PMDs even if no BSP was installed before. +compile it along with other DPDK PMDs even if no BSP was installed previously. Of course, a DPDK app will require such a BSP installed for using the NFP PMD, along with a specific NFP firmware application. -Default PMD configuration is at **common_linuxapp configuration** file: +Default PMD configuration is at the **common_linuxapp configuration** file: - **CONFIG_RTE_LIBRTE_NFP_PMD=y** -Once DPDK is built all the DPDK apps and examples include support for +Once the DPDK is built all the DPDK apps and examples include support for the NFP PMD. @@ -91,37 +92,55 @@ for details. Using the PF ------------ -NFP PMD has support for using the NFP PF as another DPDK port, but it does not +NFP PMD supports using the NFP PF as another DPDK port, but it does not have any functionality for controlling VFs. In fact, it is not possible to use the PMD with the VFs if the PF is being used by DPDK, that is, with the NFP PF -bound to ``igb_uio`` or ``vfio-pci`` kernel drivers. Future DPDK version will +bound to ``igb_uio`` or ``vfio-pci`` kernel drivers. Future DPDK versions will have a PMD able to work with the PF and VFs at the same time and with the PF implementing VF management along with other PF-only functionalities/offloads. The PMD PF has extra work to do which will delay the DPDK app initialization -like checking if a firmware is already available in the device, uploading the -firmware if necessary, and configure the Link state properly when starting or -stopping a PF port. Note that firmware upload is not always necessary which is -the main delay for NFP PF PMD initialization. +like uploading the firmware and configure the Link state properly when starting or +stopping a PF port. Since DPDK 18.05 the firmware upload happens when +a PF is initialized, which was not always true with older DPDK versions. Depending on the Netronome product installed in the system, firmware files should be available under ``/lib/firmware/netronome``. DPDK PMD supporting the -PF requires a specific link, ``/lib/firmware/netronome/nic_dpdk_default.nffw``, -which should be created automatically with Netronome's Agilio products -installation. +PF looks for a firmware file in this order: + + 1) First try to find a firmware image specific for this device using the + NFP serial number: + + serial-00-15-4d-12-20-65-10-ff.nffw + + 2) Then try the PCI name: + + pci-0000:04:00.0.nffw + + 3) Finally try the card type and media: + + nic_AMDA0099-0001_2x25.nffw + +Netronome's software packages install firmware files under ``/lib/firmware/netronome`` +to support all the Netronome's SmartNICs and different firmware applications. +This is usually done using file names based on SmartNIC type and media and with a +directory per firmware application. Options 1 and 2 for firmware filenames allow +more than one SmartNIC, same type of SmartNIC or different ones, and to upload a +different firmware to each SmartNIC. + PF multiport support -------------------- Some NFP cards support several physical ports with just one single PCI device. -DPDK core is designed with the 1:1 relationship between PCI devices and DPDK +The DPDK core is designed with a 1:1 relationship between PCI devices and DPDK ports, so NFP PMD PF support requires handling the multiport case specifically. During NFP PF initialization, the PMD will extract the information about the number of PF ports from the firmware and will create as many DPDK ports as needed. Because the unusual relationship between a single PCI device and several DPDK -ports, there are some limitations when using more than one PF DPDK ports: there +ports, there are some limitations when using more than one PF DPDK port: there is no support for RX interrupts and it is not possible either to use those PF ports with the device hotplug functionality. @@ -136,7 +155,7 @@ System configuration get the drivers from the above Github repository and follow the instructions for building and installing it. - Virtual Functions need to be enabled before they can be used with the PMD. + VFs need to be enabled before they can be used with the PMD. Before enabling the VFs it is useful to obtain information about the current NFP PCI device detected by the system: diff --git a/doc/guides/nics/octeontx.rst b/doc/guides/nics/octeontx.rst index 8e2a2b75..f8eaaa63 100644 --- a/doc/guides/nics/octeontx.rst +++ b/doc/guides/nics/octeontx.rst @@ -165,8 +165,7 @@ CRC striping ~~~~~~~~~~~~ The OCTEONTX SoC family NICs strip the CRC for every packets coming into the -host interface. So, CRC will be stripped even when the -``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``. +host interface irrespective of the offload configuration. Maximum packet length ~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/nics/overview.rst b/doc/guides/nics/overview.rst index 0df0ef81..20cd52b0 100644 --- a/doc/guides/nics/overview.rst +++ b/doc/guides/nics/overview.rst @@ -1,32 +1,6 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2016 6WIND S.A. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of 6WIND S.A. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - Overview of Networking Drivers ============================== diff --git a/doc/guides/nics/pcap_ring.rst b/doc/guides/nics/pcap_ring.rst index 7fd063c9..879e5430 100644 --- a/doc/guides/nics/pcap_ring.rst +++ b/doc/guides/nics/pcap_ring.rst @@ -71,11 +71,19 @@ The different stream types are: tx_pcap=/path/to/file.pcap * rx_iface: Defines a reception stream based on a network interface name. - The driver reads packets coming from the given interface using the Linux kernel driver for that interface. + The driver reads packets from the given interface using the Linux kernel driver for that interface. + The driver captures both the incoming and outgoing packets on that interface. The value is an interface name. rx_iface=eth0 +* rx_iface_in: Defines a reception stream based on a network interface name. + The driver reads packets from the given interface using the Linux kernel driver for that interface. + The driver captures only the incoming packets on that interface. + The value is an interface name. + + rx_iface_in=eth0 + * tx_iface: Defines a transmission stream based on a network interface name. The driver sends packets to the given interface using the Linux kernel driver for that interface. The value is an interface name. @@ -122,6 +130,21 @@ Forward packets through two network interfaces: $RTE_TARGET/app/testpmd -l 0-3 -n 4 \ --vdev 'net_pcap0,iface=eth0' --vdev='net_pcap1;iface=eth1' +Enable 2 tx queues on a network interface: + +.. code-block:: console + + $RTE_TARGET/app/testpmd -l 0-3 -n 4 \ + --vdev 'net_pcap0,rx_iface=eth1,tx_iface=eth1,tx_iface=eth1' \ + -- --txq 2 + +Read only incoming packets from a network interface and write them back to the same network interface: + +.. code-block:: console + + $RTE_TARGET/app/testpmd -l 0-3 -n 4 \ + --vdev 'net_pcap0,rx_iface_in=eth1,tx_iface=eth1' + Using libpcap-based PMD with the testpmd Application ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/guides/nics/qede.rst b/doc/guides/nics/qede.rst index 63ce9b4c..cba38868 100644 --- a/doc/guides/nics/qede.rst +++ b/doc/guides/nics/qede.rst @@ -35,15 +35,15 @@ Supported Features - N-tuple filter and flow director (limited support) - NPAR (NIC Partitioning) - SR-IOV VF -- VXLAN Tunneling offload +- GRE Tunneling offload - GENEVE Tunneling offload +- VXLAN Tunneling offload - MPLSoUDP Tx Tunneling offload Non-supported Features ---------------------- - SR-IOV PF -- GRE and NVGRE Tunneling offloads Co-existence considerations --------------------------- @@ -58,19 +58,21 @@ Supported QLogic Adapters Prerequisites ------------- -- Requires storm firmware version **8.30.12.0**. Firmware may be available +- Requires storm firmware version **8.33.12.0**. Firmware may be available inbox in certain newer Linux distros under the standard directory - ``E.g. /lib/firmware/qed/qed_init_values-8.30.12.0.bin`` + ``E.g. /lib/firmware/qed/qed_init_values-8.33.12.0.bin``. If the required firmware files are not available then download it from - `QLogic Driver Download Center <http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/DefaultNewSearch.aspx>`_. - For downloading firmware file, select adapter category, model and DPDK Poll Mode Driver. - -- Requires management firmware (MFW) version **8.30.x.x** or higher to be - flashed on to the adapter. If the required management firmware is not - available then download from - `QLogic Driver Download Center <http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/DefaultNewSearch.aspx>`_. - For downloading firmware upgrade utility, select adapter category, model and Linux distro. - To flash the management firmware refer to the instructions in the QLogic Firmware Upgrade Utility Readme document. + `linux-firmware git repository <http://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/qed>`_ + or `QLogic Driver Download Center <http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/DefaultNewSearch.aspx>`_. + To download firmware file from QLogic website, select adapter category, model and DPDK Poll Mode Driver. + +- Requires the NIC be updated minimally with **8.30.x.x** Management firmware(MFW) version supported for that NIC. + It is highly recommended that the NIC be updated with the latest available management firmware version to get latest feature set. + Management Firmware and Firmware Upgrade Utility for Cavium FastLinQ(r) branded adapters can be downloaded from + `Driver Download Center <http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/DefaultNewSearch.aspx>`_. + For downloading Firmware Upgrade Utility, select NIC category, model and Linux distro. + To update the management firmware, refer to the instructions in the Firmware Upgrade Utility Readme document. + For OEM branded adapters please follow the instruction provided by the OEM to update the Management Firmware on the NIC. - SR-IOV requires Linux PF driver version **8.20.x.x** or higher. If the required PF driver is not available then download it from @@ -104,7 +106,7 @@ enabling debugging options may affect system performance. - ``CONFIG_RTE_LIBRTE_QEDE_FW`` (default **""**) Gives absolute path of firmware file. - ``Eg: "/lib/firmware/qed/qed_init_values-8.30.12.0.bin"`` + ``Eg: "/lib/firmware/qed/qed_init_values-8.33.12.0.bin"`` Empty string indicates driver will pick up the firmware file from the default location /lib/firmware/qed. CAUTION this option is more for custom firmware, it is not @@ -121,7 +123,7 @@ SR-IOV: Prerequisites and Sample Application Notes This section provides instructions to configure SR-IOV with Linux OS. -**Note**: librte_pmd_qede will be used to bind to SR-IOV VF device and Linux native kernel driver (qede) will function as SR-IOV PF driver. Requires PF driver to be 8.10.x.x or higher. +**Note**: librte_pmd_qede will be used to bind to SR-IOV VF device and Linux native kernel driver (qede) will function as SR-IOV PF driver. Requires PF driver to be 8.20.x.x or higher. #. Verify SR-IOV and ARI capability is enabled on the adapter using ``lspci``: @@ -193,7 +195,7 @@ This section provides instructions to configure SR-IOV with Linux OS. #. Running testpmd - (Supply ``--log-level="pmd.net.qede.driver",7`` to view informational messages): + (Supply ``--log-level="pmd.net.qede.driver:info`` to view informational messages): Refer to the document :ref:`compiling and testing a PMD for a NIC <pmd_build_and_test>` to run diff --git a/doc/guides/nics/sfc_efx.rst b/doc/guides/nics/sfc_efx.rst index ccdf5ff0..63939ec8 100644 --- a/doc/guides/nics/sfc_efx.rst +++ b/doc/guides/nics/sfc_efx.rst @@ -30,7 +30,8 @@ Solarflare libefx-based Poll Mode Driver ======================================== The SFC EFX PMD (**librte_pmd_sfc_efx**) provides poll mode driver support -for **Solarflare SFN7xxx and SFN8xxx** family of 10/40 Gbps adapters. +for **Solarflare SFN7xxx and SFN8xxx** family of 10/40 Gbps adapters and +**Solarflare XtremeScale X2xxx** family of 10/25/40/50/100 Gbps adapters. SFC EFX PMD has support for the latest Linux and FreeBSD operating systems. More information can be found at `Solarflare Communications website @@ -87,6 +88,8 @@ SFC EFX PMD has support for: - Flow API +- Loopback + Non-supported Features ---------------------- @@ -97,8 +100,6 @@ The features not yet supported include: - Priority-based flow control -- Loopback - - Configurable RX CRC stripping (always stripped) - Header split on receive @@ -120,22 +121,37 @@ required in the receive buffer. It should be taken into account when mbuf pool for receive is created. +Equal stride super-buffer mode +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When the receive queue uses equal stride super-buffer DMA mode, one HW Rx +descriptor carries many Rx buffers which contiguously follow each other +with some stride (equal to total size of rte_mbuf as mempool object). +Each Rx buffer is an independent rte_mbuf. +However dedicated mempool manager must be used when mempool for the Rx +queue is created. The manager must support dequeue of the contiguous +block of objects and provide mempool info API to get the block size. + +Another limitation of a equal stride super-buffer mode, imposed by the +firmware, is that it allows for a single RSS context. + + Tunnels support --------------- -NVGRE, VXLAN and GENEVE tunnels are supported on SFN8xxx family adapters -with full-feature firmware variant running. +NVGRE, VXLAN and GENEVE tunnels are supported on SFN8xxx and X2xxx family +adapters with full-feature firmware variant running. **sfboot** should be used to configure NIC to run full-feature firmware variant. See Solarflare Server Adapter User's Guide for details. -SFN8xxx family adapters provide either inner or outer packet classes. +SFN8xxx and X2xxx family adapters provide either inner or outer packet classes. If adapter firmware advertises support for tunnels then the PMD configures the hardware to report inner classes, and outer classes are not reported in received packets. However, for VXLAN and GENEVE tunnels the PMD does report UDP as the outer layer 4 packet type. -SFN8xxx family adapters report GENEVE packets as VXLAN. +SFN8xxx and X2xxx family adapters report GENEVE packets as VXLAN. If UDP ports are configured for only one tunnel type then it is safe to treat VXLAN packet type indication as the corresponding UDP tunnel type. @@ -152,7 +168,9 @@ Supported pattern items: - VOID - ETH (exact match of source/destination addresses, individual/group match - of destination address, EtherType) + of destination address, EtherType in the outer frame and exact match of + destination addresses, individual/group match of destination address in + the inner frame) - VLAN (exact match of VID, double-tagging is supported) @@ -166,6 +184,13 @@ Supported pattern items: - UDP (exact match of source/destination ports) +- VXLAN (exact match of VXLAN network identifier) + +- GENEVE (exact match of virtual network identifier, only Ethernet (0x6558) + protocol type is supported) + +- NVGRE (exact match of virtual subnet ID) + Supported actions: - VOID @@ -174,6 +199,12 @@ Supported actions: - RSS +- DROP + +- FLAG (supported only with ef10_essb Rx datapath) + +- MARK (supported only with ef10_essb Rx datapath) + Validating flow rules depends on the firmware variant. Ethernet destinaton individual/group match @@ -184,10 +215,31 @@ in the mask of destination address. If destinaton address in the spec is multicast, it matches all multicast (and broadcast) packets, oherwise it matches unicast packets that are not filtered by other flow rules. +Exceptions to flow rules +~~~~~~~~~~~~~~~~~~~~~~~~ + +There is a list of exceptional flow rule patterns which will not be +accepted by the PMD. A pattern will be rejected if at least one of the +conditions is met: + +- Filtering by IPv4 or IPv6 EtherType without pattern items of internet + layer and above. + +- The last item is IPV4 or IPV6, and it's empty. + +- Filtering by TCP or UDP IP transport protocol without pattern items of + transport layer and above. + +- The last item is TCP or UDP, and it's empty. + Supported NICs -------------- +- Solarflare XtremeScale Adapters: + + - Solarflare X2522 Dual Port SFP28 10/25GbE Adapter + - Solarflare Flareon [Ultra] Server Adapters: - Solarflare SFN8522 Dual Port SFP+ Server Adapter @@ -258,15 +310,18 @@ whitelist option like "-w 02:00.0,arg1=value1,...". Case-insensitive 1/y/yes/on or 0/n/no/off may be used to specify boolean parameters value. -- ``rx_datapath`` [auto|efx|ef10] (default **auto**) +- ``rx_datapath`` [auto|efx|ef10|ef10_esps] (default **auto**) Choose receive datapath implementation. **auto** allows the driver itself to make a choice based on firmware features available and required by the datapath implementation. **efx** chooses libefx-based datapath which supports Rx scatter. - **ef10** chooses EF10 (SFN7xxx, SFN8xxx) native datapath which is + **ef10** chooses EF10 (SFN7xxx, SFN8xxx, X2xxx) native datapath which is more efficient than libefx-based and provides richer packet type classification, but lacks Rx scatter support. + **ef10_esps** chooses SFNX2xxx equal stride packed stream datapath + which may be used on DPDK firmware variant only + (see notes about its limitations above). - ``tx_datapath`` [auto|efx|ef10|ef10_simple] (default **auto**) @@ -277,12 +332,12 @@ boolean parameters value. (full-feature firmware variant only), TSO and multi-segment mbufs. Mbuf segments may come from different mempools, and mbuf reference counters are treated responsibly. - **ef10** chooses EF10 (SFN7xxx, SFN8xxx) native datapath which is + **ef10** chooses EF10 (SFN7xxx, SFN8xxx, X2xxx) native datapath which is more efficient than libefx-based but has no VLAN insertion and TSO support yet. Mbuf segments may come from different mempools, and mbuf reference counters are treated responsibly. - **ef10_simple** chooses EF10 (SFN7xxx, SFN8xxx) native datapath which + **ef10_simple** chooses EF10 (SFN7xxx, SFN8xxx, X2xxx) native datapath which is even more faster then **ef10** but does not support multi-segment mbufs, disallows multiple mempools and neglects mbuf reference counters. @@ -293,21 +348,73 @@ boolean parameters value. **auto** allows NIC firmware to make a choice based on installed licences and firmware variant configured using **sfboot**. -- ``debug_init`` [bool] (default **n**) - - Enable extra logging during device initialization and startup. - -- ``mcdi_logging`` [bool] (default **n**) - - Enable extra logging of the communication with the NIC's management CPU. - The logging is done using RTE_LOG() with INFO level and PMD type. - The format is consumed by the Solarflare netlogdecode cross-platform tool. - - ``stats_update_period_ms`` [long] (default **1000**) Adjust period in milliseconds to update port hardware statistics. The accepted range is 0 to 65535. The value of **0** may be used to disable periodic statistics update. One should note that it's - only possible to set an arbitrary value on SFN8xxx provided that + only possible to set an arbitrary value on SFN8xxx and X2xxx provided that firmware version is 6.2.1.1033 or higher, otherwise any positive value will select a fixed update period of **1000** milliseconds + +- ``fw_variant`` [dont-care|full-feature|ultra-low-latency| + capture-packed-stream|dpdk] (default **dont-care**) + + Choose the preferred firmware variant to use. In order for the selected + option to have an effect, the **sfboot** utility must be configured with the + **auto** firmware-variant option. The preferred firmware variant applies to + all ports on the NIC. + **dont-care** ensures that the driver can attach to an unprivileged function. + The datapath firmware type to use is controlled by the **sfboot** + utility. + **full-feature** chooses full featured firmware. + **ultra-low-latency** chooses firmware with fewer features but lower latency. + **capture-packed-stream** chooses firmware for SolarCapture packed stream + mode. + **dpdk** chooses DPDK firmware with equal stride super-buffer Rx mode + for higher Rx packet rate and packet marks support and firmware subvariant + without checksumming on transmit for higher Tx packet rate if + checksumming is not required. + +- ``rxd_wait_timeout_ns`` [long] (default **200 us**) + + Adjust timeout in nanoseconds to head-of-line block to wait for + Rx descriptors. + The accepted range is 0 to 400 ms. + Flow control should be enabled to make it work. + The value of **0** disables it and packets are dropped immediately. + When a packet is dropped because of no Rx descriptors, + ``rx_nodesc_drop_cnt`` counter grows. + The feature is supported only by the DPDK firmware variant when equal + stride super-buffer Rx mode is used. + + +Dynamic Logging Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One may leverage EAL option "--log-level" to change default levels +for the log types supported by the driver. The option is used with +an argument typically consisting of two parts separated by a colon. + +Level value is the last part which takes a symbolic name (or integer). +Log type is the former part which may shell match syntax. +Depending on the choice of the expression, the given log level may +be used either for some specific log type or for a subset of types. + +SFC EFX PMD provides the following log types available for control: + +- ``pmd.net.sfc.driver`` (default level is **notice**) + + Affects driver-wide messages unrelated to any particular devices. + +- ``pmd.net.sfc.main`` (default level is **notice**) + + Matches a subset of per-port log types registered during runtime. + A full name for a particular type may be obtained by appending a + dot and a PCI device identifier (``XXXX:XX:XX.X``) to the prefix. + +- ``pmd.net.sfc.mcdi`` (default level is **notice**) + + Extra logging of the communication with the NIC's management CPU. + The format of the log is consumed by the Solarflare netlogdecode + cross-platform tool. May be managed per-port, as explained above. diff --git a/doc/guides/nics/softnic.rst b/doc/guides/nics/softnic.rst new file mode 100644 index 00000000..6c2287a1 --- /dev/null +++ b/doc/guides/nics/softnic.rst @@ -0,0 +1,250 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Soft NIC Poll Mode Driver +========================= + +The Soft NIC allows building custom NIC pipelines in software. The Soft NIC pipeline +is DIY and reconfigurable through ``firmware`` (DPDK Packet Framework script). + +The Soft NIC leverages the DPDK Packet Framework libraries (librte_port, +librte_table and librte_pipeline) to make it modular, flexible and extensible +with new functionality. Please refer to DPDK Programmer's Guide, Chapter +``Packet Framework`` and DPDK Sample Application User Guide, +Chapter ``IP Pipeline Application`` for more details. + +The Soft NIC is configured through the standard DPDK ethdev API (ethdev, flow, +QoS, security). The internal framework is not externally visible. + +Key benefits: + - Can be used to augment missing features to HW NICs. + - Allows consumption of advanced DPDK features without application redesign. + - Allows out-of-the-box performance boost of DPDK consumers applications simply by + instantiating this type of Ethernet device. + +Flow +---- +* ``Device creation``: Each Soft NIC instance is a virtual device. + +* ``Device start``: The Soft NIC firmware script is executed every time the device + is started. The firmware script typically creates several internal objects, + such as: memory pools, SW queues, traffic manager, action profiles, pipelines, + etc. + +* ``Device stop``: All the internal objects that were previously created by the + firmware script during device start are now destroyed. + +* ``Device run``: Each Soft NIC device needs one or several CPU cores to run. + The firmware script maps each internal pipeline to a CPU core. Multiple + pipelines can be mapped to the same CPU core. In order for a given pipeline + assigned to CPU core X to run, the application needs to periodically call on + CPU core X the `rte_pmd_softnic_run()` function for the current Soft NIC + device. + +* ``Application run``: The application reads packets from the Soft NIC device RX + queues and writes packets to the Soft NIC device TX queues. + +Supported Operating Systems +--------------------------- + +Any Linux distribution fulfilling the conditions described in ``System Requirements`` +section of :ref:`the DPDK documentation <linux_gsg>` or refer to *DPDK +Release Notes*. + +Build options +------------- + +The default PMD configuration available in the common_linuxapp configuration file: + +CONFIG_RTE_LIBRTE_PMD_SOFTNIC=y + +Once the DPDK is built, all the DPDK applications include support for the +Soft NIC PMD. + +Soft NIC PMD arguments +---------------------- + +The user can specify below arguments in EAL ``--vdev`` options to create the +Soft NIC device instance: + + --vdev "net_softnic0,firmware=firmware.cli,conn_port=8086" + +#. ``firmware``: path to the firmware script used for Soft NIC configuration. + The example "firmware" script is provided at `drivers/net/softnic/`. + (Optional: No, Default = NA) + +#. ``conn_port``: tcp connection port (non-zero value) used by remote client + (for examples- telnet, netcat, etc.) to connect and configure Soft NIC device in run-time. + (Optional: yes, Default value: 0, no connection with external client) + +#. ``cpu_id``: numa node id. (Optional: yes, Default value: 0) + +#. ``tm_n_queues``: number of traffic manager's scheduler queues. The traffic manager + is based on DPDK *librte_sched* library. (Optional: yes, Default value: 65,536 queues) + +#. ``tm_qsize0``: size of scheduler queue 0 per traffic class of the pipes/subscribers. + (Optional: yes, Default: 64) + +#. ``tm_qsize1``: size of scheduler queue 1 per traffic class of the pipes/subscribers. + (Optional: yes, Default: 64) + +#. ``tm_qsize2``: size of scheduler queue 2 per traffic class of the pipes/subscribers. + (Optional: yes, Default: 64) + +#. ``tm_qsize3``: size of scheduler queue 3 per traffic class of the pipes/subscribers. + (Optional: yes, Default: 64) + + +Soft NIC testing +---------------- + +* Run testpmd application in Soft NIC forwarding mode with loopback feature + enabled on Soft NIC port: + + .. code-block:: console + + ./testpmd -c 0x3 --vdev 'net_softnic0,firmware=<script path>/firmware.cli,cpu_id=0,conn_port=8086' -- -i + --forward-mode=softnic --portmask=0x2 + + .. code-block:: console + + ... + Interactive-mode selected + Set softnic packet forwarding mode + ... + Configuring Port 0 (socket 0) + Port 0: 90:E2:BA:37:9D:DC + Configuring Port 1 (socket 0) + + ; SPDX-License-Identifier: BSD-3-Clause + ; Copyright(c) 2018 Intel Corporation + + link LINK dev 0000:02:00.0 + + pipeline RX period 10 offset_port_id 0 + pipeline RX port in bsz 32 link LINK rxq 0 + pipeline RX port out bsz 32 swq RXQ0 + pipeline RX table match stub + pipeline RX port in 0 table 0 + + pipeline TX period 10 offset_port_id 0 + pipeline TX port in bsz 32 swq TXQ0 + pipeline TX port out bsz 32 link LINK txq 0 + pipeline TX table match stub + pipeline TX port in 0 table 0 + + thread 1 pipeline RX enable + thread 1 pipeline TX enable + Port 1: 00:00:00:00:00:00 + Checking link statuses... + Done + testpmd> + +* Start forwarding + + .. code-block:: console + + testpmd> start + softnic packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP over anonymous pages disabled + Logical Core 1 (socket 0) forwards packets on 1 streams: + RX P=2/Q=0 (socket 0) -> TX P=2/Q=0 (socket 0) peer=02:00:00:00:00:02 + + softnic packet forwarding packets/burst=32 + nb forwarding cores=1 - nb forwarding ports=1 + port 0: RX queue number: 1 Tx queue number: 1 + Rx offloads=0x1000 Tx offloads=0x0 + RX queue: 0 + RX desc=512 - RX free threshold=32 + RX threshold registers: pthresh=8 hthresh=8 wthresh=0 + RX Offloads=0x0 + TX queue: 0 + TX desc=512 - TX free threshold=32 + TX threshold registers: pthresh=32 hthresh=0 wthresh=0 + TX offloads=0x0 - TX RS bit threshold=32 + port 1: RX queue number: 1 Tx queue number: 1 + Rx offloads=0x0 Tx offloads=0x0 + RX queue: 0 + RX desc=0 - RX free threshold=0 + RX threshold registers: pthresh=0 hthresh=0 wthresh=0 + RX Offloads=0x0 + TX queue: 0 + TX desc=0 - TX free threshold=0 + TX threshold registers: pthresh=0 hthresh=0 wthresh=0 + TX offloads=0x0 - TX RS bit threshold=0 + +* Start remote client (e.g. telnet) to communicate with the softnic device: + + .. code-block:: console + + $ telnet 127.0.0.1 8086 + Trying 127.0.0.1... + Connected to 127.0.0.1. + Escape character is '^]'. + + Welcome to Soft NIC! + + softnic> + +* Add/update Soft NIC pipeline table match-action entries from telnet client: + + .. code-block:: console + + softnic> pipeline RX table 0 rule add match default action fwd port 0 + softnic> pipeline TX table 0 rule add match default action fwd port 0 + +Soft NIC Firmware +----------------- + +The Soft NIC firmware, for example- `softnic/firmware.cli`, consists of following CLI commands +for creating and managing software based NIC pipelines. For more details, please refer to CLI +command description provided in `softnic/rte_eth_softnic_cli.c`. + +* Physical port for packets send/receive: + + .. code-block:: console + + link LINK dev 0000:02:00.0 + +* Pipeline create: + + .. code-block:: console + + pipeline RX period 10 offset_port_id 0 (Soft NIC rx-path pipeline) + pipeline TX period 10 offset_port_id 0 (Soft NIC tx-path pipeline) + +* Pipeline input/output port create + + .. code-block:: console + + pipeline RX port in bsz 32 link LINK rxq 0 (Soft NIC rx pipeline input port) + pipeline RX port out bsz 32 swq RXQ0 (Soft NIC rx pipeline output port) + pipeline TX port in bsz 32 swq TXQ0 (Soft NIC tx pipeline input port) + pipeline TX port out bsz 32 link LINK txq 0 (Soft NIC tx pipeline output port) + +* Pipeline table create + + .. code-block:: console + + pipeline RX table match stub (Soft NIC rx pipeline match-action table) + pipeline TX table match stub (Soft NIC tx pipeline match-action table) + +* Pipeline input port connection with table + + .. code-block:: console + + pipeline RX port in 0 table 0 (Soft NIC rx pipeline input port 0 connection with table 0) + pipeline TX port in 0 table 0 (Soft NIC tx pipeline input port 0 connection with table 0) + +* Pipeline table match-action rules add + + .. code-block:: console + + pipeline RX table 0 rule add match default action fwd port 0 (Soft NIC rx pipeline table 0 rule) + pipeline TX table 0 rule add match default action fwd port 0 (Soft NIC tx pipeline table 0 rule) + +* Enable pipeline on CPU thread + + .. code-block:: console + + thread 1 pipeline RX enable (Soft NIC rx pipeline enable on cpu thread id 1) + thread 1 pipeline TX enable (Soft NIC tx pipeline enable on cpu thread id 1) diff --git a/doc/guides/nics/szedata2.rst b/doc/guides/nics/szedata2.rst index 1a5d4138..a2092f9c 100644 --- a/doc/guides/nics/szedata2.rst +++ b/doc/guides/nics/szedata2.rst @@ -1,32 +1,5 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2015 - 2016 CESNET - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of CESNET nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. SZEDATA2 poll mode driver library ================================= @@ -70,8 +43,10 @@ separately: * **Kernel modules** + * combo6core * combov3 - * szedata2_cv3 + * szedata2 + * szedata2_cv3 or szedata2_cv3_fdt Kernel modules manage initialization of hardware, allocation and sharing of resources for user space applications. @@ -79,6 +54,15 @@ separately: Information about getting the dependencies can be found `here <http://www.netcope.com/en/company/community-support/dpdk-libsze2>`_. +Versions of the packages +~~~~~~~~~~~~~~~~~~~~~~~~ + +The minimum version of the provided packages: + +* for DPDK from 18.05: **4.4.1** + +* for DPDK up to 18.02 (including): **3.0.5** + Configuration ------------- @@ -89,45 +73,53 @@ These configuration options can be modified before compilation in the Value **y** enables compilation of szedata2 PMD. -* ``CONFIG_RTE_LIBRTE_PMD_SZEDATA2_AS`` default value: **0** - - This option defines type of firmware address space and must be set - according to the used card and mode. - Currently supported values are: - - * **0** - for cards (modes): - - * NFB-100G1 (100G1) +Using the SZEDATA2 PMD +---------------------- - * **1** - for cards (modes): +From DPDK version 16.04 the type of SZEDATA2 PMD is changed to PMD_PDEV. +SZEDATA2 device is automatically recognized during EAL initialization. +No special command line options are needed. - * NFB-100G2Q (100G1) +Kernel modules have to be loaded before running the DPDK application. - * **2** - for cards (modes): +NFB card architecture +--------------------- - * NFB-40G2 (40G2) - * NFB-100G2C (100G2) - * NFB-100G2Q (40G2) +The NFB cards are multi-port multi-queue cards, where (generally) data from any +Ethernet port may be sent to any queue. +They were historically represented in DPDK as a single port. - * **3** - for cards (modes): +However, the new NFB-200G2QL card employs an addon cable which allows to connect +it to two physical PCI-E slots at the same time (see the diagram below). +This is done to allow 200 Gbps of traffic to be transferred through the PCI-E +bus (note that a single PCI-E 3.0 x16 slot provides only 125 Gbps theoretical +throughput). - * NFB-40G2 (10G8) - * NFB-100G2Q (10G8) +Since each slot may be connected to a different CPU and therefore to a different +NUMA node, the card is represented as two ports in DPDK (each with half of the +queues), which allows DPDK to work with data from the individual queues on the +right NUMA node. - * **4** - for cards (modes): +.. figure:: img/szedata2_nfb200g_architecture.* + :align: center - * NFB-100G1 (10G10) + NFB-200G2QL high-level diagram - * **5** - for experimental firmwares and future use +Limitations +----------- -Using the SZEDATA2 PMD ----------------------- +The SZEDATA2 PMD does not support operations related to Ethernet ports +(link_up, link_down, set_mac_address, etc.). -From DPDK version 16.04 the type of SZEDATA2 PMD is changed to PMD_PDEV. -SZEDATA2 device is automatically recognized during EAL initialization. -No special command line options are needed. +NFB cards employ multiple Ethernet ports. +Until now, Ethernet port-related operations were performed on all of them +(since the whole card was represented as a single port). +With NFB-200G2QL card, this is no longer viable (see above). -Kernel modules have to be loaded before running the DPDK application. +Since there is no fixed mapping between the queues and Ethernet ports, and since +a single card can be represented as two ports in DPDK, there is no way of +telling which (if any) physical ports should be associated with individual +ports in DPDK. Example of usage ---------------- diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst index ea61be38..27148681 100644 --- a/doc/guides/nics/tap.rst +++ b/doc/guides/nics/tap.rst @@ -1,8 +1,8 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2016 Intel Corporation. -Tap Poll Mode Driver -==================== +Tun|Tap Poll Mode Driver +======================== The ``rte_eth_tap.c`` PMD creates a device using TAP interfaces on the local host. The PMD allows for DPDK and the host to communicate using a raw @@ -37,6 +37,12 @@ for each interface string containing ``mac=fixed``. The MAC address is formatted as 00:'d':'t':'a':'p':[00-FF]. Convert the characters to hex and you get the actual MAC address: ``00:64:74:61:70:[00-FF]``. + --vdev=net_tap0,mac="00:64:74:61:70:11" + +The MAC address will have a user value passed as string. The MAC address is in +format with delimeter ``:``. The string is byte converted to hex and you get +the actual MAC address: ``00:64:74:61:70:11``. + It is possible to specify a remote netdevice to capture packets from by adding ``remote=foo1``, for example:: @@ -77,6 +83,17 @@ can utilize that stack to handle the network protocols. Plus you would be able to address the interface using an IP address assigned to the internal interface. +The TUN PMD allows user to create a TUN device on host. The PMD allows user +to transmit and receive packets via DPDK API calls with L3 header and payload. +The devices in host can be accessed via ``ifconfig`` or ``ip`` command. TUN +interfaces are passed to DPDK ``rte_eal_init`` arguments as ``--vdev=net_tunX``, +where X stands for unique id, example:: + + --vdev=net_tun0 --vdev=net_tun1,iface=foo1, ... + +Unlike TAP PMD, TUN PMD does not support user arguments as ``MAC`` or ``remote`` user +options. Default interface name is ``dtunX``, where X stands for unique id. + Flow API support ---------------- @@ -91,7 +108,7 @@ The kernel support can be checked with this command:: Supported items: - eth: src and dst (with variable masks), and eth_type (0xffff mask). -- vlan: vid, pcp, tpid, but not eid. (requires kernel 4.9) +- vlan: vid, pcp, but not eid. (requires kernel 4.9) - ipv4/6: src and dst (with variable masks), and ip_proto (0xffff mask). - udp/tcp: src and dst port (0xffff) mask. @@ -149,7 +166,7 @@ Run pktgen from the pktgen directory in a terminal with a commandline like the following:: sudo ./app/app/x86_64-native-linuxapp-gcc/app/pktgen -l 1-5 -n 4 \ - --proc-type auto --log-level 8 --socket-mem 512,512 --file-prefix pg \ + --proc-type auto --log-level debug --socket-mem 512,512 --file-prefix pg \ --vdev=net_tap0 --vdev=net_tap1 -b 05:00.0 -b 05:00.1 \ -b 04:00.0 -b 04:00.1 -b 04:00.2 -b 04:00.3 \ -b 81:00.0 -b 81:00.1 -b 81:00.2 -b 81:00.3 \ @@ -241,6 +258,11 @@ Please refer to ``iproute2`` package file ``lib/bpf.c`` function An example utility for eBPF instruction generation in the format of C arrays will be added in next releases +TAP reports on supported RSS functions as part of dev_infos_get callback: +``ETH_RSS_IP``, ``ETH_RSS_UDP`` and ``ETH_RSS_TCP``. +**Known limitation:** TAP supports all of the above hash functions together +and not in partial combinations. + Systems supporting flow API --------------------------- diff --git a/doc/guides/nics/thunderx.rst b/doc/guides/nics/thunderx.rst index 5270ef23..e84eaafe 100644 --- a/doc/guides/nics/thunderx.rst +++ b/doc/guides/nics/thunderx.rst @@ -30,6 +30,7 @@ Features of the ThunderX PMD are: - SR-IOV VF - NUMA support - Multi queue set support (up to 96 queues (12 queue sets)) per port +- Skip data bytes Supported ThunderX SoCs ----------------------- @@ -312,6 +313,21 @@ We will choose four secondary queue sets from the ending of the list (0002:01:01 The nicvf thunderx driver will make use of attached secondary VFs automatically during the interface configuration stage. + +Module params +-------------- + +skip_data_bytes +~~~~~~~~~~~~~~~ +This feature is used to create a hole between HEADROOM and actual data. Size of hole is specified +in bytes as module param("skip_data_bytes") to pmd. +This scheme is useful when application would like to insert vlan header without disturbing HEADROOM. + +Example: + .. code-block:: console + + -w 0002:01:00.2,skip_data_bytes=8 + Limitations ----------- @@ -319,8 +335,7 @@ CRC striping ~~~~~~~~~~~~ The ThunderX SoC family NICs strip the CRC for every packets coming into the -host interface. So, CRC will be stripped even when the -``rxmode.hw_strip_crc`` member is set to 0 in ``struct rte_eth_conf``. +host interface irrespective of the offload configuration. Maximum packet length ~~~~~~~~~~~~~~~~~~~~~ @@ -336,3 +351,8 @@ Maximum packet segments The ThunderX SoC family NICs support up to 12 segments per packet when working in scatter/gather mode. So, setting MTU will result with ``EINVAL`` when the frame size does not fit in the maximum number of segments. + +skip_data_bytes +~~~~~~~~~~~~~~~ + +Maximum limit of skip_data_bytes is 128 bytes and number of bytes should be multiple of 8. diff --git a/doc/guides/nics/vdev_netvsc.rst b/doc/guides/nics/vdev_netvsc.rst index 55d130a3..d1da0711 100644 --- a/doc/guides/nics/vdev_netvsc.rst +++ b/doc/guides/nics/vdev_netvsc.rst @@ -1,6 +1,6 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright 2017 6WIND S.A. - Copyright 2017 Mellanox Technologies, Ltd. + Copyright 2017 Mellanox Technologies, Ltd VDEV_NETVSC driver ================== @@ -89,12 +89,16 @@ The following device parameters are supported: - ``force`` [int] If nonzero, forces the use of specified interfaces even if not detected as - NetVSC or detected as routed NETVSC. + NetVSC. - ``ignore`` [int] - If nonzero, ignores the driver runnig (actually used to disable the + If nonzero, ignores the driver running (actually used to disable the auto-detection in Hyper-V VM). -Not specifying either ``iface`` or ``mac`` makes this driver attach itself to -all unrouted NetVSC interfaces found on the system. +.. note:: + + Not specifying either ``iface`` or ``mac`` makes this driver attach itself to + all unrouted NetVSC interfaces found on the system. + Specifying the device makes this driver attach itself to the device + regardless the device routes. diff --git a/doc/guides/nics/virtio.rst b/doc/guides/nics/virtio.rst index ca09cd20..7c099fb7 100644 --- a/doc/guides/nics/virtio.rst +++ b/doc/guides/nics/virtio.rst @@ -201,7 +201,7 @@ The packet transmission flow is: Virtio PMD Rx/Tx Callbacks -------------------------- -Virtio driver has 3 Rx callbacks and 2 Tx callbacks. +Virtio driver has 4 Rx callbacks and 3 Tx callbacks. Rx callbacks: @@ -215,6 +215,9 @@ Rx callbacks: Vector version without mergeable Rx buffer support, also fixes the available ring indexes and uses vector instructions to optimize performance. +#. ``virtio_recv_mergeable_pkts_inorder``: + In-order version with mergeable Rx buffer support. + Tx callbacks: #. ``virtio_xmit_pkts``: @@ -223,6 +226,8 @@ Tx callbacks: #. ``virtio_xmit_pkts_simple``: Vector version fixes the available ring indexes to optimize performance. +#. ``virtio_xmit_pkts_inorder``: + In-order version. By default, the non-vector callbacks are used: @@ -234,7 +239,7 @@ By default, the non-vector callbacks are used: Vector callbacks will be used when: -* ``txq_flags`` is set to ``VIRTIO_SIMPLE_FLAGS`` (0xF01), which implies: +* ``txmode.offloads`` is set to ``0x0``, which implies: * Single segment is specified. @@ -252,8 +257,14 @@ The corresponding callbacks are: Example of using the vector version of the virtio poll mode driver in ``testpmd``:: - testpmd -l 0-2 -n 4 -- -i --txqflags=0xF01 --rxq=1 --txq=1 --nb-cores=1 + testpmd -l 0-2 -n 4 -- -i --tx-offloads=0x0 --rxq=1 --txq=1 --nb-cores=1 + +In-order callbacks only work on simulated virtio user vdev. + +* For Rx: If mergeable Rx buffers is enabled and in-order is enabled then + ``virtio_xmit_pkts_inorder`` is used. +* For Tx: If in-order is enabled then ``virtio_xmit_pkts_inorder`` is used. Interrupt mode -------------- @@ -318,3 +329,26 @@ Here we use l3fwd-power as an example to show how to get started. $ l3fwd-power -l 0-1 -- -p 1 -P --config="(0,0,1)" \ --no-numa --parse-ptype + + +Virtio PMD arguments +-------------------- + +The user can specify below argument in devargs. + +#. ``vdpa``: + + A virtio device could also be driven by vDPA (vhost data path acceleration) + driver, and works as a HW vhost backend. This argument is used to specify + a virtio device needs to work in vDPA mode. + (Default: 0 (disabled)) + +#. ``mrg_rxbuf``: + + It is used to enable virtio device mergeable Rx buffer feature. + (Default: 1 (enabled)) + +#. ``in_order``: + + It is used to enable virtio device in-order feature. + (Default: 1 (enabled)) diff --git a/doc/guides/platform/octeontx.rst b/doc/guides/platform/octeontx.rst index dc1aa4fe..b0a99c38 100644 --- a/doc/guides/platform/octeontx.rst +++ b/doc/guides/platform/octeontx.rst @@ -16,11 +16,11 @@ Common Offload HW Block Drivers ------------------------------- 1. **Eventdev Driver** - See :doc: `../eventdevs/octeontx.rst` for octeontx ssovf eventdev driver + See :doc:`../eventdevs/octeontx` for octeontx ssovf eventdev driver information. 2. **Mempool Driver** - See :doc: `../mempool/octeontx.rst` for octeontx fpavf mempool driver + See :doc:`../mempool/octeontx` for octeontx fpavf mempool driver information. Steps To Setup Platform @@ -52,4 +52,4 @@ OCTEONTX compatible board: SDK and related information can be obtained from: `Cavium support site <https://support.cavium.com/>`_. -- Follow the DPDK :doc: `../linux_gsg/index.rst` to setup the basic DPDK environment. +- Follow the DPDK :doc:`../linux_gsg/index` to setup the basic DPDK environment. diff --git a/doc/guides/prog_guide/bbdev.rst b/doc/guides/prog_guide/bbdev.rst index d40c7f4c..9de14443 100644 --- a/doc/guides/prog_guide/bbdev.rst +++ b/doc/guides/prog_guide/bbdev.rst @@ -6,12 +6,12 @@ Wireless Baseband Device Library The Wireless Baseband library provides a common programming framework that abstracts HW accelerators based on FPGA and/or Fixed Function Accelerators that -assist with 3gpp Physical Layer processing. Furthermore, it decouples the +assist with 3GPP Physical Layer processing. Furthermore, it decouples the application from the compute-intensive wireless functions by abstracting their optimized libraries to appear as virtual bbdev devices. The functional scope of the BBDEV library are those functions in relation to -the 3gpp Layer 1 signal processing (channel coding, modulation, ...). +the 3GPP Layer 1 signal processing (channel coding, modulation, ...). The framework currently only supports Turbo Code FEC function. @@ -42,13 +42,13 @@ From the command line using the --vdev EAL option .. code-block:: console - --vdev 'turbo_sw,max_nb_queues=8,socket_id=0' + --vdev 'baseband_turbo_sw,max_nb_queues=8,socket_id=0' Our using the rte_vdev_init API within the application code. .. code-block:: c - rte_vdev_init("turbo_sw", "max_nb_queues=2,socket_id=0") + rte_vdev_init("baseband_turbo_sw", "max_nb_queues=2,socket_id=0") All virtual bbdev devices support the following initialization parameters: @@ -167,7 +167,7 @@ Logical Cores, Memory and Queues Relationships ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The bbdev device Library as the Poll Mode Driver library support NUMA for when -a processor’s logical cores and interfaces utilize its local memory. Therefore +a processor's logical cores and interfaces utilize its local memory. Therefore baseband operations, the mbuf being operated on should be allocated from memory pools created in the local memory. The buffers should, if possible, remain on the local processor to obtain the best performance results and buffer @@ -202,7 +202,7 @@ structure in the *DPDK API Reference*. A device reports its capabilities when registering itself in the bbdev framework. With the aid of this capabilities mechanism, an application can query devices to -discover which operations within the 3gpp physical layer they are capable of +discover which operations within the 3GPP physical layer they are capable of performing. Below is an example of the capabilities for a PMD it supports in relation to Turbo Encoding and Decoding operations. @@ -216,7 +216,10 @@ relation to Turbo Encoding and Decoding operations. RTE_BBDEV_TURBO_SUBBLOCK_DEINTERLEAVE | RTE_BBDEV_TURBO_POS_LLR_1_BIT_IN | RTE_BBDEV_TURBO_NEG_LLR_1_BIT_IN | - RTE_BBDEV_TURBO_CRC_TYPE_24B, + RTE_BBDEV_TURBO_CRC_TYPE_24B | + RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP | + RTE_BBDEV_TURBO_EARLY_TERMINATION, + .max_llr_modulus = 16, .num_buffers_src = RTE_BBDEV_MAX_CODE_BLOCKS, .num_buffers_hard_out = RTE_BBDEV_MAX_CODE_BLOCKS, @@ -228,6 +231,7 @@ relation to Turbo Encoding and Decoding operations. .cap.turbo_enc = { .capability_flags = RTE_BBDEV_TURBO_CRC_24B_ATTACH | + RTE_BBDEV_TURBO_CRC_24A_ATTACH | RTE_BBDEV_TURBO_RATE_MATCH | RTE_BBDEV_TURBO_RV_INDEX_BYPASS, .num_buffers_src = RTE_BBDEV_MAX_CODE_BLOCKS, @@ -391,27 +395,101 @@ operation to its allocating pool. void rte_bbdev_enc_op_free_bulk(struct rte_bbdev_enc_op **ops, unsigned int num_ops) -BBDEV Operations -~~~~~~~~~~~~~~~~ +BBDEV Inbound/Outbound Memory +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The bbdev operation structure contains all the mutable data relating to -performing Turbo code processing on a referenced mbuf data buffer. It is used -for either encode or decode operations. +performing Turbo coding on a referenced mbuf data buffer. It is used for either +encode or decode operations. Turbo Encode operation accepts one input and one output. - Turbo Decode operation accepts one input and two outputs, called *hard-decision* and *soft-decision* outputs. *Soft-decision* output is optional. -It is expected that the application provides input and output ``mbuf`` pointers +It is expected that the application provides input and output mbuf pointers allocated and ready to use. The baseband framework supports turbo coding on Code Blocks (CB) and Transport Blocks (TB). -For the output buffer(s), the application needs only to provide an allocated and -free mbuf (containing only one mbuf segment), so that bbdev can write the -operation outcome. +For the output buffer(s), the application is required to provide an allocated +and free mbuf, so that bbdev write back the resulting output. + +The support of split "scattered" buffers is a driver-specific feature, so it is +reported individually by the supporting driver as a capability. + +Input and output data buffers are identified by ``rte_bbdev_op_data`` structure, +as follows: + +.. code-block:: c + + struct rte_bbdev_op_data { + struct rte_mbuf *data; + uint32_t offset; + uint32_t length; + }; + + +This structure has three elements: + +- ``data``: This is the mbuf data structure representing the data for BBDEV + operation. + + This mbuf pointer can point to one Code Block (CB) data buffer or multiple CBs + contiguously located next to each other. A Transport Block (TB) represents a + whole piece of data that is divided into one or more CBs. Maximum number of + CBs can be contained in one TB is defined by ``RTE_BBDEV_MAX_CODE_BLOCKS``. + + An mbuf data structure cannot represent more than one TB. The smallest piece + of data that can be contained in one mbuf is one CB. + An mbuf can include one contiguous CB, subset of contiguous CBs that are + belonging to one TB, or all contiguous CBs that are belonging to one TB. + + If a BBDEV PMD supports the extended capability "Scatter-Gather", then it is + capable of collecting (gathering) non-contiguous (scattered) data from + multiple locations in the memory. + This capability is reported by the capability flags: + + - ``RTE_BBDEV_TURBO_ENC_SCATTER_GATHER``, and -**Turbo Encode Op structure** + - ``RTE_BBDEV_TURBO_DEC_SCATTER_GATHER``. + + Only if a BBDEV PMD supports this feature, chained mbuf data structures are + accepted. A chained mbuf can represent one non-contiguous CB or multiple + non-contiguous CBs. + The first mbuf segment in the given chained mbuf represents the first piece + of the CB. Offset is only applicable to the first segment. ``length`` is the + total length of the CB. + + BBDEV driver is responsible for identifying where the split is and enqueue + the split data to its internal queues. + + If BBDEV PMD does not support this feature, it will assume inbound mbuf data + contains one segment. + + The output mbuf data though is always one segment, even if the input was a + chained mbuf. + + +- ``offset``: This is the starting point of the BBDEV (encode/decode) operation, + in bytes. + + BBDEV starts to read data past this offset. + In case of chained mbuf, this offset applies only to the first mbuf segment. + + +- ``length``: This is the total data length to be processed in one operation, + in bytes. + + In case the mbuf data is representing one CB, this is the length of the CB + undergoing the operation. + If it is for multiple CBs, this is the total length of those CBs undergoing + the operation. + If it is for one TB, this is the total length of the TB under operation. + In case of chained mbuf, this data length includes the lengths of the + "scattered" data segments undergoing the operation. + + +BBDEV Turbo Encode Operation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: c @@ -428,8 +506,78 @@ operation outcome. }; }; +The Turbo encode structure is composed of the ``input`` and ``output`` mbuf +data pointers. The provided mbuf pointer of ``input`` needs to be big enough to +stretch for extra CRC trailers. + +``op_flags`` parameter holds all operation related flags, like whether CRC24A is +included by the application or not. + +``code_block_mode`` flag identifies the mode in which bbdev is operating in. + +The encode interface works on both the code block (CB) and the transport block +(TB). An operation executes in "CB-mode" when the CB is standalone. While +"TB-mode" executes when an operation performs on one or multiple CBs that +belong to a TB. Therefore, a given data can be standalone CB, full-size TB or +partial TB. Partial TB means that only a subset of CBs belonging to a bigger TB +are being enqueued. -**Turbo Decode Op structure** + **NOTE:** It is assumed that all enqueued ops in one ``rte_bbdev_enqueue_enc_ops()`` + call belong to one mode, either CB-mode or TB-mode. + +In case that the CB is smaller than Z (6144 bits), then effectively the TB = CB. +CRC24A is appended to the tail of the CB. The application is responsible for +calculating and appending CRC24A before calling BBDEV in case that the +underlying driver does not support CRC24A generation. + +In CB-mode, CRC24A/B is an optional operation. +The input ``k`` is the size of the CB (this maps to K as described in 3GPP TS +36.212 section 5.1.2), this size is inclusive of CRC24A/B. +The ``length`` is inclusive of CRC24A/B and equals to ``k`` in this case. + +Not all BBDEV PMDs are capable of CRC24A/B calculation. Flags +``RTE_BBDEV_TURBO_CRC_24A_ATTACH`` and ``RTE_BBDEV_TURBO_CRC_24B_ATTACH`` +informs the application with relevant capability. These flags can be set in the +``op_flags`` parameter to indicate BBDEV to calculate and append CRC24A to CB +before going forward with Turbo encoding. + +Output format of the CB encode will have the encoded CB in ``e`` size output +(this maps to E described in 3GPP TS 36.212 section 5.1.4.1.2). The output mbuf +buffer size needs to be big enough to hold the encoded buffer of size ``e``. + +In TB-mode, CRC24A is assumed to be pre-calculated and appended to the inbound +TB mbuf data buffer. +The output mbuf data structure is expected to be allocated by the application +with enough room for the output data. + +The difference between the partial and full-size TB is that we need to know the +index of the first CB in this group and the number of CBs contained within. +The first CB index is given by ``r`` but the number of the remaining CBs is +calculated automatically by BBDEV before passing down to the driver. + +The number of remaining CBs should not be confused with ``c``. ``c`` is the +total number of CBs that composes the whole TB (this maps to C as +described in 3GPP TS 36.212 section 5.1.2). + +The ``length`` is total size of the CBs inclusive of any CRC24A and CRC24B in +case they were appended by the application. + +The case when one CB belongs to TB and is being enqueued individually to BBDEV, +this case is considered as a special case of partial TB where its number of CBs +is 1. Therefore, it requires to get processed in TB-mode. + +The figure below visualizes the encoding of CBs using BBDEV interface in +TB-mode. CB-mode is a reduced version, where only one CB exists: + +.. _figure_turbo_tb_encode: + +.. figure:: img/turbo_tb_encode.* + + Turbo encoding of Code Blocks in mbuf structure + + +BBDEV Turbo Decode Operation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: c @@ -452,18 +600,75 @@ operation outcome. }; }; -Input and output data buffers are identified by ``rte_bbdev_op_data`` structure. -This structure has three elements: +The Turbo decode structure is composed of the ``input`` and ``output`` mbuf +data pointers. + +``op_flags`` parameter holds all operation related flags, like whether CRC24B is +retained or not. + +``code_block_mode`` flag identifies the mode in which bbdev is operating in. + +Similarly, the decode interface works on both the code block (CB) and the +transport block (TB). An operation executes in "CB-mode" when the CB is +standalone. While "TB-mode" executes when an operation performs on one or +multiple CBs that belong to a TB. Therefore, a given data can be standalone CB, +full-size TB or partial TB. Partial TB means that only a subset of CBs belonging +to a bigger TB are being enqueued. + + **NOTE:** It is assumed that all enqueued ops in one ``rte_bbdev_enqueue_dec_ops()`` + call belong to one mode, either CB-mode or TB-mode. + +The input ``k`` is the size of the decoded CB (this maps to K as described in +3GPP TS 36.212 section 5.1.2), this size is inclusive of CRC24A/B. +The ``length`` is inclusive of CRC24A/B and equals to ``k`` in this case. + +The input encoded CB data is the Virtual Circular Buffer data stream, wk, with +the null padding included as described in 3GPP TS 36.212 section 5.1.4.1.2 and +shown in 3GPP TS 36.212 section 5.1.4.1 Figure 5.1.4-1. +The size of the virtual circular buffer is 3*Kpi, where Kpi is the 32 byte +aligned value of K, as specified in 3GPP TS 36.212 section 5.1.4.1.1. + +Each byte in the input circular buffer is the LLR value of each bit of the +original CB. + +``hard_output`` is a mandatory capability that all BBDEV PMDs support. This is +the decoded CBs of K sizes (CRC24A/B is the last 24-bit in each decoded CB). +Soft output is an optional capability for BBDEV PMDs. Setting flag +``RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP`` in ``op_flags`` directs BBDEV to retain +CRC24B at the end of each CB. This might be useful for the application in debug +mode. +An LLR rate matched output is computed in the ``soft_output`` buffer structure +for the given ``e`` size (this maps to E described in 3GPP TS 36.212 section +5.1.4.1.2). The output mbuf buffer size needs to be big enough to hold the +encoded buffer of size ``e``. + +The first CB Virtual Circular Buffer (VCB) index is given by ``r`` but the +number of the remaining CB VCBs is calculated automatically by BBDEV before +passing down to the driver. + +The number of remaining CB VCBs should not be confused with ``c``. ``c`` is the +total number of CBs that composes the whole TB (this maps to C as +described in 3GPP TS 36.212 section 5.1.2). + +The ``length`` is total size of the CBs inclusive of any CRC24A and CRC24B in +case they were appended by the application. + +The case when one CB belongs to TB and is being enqueued individually to BBDEV, +this case is considered as a special case of partial TB where its number of CBs +is 1. Therefore, it requires to get processed in TB-mode. + +The output mbuf data structure is expected to be allocated by the application +with enough room for the output data. + +The figure below visualizes the decoding of CBs using BBDEV interface in +TB-mode. CB-mode is a reduced version, where only one CB exists: + +.. _figure_turbo_tb_decode: -- ``data`` - This is the mbuf reference +.. figure:: img/turbo_tb_decode.* -- ``offset`` - The starting point for the Turbo input/output, in bytes, from the - start of the data in the data buffer. It must be smaller than data_len of the - mbuf's first segment + Turbo decoding of Code Blocks in mbuf structure -- ``length`` - The length, in bytes, of the buffer on which the Turbo operation - will or has been computed. For the input, the length is set by the application. - For the output(s), the length is computed by the bbdev PMD driver. Sample code ----------- diff --git a/doc/guides/prog_guide/bpf_lib.rst b/doc/guides/prog_guide/bpf_lib.rst new file mode 100644 index 00000000..7c08e6b2 --- /dev/null +++ b/doc/guides/prog_guide/bpf_lib.rst @@ -0,0 +1,38 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +Berkeley Packet Filter Library +============================== + +The DPDK provides an BPF library that gives the ability +to load and execute Enhanced Berkeley Packet Filter (eBPF) bytecode within +user-space dpdk application. + +It supports basic set of features from eBPF spec. +Please refer to the +`eBPF spec <https://www.kernel.org/doc/Documentation/networking/filter.txt>` +for more information. +Also it introduces basic framework to load/unload BPF-based filters +on eth devices (right now only via SW RX/TX callbacks). + +The library API provides the following basic operations: + +* Create a new BPF execution context and load user provided eBPF code into it. + +* Destroy an BPF execution context and its runtime structures and free the associated memory. + +* Execute eBPF bytecode associated with provided input parameter. + +* Provide information about natively compiled code for given BPF context. + +* Load BPF program from the ELF file and install callback to execute it on given ethdev port/queue. + +Not currently supported eBPF features +------------------------------------- + + - JIT for non X86_64 platforms + - cBPF + - tail-pointer call + - eBPF MAP + - skb + - external function calls for 32-bit platforms diff --git a/doc/guides/prog_guide/compressdev.rst b/doc/guides/prog_guide/compressdev.rst new file mode 100644 index 00000000..87e26490 --- /dev/null +++ b/doc/guides/prog_guide/compressdev.rst @@ -0,0 +1,623 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017-2018 Cavium Networks. + +Compression Device Library +=========================== + +The compression framework provides a generic set of APIs to perform compression services +as well as to query and configure compression devices both physical(hardware) and virtual(software) +to perform those services. The framework currently only supports lossless compression schemes: +Deflate and LZS. + +Device Management +----------------- + +Device Creation +~~~~~~~~~~~~~~~ + +Physical compression devices are discovered during the bus probe of the EAL function +which is executed at DPDK initialization, based on their unique device identifier. +For eg. PCI devices can be identified using PCI BDF (bus/bridge, device, function). +Specific physical compression devices, like other physical devices in DPDK can be +white-listed or black-listed using the EAL command line options. + +Virtual devices can be created by two mechanisms, either using the EAL command +line options or from within the application using an EAL API directly. + +From the command line using the --vdev EAL option + +.. code-block:: console + + --vdev '<pmd name>,socket_id=0' + +.. Note:: + + * If DPDK application requires multiple software compression PMD devices then required + number of ``--vdev`` with appropriate libraries are to be added. + + * An Application with multiple compression device instances exposed by the same PMD must + specify a unique name for each device. + + Example: ``--vdev 'pmd0' --vdev 'pmd1'`` + +Or, by using the rte_vdev_init API within the application code. + +.. code-block:: c + + rte_vdev_init("<pmd_name>","socket_id=0") + +All virtual compression devices support the following initialization parameters: + +* ``socket_id`` - socket on which to allocate the device resources on. + +Device Identification +~~~~~~~~~~~~~~~~~~~~~ + +Each device, whether virtual or physical is uniquely designated by two +identifiers: + +- A unique device index used to designate the compression device in all functions + exported by the compressdev API. + +- A device name used to designate the compression device in console messages, for + administration or debugging purposes. + +Device Configuration +~~~~~~~~~~~~~~~~~~~~ + +The configuration of each compression device includes the following operations: + +- Allocation of resources, including hardware resources if a physical device. +- Resetting the device into a well-known default state. +- Initialization of statistics counters. + +The ``rte_compressdev_configure`` API is used to configure a compression device. + +The ``rte_compressdev_config`` structure is used to pass the configuration +parameters. + +See *DPDK API Reference* for details. + +Configuration of Queue Pairs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each compression device queue pair is individually configured through the +``rte_compressdev_queue_pair_setup`` API. + +The ``max_inflight_ops`` is used to pass maximum number of +rte_comp_op that could be present in a queue at-a-time. +PMD then can allocate resources accordingly on a specified socket. + +See *DPDK API Reference* for details. + +Logical Cores, Memory and Queues Pair Relationships +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Library supports NUMA similarly as described in Cryptodev library section. + +A queue pair cannot be shared and should be exclusively used by a single processing +context for enqueuing operations or dequeuing operations on the same compression device +since sharing would require global locks and hinder performance. It is however possible +to use a different logical core to dequeue an operation on a queue pair from the logical +core on which it was enqueued. This means that a compression burst enqueue/dequeue +APIs are a logical place to transition from one logical core to another in a +data processing pipeline. + +Device Features and Capabilities +--------------------------------- + +Compression devices define their functionality through two mechanisms, global device +features and algorithm features. Global devices features identify device +wide level features which are applicable to the whole device such as supported hardware +acceleration and CPU features. List of compression device features can be seen in the +RTE_COMPDEV_FF_XXX macros. + +The algorithm features lists individual algo feature which device supports per-algorithm, +such as a stateful compression/decompression, checksums operation etc. List of algorithm +features can be seen in the RTE_COMP_FF_XXX macros. + +Capabilities +~~~~~~~~~~~~ +Each PMD has a list of capabilities, including algorithms listed in +enum ``rte_comp_algorithm`` and its associated feature flag and +sliding window range in log base 2 value. Sliding window tells +the minimum and maximum size of lookup window that algorithm uses +to find duplicates. + +See *DPDK API Reference* for details. + +Each Compression poll mode driver defines its array of capabilities +for each algorithm it supports. See PMD implementation for capability +initialization. + +Capabilities Discovery +~~~~~~~~~~~~~~~~~~~~~~ + +PMD capability and features are discovered via ``rte_compressdev_info_get`` function. + +The ``rte_compressdev_info`` structure contains all the relevant information for the device. + +See *DPDK API Reference* for details. + +Compression Operation +---------------------- + +DPDK compression supports two types of compression methodologies: + +- Stateless, data associated to a compression operation is compressed without any reference + to another compression operation. + +- Stateful, data in each compression operation is compressed with reference to previous compression + operations in the same data stream i.e. history of data is maintained between the operations. + +For more explanation, please refer RFC https://www.ietf.org/rfc/rfc1951.txt + +Operation Representation +~~~~~~~~~~~~~~~~~~~~~~~~ + +Compression operation is described via ``struct rte_comp_op``, which contains both input and +output data. The operation structure includes the operation type (stateless or stateful), +the operation status and the priv_xform/stream handle, source, destination and checksum buffer +pointers. It also contains the source mempool from which the operation is allocated. +PMD updates consumed field with amount of data read from source buffer and produced +field with amount of data of written into destination buffer along with status of +operation. See section *Produced, Consumed And Operation Status* for more details. + +Compression operations mempool also has an ability to allocate private memory with the +operation for application's purposes. Application software is responsible for specifying +all the operation specific fields in the ``rte_comp_op`` structure which are then used +by the compression PMD to process the requested operation. + + +Operation Management and Allocation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The compressdev library provides an API set for managing compression operations which +utilize the Mempool Library to allocate operation buffers. Therefore, it ensures +that the compression operation is interleaved optimally across the channels and +ranks for optimal processing. + +A ``rte_comp_op`` contains a field indicating the pool it originated from. + +``rte_comp_op_alloc()`` and ``rte_comp_op_bulk_alloc()`` are used to allocate +compression operations from a given compression operation mempool. +The operation gets reset before being returned to a user so that operation +is always in a good known state before use by the application. + +``rte_comp_op_free()`` is called by the application to return an operation to +its allocating pool. + +See *DPDK API Reference* for details. + +Passing source data as mbuf-chain +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +If input data is scattered across several different buffers, then +Application can either parse through all such buffers and make one +mbuf-chain and enqueue it for processing or, alternatively, it can +make multiple sequential enqueue_burst() calls for each of them +processing them statefully. See *Compression API Stateful Operation* +for stateful processing of ops. + +Operation Status +~~~~~~~~~~~~~~~~ +Each operation carries a status information updated by PMD after it is processed. +following are currently supported status: + +- RTE_COMP_OP_STATUS_SUCCESS, + Operation is successfully completed + +- RTE_COMP_OP_STATUS_NOT_PROCESSED, + Operation has not yet been processed by the device + +- RTE_COMP_OP_STATUS_INVALID_ARGS, + Operation failed due to invalid arguments in request + +- RTE_COMP_OP_STATUS_ERROR, + Operation failed because of internal error + +- RTE_COMP_OP_STATUS_INVALID_STATE, + Operation is invoked in invalid state + +- RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED, + Output buffer ran out of space during processing. Error case, + PMD cannot continue from here. + +- RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE, + Output buffer ran out of space before operation completed, but this + is not an error case. Output data up to op.produced can be used and + next op in the stream should continue on from op.consumed+1. + +Produced, Consumed And Operation Status +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- If status is RTE_COMP_OP_STATUS_SUCCESS, + consumed = amount of data read from input buffer, and + produced = amount of data written in destination buffer +- If status is RTE_COMP_OP_STATUS_FAILURE, + consumed = produced = 0 or undefined +- If status is RTE_COMP_OP_STATUS_OUT_OF_SPACE_TERMINATED, + consumed = 0 and + produced = usually 0, but in decompression cases a PMD may return > 0 + i.e. amount of data successfully produced until out of space condition + hit. Application can consume output data in this case, if required. +- If status is RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE, + consumed = amount of data read, and + produced = amount of data successfully produced until + out of space condition hit. PMD has ability to recover + from here, so application can submit next op from + consumed+1 and a destination buffer with available space. + +Transforms +---------- + +Compression transforms (``rte_comp_xform``) are the mechanism +to specify the details of the compression operation such as algorithm, +window size and checksum. + +Compression API Hash support +---------------------------- + +Compression API allows application to enable digest calculation +alongside compression and decompression of data. A PMD reflects its +support for hash algorithms via capability algo feature flags. +If supported, PMD calculates digest always on plaintext i.e. +before compression and after decompression. + +Currently supported list of hash algos are SHA-1 and SHA2 family +SHA256. + +See *DPDK API Reference* for details. + +If required, application should set valid hash algo in compress +or decompress xforms during ``rte_compressdev_stream_create()`` +or ``rte_compressdev_private_xform_create()`` and pass a valid +output buffer in ``rte_comp_op`` hash field struct to store the +resulting digest. Buffer passed should be contiguous and large +enough to store digest which is 20 bytes for SHA-1 and +32 bytes for SHA2-256. + +Compression API Stateless operation +------------------------------------ + +An op is processed stateless if it has +- op_type set to RTE_COMP_OP_STATELESS +- flush value set to RTE_FLUSH_FULL or RTE_FLUSH_FINAL +(required only on compression side), +- All required input in source buffer + +When all of the above conditions are met, PMD initiates stateless processing +and releases acquired resources after processing of current operation is +complete. Application can enqueue multiple stateless ops in a single burst +and must attach priv_xform handle to such ops. + +priv_xform in Stateless operation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +priv_xform is PMD internally managed private data that it maintains to do stateless processing. +priv_xforms are initialized provided a generic xform structure by an application via making call +to ``rte_comp_private_xform_create``, at an output PMD returns an opaque priv_xform reference. +If PMD support SHAREABLE priv_xform indicated via algorithm feature flag, then application can +attach same priv_xform with many stateless ops at-a-time. If not, then application needs to +create as many priv_xforms as it expects to have stateless operations in-flight. + +.. figure:: img/stateless-op.* + + Stateless Ops using Non-Shareable priv_xform + + +.. figure:: img/stateless-op-shared.* + + Stateless Ops using Shareable priv_xform + + +Application should call ``rte_compressdev_private_xform_create()`` and attach to stateless op before +enqueuing them for processing and free via ``rte_compressdev_private_xform_free()`` during termination. + +An example pseudocode to setup and process NUM_OPS stateless ops with each of length OP_LEN +using priv_xform would look like: + +.. code-block:: c + + /* + * pseudocode for stateless compression + */ + + uint8_t cdev_id = rte_compdev_get_dev_id(<pmd name>); + + /* configure the device. */ + if (rte_compressdev_configure(cdev_id, &conf) < 0) + rte_exit(EXIT_FAILURE, "Failed to configure compressdev %u", cdev_id); + + if (rte_compressdev_queue_pair_setup(cdev_id, 0, NUM_MAX_INFLIGHT_OPS, + socket_id()) < 0) + rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n"); + + if (rte_compressdev_start(cdev_id) < 0) + rte_exit(EXIT_FAILURE, "Failed to start device\n"); + + /* setup compress transform */ + struct rte_compress_compress_xform compress_xform = { + .type = RTE_COMP_COMPRESS, + .compress = { + .algo = RTE_COMP_ALGO_DEFLATE, + .deflate = { + .huffman = RTE_COMP_HUFFMAN_DEFAULT + }, + .level = RTE_COMP_LEVEL_PMD_DEFAULT, + .chksum = RTE_COMP_CHECKSUM_NONE, + .window_size = DEFAULT_WINDOW_SIZE, + .hash_algo = RTE_COMP_HASH_ALGO_NONE + } + }; + + /* create priv_xform and initialize it for the compression device. */ + void *priv_xform = NULL; + rte_compressdev_info_get(cdev_id, &dev_info); + if(dev_info.capability->comps_feature_flag & RTE_COMP_FF_SHAREABLE_PRIV_XFORM) { + rte_comp_priv_xform_create(cdev_id, &compress_xform, &priv_xform); + } else { + shareable = 0; + } + + /* create operation pool via call to rte_comp_op_pool_create and alloc ops */ + rte_comp_op_bulk_alloc(op_pool, comp_ops, NUM_OPS); + + /* prepare ops for compression operations */ + for (i = 0; i < NUM_OPS; i++) { + struct rte_comp_op *op = comp_ops[i]; + if (!shareable) + rte_priv_xform_create(cdev_id, &compress_xform, &op->priv_xform) + else + op->priv_xform = priv_xform; + op->type = RTE_COMP_OP_STATELESS; + op->flush = RTE_COMP_FLUSH_FINAL; + + op->src.offset = 0; + op->dst.offset = 0; + op->src.length = OP_LEN; + op->input_chksum = 0; + setup op->m_src and op->m_dst; + } + num_enqd = rte_compressdev_enqueue_burst(cdev_id, 0, comp_ops, NUM_OPS); + /* wait for this to complete before enqueing next*/ + do { + num_deque = rte_compressdev_dequeue_burst(cdev_id, 0 , &processed_ops, NUM_OPS); + } while (num_dqud < num_enqd); + + +Stateless and OUT_OF_SPACE +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +OUT_OF_SPACE is a condition when output buffer runs out of space and where PMD +still has more data to produce. If PMD runs into such condition, then PMD returns +RTE_COMP_OP_OUT_OF_SPACE_TERMINATED error. In such case, PMD resets itself and can set +consumed=0 and produced=amount of output it could produce before hitting out_of_space. +Application would need to resubmit the whole input with a larger output buffer, if it +wants the operation to be completed. + +Hash in Stateless +~~~~~~~~~~~~~~~~~ +If hash is enabled, digest buffer will contain valid data after op is successfully +processed i.e. dequeued with status = RTE_COMP_OP_STATUS_SUCCESS. + +Checksum in Stateless +~~~~~~~~~~~~~~~~~~~~~ +If checksum is enabled, checksum will only be available after op is successfully +processed i.e. dequeued with status = RTE_COMP_OP_STATUS_SUCCESS. + +Compression API Stateful operation +----------------------------------- + +Compression API provide RTE_COMP_FF_STATEFUL_COMPRESSION and +RTE_COMP_FF_STATEFUL_DECOMPRESSION feature flag for PMD to reflect +its support for Stateful operations. + +A Stateful operation in DPDK compression means application invokes enqueue +burst() multiple times to process related chunk of data because +application broke data into several ops. + +In such case +- ops are setup with op_type RTE_COMP_OP_STATEFUL, +- all ops except last set to flush value = RTE_COMP_NO/SYNC_FLUSH +and last set to flush value RTE_COMP_FULL/FINAL_FLUSH. + +In case of either one or all of the above conditions, PMD initiates +stateful processing and releases acquired resources after processing +operation with flush value = RTE_COMP_FLUSH_FULL/FINAL is complete. +Unlike stateless, application can enqueue only one stateful op from +a particular stream at a time and must attach stream handle +to each op. + +Stream in Stateful operation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`stream` in DPDK compression is a logical entity which identifies related set of ops, say, a one large +file broken into multiple chunks then file is represented by a stream and each chunk of that file is +represented by compression op `rte_comp_op`. Whenever application wants a stateful processing of such +data, then it must get a stream handle via making call to ``rte_comp_stream_create()`` +with xform, at an output the target PMD will return an opaque stream handle to application which +it must attach to all of the ops carrying data of that stream. In stateful processing, every op +requires previous op data for compression/decompression. A PMD allocates and set up resources such +as history, states, etc. within a stream, which are maintained during the processing of the related ops. + +Unlike priv_xforms, stream is always a NON_SHAREABLE entity. One stream handle must be attached to only +one set of related ops and cannot be reused until all of them are processed with status Success or failure. + +.. figure:: img/stateful-op.* + + Stateful Ops + + +Application should call ``rte_comp_stream_create()`` and attach to op before +enqueuing them for processing and free via ``rte_comp_stream_free()`` during +termination. All ops that are to be processed statefully should carry *same* stream. + +See *DPDK API Reference* document for details. + +An example pseudocode to set up and process a stream having NUM_CHUNKS with each chunk size of CHUNK_LEN would look like: + +.. code-block:: c + + /* + * pseudocode for stateful compression + */ + + uint8_t cdev_id = rte_compdev_get_dev_id(<pmd name>); + + /* configure the device. */ + if (rte_compressdev_configure(cdev_id, &conf) < 0) + rte_exit(EXIT_FAILURE, "Failed to configure compressdev %u", cdev_id); + + if (rte_compressdev_queue_pair_setup(cdev_id, 0, NUM_MAX_INFLIGHT_OPS, + socket_id()) < 0) + rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n"); + + if (rte_compressdev_start(cdev_id) < 0) + rte_exit(EXIT_FAILURE, "Failed to start device\n"); + + /* setup compress transform. */ + struct rte_compress_compress_xform compress_xform = { + .type = RTE_COMP_COMPRESS, + .compress = { + .algo = RTE_COMP_ALGO_DEFLATE, + .deflate = { + .huffman = RTE_COMP_HUFFMAN_DEFAULT + }, + .level = RTE_COMP_LEVEL_PMD_DEFAULT, + .chksum = RTE_COMP_CHECKSUM_NONE, + .window_size = DEFAULT_WINDOW_SIZE, + .hash_algo = RTE_COMP_HASH_ALGO_NONE + } + }; + + /* create stream */ + rte_comp_stream_create(cdev_id, &compress_xform, &stream); + + /* create an op pool and allocate ops */ + rte_comp_op_bulk_alloc(op_pool, comp_ops, NUM_CHUNKS); + + /* Prepare source and destination mbufs for compression operations */ + unsigned int i; + for (i = 0; i < NUM_CHUNKS; i++) { + if (rte_pktmbuf_append(mbufs[i], CHUNK_LEN) == NULL) + rte_exit(EXIT_FAILURE, "Not enough room in the mbuf\n"); + comp_ops[i]->m_src = mbufs[i]; + if (rte_pktmbuf_append(dst_mbufs[i], CHUNK_LEN) == NULL) + rte_exit(EXIT_FAILURE, "Not enough room in the mbuf\n"); + comp_ops[i]->m_dst = dst_mbufs[i]; + } + + /* Set up the compress operations. */ + for (i = 0; i < NUM_CHUNKS; i++) { + struct rte_comp_op *op = comp_ops[i]; + op->stream = stream; + op->m_src = src_buf[i]; + op->m_dst = dst_buf[i]; + op->type = RTE_COMP_OP_STATEFUL; + if(i == NUM_CHUNKS-1) { + /* set to final, if last chunk*/ + op->flush = RTE_COMP_FLUSH_FINAL; + } else { + /* set to NONE, for all intermediary ops */ + op->flush = RTE_COMP_FLUSH_NONE; + } + op->src.offset = 0; + op->dst.offset = 0; + op->src.length = CHUNK_LEN; + op->input_chksum = 0; + num_enqd = rte_compressdev_enqueue_burst(cdev_id, 0, &op[i], 1); + /* wait for this to complete before enqueing next*/ + do { + num_deqd = rte_compressdev_dequeue_burst(cdev_id, 0 , &processed_ops, 1); + } while (num_deqd < num_enqd); + /* push next op*/ + } + + +Stateful and OUT_OF_SPACE +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If PMD supports stateful operation, then OUT_OF_SPACE status is not an actual +error for the PMD. In such case, PMD returns with status +RTE_COMP_OP_STATUS_OUT_OF_SPACE_RECOVERABLE with consumed = number of input bytes +read and produced = length of complete output buffer. +Application should enqueue next op with source starting at consumed+1 and an +output buffer with available space. + +Hash in Stateful +~~~~~~~~~~~~~~~~ +If enabled, digest buffer will contain valid digest after last op in stream +(having flush = RTE_COMP_OP_FLUSH_FINAL) is successfully processed i.e. dequeued +with status = RTE_COMP_OP_STATUS_SUCCESS. + +Checksum in Stateful +~~~~~~~~~~~~~~~~~~~~ +If enabled, checksum will only be available after last op in stream +(having flush = RTE_COMP_OP_FLUSH_FINAL) is successfully processed i.e. dequeued +with status = RTE_COMP_OP_STATUS_SUCCESS. + +Burst in compression API +------------------------- + +Scheduling of compression operations on DPDK's application data path is +performed using a burst oriented asynchronous API set. A queue pair on a compression +device accepts a burst of compression operations using enqueue burst API. On physical +devices the enqueue burst API will place the operations to be processed +on the device's hardware input queue, for virtual devices the processing of the +operations is usually completed during the enqueue call to the compression +device. The dequeue burst API will retrieve any processed operations available +from the queue pair on the compression device, from physical devices this is usually +directly from the devices processed queue, and for virtual device's from a +``rte_ring`` where processed operations are place after being processed on the +enqueue call. + +A burst in DPDK compression can be a combination of stateless and stateful operations with a condition +that for stateful ops only one op at-a-time should be enqueued from a particular stream i.e. no-two ops +should belong to same stream in a single burst. However a burst may contain multiple stateful ops as long +as each op is attached to a different stream i.e. a burst can look like: + ++---------------+--------------+--------------+-----------------+--------------+--------------+ +| enqueue_burst | op1.no_flush | op2.no_flush | op3.flush_final | op4.no_flush | op5.no_flush | ++---------------+--------------+--------------+-----------------+--------------+--------------+ + +Where, op1 .. op5 all belong to different independent data units. op1, op2, op4, op5 must be stateful +as stateless ops can only use flush full or final and op3 can be of type stateless or stateful. +Every op with type set to RTE_COMP_OP_TYPE_STATELESS must be attached to priv_xform and +Every op with type set to RTE_COMP_OP_TYPE_STATEFUL *must* be attached to stream. + +Since each operation in a burst is independent and thus can be completed +out-of-order, applications which need ordering, should setup per-op user data +area with reordering information so that it can determine enqueue order at +dequeue. + +Also if multiple threads calls enqueue_burst() on same queue pair then it’s +application onus to use proper locking mechanism to ensure exclusive enqueuing +of operations. + +Enqueue / Dequeue Burst APIs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The burst enqueue API uses a compression device identifier and a queue pair +identifier to specify the compression device queue pair to schedule the processing on. +The ``nb_ops`` parameter is the number of operations to process which are +supplied in the ``ops`` array of ``rte_comp_op`` structures. +The enqueue function returns the number of operations it actually enqueued for +processing, a return value equal to ``nb_ops`` means that all packets have been +enqueued. + +The dequeue API uses the same format as the enqueue API but +the ``nb_ops`` and ``ops`` parameters are now used to specify the max processed +operations the user wishes to retrieve and the location in which to store them. +The API call returns the actual number of processed operations returned, this +can never be larger than ``nb_ops``. + +Sample code +----------- + +There are unit test applications that show how to use the compressdev library inside +test/test/test_compressdev.c + +Compression Device API +~~~~~~~~~~~~~~~~~~~~~~ + +The compressdev Library API is described in the *DPDK API Reference* document. diff --git a/doc/guides/prog_guide/cryptodev_lib.rst b/doc/guides/prog_guide/cryptodev_lib.rst index 066fe2d2..90d01e93 100644 --- a/doc/guides/prog_guide/cryptodev_lib.rst +++ b/doc/guides/prog_guide/cryptodev_lib.rst @@ -8,7 +8,7 @@ The cryptodev library provides a Crypto device framework for management and provisioning of hardware and software Crypto poll mode drivers, defining generic APIs which support a number of different Crypto operations. The framework currently only supports cipher, authentication, chained cipher/authentication -and AEAD symmetric Crypto operations. +and AEAD symmetric and asymmetric Crypto operations. Design Principles @@ -41,7 +41,7 @@ From the command line using the --vdev EAL option .. code-block:: console - --vdev 'crypto_aesni_mb0,max_nb_queue_pairs=2,max_nb_sessions=1024,socket_id=0' + --vdev 'crypto_aesni_mb0,max_nb_queue_pairs=2,socket_id=0' .. Note:: @@ -57,12 +57,11 @@ Our using the rte_vdev_init API within the application code. .. code-block:: c rte_vdev_init("crypto_aesni_mb", - "max_nb_queue_pairs=2,max_nb_sessions=1024,socket_id=0") + "max_nb_queue_pairs=2,socket_id=0") All virtual Crypto devices support the following initialization parameters: * ``max_nb_queue_pairs`` - maximum number of queue pairs supported by the device. -* ``max_nb_sessions`` - maximum number of sessions supported by the device * ``socket_id`` - socket on which to allocate the device resources on. @@ -159,8 +158,8 @@ Device Features and Capabilities Crypto devices define their functionality through two mechanisms, global device features and algorithm capabilities. Global devices features identify device wide level features which are applicable to the whole device such as -the device having hardware acceleration or supporting symmetric Crypto -operations, +the device having hardware acceleration or supporting symmetric and/or asymmetric +Crypto operations. The capabilities mechanism defines the individual algorithms/functions which the device supports, such as a specific symmetric Crypto cipher, @@ -269,7 +268,7 @@ relevant information for the device. struct rte_cryptodev_info { const char *driver_name; uint8_t driver_id; - struct rte_pci_device *pci_dev; + struct rte_device *device; uint64_t feature_flags; @@ -299,6 +298,33 @@ directly from the devices processed queue, and for virtual device's from a enqueue call. +Private data +~~~~~~~~~~~~ +For session-based operations, the set and get API provides a mechanism for an +application to store and retrieve the private user data information stored along +with the crypto session. + +For example, suppose an application is submitting a crypto operation with a session +associated and wants to indicate private user data information which is required to be +used after completion of the crypto operation. In this case, the application can use +the set API to set the user data and retrieve it using get API. + +.. code-block:: c + + int rte_cryptodev_sym_session_set_user_data( + struct rte_cryptodev_sym_session *sess, void *data, uint16_t size); + + void * rte_cryptodev_sym_session_get_user_data( + struct rte_cryptodev_sym_session *sess); + + +For session-less mode, the private user data information can be placed along with the +``struct rte_crypto_op``. The ``rte_crypto_op::private_data_offset`` indicates the +start of private data information. The offset is counted from the start of the +rte_crypto_op including other crypto information such as the IVs (since there can +be an IV also for authentication). + + Enqueue / Dequeue Burst APIs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -419,7 +445,7 @@ Crypto workloads. .. figure:: img/cryptodev_sym_sess.* -The Crypto device framework provides APIs to allocate and initizalize sessions +The Crypto device framework provides APIs to allocate and initialize sessions for crypto devices, where sessions are mempool objects. It is the application's responsibility to create and manage the session mempools. This approach allows for different scenarios such as having a single session @@ -427,12 +453,12 @@ mempool for all crypto devices (where the mempool object size is big enough to hold the private session of any crypto device), as well as having multiple session mempools of different sizes for better memory usage. -An application can use ``rte_cryptodev_get_private_session_size()`` to +An application can use ``rte_cryptodev_sym_get_private_session_size()`` to get the private session size of given crypto device. This function would allow an application to calculate the max device session size of all crypto devices to create a single session mempool. If instead an application creates multiple session mempools, the Crypto device -framework also provides ``rte_cryptodev_get_header_session_size`` to get +framework also provides ``rte_cryptodev_sym_get_header_session_size`` to get the size of an uninitialized session. Once the session mempools have been created, ``rte_cryptodev_sym_session_create()`` @@ -635,7 +661,7 @@ using one of the crypto PMDs available in DPDK. uint8_t cdev_id = rte_cryptodev_get_dev_id(crypto_name); /* Get private session data size. */ - session_size = rte_cryptodev_get_private_session_size(cdev_id); + session_size = rte_cryptodev_sym_get_private_session_size(cdev_id); /* * Create session mempool, with two objects per session, @@ -761,14 +787,260 @@ using one of the crypto PMDs available in DPDK. num_dequeued_ops); } while (total_num_dequeued_ops < num_enqueued_ops); - Asymmetric Cryptography ----------------------- -Asymmetric functionality is currently not supported by the cryptodev API. +The cryptodev library currently provides support for the following asymmetric +Crypto operations; RSA, Modular exponentiation and inversion, Diffie-Hellman +public and/or private key generation and shared secret compute, DSA Signature +generation and verification. + +Session and Session Management +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sessions are used in asymmetric cryptographic processing to store the immutable +data defined in asymmetric cryptographic transform which is further used in the +operation processing. Sessions typically stores information, such as, public +and private key information or domain params or prime modulus data i.e. immutable +across data sets. Crypto sessions cache this immutable data in a optimal way for the +underlying PMD and this allows further acceleration of the offload of Crypto workloads. + +Like symmetric, the Crypto device framework provides APIs to allocate and initialize +asymmetric sessions for crypto devices, where sessions are mempool objects. +It is the application's responsibility to create and manage the session mempools. +Application using both symmetric and asymmetric sessions should allocate and maintain +different sessions pools for each type. + +An application can use ``rte_cryptodev_get_asym_session_private_size()`` to +get the private size of asymmetric session on a given crypto device. This +function would allow an application to calculate the max device asymmetric +session size of all crypto devices to create a single session mempool. +If instead an application creates multiple asymmetric session mempools, +the Crypto device framework also provides ``rte_cryptodev_asym_get_header_session_size()`` to get +the size of an uninitialized session. + +Once the session mempools have been created, ``rte_cryptodev_asym_session_create()`` +is used to allocate an uninitialized asymmetric session from the given mempool. +The session then must be initialized using ``rte_cryptodev_asym_session_init()`` +for each of the required crypto devices. An asymmetric transform chain +is used to specify the operation and its parameters. See the section below for +details on transforms. + +When a session is no longer used, user must call ``rte_cryptodev_asym_session_clear()`` +for each of the crypto devices that are using the session, to free all driver +private asymmetric session data. Once this is done, session should be freed using +``rte_cryptodev_asym_session_free()`` which returns them to their mempool. + +Asymmetric Sessionless Support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Currently asymmetric crypto framework does not support sessionless. + +Transforms and Transform Chaining +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Asymmetric Crypto transforms (``rte_crypto_asym_xform``) are the mechanism used +to specify the details of the asymmetric Crypto operation. Next pointer within +xform allows transform to be chained together. Also it is important to note that +the order in which the transforms are passed indicates the order of the chaining. + +Not all asymmetric crypto xforms are supported for chaining. Currently supported +asymmetric crypto chaining is Diffie-Hellman private key generation followed by +public generation. Also, currently API does not support chaining of symmetric and +asymmetric crypto xfroms. + +Each xform defines specific asymmetric crypto algo. Currently supported are: +* RSA +* Modular operations (Exponentiation and Inverse) +* Diffie-Hellman +* DSA +* None - special case where PMD may support a passthrough mode. More for diagnostic purpose + +See *DPDK API Reference* for details on each rte_crypto_xxx_xform struct + +Asymmetric Operations +~~~~~~~~~~~~~~~~~~~~~ + +The asymmetric Crypto operation structure contains all the mutable data relating +to asymmetric cryptographic processing on an input data buffer. It uses either +RSA, Modular, Diffie-Hellman or DSA operations depending upon session it is attached +to. + +Every operation must carry a valid session handle which further carries information +on xform or xform-chain to be performed on op. Every xform type defines its own set +of operational params in their respective rte_crypto_xxx_op_param struct. Depending +on xform information within session, PMD picks up and process respective op_param +struct. +Unlike symmetric, asymmetric operations do not use mbufs for input/output. +They operate on data buffer of type ``rte_crypto_param``. + +See *DPDK API Reference* for details on each rte_crypto_xxx_op_param struct + +Asymmetric crypto Sample code +----------------------------- + +There's a unit test application test_cryptodev_asym.c inside unit test framework that +show how to setup and process asymmetric operations using cryptodev library. + +The following sample code shows the basic steps to compute modular exponentiation +using 1024-bit modulus length using openssl PMD available in DPDK (performing other +crypto operations is similar except change to respective op and xform setup). + +.. code-block:: c + + /* + * Simple example to compute modular exponentiation with 1024-bit key + * + */ + #define MAX_ASYM_SESSIONS 10 + #define NUM_ASYM_BUFS 10 + + struct rte_mempool *crypto_op_pool, *asym_session_pool; + unsigned int asym_session_size; + int ret; + + /* Initialize EAL. */ + ret = rte_eal_init(argc, argv); + if (ret < 0) + rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); + + uint8_t socket_id = rte_socket_id(); + + /* Create crypto operation pool. */ + crypto_op_pool = rte_crypto_op_pool_create( + "crypto_op_pool", + RTE_CRYPTO_OP_TYPE_ASYMMETRIC, + NUM_ASYM_BUFS, 0, 0, + socket_id); + if (crypto_op_pool == NULL) + rte_exit(EXIT_FAILURE, "Cannot create crypto op pool\n"); + /* Create the virtual crypto device. */ + char args[128]; + const char *crypto_name = "crypto_openssl"; + snprintf(args, sizeof(args), "socket_id=%d", socket_id); + ret = rte_vdev_init(crypto_name, args); + if (ret != 0) + rte_exit(EXIT_FAILURE, "Cannot create virtual device"); -Crypto Device API -~~~~~~~~~~~~~~~~~ + uint8_t cdev_id = rte_cryptodev_get_dev_id(crypto_name); + + /* Get private asym session data size. */ + asym_session_size = rte_cryptodev_get_asym_private_session_size(cdev_id); + + /* + * Create session mempool, with two objects per session, + * one for the session header and another one for the + * private asym session data for the crypto device. + */ + asym_session_pool = rte_mempool_create("asym_session_pool", + MAX_ASYM_SESSIONS * 2, + asym_session_size, + 0, + 0, NULL, NULL, NULL, + NULL, socket_id, + 0); + + /* Configure the crypto device. */ + struct rte_cryptodev_config conf = { + .nb_queue_pairs = 1, + .socket_id = socket_id + }; + struct rte_cryptodev_qp_conf qp_conf = { + .nb_descriptors = 2048 + }; + + if (rte_cryptodev_configure(cdev_id, &conf) < 0) + rte_exit(EXIT_FAILURE, "Failed to configure cryptodev %u", cdev_id); + + if (rte_cryptodev_queue_pair_setup(cdev_id, 0, &qp_conf, + socket_id, asym_session_pool) < 0) + rte_exit(EXIT_FAILURE, "Failed to setup queue pair\n"); + + if (rte_cryptodev_start(cdev_id) < 0) + rte_exit(EXIT_FAILURE, "Failed to start device\n"); + + /* Setup crypto xform to do modular exponentiation with 1024 bit + * length modulus + */ + struct rte_crypto_asym_xform modex_xform = { + .next = NULL, + .xform_type = RTE_CRYPTO_ASYM_XFORM_MODEX, + .modex = { + .modulus = { + .data = + (uint8_t *) + ("\xb3\xa1\xaf\xb7\x13\x08\x00\x0a\x35\xdc\x2b\x20\x8d" + "\xa1\xb5\xce\x47\x8a\xc3\x80\xf4\x7d\x4a\xa2\x62\xfd\x61\x7f" + "\xb5\xa8\xde\x0a\x17\x97\xa0\xbf\xdf\x56\x5a\x3d\x51\x56\x4f" + "\x70\x70\x3f\x63\x6a\x44\x5b\xad\x84\x0d\x3f\x27\x6e\x3b\x34" + "\x91\x60\x14\xb9\xaa\x72\xfd\xa3\x64\xd2\x03\xa7\x53\x87\x9e" + "\x88\x0b\xc1\x14\x93\x1a\x62\xff\xb1\x5d\x74\xcd\x59\x63\x18" + "\x11\x3d\x4f\xba\x75\xd4\x33\x4e\x23\x6b\x7b\x57\x44\xe1\xd3" + "\x03\x13\xa6\xf0\x8b\x60\xb0\x9e\xee\x75\x08\x9d\x71\x63\x13" + "\xcb\xa6\x81\x92\x14\x03\x22\x2d\xde\x55"), + .length = 128 + }, + .exponent = { + .data = (uint8_t *)("\x01\x00\x01"), + .length = 3 + } + } + }; + /* Create asym crypto session and initialize it for the crypto device. */ + struct rte_cryptodev_asym_session *asym_session; + asym_session = rte_cryptodev_asym_session_create(asym_session_pool); + if (asym_session == NULL) + rte_exit(EXIT_FAILURE, "Session could not be created\n"); + + if (rte_cryptodev_asym_session_init(cdev_id, asym_session, + &modex_xform, asym_session_pool) < 0) + rte_exit(EXIT_FAILURE, "Session could not be initialized " + "for the crypto device\n"); + + /* Get a burst of crypto operations. */ + struct rte_crypto_op *crypto_ops[1]; + if (rte_crypto_op_bulk_alloc(crypto_op_pool, + RTE_CRYPTO_OP_TYPE_ASYMMETRIC, + crypto_ops, 1) == 0) + rte_exit(EXIT_FAILURE, "Not enough crypto operations available\n"); + + /* Set up the crypto operations. */ + struct rte_crypto_asym_op *asym_op = crypto_ops[0]->asym; + + /* calculate mod exp of value 0xf8 */ + static unsigned char base[] = {0xF8}; + asym_op->modex.base.data = base; + asym_op->modex.base.length = sizeof(base); + asym_op->modex.base.iova = base; + + /* Attach the asym crypto session to the operation */ + rte_crypto_op_attach_asym_session(op, asym_session); + + /* Enqueue the crypto operations in the crypto device. */ + uint16_t num_enqueued_ops = rte_cryptodev_enqueue_burst(cdev_id, 0, + crypto_ops, 1); + + /* + * Dequeue the crypto operations until all the operations + * are processed in the crypto device. + */ + uint16_t num_dequeued_ops, total_num_dequeued_ops = 0; + do { + struct rte_crypto_op *dequeued_ops[1]; + num_dequeued_ops = rte_cryptodev_dequeue_burst(cdev_id, 0, + dequeued_ops, 1); + total_num_dequeued_ops += num_dequeued_ops; + + /* Check if operation was processed successfully */ + if (dequeued_ops[0]->status != RTE_CRYPTO_OP_STATUS_SUCCESS) + rte_exit(EXIT_FAILURE, + "Some operations were not processed correctly"); + + } while (total_num_dequeued_ops < num_enqueued_ops); + + +Asymmetric Crypto Device API +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The cryptodev Library API is described in the *DPDK API Reference* document. +The cryptodev Library API is described in the +`DPDK API Reference <http://dpdk.org/doc/api/>`_ diff --git a/doc/guides/prog_guide/env_abstraction_layer.rst b/doc/guides/prog_guide/env_abstraction_layer.rst index 9a2fab1e..d362c920 100644 --- a/doc/guides/prog_guide/env_abstraction_layer.rst +++ b/doc/guides/prog_guide/env_abstraction_layer.rst @@ -94,10 +94,125 @@ The allocation of large contiguous physical memory is done using the hugetlbfs k The EAL provides an API to reserve named memory zones in this contiguous memory. The physical address of the reserved memory for that memory zone is also returned to the user by the memory zone reservation API. +There are two modes in which DPDK memory subsystem can operate: dynamic mode, +and legacy mode. Both modes are explained below. + .. note:: Memory reservations done using the APIs provided by rte_malloc are also backed by pages from the hugetlbfs filesystem. ++ Dynamic memory mode + +Currently, this mode is only supported on Linux. + +In this mode, usage of hugepages by DPDK application will grow and shrink based +on application's requests. Any memory allocation through ``rte_malloc()``, +``rte_memzone_reserve()`` or other methods, can potentially result in more +hugepages being reserved from the system. Similarly, any memory deallocation can +potentially result in hugepages being released back to the system. + +Memory allocated in this mode is not guaranteed to be IOVA-contiguous. If large +chunks of IOVA-contiguous are required (with "large" defined as "more than one +page"), it is recommended to either use VFIO driver for all physical devices (so +that IOVA and VA addresses can be the same, thereby bypassing physical addresses +entirely), or use legacy memory mode. + +For chunks of memory which must be IOVA-contiguous, it is recommended to use +``rte_memzone_reserve()`` function with ``RTE_MEMZONE_IOVA_CONTIG`` flag +specified. This way, memory allocator will ensure that, whatever memory mode is +in use, either reserved memory will satisfy the requirements, or the allocation +will fail. + +There is no need to preallocate any memory at startup using ``-m`` or +``--socket-mem`` command-line parameters, however it is still possible to do so, +in which case preallocate memory will be "pinned" (i.e. will never be released +by the application back to the system). It will be possible to allocate more +hugepages, and deallocate those, but any preallocated pages will not be freed. +If neither ``-m`` nor ``--socket-mem`` were specified, no memory will be +preallocated, and all memory will be allocated at runtime, as needed. + +Another available option to use in dynamic memory mode is +``--single-file-segments`` command-line option. This option will put pages in +single files (per memseg list), as opposed to creating a file per page. This is +normally not needed, but can be useful for use cases like userspace vhost, where +there is limited number of page file descriptors that can be passed to VirtIO. + +If the application (or DPDK-internal code, such as device drivers) wishes to +receive notifications about newly allocated memory, it is possible to register +for memory event callbacks via ``rte_mem_event_callback_register()`` function. +This will call a callback function any time DPDK's memory map has changed. + +If the application (or DPDK-internal code, such as device drivers) wishes to be +notified about memory allocations above specified threshold (and have a chance +to deny them), allocation validator callbacks are also available via +``rte_mem_alloc_validator_callback_register()`` function. + +A default validator callback is provided by EAL, which can be enabled with a +``--socket-limit`` command-line option, for a simple way to limit maximum amount +of memory that can be used by DPDK application. + +.. note:: + + In multiprocess scenario, all related processes (i.e. primary process, and + secondary processes running with the same prefix) must be in the same memory + modes. That is, if primary process is run in dynamic memory mode, all of its + secondary processes must be run in the same mode. The same is applicable to + ``--single-file-segments`` command-line option - both primary and secondary + processes must shared this mode. + ++ Legacy memory mode + +This mode is enabled by specifying ``--legacy-mem`` command-line switch to the +EAL. This switch will have no effect on FreeBSD as FreeBSD only supports +legacy mode anyway. + +This mode mimics historical behavior of EAL. That is, EAL will reserve all +memory at startup, sort all memory into large IOVA-contiguous chunks, and will +not allow acquiring or releasing hugepages from the system at runtime. + +If neither ``-m`` nor ``--socket-mem`` were specified, the entire available +hugepage memory will be preallocated. + ++ 32-bit support + +Additional restrictions are present when running in 32-bit mode. In dynamic +memory mode, by default maximum of 2 gigabytes of VA space will be preallocated, +and all of it will be on master lcore NUMA node unless ``--socket-mem`` flag is +used. + +In legacy mode, VA space will only be preallocated for segments that were +requested (plus padding, to keep IOVA-contiguousness). + ++ Maximum amount of memory + +All possible virtual memory space that can ever be used for hugepage mapping in +a DPDK process is preallocated at startup, thereby placing an upper limit on how +much memory a DPDK application can have. DPDK memory is stored in segment lists, +each segment is strictly one physical page. It is possible to change the amount +of virtual memory being preallocated at startup by editing the following config +variables: + +* ``CONFIG_RTE_MAX_MEMSEG_LISTS`` controls how many segment lists can DPDK have +* ``CONFIG_RTE_MAX_MEM_MB_PER_LIST`` controls how much megabytes of memory each + segment list can address +* ``CONFIG_RTE_MAX_MEMSEG_PER_LIST`` controls how many segments each segment can + have +* ``CONFIG_RTE_MAX_MEMSEG_PER_TYPE`` controls how many segments each memory type + can have (where "type" is defined as "page size + NUMA node" combination) +* ``CONFIG_RTE_MAX_MEM_MB_PER_TYPE`` controls how much megabytes of memory each + memory type can address +* ``CONFIG_RTE_MAX_MEM_MB`` places a global maximum on the amount of memory + DPDK can reserve + +Normally, these options do not need to be changed. + +.. note:: + + Preallocated virtual memory is not to be confused with preallocated hugepage + memory! All DPDK processes preallocate virtual memory at startup. Hugepages + can later be mapped into that preallocated VA space (if dynamic memory mode + is enabled), and can optionally be mapped into it at startup. + PCI Access ~~~~~~~~~~ @@ -211,7 +326,7 @@ Memory Segments and Memory Zones (memzone) The mapping of physical memory is provided by this feature in the EAL. As physical memory can have gaps, the memory is described in a table of descriptors, -and each descriptor (called rte_memseg ) describes a contiguous portion of memory. +and each descriptor (called rte_memseg ) describes a physical page. On top of this, the memzone allocator's role is to reserve contiguous portions of physical memory. These zones are identified by a unique name when the memory is reserved. @@ -225,6 +340,9 @@ Memory zones can be reserved with specific start address alignment by supplying The alignment value should be a power of two and not less than the cache line size (64 bytes). Memory zones can also be reserved from either 2 MB or 1 GB hugepages, provided that both are available on the system. +Both memsegs and memzones are stored using ``rte_fbarray`` structures. Please +refer to *DPDK API Reference* for more information. + Multiple pthread ---------------- @@ -331,17 +449,21 @@ Known Issues Bypassing this constraint may cause the 2nd pthread to spin until the 1st one is scheduled again. Moreover, if the 1st pthread is preempted by a context that has an higher priority, it may even cause a dead lock. - This does not mean it cannot be used, simply, there is a need to narrow down the situation when it is used by multi-pthread on the same core. + This means, use cases involving preemptible pthreads should consider using rte_ring carefully. + + 1. It CAN be used for preemptible single-producer and single-consumer use case. - 1. It CAN be used for any single-producer or single-consumer situation. + 2. It CAN be used for non-preemptible multi-producer and preemptible single-consumer use case. - 2. It MAY be used by multi-producer/consumer pthread whose scheduling policy are all SCHED_OTHER(cfs). User SHOULD be aware of the performance penalty before using it. + 3. It CAN be used for preemptible single-producer and non-preemptible multi-consumer use case. - 3. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + 4. It MAY be used by preemptible multi-producer and/or preemptible multi-consumer pthreads whose scheduling policy are all SCHED_OTHER(cfs), SCHED_IDLE or SCHED_BATCH. User SHOULD be aware of the performance penalty before using it. + + 5. It MUST not be used by multi-producer/consumer pthreads, whose scheduling policies are SCHED_FIFO or SCHED_RR. + rte_timer - Running ``rte_timer_manager()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed. + Running ``rte_timer_manage()`` on a non-EAL pthread is not allowed. However, resetting/stopping the timer from a non-EAL pthread is allowed. + rte_log @@ -453,11 +575,9 @@ The key fields of the heap structure and their function are described below * free_head - this points to the first element in the list of free nodes for this malloc heap. -.. note:: +* first - this points to the first element in the heap. - The malloc_heap structure does not keep track of in-use blocks of memory, - since these are never touched except when they are to be freed again - - at which point the pointer to the block is an input to the free() function. +* last - this points to the last element in the heap. .. _figure_malloc_heap: @@ -473,16 +593,21 @@ Structure: malloc_elem The malloc_elem structure is used as a generic header structure for various blocks of memory. -It is used in three different ways - all shown in the diagram above: +It is used in two different ways - all shown in the diagram above: #. As a header on a block of free or allocated memory - normal case #. As a padding header inside a block of memory -#. As an end-of-memseg marker - The most important fields in the structure and how they are used are described below. +Malloc heap is a doubly-linked list, where each element keeps track of its +previous and next elements. Due to the fact that hugepage memory can come and +go, neighbouring malloc elements may not necessarily be adjacent in memory. +Also, since a malloc element may span multiple pages, its contents may not +necessarily be IOVA-contiguous either - each malloc element is only guaranteed +to be virtually contiguous. + .. note:: If the usage of a particular field in one of the above three usages is not @@ -495,13 +620,20 @@ The most important fields in the structure and how they are used are described b It is used for normal memory blocks when they are being freed, to add the newly-freed block to the heap's free-list. -* prev - this pointer points to the header element/block in the memseg - immediately behind the current one. When freeing a block, this pointer is - used to reference the previous block to check if that block is also free. - If so, then the two free blocks are merged to form a single larger block. +* prev - this pointer points to previous header element/block in memory. When + freeing a block, this pointer is used to reference the previous block to + check if that block is also free. If so, and the two blocks are immediately + adjacent to each other, then the two free blocks are merged to form a single + larger block. -* next_free - this pointer is used to chain the free-list of unallocated - memory blocks together. +* next - this pointer points to next header element/block in memory. When + freeing a block, this pointer is used to reference the next block to check + if that block is also free. If so, and the two blocks are immediately + adjacent to each other, then the two free blocks are merged to form a single + larger block. + +* free_list - this is a structure pointing to previous and next elements in + this heap's free list. It is only used in normal memory blocks; on ``malloc()`` to find a suitable free block to allocate and on ``free()`` to add the newly freed element to the free-list. @@ -515,9 +647,6 @@ The most important fields in the structure and how they are used are described b constraints. In that case, the pad header is used to locate the actual malloc element header for the block. - For the end-of-memseg structure, this is always a ``BUSY`` value, which - ensures that no element, on being freed, searches beyond the end of the - memseg for other blocks to merge with into a larger free area. * pad - this holds the length of the padding present at the start of the block. In the case of a normal block header, it is added to the address of the end @@ -528,22 +657,19 @@ The most important fields in the structure and how they are used are described b actual block header. * size - the size of the data block, including the header itself. - For end-of-memseg structures, this size is given as zero, though it is never - actually checked. - For normal blocks which are being freed, this size value is used in place of - a "next" pointer to identify the location of the next block of memory that - in the case of being ``FREE``, the two free blocks can be merged into one. Memory Allocation ^^^^^^^^^^^^^^^^^ -On EAL initialization, all memsegs are setup as part of the malloc heap. -This setup involves placing a dummy structure at the end with ``BUSY`` state, -which may contain a sentinel value if ``CONFIG_RTE_MALLOC_DEBUG`` is enabled, -and a proper :ref:`element header<malloc_elem>` with ``FREE`` at the start -for each memseg. +On EAL initialization, all preallocated memory segments are setup as part of the +malloc heap. This setup involves placing an :ref:`element header<malloc_elem>` +with ``FREE`` at the start of each virtually contiguous segment of memory. The ``FREE`` element is then added to the ``free_list`` for the malloc heap. +This setup also happens whenever memory is allocated at runtime (if supported), +in which case newly allocated pages are also added to the heap, merging with any +adjacent free segments if there are any. + When an application makes a call to a malloc-like function, the malloc function will first index the ``lcore_config`` structure for the calling thread, and determine the NUMA node of that thread. @@ -574,8 +700,34 @@ the start and/or end of the element, resulting in the following behavior: The advantage of allocating the memory from the end of the existing element is that no adjustment of the free list needs to take place - the existing element -on the free list just has its size pointer adjusted, and the following element -has its "prev" pointer redirected to the newly created element. +on the free list just has its size value adjusted, and the next/previous elements +have their "prev"/"next" pointers redirected to the newly created element. + +In case when there is not enough memory in the heap to satisfy allocation +request, EAL will attempt to allocate more memory from the system (if supported) +and, following successful allocation, will retry reserving the memory again. In +a multiprocessing scenario, all primary and secondary processes will synchronize +their memory maps to ensure that any valid pointer to DPDK memory is guaranteed +to be valid at all times in all currently running processes. + +Failure to synchronize memory maps in one of the processes will cause allocation +to fail, even though some of the processes may have allocated the memory +successfully. The memory is not added to the malloc heap unless primary process +has ensured that all other processes have mapped this memory successfully. + +Any successful allocation event will trigger a callback, for which user +applications and other DPDK subsystems can register. Additionally, validation +callbacks will be triggered before allocation if the newly allocated memory will +exceed threshold set by the user, giving a chance to allow or deny allocation. + +.. note:: + + Any allocation of new pages has to go through primary process. If the + primary process is not active, no memory will be allocated even if it was + theoretically possible to do so. This is because primary's process map acts + as an authority on what should or should not be mapped, while each secondary + process has its own, local memory map. Secondary processes do not update the + shared memory map, they only copy its contents to their local memory map. Freeing Memory ^^^^^^^^^^^^^^ @@ -589,8 +741,17 @@ the pointer to get the proper element header for the entire block. From this element header, we get pointers to the heap from which the block was allocated and to where it must be freed, as well as the pointer to the previous -element, and via the size field, we can calculate the pointer to the next element. -These next and previous elements are then checked to see if they are also -``FREE``, and if so, they are merged with the current element. -This means that we can never have two ``FREE`` memory blocks adjacent to one -another, as they are always merged into a single block. +and next elements. These next and previous elements are then checked to see if +they are also ``FREE`` and are immediately adjacent to the current one, and if +so, they are merged with the current element. This means that we can never have +two ``FREE`` memory blocks adjacent to one another, as they are always merged +into a single block. + +If deallocating pages at runtime is supported, and the free element encloses +one or more pages, those pages can be deallocated and be removed from the heap. +If DPDK was started with command-line parameters for preallocating memory +(``-m`` or ``--socket-mem``), then those pages that were allocated at startup +will not be deallocated. + +Any successful deallocation event will trigger a callback, for which user +applications and other DPDK subsystems can register. diff --git a/doc/guides/prog_guide/event_crypto_adapter.rst b/doc/guides/prog_guide/event_crypto_adapter.rst new file mode 100644 index 00000000..9fe09c80 --- /dev/null +++ b/doc/guides/prog_guide/event_crypto_adapter.rst @@ -0,0 +1,296 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. All rights reserved. + +Event Crypto Adapter Library +============================ + +The DPDK :doc:`Eventdev library <eventdev>` provides event driven +programming model with features to schedule events. +The :doc:`Cryptodev library <cryptodev_lib>` provides an interface to +the crypto poll mode drivers which supports different crypto operations. +The Event Crypto Adapter is one of the adapter which is intended to +bridge between the event device and the crypto device. + +The packet flow from crypto device to the event device can be accomplished +using SW and HW based transfer mechanism. +The Adapter queries an eventdev PMD to determine which mechanism to be used. +The adapter uses an EAL service core function for SW based packet transfer +and uses the eventdev PMD functions to configure HW based packet transfer +between the crypto device and the event device. The crypto adapter uses a new +event type called ``RTE_EVENT_TYPE_CRYPTODEV`` to indicate the event source. + +The application can choose to submit a crypto operation directly to +crypto device or send it to the crypto adapter via eventdev based on +RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD capability. +The first mode is known as the event new(RTE_EVENT_CRYPTO_ADAPTER_OP_NEW) +mode and the second as the event forward(RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD) +mode. The choice of mode can be specified while creating the adapter. +In the former mode, it is an application responsibility to enable ingress +packet ordering. In the latter mode, it is the adapter responsibility to +enable the ingress packet ordering. + + +Adapter Mode +------------ + +RTE_EVENT_CRYPTO_ADAPTER_OP_NEW mode +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the RTE_EVENT_CRYPTO_ADAPTER_OP_NEW mode, application submits crypto +operations directly to crypto device. The adapter then dequeues crypto +completions from crypto device and enqueues them as events to the event device. +This mode does not ensure ingress ordering, if the application directly +enqueues to the cryptodev without going through crypto/atomic stage. +In this mode, events dequeued from the adapter will be treated as new events. +The application needs to specify event information (response information) +which is needed to enqueue an event after the crypto operation is completed. + +.. _figure_event_crypto_adapter_op_new: + +.. figure:: img/event_crypto_adapter_op_new.* + + Working model of ``RTE_EVENT_CRYPTO_ADAPTER_OP_NEW`` mode + + +RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode, if HW supports +RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_OP_FWD capability the application +can directly submit the crypto operations to the cryptodev. +If not, application retrieves crypto adapter's event port using +rte_event_crypto_adapter_event_port_get() API. Then, links its event +queue to this port and starts enqueuing crypto operations as events +to the eventdev. The adapter then dequeues the events and submits the +crypto operations to the cryptodev. After the crypto completions, the +adapter enqueues events to the event device. +Application can use this mode, when ingress packet ordering is needed. +In this mode, events dequeued from the adapter will be treated as +forwarded events. The application needs to specify the cryptodev ID +and queue pair ID (request information) needed to enqueue a crypto +operation in addition to the event information (response information) +needed to enqueue an event after the crypto operation has completed. + +.. _figure_event_crypto_adapter_op_forward: + +.. figure:: img/event_crypto_adapter_op_forward.* + + Working model of ``RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD`` mode + + +API Overview +------------ + +This section has a brief introduction to the event crypto adapter APIs. +The application is expected to create an adapter which is associated with +a single eventdev, then add cryptodev and queue pair to the adapter instance. + +Create an adapter instance +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An adapter instance is created using ``rte_event_crypto_adapter_create()``. This +function is called with event device to be associated with the adapter and port +configuration for the adapter to setup an event port(if the adapter needs to use +a service function). + +Adapter can be started in ``RTE_EVENT_CRYPTO_ADAPTER_OP_NEW`` or +``RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD`` mode. + +.. code-block:: c + + int err; + uint8_t dev_id, id; + struct rte_event_dev_info dev_info; + struct rte_event_port_conf conf; + enum rte_event_crypto_adapter_mode mode; + + err = rte_event_dev_info_get(id, &dev_info); + + conf.new_event_threshold = dev_info.max_num_events; + conf.dequeue_depth = dev_info.max_event_port_dequeue_depth; + conf.enqueue_depth = dev_info.max_event_port_enqueue_depth; + mode = RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD; + err = rte_event_crypto_adapter_create(id, dev_id, &conf, mode); + +If the application desires to have finer control of eventdev port allocation +and setup, it can use the ``rte_event_crypto_adapter_create_ext()`` function. +The ``rte_event_crypto_adapter_create_ext()`` function is passed as a callback +function. The callback function is invoked if the adapter needs to use a +service function and needs to create an event port for it. The callback is +expected to fill the ``struct rte_event_crypto_adapter_conf`` structure +passed to it. + +For RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode, the event port created by adapter +can be retrieved using ``rte_event_crypto_adapter_event_port_get()`` API. +Application can use this event port to link with event queue on which it +enqueues events towards the crypto adapter. + +.. code-block:: c + + uint8_t id, evdev, crypto_ev_port_id, app_qid; + struct rte_event ev; + int ret; + + ret = rte_event_crypto_adapter_event_port_get(id, &crypto_ev_port_id); + ret = rte_event_queue_setup(evdev, app_qid, NULL); + ret = rte_event_port_link(evdev, crypto_ev_port_id, &app_qid, NULL, 1); + + // Fill in event info and update event_ptr with rte_crypto_op + memset(&ev, 0, sizeof(ev)); + ev.queue_id = app_qid; + . + . + ev.event_ptr = op; + ret = rte_event_enqueue_burst(evdev, app_ev_port_id, ev, nb_events); + +Querying adapter capabilities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``rte_event_crypto_adapter_caps_get()`` function allows +the application to query the adapter capabilities for an eventdev and cryptodev +combination. This API provides whether cryptodev and eventdev are connected using +internal HW port or not. + +.. code-block:: c + + rte_event_crypto_adapter_caps_get(dev_id, cdev_id, &cap); + +Adding queue pair to the adapter instance +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Cryptodev device id and queue pair are created using cryptodev APIs. +For more information see :doc:`here <cryptodev_lib>`. + +.. code-block:: c + + struct rte_cryptodev_config conf; + struct rte_cryptodev_qp_conf qp_conf; + uint8_t cdev_id = 0; + uint16_t qp_id = 0; + + rte_cryptodev_configure(cdev_id, &conf); + rte_cryptodev_queue_pair_setup(cdev_id, qp_id, &qp_conf); + +These cryptodev id and queue pair are added to the instance using the +``rte_event_crypto_adapter_queue_pair_add()`` API. +The same is removed using ``rte_event_crypto_adapter_queue_pair_del()`` API. +If HW supports RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_QP_EV_BIND +capability, event information must be passed to the add API. + +.. code-block:: c + + uint32_t cap; + int ret; + + ret = rte_event_crypto_adapter_caps_get(id, evdev, &cap); + if (cap & RTE_EVENT_CRYPTO_ADAPTER_CAP_INTERNAL_PORT_QP_EV_BIND) { + struct rte_event event; + + // Fill in event information & pass it to add API + rte_event_crypto_adapter_queue_pair_add(id, cdev_id, qp_id, &event); + } else + rte_event_crypto_adapter_queue_pair_add(id, cdev_id, qp_id, NULL); + +Configure the service function +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the adapter uses a service function, the application is required to assign +a service core to the service function as show below. + +.. code-block:: c + + uint32_t service_id; + + if (rte_event_crypto_adapter_service_id_get(id, &service_id) == 0) + rte_service_map_lcore_set(service_id, CORE_ID); + +Set event request/response information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the RTE_EVENT_CRYPTO_ADAPTER_OP_FORWARD mode, the application needs +to specify the cryptodev ID and queue pair ID (request information) in +addition to the event information (response information) needed to enqueue +an event after the crypto operation has completed. The request and response +information are specified in the ``struct rte_crypto_op`` private data or +session's private data. + +In the RTE_EVENT_CRYPTO_ADAPTER_OP_NEW mode, the application is required +to provide only the response information. + +The SW adapter or HW PMD uses ``rte_crypto_op::sess_type`` to +decide whether request/response data is located in the crypto session/ +crypto security session or at an offset in the ``struct rte_crypto_op``. +The ``rte_crypto_op::private_data_offset`` is used to locate the request/ +response in the ``rte_crypto_op``. + +For crypto session, ``rte_cryptodev_sym_session_set_user_data()`` API +will be used to set request/response data. The same data will be obtained +by ``rte_cryptodev_sym_session_get_user_data()`` API. The +RTE_EVENT_CRYPTO_ADAPTER_CAP_SESSION_PRIVATE_DATA capability indicates +whether HW or SW supports this feature. + +For security session, ``rte_security_session_set_private_data()`` API +will be used to set request/response data. The same data will be obtained +by ``rte_security_session_get_private_data()`` API. + +For session-less it is mandatory to place the request/response data with +the ``rte_crypto_op``. + +.. code-block:: c + + union rte_event_crypto_metadata m_data; + struct rte_event ev; + struct rte_crypto_op *op; + + /* Allocate & fill op structure */ + op = rte_crypto_op_alloc(); + + memset(&m_data, 0, sizeof(m_data)); + memset(&ev, 0, sizeof(ev)); + /* Fill event information and update event_ptr to rte_crypto_op */ + ev.event_ptr = op; + + if (op->sess_type == RTE_CRYPTO_OP_WITH_SESSION) { + /* Copy response information */ + rte_memcpy(&m_data.response_info, &ev, sizeof(ev)); + /* Copy request information */ + m_data.request_info.cdev_id = cdev_id; + m_data.request_info.queue_pair_id = qp_id; + /* Call set API to store private data information */ + rte_cryptodev_sym_session_set_user_data( + op->sym->session, + &m_data, + sizeof(m_data)); + } if (op->sess_type == RTE_CRYPTO_OP_SESSIONLESS) { + uint32_t len = IV_OFFSET + MAXIMUM_IV_LENGTH + + (sizeof(struct rte_crypto_sym_xform) * 2); + op->private_data_offset = len; + /* Copy response information */ + rte_memcpy(&m_data.response_info, &ev, sizeof(ev)); + /* Copy request information */ + m_data.request_info.cdev_id = cdev_id; + m_data.request_info.queue_pair_id = qp_id; + /* Store private data information along with rte_crypto_op */ + rte_memcpy(op + len, &m_data, sizeof(m_data)); + } + +Start the adapter instance +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The application calls ``rte_event_crypto_adapter_start()`` to start the adapter. +This function calls the start callbacks of the eventdev PMDs for hardware based +eventdev-cryptodev connections and ``rte_service_run_state_set()`` to enable the +service function if one exists. + +.. code-block:: c + + rte_event_crypto_adapter_start(id, mode); + +Get adapter statistics +~~~~~~~~~~~~~~~~~~~~~~ + +The ``rte_event_crypto_adapter_stats_get()`` function reports counters defined +in struct ``rte_event_crypto_adapter_stats``. The received packet and +enqueued event counts are a sum of the counts from the eventdev PMD callbacks +if the callback is supported, and the counts maintained by the service function, +if one exists. diff --git a/doc/guides/prog_guide/event_ethernet_rx_adapter.rst b/doc/guides/prog_guide/event_ethernet_rx_adapter.rst index 4ab87a37..0166bb45 100644 --- a/doc/guides/prog_guide/event_ethernet_rx_adapter.rst +++ b/doc/guides/prog_guide/event_ethernet_rx_adapter.rst @@ -12,7 +12,11 @@ be supported in hardware or require a software thread to receive packets from the ethdev port using ethdev poll mode APIs and enqueue these as events to the event device using the eventdev API. Both transfer mechanisms may be present on the same platform depending on the particular combination of the ethdev and -the event device. +the event device. For SW based packet transfer, if the mbuf does not have a +timestamp set, the adapter adds a timestamp to the mbuf using +rte_get_tsc_cycles(), this provides a more accurate timestamp as compared to +if the application were to set the timestamp since it avoids event device +schedule latency. The Event Ethernet Rx Adapter library is intended for the application code to configure both transfer mechanisms using a common API. A capability API allows @@ -140,3 +144,44 @@ enqueued event counts are a sum of the counts from the eventdev PMD callbacks if the callback is supported, and the counts maintained by the service function, if one exists. The service function also maintains a count of cycles for which it was not able to enqueue to the event device. + +Interrupt Based Rx Queues +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The service core function is typically set up to poll ethernet Rx queues for +packets. Certain queues may have low packet rates and it would be more +efficient to enable the Rx queue interrupt and read packets after receiving +the interrupt. + +The servicing_weight member of struct rte_event_eth_rx_adapter_queue_conf +is applicable when the adapter uses a service core function. The application +has to enable Rx queue interrupts when configuring the ethernet device +using the ``rte_eth_dev_configure()`` function and then use a servicing_weight +of zero when addding the Rx queue to the adapter. + +The adapter creates a thread blocked on the interrupt, on an interrupt this +thread enqueues the port id and the queue id to a ring buffer. The adapter +service function dequeues the port id and queue id from the ring buffer, +invokes the ``rte_eth_rx_burst()`` to receive packets on the queue and +converts the received packets to events in the same manner as packets +received on a polled Rx queue. The interrupt thread is affinitized to the same +CPUs as the lcores of the Rx adapter service function, if the Rx adapter +service function has not been mapped to any lcores, the interrupt thread +is mapped to the master lcore. + +Rx Callback for SW Rx Adapter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For SW based packet transfers, i.e., when the +``RTE_EVENT_ETH_RX_ADAPTER_CAP_INTERNAL_PORT`` is not set in the adapter's +capabilities flags for a particular ethernet device, the service function +temporarily enqueues mbufs to an event buffer before batch enqueueing these +to the event device. If the buffer fills up, the service function stops +dequeueing packets from the ethernet device. The application may want to +monitor the buffer fill level and instruct the service function to selectively +enqueue packets to the event device. The application may also use some other +criteria to decide which packets should enter the event device even when +the event buffer fill level is low. The +``rte_event_eth_rx_adapter_cb_register()`` function allow the application +to register a callback that selects which packets to enqueue to the event +device. diff --git a/doc/guides/prog_guide/event_timer_adapter.rst b/doc/guides/prog_guide/event_timer_adapter.rst new file mode 100644 index 00000000..7bbbdfe9 --- /dev/null +++ b/doc/guides/prog_guide/event_timer_adapter.rst @@ -0,0 +1,296 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017 Intel Corporation. All rights reserved. + +Event Timer Adapter Library +=========================== + +The DPDK +`Event Device library <http://dpdk.org/doc/guides/prog_guide/eventdev.html>`_ +introduces an event driven programming model which presents applications with +an alternative to the polling model traditionally used in DPDK +applications. Event devices can be coupled with arbitrary components to provide +new event sources by using **event adapters**. The Event Timer Adapter is one +such adapter; it bridges event devices and timer mechanisms. + +The Event Timer Adapter library extends the event driven model +by introducing a :ref:`new type of event <timer_expiry_event>` that represents +a timer expiration, and providing an API with which adapters can be created or +destroyed, and :ref:`event timers <event_timer>` can be armed and canceled. + +The Event Timer Adapter library is designed to interface with hardware or +software implementations of the timer mechanism; it will query an eventdev PMD +to determine which implementation should be used. The default software +implementation manages timers using the DPDK +`Timer library <http://dpdk.org/doc/guides/prog_guide/timer_lib.html>`_. + +Examples of using the API are presented in the `API Overview`_ and +`Processing Timer Expiry Events`_ sections. Code samples are abstracted and +are based on the example of handling a TCP retransmission. + +.. _event_timer: + +Event Timer struct +------------------ +Event timers are timers that enqueue a timer expiration event to an event +device upon timer expiration. + +The Event Timer Adapter API represents each event timer with a generic struct, +which contains an event and user metadata. The ``rte_event_timer`` struct is +defined in ``lib/librte_event/librte_event_timer_adapter.h``. + +.. _timer_expiry_event: + +Timer Expiry Event +~~~~~~~~~~~~~~~~~~ + +The event contained by an event timer is enqueued in the event device when the +timer expires, and the event device uses the attributes below when scheduling +it: + +* ``event_queue_id`` - Application should set this to specify an event queue to + which the timer expiry event should be enqueued +* ``event_priority`` - Application can set this to indicate the priority of the + timer expiry event in the event queue relative to other events +* ``sched_type`` - Application can set this to specify the scheduling type of + the timer expiry event +* ``flow_id`` - Application can set this to indicate which flow this timer + expiry event corresponds to +* ``op`` - Will be set to ``RTE_EVENT_OP_NEW`` by the event timer adapter +* ``event_type`` - Will be set to ``RTE_EVENT_TYPE_TIMER`` by the event timer + adapter + +Timeout Ticks +~~~~~~~~~~~~~ + +The number of ticks from now in which the timer will expire. The ticks value +has a resolution (``timer_tick_ns``) that is specified in the event timer +adapter configuration. + +State +~~~~~ + +Before arming an event timer, the application should initialize its state to +RTE_EVENT_TIMER_NOT_ARMED. The event timer's state will be updated when a +request to arm or cancel it takes effect. + +If the application wishes to rearm the timer after it has expired, it should +reset the state back to RTE_EVENT_TIMER_NOT_ARMED before doing so. + +User Metadata +~~~~~~~~~~~~~ + +Memory to store user specific metadata. The event timer adapter implementation +will not modify this area. + +API Overview +------------ + +This section will introduce the reader to the event timer adapter API, showing +how to create and configure an event timer adapter and use it to manage event +timers. + +From a high level, the setup steps are: + +* rte_event_timer_adapter_create() +* rte_event_timer_adapter_start() + +And to start and stop timers: + +* rte_event_timer_arm_burst() +* rte_event_timer_cancel_burst() + +Create and Configure an Adapter Instance +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To create an event timer adapter instance, initialize an +``rte_event_timer_adapter_conf`` struct with the desired values, and pass it +to ``rte_event_timer_adapter_create()``. + +.. code-block:: c + + #define NSECPERSEC 1E9 // No of ns in 1 sec + const struct rte_event_timer_adapter_conf adapter_config = { + .event_dev_id = event_dev_id, + .timer_adapter_id = 0, + .clk_src = RTE_EVENT_TIMER_ADAPTER_CPU_CLK, + .timer_tick_ns = NSECPERSEC / 10, // 100 milliseconds + .max_tmo_nsec = 180 * NSECPERSEC // 2 minutes + .nb_timers = 40000, + .timer_adapter_flags = 0, + }; + + struct rte_event_timer_adapter *adapter = NULL; + adapter = rte_event_timer_adapter_create(&adapter_config); + + if (adapter == NULL) { ... }; + +Before creating an instance of a timer adapter, the application should create +and configure an event device along with its event ports. Based on the event +device capability, it might require creating an additional event port to be +used by the timer adapter. If required, the +``rte_event_timer_adapter_create()`` function will use a default method to +configure an event port; it will examine the current event device +configuration, determine the next available port identifier number, and create +a new event port with a default port configuration. + +If the application desires to have finer control of event port allocation +and setup, it can use the ``rte_event_timer_adapter_create_ext()`` function. +This function is passed a callback function that will be invoked if the +adapter needs to create an event port, giving the application the opportunity +to control how it is done. + +Retrieve Event Timer Adapter Contextual Information +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The event timer adapter implementation may have constraints on tick resolution +or maximum timer expiry timeout based on the given event timer adapter or +system. In this case, the implementation may adjust the tick resolution or +maximum timeout to the best possible configuration. + +Upon successful event timer adapter creation, the application can get the +configured resolution and max timeout with +``rte_event_timer_adapter_get_info()``. This function will return an +``rte_event_timer_adapter_info`` struct, which contains the following members: + +* ``min_resolution_ns`` - Minimum timer adapter tick resolution in ns. +* ``max_tmo_ns`` - Maximum timer timeout(expiry) in ns. +* ``adapter_conf`` - Configured event timer adapter attributes + +Configuring the Service Component +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the adapter uses a service component, the application is required to map +the service to a service core before starting the adapter: + +.. code-block:: c + + uint32_t service_id; + + if (rte_event_timer_adapter_service_id_get(adapter, &service_id) == 0) + rte_service_map_lcore_set(service_id, EVTIM_CORE_ID); + +An event timer adapter uses a service component if the event device PMD +indicates that the adapter should use a software implementation. + +Starting the Adapter Instance +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The application should call ``rte_event_timer_adapter_start()`` to start +running the event timer adapter. This function calls the start entry points +defined by eventdev PMDs for hardware implementations or puts a service +component into the running state in the software implementation. + +Arming Event Timers +~~~~~~~~~~~~~~~~~~~ + +Once an event timer adapter has been started, an application can begin to +manage event timers with it. + +The application should allocate ``struct rte_event_timer`` objects from a +mempool or huge-page backed application buffers of required size. Upon +successful allocation, the application should initialize the event timer, and +then set any of the necessary event attributes described in the +`Timer Expiry Event`_ section. In the following example, assume ``conn`` +represents a TCP connection and that ``event_timer_pool`` is a mempool that +was created previously: + +.. code-block:: c + + rte_mempool_get(event_timer_pool, (void **)&conn->evtim); + if (conn->evtim == NULL) { ... } + + /* Set up the event timer. */ + conn->evtim->ev.op = RTE_EVENT_OP_NEW; + conn->evtim->ev.queue_id = event_queue_id; + conn->evtim->ev.sched_type = RTE_SCHED_TYPE_ATOMIC; + conn->evtim->ev.priority = RTE_EVENT_DEV_PRIORITY_NORMAL; + conn->evtim->ev.event_type = RTE_EVENT_TYPE_TIMER; + conn->evtim->ev.event_ptr = conn; + conn->evtim->state = RTE_EVENT_TIMER_NOT_ARMED; + conn->evtim->timeout_ticks = 30; //3 sec Per RFC1122(TCP returns) + +Note that it is necessary to initialize the event timer state to +RTE_EVENT_TIMER_NOT_ARMED. Also note that we have saved a pointer to the +``conn`` object in the timer's event payload. This will allow us to locate +the connection object again once we dequeue the timer expiry event from the +event device later. As a convenience, the application may specify no value for +ev.event_ptr, and the adapter will by default set it to point at the event +timer itself. + +Now we can arm the event timer with ``rte_event_timer_arm_burst()``: + +.. code-block:: c + + ret = rte_event_timer_arm_burst(adapter, &conn->evtim, 1); + if (ret != 1) { ... } + +Once an event timer expires, the application may free it or rearm it as +necessary. If the application will rearm the timer, the state should be reset +to RTE_EVENT_TIMER_NOT_ARMED by the application before rearming it. + +Multiple Event Timers with Same Expiry Value +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In the special case that there is a set of event timers that should all expire +at the same time, the application may call +``rte_event_timer_arm_tmo_tick_burst()``, which allows the implementation to +optimize the operation if possible. + +Canceling Event Timers +~~~~~~~~~~~~~~~~~~~~~~ + +An event timer that has been armed as described in `Arming Event Timers`_ can +be canceled by calling ``rte_event_timer_cancel_burst()``: + +.. code-block:: c + + /* Ack for the previous tcp data packet has been received; + * cancel the retransmission timer + */ + rte_event_timer_cancel_burst(adapter, &conn->timer, 1); + +Processing Timer Expiry Events +------------------------------ + +Once an event timer has successfully enqueued a timer expiry event in the event +device, the application will subsequently dequeue it from the event device. +The application can use the event payload to retrieve a pointer to the object +associated with the event timer. It can then re-arm the event timer or free the +event timer object as desired: + +.. code-block:: c + + void + event_processing_loop(...) + { + while (...) { + /* Receive events from the configured event port. */ + rte_event_dequeue_burst(event_dev_id, event_port, &ev, 1, 0); + ... + switch(ev.event_type) { + ... + case RTE_EVENT_TYPE_TIMER: + process_timer_event(ev); + ... + break; + } + } + } + + uint8_t + process_timer_event(...) + { + /* A retransmission timeout for the connection has been received. */ + conn = ev.event_ptr; + /* Retransmit last packet (e.g. TCP segment). */ + ... + /* Re-arm timer using original values. */ + rte_event_timer_arm_burst(adapter_id, &conn->timer, 1); + } + +Summary +------- + +The Event Timer Adapter library extends the DPDK event-based programming model +by representing timer expirations as events in the system and allowing +applications to use existing event processing loops to arm and cancel event +timers or handle timer expiry events. diff --git a/doc/guides/prog_guide/eventdev.rst b/doc/guides/prog_guide/eventdev.rst index ce19997d..8fcae546 100644 --- a/doc/guides/prog_guide/eventdev.rst +++ b/doc/guides/prog_guide/eventdev.rst @@ -1,5 +1,6 @@ .. SPDX-License-Identifier: BSD-3-Clause Copyright(c) 2017 Intel Corporation. + Copyright(c) 2018 Arm Limited. Event Device Library ==================== @@ -129,8 +130,10 @@ API Walk-through This section will introduce the reader to the eventdev API, showing how to create and configure an eventdev and use it for a two-stage atomic pipeline -with a single core for TX. The diagram below shows the final state of the -application after this walk-through: +with one core each for RX and TX. RX and TX cores are shown here for +illustration, refer to Eventdev Adapter documentation for further details. +The diagram below shows the final state of the application after this +walk-through: .. _figure_eventdev-usage1: @@ -196,23 +199,29 @@ calling the setup function. Repeat this step for each queue, starting from .nb_atomic_flows = 1024, .nb_atomic_order_sequences = 1024, }; + struct rte_event_queue_conf single_link_conf = { + .event_queue_cfg = RTE_EVENT_QUEUE_CFG_SINGLE_LINK, + }; int dev_id = 0; - int queue_id = 0; - int err = rte_event_queue_setup(dev_id, queue_id, &atomic_conf); + int atomic_q_1 = 0; + int atomic_q_2 = 1; + int single_link_q = 2; + int err = rte_event_queue_setup(dev_id, atomic_q_1, &atomic_conf); + int err = rte_event_queue_setup(dev_id, atomic_q_2, &atomic_conf); + int err = rte_event_queue_setup(dev_id, single_link_q, &single_link_conf); -The remainder of this walk-through assumes that the queues are configured as -follows: +As shown above, queue IDs are as follows: * id 0, atomic queue #1 * id 1, atomic queue #2 * id 2, single-link queue +These queues are used for the remainder of this walk-through. + Setting up Ports ~~~~~~~~~~~~~~~~ -Once queues are set up successfully, create the ports as required. Each port -should be set up with its corresponding port_conf type, worker for worker cores, -rx and tx for the RX and TX cores: +Once queues are set up successfully, create the ports as required. .. code-block:: c @@ -232,15 +241,24 @@ rx and tx for the RX and TX cores: .new_event_threshold = 4096, }; int dev_id = 0; - int port_id = 0; - int err = rte_event_port_setup(dev_id, port_id, &CORE_FUNCTION_conf); + int rx_port_id = 0; + int err = rte_event_port_setup(dev_id, rx_port_id, &rx_conf); + + for(int worker_port_id = 1; worker_port_id <= 4; worker_port_id++) { + int err = rte_event_port_setup(dev_id, worker_port_id, &worker_conf); + } -It is now assumed that: + int tx_port_id = 5; + int err = rte_event_port_setup(dev_id, tx_port_id, &tx_conf); + +As shown above: * port 0: RX core * ports 1,2,3,4: Workers * port 5: TX core +These ports are used for the remainder of this walk-through. + Linking Queues and Ports ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -254,15 +272,14 @@ can be achieved like this: .. code-block:: c - uint8_t port_id = 0; + uint8_t rx_port_id = 0; + uint8_t tx_port_id = 5; uint8_t atomic_qs[] = {0, 1}; uint8_t single_link_q = 2; - uint8_t tx_port_id = 5; uin8t_t priority = RTE_EVENT_DEV_PRIORITY_NORMAL; - for(int i = 0; i < 4; i++) { - int worker_port = i + 1; - int links_made = rte_event_port_link(dev_id, worker_port, atomic_qs, NULL, 2); + for(int worker_port_id = 1; worker_port_id <= 4; worker_port_id++) { + int links_made = rte_event_port_link(dev_id, worker_port_id, atomic_qs, NULL, 2); } int links_made = rte_event_port_link(dev_id, tx_port_id, &single_link_q, &priority, 1); @@ -295,14 +312,14 @@ The following code shows how those packets can be enqueued into the eventdev: ev[i].flow_id = mbufs[i]->hash.rss; ev[i].op = RTE_EVENT_OP_NEW; ev[i].sched_type = RTE_SCHED_TYPE_ATOMIC; - ev[i].queue_id = 0; + ev[i].queue_id = atomic_q_1; ev[i].event_type = RTE_EVENT_TYPE_ETHDEV; ev[i].sub_event_type = 0; ev[i].priority = RTE_EVENT_DEV_PRIORITY_NORMAL; ev[i].mbuf = mbufs[i]; } - const int nb_tx = rte_event_enqueue_burst(dev_id, port_id, ev, nb_rx); + const int nb_tx = rte_event_enqueue_burst(dev_id, rx_port_id, ev, nb_rx); if (nb_tx != nb_rx) { for(i = nb_tx; i < nb_rx; i++) rte_pktmbuf_free(mbufs[i]); @@ -334,7 +351,7 @@ the event to the next stage in the pipeline. events[i].queue_id++; } - uint16_t nb_tx = rte_event_enqueue_burst(dev_id, port_id, events, nb_rx); + uint16_t nb_tx = rte_event_enqueue_burst(dev_id, worker_port_id, events, nb_rx); Egress of Events diff --git a/doc/guides/prog_guide/generic_segmentation_offload_lib.rst b/doc/guides/prog_guide/generic_segmentation_offload_lib.rst index 9959f0d2..0cfc1198 100644 --- a/doc/guides/prog_guide/generic_segmentation_offload_lib.rst +++ b/doc/guides/prog_guide/generic_segmentation_offload_lib.rst @@ -43,6 +43,7 @@ Limitations #. Currently, the GSO library supports the following IPv4 packet types: - TCP + - UDP - VxLAN - GRE @@ -146,6 +147,15 @@ TCP/IPv4 GSO TCP/IPv4 GSO supports segmentation of suitably large TCP/IPv4 packets, which may also contain an optional VLAN tag. +UDP/IPv4 GSO +~~~~~~~~~~~~ +UDP/IPv4 GSO supports segmentation of suitably large UDP/IPv4 packets, which +may also contain an optional VLAN tag. UDP GSO is the same as IP fragmentation. +Specifically, UDP GSO treats the UDP header as a part of the payload and +does not modify it during segmentation. Therefore, after UDP GSO, only the +first output packet has the original UDP header, and others just have l2 +and l3 headers. + VxLAN GSO ~~~~~~~~~ VxLAN packets GSO supports segmentation of suitably large VxLAN packets, diff --git a/doc/guides/prog_guide/glossary.rst b/doc/guides/prog_guide/glossary.rst index e101bc02..dda45bd1 100644 --- a/doc/guides/prog_guide/glossary.rst +++ b/doc/guides/prog_guide/glossary.rst @@ -41,9 +41,6 @@ CPU CRC Cyclic Redundancy Check -ctrlmbuf - An *mbuf* carrying control data. - Data Plane In contrast to the control plane, the data plane in a network architecture are the layers involved when forwarding packets. These layers must be diff --git a/doc/guides/prog_guide/hash_lib.rst b/doc/guides/prog_guide/hash_lib.rst index 180c776e..76a1f323 100644 --- a/doc/guides/prog_guide/hash_lib.rst +++ b/doc/guides/prog_guide/hash_lib.rst @@ -19,6 +19,8 @@ The main configuration parameters for the hash are: * Size of the key in bytes +* An extra flag used to describe additional settings, for example the multithreading mode of operation (as will be described later) + The hash also allows the configuration of some low-level implementation related parameters such as: * Hash function to translate the key into a bucket index @@ -49,8 +51,7 @@ Apart from these method explained above, the API allows the user three more opti Also, the API contains a method to allow the user to look up entries in bursts, achieving higher performance than looking up individual entries, as the function prefetches next entries at the time it is operating with the first ones, which reduces significantly the impact of the necessary memory accesses. -Notice that this method uses a pipeline of 8 entries (4 stages of 2 entries), so it is highly recommended -to use at least 8 entries per burst. + The actual data associated with each key can be either managed by the user using a separate table that mirrors the hash in terms of number of entries and position of each entry, @@ -63,11 +64,33 @@ However, this table could also be used for more sophisticated features and provi Multi-process support --------------------- -The hash library can be used in a multi-process environment, minding that only lookups are thread-safe. +The hash library can be used in a multi-process environment. The only function that can only be used in single-process mode is rte_hash_set_cmp_func(), which sets up a custom compare function, which is assigned to a function pointer (therefore, it is not supported in multi-process mode). + +Multi-thread support +--------------------- + +The hash library supports multithreading, and the user specifies the needed mode of operation at the creation time of the hash table +by appropriately setting the flag. In all modes of operation lookups are thread-safe meaning lookups can be called from multiple +threads concurrently. + +For concurrent writes, and concurrent reads and writes the following flag values define the corresponding modes of operation: + +* If the multi-writer flag (RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD) is set, multiple threads writing to the table is allowed. + Key add, delete, and table reset are protected from other writer threads. With only this flag set, readers are not protected from ongoing writes. + +* If the read/write concurrency (RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY) is set, multithread read/write operation is safe + (i.e., no need to stop the readers from accessing the hash table until writers finish their updates. Reads and writes can operate table concurrently). + +* In addition to these two flag values, if the transactional memory flag (RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) is also set, + hardware transactional memory will be used to guarantee the thread safety as long as it is supported by the hardware (for example the Intel® TSX support). + +If the platform supports Intel® TSX, it is advised to set the transactional memory flag, as this will speed up concurrent table operations. +Otherwise concurrent operations will be slower because of the overhead associated with the software locking mechanisms. + Implementation Details ---------------------- diff --git a/doc/guides/prog_guide/img/event_crypto_adapter_op_forward.svg b/doc/guides/prog_guide/img/event_crypto_adapter_op_forward.svg new file mode 100644 index 00000000..54466f2e --- /dev/null +++ b/doc/guides/prog_guide/img/event_crypto_adapter_op_forward.svg @@ -0,0 +1,1078 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="720px" + height="486px" + id="svg13237" + version="1.1" + inkscape:version="0.48.4 r9939" + sodipodi:docname="event_crypto_adapter_enq_deq.svg"> + <defs + id="defs13239"> + <marker + inkscape:stockid="Arrow1Sstart" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Sstart" + style="overflow:visible"> + <path + id="path8416" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.2) translate(6,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Send" + style="overflow:visible;"> + <path + id="path8419" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" + transform="scale(0.2) rotate(180) translate(6,0)" /> + </marker> + <marker + inkscape:stockid="DiamondL" + orient="auto" + refY="0.0" + refX="0.0" + id="DiamondL" + style="overflow:visible"> + <path + id="path8483" + d="M 0,-7.0710768 L -7.0710894,0 L 0,7.0710589 L 7.0710462,0 L 0,-7.0710768 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8)" /> + </marker> + <marker + inkscape:stockid="DotL" + orient="auto" + refY="0.0" + refX="0.0" + id="DotL" + style="overflow:visible"> + <path + id="path8465" + d="M -2.5,-1.0 C -2.5,1.7600000 -4.7400000,4.0 -7.5,4.0 C -10.260000,4.0 -12.5,1.7600000 -12.5,-1.0 C -12.5,-3.7600000 -10.260000,-6.0 -7.5,-6.0 C -4.7400000,-6.0 -2.5,-3.7600000 -2.5,-1.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8) translate(7.4, 1)" /> + </marker> + <marker + inkscape:stockid="SquareL" + orient="auto" + refY="0.0" + refX="0.0" + id="SquareL" + style="overflow:visible"> + <path + id="path8474" + d="M -5.0,-5.0 L -5.0,5.0 L 5.0,5.0 L 5.0,-5.0 L -5.0,-5.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8)" /> + </marker> + <marker + inkscape:stockid="TriangleOutL" + orient="auto" + refY="0.0" + refX="0.0" + id="TriangleOutL" + style="overflow:visible"> + <path + id="path8546" + d="M 5.77,0.0 L -2.88,5.0 L -2.88,-5.0 L 5.77,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Lstart" + style="overflow:visible"> + <path + id="path8404" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8) translate(12.5,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Mend" + style="overflow:visible;"> + <path + id="path8413" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" + transform="scale(0.4) rotate(180) translate(10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow2Lend" + style="overflow:visible;"> + <path + id="path8425" + style="fill-rule:evenodd;stroke-width:0.62500000;stroke-linejoin:round;" + d="M 8.7185878,4.0337352 L -2.2072895,0.016013256 L 8.7185884,-4.0017078 C 6.9730900,-1.6296469 6.9831476,1.6157441 8.7185878,4.0337352 z " + transform="scale(1.1) rotate(180) translate(1,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Lend" + style="overflow:visible;"> + <path + id="path8407" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" + transform="scale(0.8) rotate(180) translate(12.5,0)" /> + </marker> + <filter + id="filter_2" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15" /> + </filter> + <filter + id="filter_2-3" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1" /> + </filter> + <filter + id="filter_2-0" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-7" /> + </filter> + <filter + id="filter_2-0-8" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-7-7" /> + </filter> + <filter + id="filter_2-3-9" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-6" /> + </filter> + <filter + id="filter_2-3-6" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-63" /> + </filter> + <filter + id="filter_2-3-91" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-3" /> + </filter> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-5" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-7" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8404-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-51" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-3" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-6" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-62" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-9" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-2" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-7" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-7-9" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8404-0-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <filter + id="filter_2-3-6-1" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-63-8" /> + </filter> + <filter + id="filter_2-3-92" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-2" /> + </filter> + <filter + id="filter_2-3-94" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-7" /> + </filter> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-7-6" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8404-0-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-55" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-4" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1" + inkscape:cx="359.77003" + inkscape:cy="287.74194" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:window-width="1200" + inkscape:window-height="898" + inkscape:window-x="0" + inkscape:window-y="31" + inkscape:window-maximized="1" + inkscape:snap-nodes="false"> + <inkscape:grid + type="xygrid" + id="grid13454" /> + </sodipodi:namedview> + <metadata + id="metadata13242"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title /> + </cc:Work> + </rdf:RDF> + </metadata> + <g + id="layer1" + inkscape:label="Layer 1" + inkscape:groupmode="layer"> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-4" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,88.874699,-812.39909)"> + <title + id="title22-7-5">Square</title> + <desc + id="desc24-7-8">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-5" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-7" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-91)" /> + </g> + <g + id="g13515-33"> + <g + id="g13534-8"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-95" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.71226478;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Lstart-7);marker-end:none" + d="m 312.28671,240.74335 -84.28774,0" + id="path17209" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.71898615px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-mid:none;marker-end:url(#Arrow1Lend)" + d="m 221.6484,77.57125 94.28101,0" + id="path17209-8" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,314.24227,-811.89589)"> + <title + id="title22-7">Square</title> + <desc + id="desc24-7">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3)" /> + </g> + <g + id="g13515"> + <g + id="g13534"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.72471404;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Lstart);marker-end:none" + d="m 89.025329,74.39932 -64.275286,0" + id="path17209-3" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-4.325033,28.642983)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-3" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-1.93108,192.80833)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-1" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.75141162;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Lstart-7);marker-end:none" + d="m 18.763392,120.7432 68.995153,0" + id="path17209-3-0" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" + transform="matrix(0.73232502,0,0,0.75477602,-218.16394,72.68276)" /> + <path + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-2" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" + transform="matrix(0.73232502,0,0,0.75477602,-217.40136,26.716271)" /> + <g + id="g29167-4" + transform="matrix(0.73232502,0,0,0.75477602,-217.31662,28.007562)"> + <text + sodipodi:linespacing="125%" + id="text29163-9" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-9" + sodipodi:role="line">1</tspan></text> + </g> + <g + id="g29167-9" + transform="matrix(0.73232502,0,0,0.75477602,-4.9726112,28.689051)"> + <text + sodipodi:linespacing="125%" + id="text29163-3" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-3" + sodipodi:role="line">2</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.67803264px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-7);marker-end:none" + d="m 181,214.66098 0,-69.32196" + id="path17211-7-1-6" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <g + id="g29167" + transform="matrix(0.73232502,0,0,0.75477602,-218.07919,73.10621)"> + <text + sodipodi:linespacing="125%" + id="text29163" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165" + sodipodi:role="line">8</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.67803264px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#Arrow1Lstart-7);marker-end:none" + d="m 131,145.8531 0,69.32197" + id="path17211-7-1" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-140.37076,129.97088)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-8" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <g + id="g29167-2" + transform="matrix(0.73232502,0,0,0.75477602,-140.28602,131.01695)"> + <text + sodipodi:linespacing="125%" + id="text29163-92" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-8" + sodipodi:role="line">7</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.71898615px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-mid:none;marker-end:url(#Arrow1Lend)" + d="m 317.1405,116 -94.281,0" + id="path17209-8-0" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-3.4914,66.68745)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-6" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <g + id="g29167-46" + transform="matrix(0.73232502,0,0,0.75477602,-4.40666,67.48829)"> + <text + sodipodi:linespacing="125%" + id="text29163-1" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-5" + sodipodi:role="line">3</tspan></text> + </g> + <path + transform="matrix(0.73232502,0,0,0.75477602,-90.692582,130.31695)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-8-6" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <g + id="g29167-6" + transform="matrix(0.73232502,0,0,0.75477602,-90.84634,131.60918)"> + <text + sodipodi:linespacing="125%" + id="text29163-17" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-2" + sodipodi:role="line">4</tspan></text> + </g> + <g + id="g29167-2-0" + transform="matrix(0.73232502,0,0,0.75477602,-2.424397,194.0216)"> + <text + sodipodi:linespacing="125%" + id="text29163-92-6" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-8-2" + sodipodi:role="line">5</tspan></text> + </g> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-8" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,93.82055,-648.98949)"> + <title + id="title22-7-97">Square</title> + <desc + id="desc24-7-3">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-6" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-12" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-92)" /> + </g> + <g + id="g13515-9"> + <g + id="g13534-3"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-1" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-84" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,314.82055,-648.98949)"> + <title + id="title22-7-50">Square</title> + <desc + id="desc24-7-36">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-1" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-0" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-94)" /> + </g> + <g + id="g13515-6"> + <g + id="g13534-32"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-0" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.71226478;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:none;marker-end:url(#Arrow1Lend)" + d="m 313.14387,285 -84.28774,0" + id="path17209-7" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-2.692582,236.31695)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-1-6" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <g + id="g29167-2-0-5" + transform="matrix(0.73232502,0,0,0.75477602,-2.424397,237.0216)"> + <text + sodipodi:linespacing="125%" + id="text29163-92-6-6" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-8-2-9" + sodipodi:role="line">6</tspan></text> + </g> + <g + id="g29167-4-3" + transform="matrix(0.73232502,0,0,0.75477602,-154.60784,51.117791)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-9-7" + sodipodi:role="line">Eventdev</tspan></text> + </g> + <g + id="g29167-4-3-5" + transform="matrix(0.73232502,0,0,0.75477602,-144.65044,201.97821)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-3" + y="70" + x="412.93716" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="412.93716" + id="tspan29165-9-7-5" + sodipodi:role="line">Crypto</tspan><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="100.26366" + x="412.93716" + sodipodi:role="line" + id="tspan3201">Adapter</tspan></text> + </g> + <g + id="g29167-4-3-5-6" + transform="matrix(0.73232502,0,0,0.75477602,79.53518,46.62529)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-3-2" + y="48.801659" + x="412.93716" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="48.801659" + x="412.93716" + sodipodi:role="line" + id="tspan3155">Application</tspan><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="79.065323" + x="412.93716" + sodipodi:role="line" + id="tspan3201-1">in ordered</tspan><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="109.32899" + x="412.93716" + sodipodi:role="line" + id="tspan3161">stage</tspan></text> + </g> + <g + id="g29167-4-3-5-2" + transform="matrix(0.73232502,0,0,0.75477602,77.535182,213.62529)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-3-7" + y="70" + x="412.93716" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="412.93716" + sodipodi:role="line" + id="tspan3201-9">Cryptodev</tspan></text> + </g> + <g + id="g29167-4-3-5-3" + transform="matrix(0.73232502,0,0,0.75477602,188.53518,-3.37471)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-3-6" + y="70" + x="375.65271" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="375.65271" + sodipodi:role="line" + id="tspan3201-6">1. Events from the previous stage.</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="93.538406" + x="375.65271" + sodipodi:role="line" + id="tspan3260" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="117.07681" + x="375.65271" + sodipodi:role="line" + id="tspan3262">2. Application in ordered stage</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="140.61522" + x="375.65271" + sodipodi:role="line" + id="tspan3288"> dequeues events from eventdev.</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="164.15363" + x="375.65271" + sodipodi:role="line" + id="tspan3264" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="187.69203" + x="375.65271" + sodipodi:role="line" + id="tspan3266">3. Application enqueues crypto</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="211.23044" + x="375.65271" + sodipodi:role="line" + id="tspan3290"> operations as events to eventdev.</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="234.76884" + x="375.65271" + sodipodi:role="line" + id="tspan3268" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="258.30725" + x="375.65271" + sodipodi:role="line" + id="tspan3270">4. Crypto adapter dequeues event</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="281.84564" + x="375.65271" + sodipodi:role="line" + id="tspan3292"> from eventdev.</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="305.38406" + x="375.65271" + sodipodi:role="line" + id="tspan3272" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="328.92245" + x="375.65271" + sodipodi:role="line" + id="tspan3274">5. Crypto adapter submits crypto</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="352.46088" + x="375.65271" + sodipodi:role="line" + id="tspan3294"> operations to cryptodev (Atomic</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="375.99927" + x="375.65271" + sodipodi:role="line" + id="tspan3296"> stage)</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="399.53769" + x="375.65271" + sodipodi:role="line" + id="tspan3276" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="423.07608" + x="375.65271" + sodipodi:role="line" + id="tspan3278">6. Crypto adapter dequeues crypto</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="446.6145" + x="375.65271" + sodipodi:role="line" + id="tspan3298"> completions from cryptodev</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="470.15289" + x="375.65271" + sodipodi:role="line" + id="tspan3280" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="493.69131" + x="375.65271" + sodipodi:role="line" + id="tspan3282">7. Crypto adapter enqueues events</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="517.22974" + x="375.65271" + sodipodi:role="line" + id="tspan3300"> to the eventdev</tspan><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="540.76813" + x="375.65271" + sodipodi:role="line" + id="tspan3284" /><tspan + style="font-size:18.83072472px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="564.30652" + x="375.65271" + sodipodi:role="line" + id="tspan3286">8. Events to the next stage</tspan></text> + </g> + </g> +</svg> diff --git a/doc/guides/prog_guide/img/event_crypto_adapter_op_new.svg b/doc/guides/prog_guide/img/event_crypto_adapter_op_new.svg new file mode 100644 index 00000000..256862ed --- /dev/null +++ b/doc/guides/prog_guide/img/event_crypto_adapter_op_new.svg @@ -0,0 +1,1061 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<svg + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="720px" + height="486px" + id="svg13237" + version="1.1" + inkscape:version="0.48.4 r9939" + sodipodi:docname="event_crypto_adapter_deq_only.svg"> + <defs + id="defs13239"> + <marker + inkscape:stockid="Arrow1Sstart" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Sstart" + style="overflow:visible"> + <path + id="path8416" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.2) translate(6,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Send" + style="overflow:visible;"> + <path + id="path8419" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" + transform="scale(0.2) rotate(180) translate(6,0)" /> + </marker> + <marker + inkscape:stockid="DiamondL" + orient="auto" + refY="0.0" + refX="0.0" + id="DiamondL" + style="overflow:visible"> + <path + id="path8483" + d="M 0,-7.0710768 L -7.0710894,0 L 0,7.0710589 L 7.0710462,0 L 0,-7.0710768 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8)" /> + </marker> + <marker + inkscape:stockid="DotL" + orient="auto" + refY="0.0" + refX="0.0" + id="DotL" + style="overflow:visible"> + <path + id="path8465" + d="M -2.5,-1.0 C -2.5,1.7600000 -4.7400000,4.0 -7.5,4.0 C -10.260000,4.0 -12.5,1.7600000 -12.5,-1.0 C -12.5,-3.7600000 -10.260000,-6.0 -7.5,-6.0 C -4.7400000,-6.0 -2.5,-3.7600000 -2.5,-1.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8) translate(7.4, 1)" /> + </marker> + <marker + inkscape:stockid="SquareL" + orient="auto" + refY="0.0" + refX="0.0" + id="SquareL" + style="overflow:visible"> + <path + id="path8474" + d="M -5.0,-5.0 L -5.0,5.0 L 5.0,5.0 L 5.0,-5.0 L -5.0,-5.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8)" /> + </marker> + <marker + inkscape:stockid="TriangleOutL" + orient="auto" + refY="0.0" + refX="0.0" + id="TriangleOutL" + style="overflow:visible"> + <path + id="path8546" + d="M 5.77,0.0 L -2.88,5.0 L -2.88,-5.0 L 5.77,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Lstart" + style="overflow:visible"> + <path + id="path8404" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" + transform="scale(0.8) translate(12.5,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Mend" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Mend" + style="overflow:visible;"> + <path + id="path8413" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" + transform="scale(0.4) rotate(180) translate(10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow2Lend" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow2Lend" + style="overflow:visible;"> + <path + id="path8425" + style="fill-rule:evenodd;stroke-width:0.62500000;stroke-linejoin:round;" + d="M 8.7185878,4.0337352 L -2.2072895,0.016013256 L 8.7185884,-4.0017078 C 6.9730900,-1.6296469 6.9831476,1.6157441 8.7185878,4.0337352 z " + transform="scale(1.1) rotate(180) translate(1,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0.0" + refX="0.0" + id="Arrow1Lend" + style="overflow:visible;"> + <path + id="path8407" + d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " + style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" + transform="scale(0.8) rotate(180) translate(12.5,0)" /> + </marker> + <filter + id="filter_2" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15" /> + </filter> + <filter + id="filter_2-3" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1" /> + </filter> + <filter + id="filter_2-0" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-7" /> + </filter> + <filter + id="filter_2-0-8" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-7-7" /> + </filter> + <filter + id="filter_2-3-9" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-6" /> + </filter> + <filter + id="filter_2-3-6" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-63" /> + </filter> + <filter + id="filter_2-3-91" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-3" /> + </filter> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-5" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-3" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-6" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart-7" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8404-0" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(0.8,0,0,0.8,10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-51" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-1" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-3" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-6" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-62" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-9" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend-2" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path8407-7" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" /> + </marker> + <filter + id="filter_2-3-91-3" + color-interpolation-filters="sRGB"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur15-1-3-6" /> + </filter> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1" + inkscape:cx="511.66308" + inkscape:cy="233.69667" + inkscape:document-units="px" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:window-width="1200" + inkscape:window-height="898" + inkscape:window-x="0" + inkscape:window-y="31" + inkscape:window-maximized="1" + inkscape:snap-nodes="false"> + <inkscape:grid + type="xygrid" + id="grid13454" /> + </sodipodi:namedview> + <metadata + id="metadata13242"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title /> + </cc:Work> + </rdf:RDF> + </metadata> + <g + id="layer1" + inkscape:label="Layer 1" + inkscape:groupmode="layer"> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-0" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,323.2187,-540.25927)"> + <title + id="title22-7-6">Square</title> + <desc + id="desc24-7-4">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-0" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-9" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-9)" /> + </g> + <g + id="g13515-4"> + <g + id="g13534-5"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-4" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-4" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.165886,88.874699,-447.8809)"> + <title + id="title22-7-5">Square</title> + <desc + id="desc24-7-8">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-5" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-7" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-91)" /> + </g> + <g + id="g13515-33"> + <g + id="g13534-8"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-95" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-9" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,88.874699,-538.24651)"> + <title + id="title22-7-9">Square</title> + <desc + id="desc24-7-5">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-2" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-1" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-6)" /> + </g> + <g + id="g13515-3"> + <g + id="g13534-0"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-9" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.74346578px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-mid:none;marker-end:url(#Arrow1Lend)" + d="m 220.66064,98.57125 101.22528,0" + id="path17209-8" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.5671361,322.24227,-811.89589)"> + <title + id="title22-7">Square</title> + <desc + id="desc24-7">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3)" /> + </g> + <g + id="g13515"> + <g + id="g13534"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <g + id="g13518" + transform="matrix(0.73232502,0,0,0.75477602,25.29268,348.89752)"> + <g + id="g13526"> + <flowRoot + xml:space="preserve" + id="flowRoot13464-9" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + transform="translate(-12.00521,-129.65179)"><flowRegion + id="flowRegion13466-1"><rect + id="rect13468-2" + width="195.99997" + height="112.00001" + x="273.33334" + y="175.33333" + style="text-align:center;text-anchor:middle" /></flowRegion><flowPara + style="font-size:20px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + id="flowPara13511" /></flowRoot> </g> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:0.75145501;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Lstart);marker-end:none" + d="m 176.26096,124.64833 0,69.24854" + id="path17209-3" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-4.325033,50.642983)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-3" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.74346578px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-mid:none;marker-end:url(#Arrow1Lend)" + d="m 322.61264,375 -101.22528,0" + id="path17209-8-0" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,0.0689171,324.80833)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-1" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.62908167px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)" + d="m 155,324.19955 0,-59.37092" + id="path17211-7-1" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,-116.37076,245.97088)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-8" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <path + style="fill:none;stroke:#000000;stroke-width:0.75058991;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow1Lstart-7);marker-end:none" + d="m 126.26097,124.99178 0,69.24941" + id="path17209-3-0" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" + transform="matrix(0.73232502,0,0,0.75477602,-146.16394,110.68276)" /> + <path + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-2" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" + transform="matrix(0.73232502,0,0,0.75477602,-95.40136,110.71627)" /> + <g + id="g29167-4" + transform="matrix(0.73232502,0,0,0.75477602,-95.31662,112.00756)"> + <text + sodipodi:linespacing="125%" + id="text29163-9" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-9" + sodipodi:role="line">1</tspan></text> + </g> + <g + id="g29167-9" + transform="matrix(0.73232502,0,0,0.75477602,-4.9726112,50.689051)"> + <text + sodipodi:linespacing="125%" + id="text29163-3" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-3" + sodipodi:role="line">2</tspan></text> + </g> + <path + style="fill:none;stroke:#000000;stroke-width:1.04032874px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow1Lend)" + d="m 388.20118,147.93341 0,173.95967" + id="path17211-7" + inkscape:connector-type="orthogonal" + inkscape:connector-curvature="0" /> + <path + transform="matrix(0.73232502,0,0,0.75477602,116.5086,136.68745)" + sodipodi:type="arc" + style="fill:#539de6;fill-opacity:1;stroke:#0000ea;stroke-width:1;stroke-linecap:square;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" + id="path29161-6" + sodipodi:cx="371" + sodipodi:cy="64.5" + sodipodi:rx="17" + sodipodi:ry="15.5" + d="m 388,64.5 a 17,15.5 0 1 1 -34,0 17,15.5 0 1 1 34,0 z" /> + <g + id="g29167-46" + transform="matrix(0.73232502,0,0,0.75477602,116.59334,137.48829)"> + <text + sodipodi:linespacing="125%" + id="text29163-1" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-5" + sodipodi:role="line">3</tspan></text> + </g> + <g + id="g29167-6" + transform="matrix(0.73232502,0,0,0.75477602,0.1536639,325.60918)"> + <text + sodipodi:linespacing="125%" + id="text29163-17" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-2" + sodipodi:role="line">4</tspan></text> + </g> + <g + id="g29167" + transform="matrix(0.73232502,0,0,0.75477602,-146.07919,111.10621)"> + <text + sodipodi:linespacing="125%" + id="text29163" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165" + sodipodi:role="line">6</tspan></text> + </g> + <g + id="g29167-4-3" + transform="matrix(0.73232502,0,0,0.75477602,-117.60784,180.11779)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6" + y="70" + x="321.30356" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="321.30356" + id="tspan29165-9-7" + sodipodi:role="line">Eventdev</tspan></text> + </g> + <g + id="g29167-4-3-5" + transform="matrix(0.73232502,0,0,0.75477602,55.34956,26.97821)"> + <text + sodipodi:linespacing="100%" + id="text29163-9-6-3" + y="70" + x="454.74152" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="454.74152" + id="tspan29165-9-7-5" + sodipodi:role="line">A<tspan + style="line-height:100%" + id="tspan3374">tomic stage</tspan></tspan><tspan + style="font-size:21.52082825000000099px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:100%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="96.901031" + x="454.74152" + sodipodi:role="line" + id="tspan3320">+</tspan><tspan + style="font-size:21.52082825000000099px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:100%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="123.80207" + x="454.74152" + sodipodi:role="line" + id="tspan3322">enqueue to</tspan><tspan + style="font-size:21.52082825000000099px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:100%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="150.70311" + x="454.74152" + sodipodi:role="line" + id="tspan3324">cryptodev</tspan></text> + </g> + <g + id="g29167-2" + transform="matrix(0.73232502,0,0,0.75477602,-116.28602,248.01695)"> + <text + sodipodi:linespacing="125%" + id="text29163-92" + y="70" + x="365" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="365" + id="tspan29165-8" + sodipodi:role="line">5</tspan></text> + </g> + <flowRoot + xml:space="preserve" + id="flowRoot3376" + style="fill:black;stroke:none;stroke-opacity:1;stroke-width:1px;stroke-linejoin:miter;stroke-linecap:butt;fill-opacity:1;font-family:Sans;font-style:normal;font-weight:normal;font-size:18px;line-height:125%;letter-spacing:0px;word-spacing:0px"><flowRegion + id="flowRegion3378"><rect + id="rect3380" + width="100" + height="37" + x="109" + y="259" /></flowRegion><flowPara + id="flowPara3382" /></flowRoot> <g + id="g29167-4-3-1" + transform="matrix(0.73232502,0,0,0.75477602,109.34956,323.97821)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-8" + y="70" + x="321.30356" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="321.30356" + id="tspan29165-9-7-7" + sodipodi:role="line">Cryptodev</tspan></text> + </g> + <g + id="g29167-4-3-1-9" + transform="matrix(0.73232502,0,0,0.75477602,-114.48565,314.20704)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-8-2" + y="70" + x="368.01718" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="368.01718" + id="tspan29165-9-7-7-0" + sodipodi:role="line">Crypto</tspan><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;font-family:Sans;-inkscape-font-specification:Sans" + y="100.26366" + x="368.01718" + sodipodi:role="line" + id="tspan3488">adapter</tspan></text> + </g> + <g + id="g29167-4-3-1-9-2" + transform="matrix(0.73232502,0,0,0.75477602,250.96804,192.62529)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-8-2-3" + y="-188.35481" + x="318.61978" + style="font-size:40px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-188.35481" + x="318.61978" + sodipodi:role="line" + id="tspan3543">1. Application dequeues</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-161.45378" + x="318.61978" + sodipodi:role="line" + id="tspan3196"> events from the previous</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-134.55275" + x="318.61978" + sodipodi:role="line" + id="tspan3232"> stage</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-107.65171" + x="318.61978" + sodipodi:role="line" + id="tspan3519" /><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-80.750671" + x="318.61978" + sodipodi:role="line" + id="tspan3551">2. Application prepares the</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-53.84964" + x="318.61978" + sodipodi:role="line" + id="tspan3203"> crypto operations.</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-26.948603" + x="318.61978" + sodipodi:role="line" + id="tspan3523" /><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="-0.047568277" + x="318.61978" + sodipodi:role="line" + id="tspan3541">3. Crypto operations are</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="26.853468" + x="318.61978" + sodipodi:role="line" + id="tspan3207"> submitted to cryptodev</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="53.754501" + x="318.61978" + sodipodi:role="line" + id="tspan3209"> by application..</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="80.65554" + x="318.61978" + sodipodi:role="line" + id="tspan3527" /><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="107.55657" + x="318.61978" + sodipodi:role="line" + id="tspan3547">4. Crypto adapter dequeues</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="134.45761" + x="318.61978" + sodipodi:role="line" + id="tspan3216"> crypto completions from</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="161.35864" + x="318.61978" + sodipodi:role="line" + id="tspan3218"> cryptodev.</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="188.25969" + x="318.61978" + sodipodi:role="line" + id="tspan3531" /><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="215.16072" + x="318.61978" + sodipodi:role="line" + id="tspan3549">5. Crypto adapter enqueues</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="242.06175" + x="318.61978" + sodipodi:role="line" + id="tspan3222"> events to the eventdev.</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="268.96277" + x="318.61978" + sodipodi:role="line" + id="tspan3535" /><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="295.8638" + x="318.61978" + sodipodi:role="line" + id="tspan3537">6. Application dequeues from</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="322.76483" + x="318.61978" + sodipodi:role="line" + id="tspan3224"> eventdev and prepare for</tspan><tspan + style="font-size:21.52082825px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="349.66586" + x="318.61978" + sodipodi:role="line" + id="tspan3226"> further processing</tspan></text> + </g> + <g + style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" + id="shape1-1-2-4-7" + v:mID="1" + v:groupContext="shape" + transform="matrix(2.1604167,0,0,1.165886,90.820551,-587.97129)"> + <title + id="title22-7-5-5">Square</title> + <desc + id="desc24-7-8-3">Atomic Queue #1</desc> + <v:userDefs> + <v:ud + v:nameU="visVersion" + v:val="VT0(15):26" /> + </v:userDefs> + <v:textBlock + v:margins="rect(4,4,4,4)" /> + <v:textRect + cx="30.75" + cy="581.25" + width="61.5" + height="61.5" /> + <g + id="shadow1-2-9-5-5" + v:groupContext="shadow" + v:shadowOffsetX="0.345598" + v:shadowOffsetY="-1.97279" + v:shadowType="1" + transform="translate(0.345598,1.97279)" + class="st1" + style="visibility:visible"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st2" + id="rect27-8-7-6" + style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-3-91-3)" /> + </g> + <g + id="g13515-33-2"> + <g + id="g13534-8-9"> + <rect + x="0" + y="550.5" + width="61.5" + height="61.5" + class="st3" + id="rect29-1-95-1" + style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> + </g> + </g> + </g> + <g + id="g29167-4-3-2" + transform="matrix(0.73232502,0,0,0.75477602,-125.66199,44.027402)"> + <text + sodipodi:linespacing="125%" + id="text29163-9-6-7" + y="70" + x="321.30356" + style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" + xml:space="preserve"><tspan + style="font-size:24.21093369px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" + y="70" + x="321.30356" + id="tspan29165-9-7-0" + sodipodi:role="line">Application</tspan></text> + </g> + </g> +</svg> diff --git a/doc/guides/prog_guide/img/eventdev_usage.svg b/doc/guides/prog_guide/img/eventdev_usage.svg index 7765649b..c19818b9 100644 --- a/doc/guides/prog_guide/img/eventdev_usage.svg +++ b/doc/guides/prog_guide/img/eventdev_usage.svg @@ -1,994 +1,549 @@ <?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export eventdev_usage.svg Page-1 --> +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2018 Arm --> -<svg - xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" - xmlns:dc="http://purl.org/dc/elements/1.1/" - xmlns:cc="http://creativecommons.org/ns#" - xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" - xmlns:svg="http://www.w3.org/2000/svg" - xmlns="http://www.w3.org/2000/svg" - xmlns:xlink="http://www.w3.org/1999/xlink" - xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" - xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" - width="683.12061" - height="184.672" - viewBox="0 0 546.49648 147.7376" - xml:space="preserve" - color-interpolation-filters="sRGB" - class="st9" - id="svg2" - version="1.1" - inkscape:version="0.48.4 r9939" - sodipodi:docname="eventdev_usage.svg" - style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"><metadata - id="metadata214"><rdf:RDF><cc:Work - rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type - rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><sodipodi:namedview - pagecolor="#ffffff" - bordercolor="#666666" - borderopacity="1" - objecttolerance="10" - gridtolerance="10" - guidetolerance="10" - inkscape:pageopacity="0" - inkscape:pageshadow="2" - inkscape:window-width="1920" - inkscape:window-height="1017" - id="namedview212" - showgrid="false" - fit-margin-top="2" - fit-margin-left="2" - fit-margin-right="2" - fit-margin-bottom="2" - inkscape:zoom="1.2339869" - inkscape:cx="501.15554" - inkscape:cy="164.17693" - inkscape:window-x="-8" - inkscape:window-y="406" - inkscape:window-maximized="1" - inkscape:current-layer="g17" /> - <v:documentProperties - v:langID="1033" - v:viewMarkup="false"> - <v:userDefs> - <v:ud - v:nameU="msvSubprocessMaster" - v:prompt="" - v:val="VT4(Rectangle)" /> - <v:ud - v:nameU="msvNoAutoConnect" - v:val="VT0(1):26" /> - </v:userDefs> - </v:documentProperties> - - <style - type="text/css" - id="style4"> - - .st1 {visibility:visible} - .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} - .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25} - .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em} - .st5 {font-size:1em} - .st6 {fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25} - .st7 {marker-end:url(#mrkr4-33);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} - .st8 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091} - .st9 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="9.04167in" height="1.84602in" + viewBox="0 0 651 132.913" xml:space="preserve" color-interpolation-filters="sRGB" class="st12"> + <v:documentProperties v:langID="1033" v:viewMarkup="false"/> + <style type="text/css"> + <![CDATA[ + .st1 {fill:#3c63ac;stroke:#30518f;stroke-width:0.75} + .st2 {fill:#feffff;font-family:Calibri;font-size:0.833336em} + .st3 {fill:none;stroke:#203864;stroke-width:0.25} + .st4 {stroke:#203864;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st5 {fill:#ffd965;stroke:#203864;stroke-width:0.25} + .st6 {fill:#000000;font-family:Calibri;font-size:1.16666em} + .st7 {font-size:1em} + .st8 {fill:none;stroke:none;stroke-width:0.25} + .st9 {fill:#000000;font-family:Calibri;font-size:0.833336em} + .st10 {fill:#000000;font-family:Calibri;font-size:1.00001em} + .st11 {font-size:1.16665em} + .st12 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> </style> - <defs - id="Markers"> - <g - id="lend4"> - <path - d="M 2,1 0,0 2,-1 2,1" - style="stroke:none" - id="path8" - inkscape:connector-curvature="0" /> - </g> - <marker - id="mrkr4-33" - class="st8" - v:arrowType="4" - v:arrowSize="2" - v:setback="7.04" - refX="-7.04" - orient="auto" - markerUnits="strokeWidth" - overflow="visible" - style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"> - <use - xlink:href="#lend4" - transform="scale(-3.52,-3.52)" - id="use11" - x="0" - y="0" - width="3" - height="3" /> - </marker> - <filter - id="filter_2-7" - color-interpolation-filters="sRGB"><feGaussianBlur - stdDeviation="2" - id="feGaussianBlur15-1" /></filter><marker - id="mrkr4-33-2" - class="st8" - v:arrowType="4" - v:arrowSize="2" - v:setback="7.04" - refX="-7.04" - orient="auto" - markerUnits="strokeWidth" - overflow="visible" - style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"><use - xlink:href="#lend4" - transform="scale(-3.52,-3.52)" - id="use11-3" - x="0" - y="0" - width="3" - height="3" /></marker><marker - id="mrkr4-33-6" - class="st8" - v:arrowType="4" - v:arrowSize="2" - v:setback="7.04" - refX="-7.04" - orient="auto" - markerUnits="strokeWidth" - overflow="visible" - style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"><use - xlink:href="#lend4" - transform="scale(-3.52,-3.52)" - id="use11-8" - x="0" - y="0" - width="3" - height="3" /></marker></defs> - <defs - id="Filters"> - <filter - id="filter_2" - color-interpolation-filters="sRGB"> - <feGaussianBlur - stdDeviation="2" - id="feGaussianBlur15" /> - </filter> - </defs> - <g - v:mID="0" - v:index="1" - v:groupContext="foregroundPage" - id="g17" - transform="translate(-47.323579,-90.784072)"> - <v:userDefs> - <v:ud - v:nameU="msvThemeOrder" - v:val="VT0(0):26" /> - </v:userDefs> - <title - id="title19">Page-1</title> - <v:pageProperties - v:drawingScale="1" - v:pageScale="1" - v:drawingUnits="0" - v:shadowOffsetX="9" - v:shadowOffsetY="-9" /> - <v:layer - v:name="Connector" - v:index="0" /> - <g - id="shape1-1" - v:mID="1" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,128.62352,-288.18843)"> - <title - id="title22">Square</title> - <desc - id="desc24">Atomic Queue #1</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="30.75" - cy="581.25" - width="61.5" - height="61.5" /> - <g - id="shadow1-2" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <rect - x="0" - y="550.5" - width="61.5" - height="61.5" - class="st2" - id="rect27" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> - </g> - <rect - x="0" - y="550.5" - width="61.5" - height="61.5" - class="st3" - id="rect29" - style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> - - </g> - <g - id="shape3-8" - v:mID="3" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,297.37175,-288.18843)"> - <title - id="title36">Square.3</title> - <desc - id="desc38">Atomic Queue #2</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="30.75" - cy="581.25" - width="61.5" - height="61.5" /> - <g - id="shadow3-9" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <rect - x="0" - y="550.5" - width="61.5" - height="61.5" - class="st2" - id="rect41" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> - </g> - <rect - x="0" - y="550.5" - width="61.5" - height="61.5" - class="st3" - id="rect43" - style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> - - </g> - <g - id="shape4-15" - v:mID="4" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,466.1192,-288.18843)"> - <title - id="title50">Square.4</title> - <desc - id="desc52">Single Link Queue # 1</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="30.75" - cy="581.25" - width="61.5" - height="61.5" /> - <g - id="shadow4-16" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <rect - x="0" - y="550.5" - width="61.5" - height="61.5" - class="st2" - id="rect55" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> - </g> - <rect - x="0" - y="550.5" - width="61.5" - height="61.5" - class="st3" - id="rect57" - style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" /> - - </g> - <g - id="shape5-22" - v:mID="5" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,52.208527,-296.14701)"> - <title - id="title64">Circle</title> - <desc - id="desc66">RX</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow5-23" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path69" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> - </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path71" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="15.19" - y="594.5" - class="st4" - v:langID="1033" - id="text73" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />RX</text> - - </g> - <g - id="shape6-28" - v:mID="6" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,84.042834,-305.07614)"> - <title - id="title76">Dynamic connector</title> - <path - d="m 0,603 50.38,0" - class="st7" - id="path78" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape7-34" - v:mID="7" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,220.95621,-296.14701)"> - <title - id="title81">Circle.7</title> - <desc - id="desc83">W ..</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow7-35" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path86" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> - </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path88" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="12.4" - y="594.5" - class="st4" - v:langID="1033" - id="text90" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />W ..</text> - - </g> - <g - id="shape9-40" - v:mID="9" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,220.95621,-243.34865)"> - <title - id="title93">Circle.9</title> - <desc - id="desc95">W N</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow9-41" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path98" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="1" v:pageScale="1" v:drawingUnits="0" v:shadowOffsetX="9" v:shadowOffsetY="-9"/> + <v:layer v:name="Connector" v:index="0"/> + <g id="group1068-1" transform="translate(0.75,-0.25)" v:mID="1068" v:groupContext="group"> + <title>Sheet.1068</title> + <g id="shape3-2" v:mID="3" v:groupContext="shape" transform="translate(63,184.827) rotate(180)"> + <title>Simple Arrow</title> + <desc>In Intf</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + <v:ud v:nameU="ArrowType" v:prompt="" v:val="VT0(2):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="31.5" cy="132.913" width="63.01" height="0" transform="rotate(180)"/> + <path d="M0 132.91 L12 120.92 L12 126.92 L63 126.92 L63 132.91 L63 138.91 L12 138.91 L12 144.91 L0 132.91 Z" + class="st1"/> + <text x="-43.6" y="-129.91" transform="rotate(180)" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>In Intf</text> </g> + <g id="group1010-5" transform="translate(130.5,-73.3167)" v:mID="1010" v:groupContext="group"> + <title>Sheet.1010</title> + <g id="group1000-6" transform="translate(-2.19824E-14,-0.0534178)" v:mID="1000" v:groupContext="group"> + <title>Sheet.1000</title> + <g id="shape1001-7" v:mID="1001" v:groupContext="shape" transform="translate(0,-4.86)"> + <title>Rectangle.38</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <rect x="0" y="123.913" width="22.3966" height="8.99999" class="st3"/> + </g> + <g id="shape1002-9" v:mID="1002" v:groupContext="shape" v:layerMember="0" transform="translate(2.19832,-18.18)"> + <title>Dynamic connector.162</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1003-12" v:mID="1003" v:groupContext="shape" v:layerMember="0" + transform="translate(7.79747,-18.18)"> + <title>Dynamic connector.163</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1004-15" v:mID="1004" v:groupContext="shape" v:layerMember="0" transform="translate(-3.40084,-18)"> + <title>Dynamic connector.164</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + </g> + <g id="group1005-18" transform="translate(22.6034,0)" v:mID="1005" v:groupContext="group"> + <title>Sheet.1005</title> + <g id="shape1006-19" v:mID="1006" v:groupContext="shape" transform="translate(0,-4.86)"> + <title>Rectangle.38</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <rect x="0" y="123.913" width="22.3966" height="8.99999" class="st3"/> + </g> + <g id="shape1007-21" v:mID="1007" v:groupContext="shape" v:layerMember="0" + transform="translate(2.19832,-18.18)"> + <title>Dynamic connector.162</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1008-24" v:mID="1008" v:groupContext="shape" v:layerMember="0" + transform="translate(7.79747,-18.18)"> + <title>Dynamic connector.163</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1009-27" v:mID="1009" v:groupContext="shape" v:layerMember="0" transform="translate(-3.40084,-18)"> + <title>Dynamic connector.164</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + </g> </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path100" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="11.69" - y="594.5" - class="st4" - v:langID="1033" - id="text102" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />W N</text> - - </g> - <g - id="shape10-46" - v:mID="10" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,220.95621,-348.94537)"> - <title - id="title105">Circle.10</title> - <desc - id="desc107">W 1</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow10-47" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path110" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> + <g id="group1016-30" transform="translate(301.5,-73.3167)" v:mID="1016" v:groupContext="group"> + <title>Sheet.1016</title> + <g id="group1017-31" transform="translate(-2.19824E-14,-0.0534178)" v:mID="1017" v:groupContext="group"> + <title>Sheet.1017</title> + <g id="shape1018-32" v:mID="1018" v:groupContext="shape" transform="translate(0,-4.86)"> + <title>Rectangle.38</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <rect x="0" y="123.913" width="22.3966" height="8.99999" class="st3"/> + </g> + <g id="shape1019-34" v:mID="1019" v:groupContext="shape" v:layerMember="0" + transform="translate(2.19832,-18.18)"> + <title>Dynamic connector.162</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1020-37" v:mID="1020" v:groupContext="shape" v:layerMember="0" + transform="translate(7.79747,-18.18)"> + <title>Dynamic connector.163</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1021-40" v:mID="1021" v:groupContext="shape" v:layerMember="0" transform="translate(-3.40084,-18)"> + <title>Dynamic connector.164</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + </g> + <g id="group1022-43" transform="translate(22.6034,0)" v:mID="1022" v:groupContext="group"> + <title>Sheet.1022</title> + <g id="shape1023-44" v:mID="1023" v:groupContext="shape" transform="translate(0,-4.86)"> + <title>Rectangle.38</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <rect x="0" y="123.913" width="22.3966" height="8.99999" class="st3"/> + </g> + <g id="shape1024-46" v:mID="1024" v:groupContext="shape" v:layerMember="0" + transform="translate(2.19832,-18.18)"> + <title>Dynamic connector.162</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1025-49" v:mID="1025" v:groupContext="shape" v:layerMember="0" + transform="translate(7.79747,-18.18)"> + <title>Dynamic connector.163</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1026-52" v:mID="1026" v:groupContext="shape" v:layerMember="0" transform="translate(-3.40084,-18)"> + <title>Dynamic connector.164</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + </g> </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path112" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="12.39" - y="594.5" - class="st4" - v:langID="1033" - id="text114" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />W 1</text> - - </g> - <g - id="shape11-52" - v:mID="11" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,195.91581,-312.06416)"> - <title - id="title117">Dynamic connector.11</title> - <path - d="m 0,612 0,-68 25.21,0" - class="st7" - id="path119" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape12-57" - v:mID="12" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,176.37498,-305.07614)"> - <title - id="title122">Dynamic connector.12</title> - <path - d="m 0,603 50.38,0" - class="st7" - id="path124" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape13-62" - v:mID="13" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,176.37498,-312.06416)"> - <title - id="title127">Dynamic connector.13</title> - <path - d="m 0,612 25.17,0 0,68 25.21,0" - class="st7" - id="path129" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape14-67" - v:mID="14" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,252.79052,-259.2658)"> - <title - id="title132">Dynamic connector.14</title> - <path - d="m 0,612 26.88,0 0,-68 23.5,0" - class="st7" - id="path134" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape15-72" - v:mID="15" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,252.79052,-305.07614)"> - <title - id="title137">Dynamic connector.15</title> - <path - d="m 0,603 50.38,0" - class="st7" - id="path139" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape19-77" - v:mID="19" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,389.70366,-296.14701)"> - <title - id="title142">Circle.19</title> - <desc - id="desc144">W ..</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow19-78" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path147" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> + <g id="group1032-55" transform="translate(468,-73.2)" v:mID="1032" v:groupContext="group"> + <title>Sheet.1032</title> + <g id="group1033-56" transform="translate(-2.19824E-14,-0.0534178)" v:mID="1033" v:groupContext="group"> + <title>Sheet.1033</title> + <g id="shape1034-57" v:mID="1034" v:groupContext="shape" transform="translate(0,-4.86)"> + <title>Rectangle.38</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <rect x="0" y="123.913" width="22.3966" height="8.99999" class="st3"/> + </g> + <g id="shape1035-59" v:mID="1035" v:groupContext="shape" v:layerMember="0" + transform="translate(2.19832,-18.18)"> + <title>Dynamic connector.162</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1036-62" v:mID="1036" v:groupContext="shape" v:layerMember="0" + transform="translate(7.79747,-18.18)"> + <title>Dynamic connector.163</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1037-65" v:mID="1037" v:groupContext="shape" v:layerMember="0" transform="translate(-3.40084,-18)"> + <title>Dynamic connector.164</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + </g> + <g id="group1038-68" transform="translate(22.6034,0)" v:mID="1038" v:groupContext="group"> + <title>Sheet.1038</title> + <g id="shape1039-69" v:mID="1039" v:groupContext="shape" transform="translate(0,-4.86)"> + <title>Rectangle.38</title> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <rect x="0" y="123.913" width="22.3966" height="8.99999" class="st3"/> + </g> + <g id="shape1040-71" v:mID="1040" v:groupContext="shape" v:layerMember="0" + transform="translate(2.19832,-18.18)"> + <title>Dynamic connector.162</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1041-74" v:mID="1041" v:groupContext="shape" v:layerMember="0" + transform="translate(7.79747,-18.18)"> + <title>Dynamic connector.163</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + <g id="shape1042-77" v:mID="1042" v:groupContext="shape" v:layerMember="0" transform="translate(-3.40084,-18)"> + <title>Dynamic connector.164</title> + <path d="M9 137.41 L9 146.41" class="st4"/> + </g> + </g> </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path149" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="12.4" - y="594.5" - class="st4" - v:langID="1033" - id="text151" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />W ..</text> - - </g> - <g - id="shape20-83" - v:mID="20" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,389.70366,-243.34865)"> - <title - id="title154">Circle.20</title> - <desc - id="desc156">W N</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow20-84" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path159" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> + <g id="shape1044-80" v:mID="1044" v:groupContext="shape" transform="translate(651.291,179.381) rotate(179.228)"> + <title>Simple Arrow.1044</title> + <desc>Out Intf</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + <v:ud v:nameU="ArrowType" v:prompt="" v:val="VT0(2):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="30.3028" cy="132.913" width="60.61" height="0" transform="rotate(180)"/> + <path d="M0 132.91 L12 120.92 L12 126.92 L60.61 126.92 L60.61 132.91 L60.61 138.91 L12 138.91 L12 144.91 L0 132.91 + Z" class="st1"/> + <text x="-46.13" y="-129.91" transform="rotate(180)" class="st2" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Out Intf</text> </g> + <g id="shape1045-83" v:mID="1045" v:groupContext="shape" transform="translate(67.8,-50.9334)"> + <title>Rounded Rectangle.1045</title> + <desc>RX Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="25.3125" cy="101.413" width="50.63" height="63"/> + <path d="M5.06 132.91 L45.56 132.91 A5.06242 5.06242 -180 0 0 50.62 127.85 L50.62 74.98 A5.06242 5.06242 -180 0 0 + 45.56 69.91 L5.06 69.91 A5.06242 5.06242 -180 0 0 0 74.98 L0 127.85 A5.06242 5.06242 -180 0 0 5.06 132.91 + Z" class="st5"/> + <text x="17.88" y="88.81" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>RX<v:newlineChar/><v:newlineChar/><tspan + x="11.97" dy="2.4em" class="st7">Core</tspan></text> </g> + <g id="shape1056-87" v:mID="1056" v:groupContext="shape" transform="translate(532.5,-54)"> + <title>Rounded Rectangle.1056</title> + <desc>TX Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.0703125):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="25.3125" cy="101.413" width="50.63" height="63"/> + <path d="M5.06 132.91 L45.56 132.91 A5.06242 5.06242 -180 0 0 50.62 127.85 L50.62 74.98 A5.06242 5.06242 -180 0 0 + 45.56 69.91 L5.06 69.91 A5.06242 5.06242 -180 0 0 0 74.98 L0 127.85 A5.06242 5.06242 -180 0 0 5.06 132.91 + Z" class="st5"/> + <text x="18.27" y="88.81" class="st6" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>TX<v:newlineChar/><v:newlineChar/><tspan + x="11.97" dy="2.4em" class="st7">Core</tspan></text> </g> + <g id="shape1057-91" v:mID="1057" v:groupContext="shape" transform="translate(123.188,-59.0334)"> + <title>Rectangle.1057</title> + <desc>Atomic Q 1</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="29.8125" cy="123.797" width="59.63" height="18.2334"/> + <rect x="0" y="114.68" width="59.625" height="18.2334" class="st8"/> + <text x="7.19" y="126.8" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Atomic Q 1</text> </g> + <g id="shape1058-94" v:mID="1058" v:groupContext="shape" transform="translate(295.5,-59.4)"> + <title>Rectangle.1058</title> + <desc>Atomic Q 2</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="29.8125" cy="123.797" width="59.63" height="18.2334"/> + <rect x="0" y="114.68" width="59.625" height="18.2334" class="st8"/> + <text x="7.19" y="126.8" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Atomic Q 2</text> </g> + <g id="shape1059-97" v:mID="1059" v:groupContext="shape" transform="translate(460.687,-58.3167)"> + <title>Rectangle.1059</title> + <desc>Single Link</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="29.8125" cy="123.797" width="59.63" height="18.2334"/> + <rect x="0" y="114.68" width="59.625" height="18.2334" class="st8"/> + <text x="8.47" y="126.8" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Single Link</text> </g> + <g id="shape1060-100" v:mID="1060" v:groupContext="shape" transform="translate(198,-1.2)"> + <title>Rectangle.1060</title> + <desc>Stage 1</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="29.8125" cy="123.797" width="59.63" height="18.2334"/> + <rect x="0" y="114.68" width="59.625" height="18.2334" class="st8"/> + <text x="14.94" y="126.8" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Stage 1</text> </g> + <g id="shape1061-103" v:mID="1061" v:groupContext="shape" transform="translate(366.188,0)"> + <title>Rectangle.1061</title> + <desc>Stage 2</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)"/> + <v:textRect cx="29.8125" cy="123.797" width="59.63" height="18.2334"/> + <rect x="0" y="114.68" width="59.625" height="18.2334" class="st8"/> + <text x="14.94" y="126.8" class="st9" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Stage 2</text> </g> + <g id="group1062-106" transform="translate(199.2,-18.7134)" v:mID="1062" v:groupContext="group"> + <title>Sheet.1062</title> + <g id="shape1052-107" v:mID="1052" v:groupContext="shape" transform="translate(18.66,-50.7)"> + <title>Rounded Rectangle.1049</title> + <desc>Worker4 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.07" cy="101.413" width="70.14" height="63"/> + <path d="M7.01 132.91 L63.13 132.91 A7.01389 7.01389 -180 0 0 70.14 125.9 L70.14 76.93 A7.01389 7.01389 -180 + 0 0 63.13 69.91 L7.01 69.91 A7.01389 7.01389 -180 0 0 0 76.93 L0 125.9 A7.01389 7.01389 -180 0 0 + 7.01 132.91 Z" class="st5"/> + <text x="13.63" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker4<v:newlineChar/><v:newlineChar/><tspan + x="21.72" dy="2.357em" class="st11">Core</tspan></text> </g> + <g id="shape1053-111" v:mID="1053" v:groupContext="shape" transform="translate(12.9,-33.6)"> + <title>Rounded Rectangle.1048</title> + <desc>Worker3 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.7" cy="101.413" width="71.4" height="63"/> + <path d="M7.14 132.91 L64.26 132.91 A7.13988 7.13988 -180 0 0 71.4 125.77 L71.4 77.05 A7.13988 7.13988 -180 0 + 0 64.26 69.91 L7.14 69.91 A7.13988 7.13988 -180 0 0 0 77.05 L0 125.77 A7.13988 7.13988 -180 0 0 + 7.14 132.91 Z" class="st5"/> + <text x="14.26" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker3<v:newlineChar/><v:newlineChar/><tspan + x="22.35" dy="2.357em" class="st11">Core</tspan></text> </g> + <g id="shape1054-115" v:mID="1054" v:groupContext="shape" transform="translate(5.89875,-16.8)"> + <title>Rounded Rectangle.1047</title> + <desc>Worker2 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.2631" cy="101.413" width="70.53" height="63"/> + <path d="M7.05 132.91 L63.47 132.91 A7.05251 7.05251 -180 0 0 70.53 125.86 L70.53 76.97 A7.05251 7.05251 -180 + 0 0 63.47 69.91 L7.05 69.91 A7.05251 7.05251 -180 0 0 0 76.97 L0 125.86 A7.05251 7.05251 -180 0 + 0 7.05 132.91 Z" class="st5"/> + <text x="13.82" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker2<v:newlineChar/><v:newlineChar/><tspan + x="21.92" dy="2.357em" class="st11">Core</tspan></text> </g> + <g id="shape1055-119" v:mID="1055" v:groupContext="shape"> + <title>Rounded Rectangle.1046</title> + <desc>Worker1 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.4" cy="101.413" width="70.8" height="63"/> + <path d="M7.08 132.91 L63.72 132.91 A7.07988 7.07988 -180 0 0 70.8 125.83 L70.8 76.99 A7.07988 7.07988 -180 0 + 0 63.72 69.91 L7.08 69.91 A7.07988 7.07988 -180 0 0 0 76.99 L0 125.83 A7.07988 7.07988 -180 0 0 + 7.08 132.91 Z" class="st5"/> + <text x="13.96" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker1<v:newlineChar/><v:newlineChar/><tspan + x="22.05" dy="2.357em" class="st11">Core</tspan></text> </g> </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path161" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="11.69" - y="594.5" - class="st4" - v:langID="1033" - id="text163" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />W N</text> - - </g> - <g - id="shape21-89" - v:mID="21" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,389.70366,-348.94537)"> - <title - id="title166">Circle.21</title> - <desc - id="desc168">W 1</desc> - <v:userDefs> - <v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /> - </v:userDefs> - <v:textBlock - v:margins="rect(4,4,4,4)" /> - <v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /> - <g - id="shadow21-90" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path171" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" /> + <g id="group1063-123" transform="translate(369.6,-18.6)" v:mID="1063" v:groupContext="group"> + <title>Sheet.1063</title> + <g id="shape1064-124" v:mID="1064" v:groupContext="shape" transform="translate(18.66,-50.7)"> + <title>Rounded Rectangle.1049</title> + <desc>Worker4 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.097416666666665):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.07" cy="101.413" width="70.14" height="63"/> + <path d="M7.01 132.91 L63.13 132.91 A7.01389 7.01389 -180 0 0 70.14 125.9 L70.14 76.93 A7.01389 7.01389 -180 + 0 0 63.13 69.91 L7.01 69.91 A7.01389 7.01389 -180 0 0 0 76.93 L0 125.9 A7.01389 7.01389 -180 0 0 + 7.01 132.91 Z" class="st5"/> + <text x="13.63" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker4<v:newlineChar/><v:newlineChar/><tspan + x="21.72" dy="2.357em" class="st11">Core</tspan></text> </g> + <g id="shape1065-128" v:mID="1065" v:groupContext="shape" transform="translate(12.9,-33.6)"> + <title>Rounded Rectangle.1048</title> + <desc>Worker3 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.099166666666667):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.7" cy="101.413" width="71.4" height="63"/> + <path d="M7.14 132.91 L64.26 132.91 A7.13988 7.13988 -180 0 0 71.4 125.77 L71.4 77.05 A7.13988 7.13988 -180 0 + 0 64.26 69.91 L7.14 69.91 A7.13988 7.13988 -180 0 0 0 77.05 L0 125.77 A7.13988 7.13988 -180 0 0 + 7.14 132.91 Z" class="st5"/> + <text x="14.26" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker3<v:newlineChar/><v:newlineChar/><tspan + x="22.35" dy="2.357em" class="st11">Core</tspan></text> </g> + <g id="shape1066-132" v:mID="1066" v:groupContext="shape" transform="translate(5.89875,-16.8)"> + <title>Rounded Rectangle.1047</title> + <desc>Worker2 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.097953125):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.2631" cy="101.413" width="70.53" height="63"/> + <path d="M7.05 132.91 L63.47 132.91 A7.05251 7.05251 -180 0 0 70.53 125.86 L70.53 76.97 A7.05251 7.05251 -180 + 0 0 63.47 69.91 L7.05 69.91 A7.05251 7.05251 -180 0 0 0 76.97 L0 125.86 A7.05251 7.05251 -180 0 + 0 7.05 132.91 Z" class="st5"/> + <text x="13.82" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker2<v:newlineChar/><v:newlineChar/><tspan + x="21.92" dy="2.357em" class="st11">Core</tspan></text> </g> + <g id="shape1067-136" v:mID="1067" v:groupContext="shape"> + <title>Rounded Rectangle.1046</title> + <desc>Worker1 Core</desc> + <v:userDefs> + <v:ud v:nameU="CTypeTopLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeTopRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotLeftSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CTypeBotRightSnip" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="CornerLockHoriz" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockVert" v:prompt="" v:val="VT0(1):5"/> + <v:ud v:nameU="CornerLockDiag" v:prompt="" v:val="VT0(0):5"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.15):1"/> + <v:ud v:nameU="visVersion" v:prompt="" v:val="VT0(15):26"/> + <v:ud v:nameU="TopLeftOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + <v:ud v:nameU="TopRightOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + <v:ud v:nameU="BotLeftOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + <v:ud v:nameU="BotRightOffset" v:prompt="" v:val="VT0(0.098333333333333):1"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:verticalAlign="0"/> + <v:textRect cx="35.4" cy="101.413" width="70.8" height="63"/> + <path d="M7.08 132.91 L63.72 132.91 A7.07988 7.07988 -180 0 0 70.8 125.83 L70.8 76.99 A7.07988 7.07988 -180 0 + 0 63.72 69.91 L7.08 69.91 A7.07988 7.07988 -180 0 0 0 76.99 L0 125.83 A7.07988 7.07988 -180 0 0 + 7.08 132.91 Z" class="st5"/> + <text x="13.96" y="84.71" class="st10" v:langID="1033"><v:paragraph v:horizAlign="1"/><v:tabList/>Worker1<v:newlineChar/><v:newlineChar/><tspan + x="22.05" dy="2.357em" class="st11">Core</tspan></text> </g> </g> - <path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path173" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /> - <text - x="12.39" - y="594.5" - class="st4" - v:langID="1033" - id="text175" - style="fill:#feffff;font-family:Calibri"><v:paragraph - v:horizAlign="1" /><v:tabList />W 1</text> - - </g> - <g - id="shape28-95" - v:mID="28" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,345.12321,-305.07614)"> - <title - id="title178">Dynamic connector.28</title> - <path - d="m 0,603 50.38,0" - class="st7" - id="path180" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape29-100" - v:mID="29" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,345.12321,-312.06416)"> - <title - id="title183">Dynamic connector.29</title> - <path - d="m 0,612 28.33,0 0,-68 22.05,0" - class="st7" - id="path185" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape30-105" - v:mID="30" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,345.12321,-312.06416)"> - <title - id="title188">Dynamic connector.30</title> - <path - d="m 0,612 28.33,0 0,68 22.05,0" - class="st7" - id="path190" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape31-110" - v:mID="31" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,421.53797,-259.2658)"> - <title - id="title193">Dynamic connector.31</title> - <path - d="m 0,612 24.42,0 0,-68 25.96,0" - class="st7" - id="path195" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape32-115" - v:mID="32" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,421.53797,-305.07614)"> - <title - id="title198">Dynamic connector.32</title> - <path - d="m 0,603 50.38,0" - class="st7" - id="path200" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape33-120" - v:mID="33" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,421.53797,-364.86253)"> - <title - id="title203">Dynamic connector.33</title> - <path - d="m 0,612 24.42,0 0,68 25.96,0" - class="st7" - id="path205" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> - </g> - <g - id="shape34-125" - v:mID="34" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,252.79052,-364.86253)"> - <title - id="title208">Dynamic connector.34</title> - <path - d="m 0,612 26.88,0 0,68 23.5,0" - class="st7" - id="path210" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /> </g> - <text - xml:space="preserve" - style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="153.38116" - y="165.90149" - id="text3106" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="153.38116" - y="165.90149" - id="tspan3110" - style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Atomic #1</tspan></text> -<text - xml:space="preserve" - style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans" - x="322.12939" - y="165.90149" - id="text3106-1" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="322.12939" - y="165.90149" - id="tspan3110-4" - style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Atomic #2</tspan></text> -<text - xml:space="preserve" - style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans" - x="491.82089" - y="172.79289" - id="text3106-0" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="491.82089" - y="172.79289" - style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan3923" /></text> -<text - xml:space="preserve" - style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans" - x="491.02899" - y="165.03951" - id="text3106-8-5" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="491.02899" - y="165.03951" - id="tspan3110-2-1" - style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Single Link</tspan></text> -<g - style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" - id="shape5-22-1" - v:mID="5" - v:groupContext="shape" - transform="matrix(0.77644652,0,0,0.77644652,556.00223,-296.89447)"><title - id="title64-5">Circle</title><desc - id="desc66-2">RX</desc><v:userDefs><v:ud - v:nameU="visVersion" - v:val="VT0(15):26" /></v:userDefs><v:textBlock - v:margins="rect(4,4,4,4)" /><v:textRect - cx="20.5" - cy="591.5" - width="35.88" - height="30.75" /><g - id="shadow5-23-7" - v:groupContext="shadow" - v:shadowOffsetX="0.345598" - v:shadowOffsetY="-1.97279" - v:shadowType="1" - transform="translate(0.345598,1.97279)" - class="st1" - style="visibility:visible"><path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st2" - id="path69-6" - inkscape:connector-curvature="0" - style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-7)" /></g><path - d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z" - class="st6" - id="path71-1" - inkscape:connector-curvature="0" - style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /><text - x="11.06866" - y="596.56067" - class="st4" - v:langID="1033" - id="text73-4" - style="fill:#feffff;font-family:Calibri"> TX</text> -</g><g - style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible" - id="shape28-95-5" - v:mID="28" - v:groupContext="shape" - v:layerMember="0" - transform="matrix(0.77644652,0,0,0.77644652,512.00213,-305.42637)"><title - id="title178-7">Dynamic connector.28</title><path - d="m 0,603 50.38,0" - class="st7" - id="path180-6" - inkscape:connector-curvature="0" - style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /></g></g> + </g> </svg> diff --git a/doc/guides/prog_guide/img/malloc_heap.svg b/doc/guides/prog_guide/img/malloc_heap.svg index 14e50088..f70bd666 100644 --- a/doc/guides/prog_guide/img/malloc_heap.svg +++ b/doc/guides/prog_guide/img/malloc_heap.svg @@ -1,1021 +1,333 @@ <?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Created with Inkscape (http://www.inkscape.org/) --> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export malloc_heap.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + width="11in" height="8.5in" viewBox="0 0 792 612" xml:space="preserve" color-interpolation-filters="sRGB" class="st34"> + <style type="text/css"> + <![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25} + .st4 {fill:#deebf6;stroke:#c7c8c8;stroke-width:0.25} + .st5 {fill:#ed7d31;stroke:#c7c8c8;stroke-width:0.25} + .st6 {fill:#fbe5d5;stroke:#c7c8c8;stroke-width:0.25} + .st7 {fill:#e2efd9;stroke:#c7c8c8;stroke-width:0.25} + .st8 {fill:#a8d08d;stroke:#c7c8c8;stroke-width:0.25} + .st9 {fill:url(#ptrn2-71);shape-rendering:crispEdges;stroke:#c7c8c8;stroke-width:0.25} + .st10 {fill:#5b9bd5;stroke:#2e75b5;stroke-width:0.25} + .st11 {fill:none;stroke:none;stroke-width:0.25} + .st12 {fill:#000000;font-family:Calibri;font-size:1.00001em} + .st13 {fill:#ed7d31;stroke:#2e75b5;stroke-width:0.25} + .st14 {fill:#deebf6;stroke:#2e75b5;stroke-width:0.25} + .st15 {fill:#fbe5d5;stroke:#2e75b5;stroke-width:0.25} + .st16 {fill:#a8d08d;stroke:#2e75b5;stroke-width:0.25} + .st17 {fill:#e2efd9;stroke:#2e75b5;stroke-width:0.25} + .st18 {fill:url(#ptrn2-71);shape-rendering:crispEdges;stroke:#2e75b5;stroke-width:0.25} + .st19 {fill:#f4b183;stroke:#4f87bb;stroke-width:0.75} + .st20 {fill:#305497;font-family:Calibri;font-size:0.833336em} + .st21 {fill:#5b9bd5;fill-opacity:0.25;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.25} + .st22 {fill:#538135;stroke:#40709c;stroke-width:0.75} + .st23 {fill:#e2efd9;font-family:Calibri;font-size:0.833336em} + .st24 {marker-end:url(#mrkr10-146);marker-start:url(#mrkr10-144);stroke:#70ad47;stroke-width:0.75} + .st25 {fill:#70ad47;fill-opacity:1;stroke:#70ad47;stroke-opacity:1;stroke-width:0.22935779816514} + .st26 {fill:#ffffff;stroke:none;stroke-linecap:butt;stroke-width:7.2} + .st27 {fill:#538135;font-family:Calibri;font-size:1.00001em} + .st28 {fill:#ffffff;stroke:none;stroke-linecap:butt} + .st29 {fill:#bdd7ee;stroke:#40709c;stroke-width:0.75} + .st30 {fill:#1e4e79;font-family:Calibri;font-size:0.833336em} + .st31 {marker-end:url(#mrkr5-171);stroke:#4f87bb;stroke-dasharray:11.25,6.75;stroke-width:0.75} + .st32 {fill:#4f87bb;fill-opacity:1;stroke:#4f87bb;stroke-opacity:1;stroke-width:0.22935779816514} + .st33 {fill:#1e4e79;font-family:Calibri;font-size:1.00001em} + .st34 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> -<!-- SPDX-License-Identifier: BSD-3-Clause --> -<!-- Copyright(c) 2015 Intel Corporation --> - -<svg - xmlns:dc="http://purl.org/dc/elements/1.1/" - xmlns:cc="http://creativecommons.org/ns#" - xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" - xmlns:svg="http://www.w3.org/2000/svg" - xmlns="http://www.w3.org/2000/svg" - xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" - xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" - id="svg2985" - version="1.1" - inkscape:version="0.48.4 r9939" - width="983.76233" - height="643.91644" - sodipodi:docname="malloc_heap_svg.svg"> - <metadata - id="metadata2991"> - <rdf:RDF> - <cc:Work - rdf:about=""> - <dc:format>image/svg+xml</dc:format> - <dc:type - rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> - <dc:title></dc:title> - </cc:Work> - </rdf:RDF> - </metadata> - <defs - id="defs2989"> - <marker - inkscape:stockid="Arrow2Mstart" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mstart" - style="overflow:visible"> - <path - id="path4265" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(0.6,0.6)" - inkscape:connector-curvature="0" /> - </marker> - <marker - inkscape:stockid="Arrow2Lstart" - orient="auto" - refY="0" - refX="0" - id="Arrow2Lstart" - style="overflow:visible"> - <path - id="path4259" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="matrix(1.1,0,0,1.1,1.1,0)" - inkscape:connector-curvature="0" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend" - style="overflow:visible"> - <path - id="path4268" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" - inkscape:connector-curvature="0" /> - </marker> - <marker - inkscape:stockid="Arrow2Lend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Lend" - style="overflow:visible"> - <path - id="path4262" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="matrix(-1.1,0,0,-1.1,-1.1,0)" - inkscape:connector-curvature="0" /> - </marker> - <marker - inkscape:stockid="Arrow1Lend" - orient="auto" - refY="0" - refX="0" - id="Arrow1Lend" - style="overflow:visible"> - <path - id="path4244" - d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" - style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" - transform="matrix(-0.8,0,0,-0.8,-10,0)" - inkscape:connector-curvature="0" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend-1" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4268-4" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend-1-1" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4268-4-8" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend-1-9" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4268-4-6" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mstart" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mstart-7" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4265-8" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(0.6,0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend-1-8" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4268-4-2" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend-1-2" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4268-4-0" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mstart" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mstart-5" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4265-7" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(0.6,0.6)" /> - </marker> - <marker - inkscape:stockid="Arrow2Mend" - orient="auto" - refY="0" - refX="0" - id="Arrow2Mend-1-5" - style="overflow:visible"> - <path - inkscape:connector-curvature="0" - id="path4268-4-4" - style="fill-rule:evenodd;stroke-width:0.625;stroke-linejoin:round" - d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" - transform="scale(-0.6,-0.6)" /> - </marker> - </defs> - <sodipodi:namedview - pagecolor="#ffffff" - bordercolor="#30ff00" - borderopacity="1" - objecttolerance="10" - gridtolerance="10" - guidetolerance="10" - inkscape:pageopacity="0" - inkscape:pageshadow="2" - inkscape:window-width="1920" - inkscape:window-height="1139" - id="namedview2987" - showgrid="false" - inkscape:zoom="0.8" - inkscape:cx="346.31962" - inkscape:cy="474.02351" - inkscape:window-x="-8" - inkscape:window-y="-8" - inkscape:window-maximized="1" - inkscape:current-layer="layer4" - borderlayer="false" - fit-margin-top="-100.6" - fit-margin-left="0" - fit-margin-right="0" - fit-margin-bottom="0" - showborder="true" - inkscape:showpageshadow="false" /> - <g - inkscape:groupmode="layer" - id="layer4" - inkscape:label="bg" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <rect - style="fill:#d1d1d1;fill-opacity:1;stroke-width:1.79999995;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0" - id="rect13505-6" - width="98.575218" - height="70.808708" - x="328.8374" - y="317.09564" /> - </g> - <g - inkscape:groupmode="layer" - id="layer2" - inkscape:label="boxes" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - id="rect2996-1" - width="187.88171" - height="52.881706" - x="75.764778" - y="5.5253706" /> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7" - width="634.0592" - height="73.027374" - x="60.830574" - y="130.24477" /> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2.02648067;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-4" - width="635.80048" - height="74.768661" - x="62.169655" - y="315.43158" /> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2.85834479;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-0" - width="886.87543" - height="106.64049" - x="-48.78373" - y="540.24988" /> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:3.13159013;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:6.26318017, 3.13159009;stroke-dashoffset:0;display:inline" - id="rect2996-1-5" - width="223.0157" - height="109.20289" - x="409.68008" - y="420.63235" /> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2.90856051;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:5.81712091, 2.90856046;stroke-dashoffset:0;display:inline" - id="rect2996-1-5-4" - width="191.98872" - height="109.42592" - x="644.63062" - y="419.66205" /> - <rect - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2.08755708;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:4.17511403, 2.08755702;stroke-dashoffset:0;display:inline" - id="rect2996-1-5-4-6" - width="154.05972" - height="70.246925" - x="678.59509" - y="214.87654" /> - </g> - <g - inkscape:groupmode="layer" - id="layer3" - inkscape:label="blue headers" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <rect - style="fill:#749aba;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.85091281;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9" - width="16.994427" - height="73.79715" - x="59.561817" - y="129.601" /> - <rect - style="fill:#749aba;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.83000004;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-4" - width="17.015339" - height="72.050293" - x="384.61731" - y="130.22485" /> - <rect - style="fill:#749aba;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.86642051;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-8" - width="16.978918" - height="75.107468" - x="261.76944" - y="315.16946" /> - <rect - style="fill:#749aba;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.36914372;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-82" - width="48.412117" - height="14.17484" - x="-42.956367" - y="549.14984" /> - <rect - style="fill:#97ffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.83000004;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-4-1" - width="17.015339" - height="72.050293" - x="241.39912" - y="131.17525" /> - <rect - style="fill:#97ffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.36399999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-4-1-3" - width="16.981569" - height="74.882637" - x="568.40881" - y="315.33447" /> - <rect - style="fill:#97ffff;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.95599997;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-4-1-3-7" - width="49.319912" - height="12.752681" - x="-43.016232" - y="595.7439" /> - </g> - <g - inkscape:groupmode="layer" - id="layer5" - inkscape:label="red headers" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.83000004;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45" - width="17.015339" - height="72.050293" - x="501.49307" - y="130.29137" /> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.84049058;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-5" - width="17.004848" - height="72.923683" - x="678.04279" - y="130.29662" /> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.85091281;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-1" - width="16.994427" - height="73.79715" - x="681.8158" - y="316.14957" /> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.86126781;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-7" - width="16.984072" - height="74.670677" - x="500.62485" - y="315.92252" /> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.82472873;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-11" - width="17.020611" - height="71.613625" - x="175.33748" - y="131.40486" /> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.86642051;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-52" - width="16.978918" - height="75.107468" - x="62.221222" - y="315.0412" /> - <rect - style="fill:#ff7b6d;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.39574718;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-76" - width="48.805244" - height="14.612387" - x="-42.996674" - y="572.61749" /> - </g> - <g - inkscape:groupmode="layer" - id="layer9" - inkscape:label="unused space" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <rect - style="fill:#dddddd;fill-opacity:1;stroke-width:1.79999995;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0" - id="rect13505" - width="98.575218" - height="70.808708" - x="402.22061" - y="131.06841" /> - <rect - style="fill:#dddddd;fill-opacity:1;stroke-width:1.79999995;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0" - id="rect13505-8" - width="96.700218" - height="70.808708" - x="77.587402" - y="131.47064" /> - <rect - style="fill:#dddddd;fill-opacity:1;stroke-width:1.79999995;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;display:inline" - id="rect13505-5" - width="220.21585" - height="72.839958" - x="279.26709" - y="316.08002" /> - <rect - style="fill:#dddddd;fill-opacity:1;stroke:#000000;stroke-width:1.12016988;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;display:inline" - id="rect13505-59" - width="51.879829" - height="15.10388" - x="445.6301" - y="550.76691" /> - <rect - style="fill:none;stroke:#000000;stroke-width:1.12016988;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;display:inline" - id="rect13505-59-3" - width="51.879829" - height="15.10388" - x="445.62964" - y="574.00262" /> - </g> - <g - inkscape:groupmode="layer" - id="layer8" - inkscape:label="pad headers" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <rect - style="fill:#fffec5;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-7-3" - width="49.88493" - height="73.447571" - x="518.21405" - y="316.16635" /> - <rect - style="fill:#fffec5;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.86126781;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-7-3-2" - width="16.98407" - height="74.670677" - x="245.17551" - y="315.48059" /> - <rect - style="fill:#fffec5;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:2.02099991;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-7-3-4" - width="49.474121" - height="72.084908" - x="193.07074" - y="130.93698" /> - <rect - style="fill:#fffec5;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;display:inline" - id="rect2996-1-7-9-45-7-3-6" - width="51.75993" - height="14.072571" - x="445.05756" - y="596.40125" /> - </g> - <g - inkscape:groupmode="layer" - id="layer6" - inkscape:label="arrows" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <path - style="fill:none;stroke:#000000;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:9, 9;stroke-dashoffset:0;marker-mid:none;marker-end:url(#Arrow2Mend)" - d="m 262.87951,51.152779 c 0,0 148.12631,-3.276651 187.01718,76.272861" - id="path3973" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend)" - d="m 681.9161,128.72302 c -22.09709,-49.497478 -148.13393,-45.873109 -179.42835,0" - id="path3988" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="M 386.69903,129.58525 C 361.95029,80.971668 231.48641,62.20327 177.21864,130.46914" - id="path3990" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend)" - d="m 60.546017,172.89554 c 0,0 -32.703692,23.86486 -60.10407166,-3.53553" - id="path3992" - inkscape:connector-curvature="0" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="m 176.82896,203.22242 c -47.24941,74.32926 -107.438064,49.90804 -116.0476,3.53553" - id="path4035" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="m 502.04581,203.43962 c -25.63262,33.58757 -82.31601,45.11485 -116.67261,2.65165" - id="path4037" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="M 763.23339,214.04621 C 748.83403,184.37018 738.54555,166.795 699.15183,161.8971" - id="path4039" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-mid:none;marker-end:url(#Arrow2Mend-1)" - d="m 769.42057,285.19885 c -0.88389,83.96892 -68.50098,75.57203 -68.50098,75.57203" - id="path4041" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="M 682.35804,313.04117 C 652.306,280.33749 539.16892,270.61477 501.16193,313.92506" - id="path4043" - inkscape:connector-curvature="0" /> - <path - style="fill:none;stroke:#000000;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:9, 9;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" - d="m 415.42523,202.55574 c 0,36.23922 -4.41941,88.38835 -35.35533,109.60155" - id="path4045" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:3;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:9, 9;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" - d="M 375.65048,315.69282 C 336.75961,232.60777 166.1701,311.27341 143.18912,205.20739" - id="path4047" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="M 263.39727,315.69282 C 245.7196,288.29244 86.62058,275.91807 62.755726,313.04117" - id="path4051" - inkscape:connector-curvature="0" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend)" - d="m 61.790091,352.05822 c -25.819377,20.1091 -49.573204,20.1091 -61.96650422,1.43636" - id="path4053" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:2.54999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:7.65, 7.65;stroke-dashoffset:0;marker-end:url(#Arrow2Mend)" - d="m 448.12892,630.25126 48.61359,0" - id="path5241" - inkscape:connector-curvature="0" /> - <path - style="fill:none;stroke:#000000;stroke-width:2.09116507px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend);display:inline" - d="m -39.741559,626.33548 c 10.599699,-0.12345 25.528414,-0.12564 43.719789,-0.81161" - id="path4053-2" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1)" - d="m 499.39416,389.93904 c -46.84583,17.67767 -206.82873,31.8198 -238.64854,1.76776" - id="path13236" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1);display:inline" - d="m 502.12201,419.58783 c 2.37436,-10.40132 1.73096,-5.65101 4.38262,-26.86421" - id="path4043-4" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow2Mstart);marker-end:url(#Arrow2Mend-1);display:inline" - d="m 517.94842,353.38466 c 19.7099,0 43.91577,-0.61421 66.57012,-0.61421" - id="path4043-4-3" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow2Mstart);marker-end:url(#Arrow2Mend-1);display:inline" - d="m 501.71494,363.4321 c 19.7099,0 157.04077,-0.61421 179.69512,-0.61421" - id="path4043-4-3-9" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow2Mend-1);display:inline" - d="M 728.67747,419.79091 C 702.92683,395.63959 592.90843,427.2649 577.43509,389.1767" - id="path4043-4-9" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - <path - style="fill:none;stroke:#000000;stroke-width:1.79999995;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-start:url(#Arrow2Mstart);marker-end:url(#Arrow2Mend-1);display:inline" - d="m 60.975741,169.05711 c 19.709901,0 90.307569,-0.61421 112.961919,-0.61421" - id="path4043-4-3-9-1" - inkscape:connector-curvature="0" - sodipodi:nodetypes="cc" /> - </g> - <g - inkscape:groupmode="layer" - id="layer7" - inkscape:label="text" - style="display:inline" - transform="translate(79.549515,-4.4031235)"> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="90.732231" - y="36.767765" - id="text10506" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508" - x="90.732231" - y="36.767765" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">struct malloc_heap</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="580.66718" - y="107.47876" - id="text10506-2" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1" - x="580.66718" - y="107.47876" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="438.12686" - y="223.50792" - id="text10506-2-5" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-7" - x="438.12686" - y="223.50792" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="581.31598" - y="298.638" - id="text10506-2-61" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-89" - x="581.31598" - y="298.638" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="274.6084" - y="99.764236" - id="text10506-2-2" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-79" - x="274.6084" - y="99.764236" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="301.12491" - y="423.26556" - id="text10506-2-54" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-3" - x="301.12491" - y="423.26556" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="133.18704" - y="303.94128" - id="text10506-2-1" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-2" - x="133.18704" - y="303.94128" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="16.340637" - y="561.27954" - id="text10506-2-3" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34" - x="16.340637" - y="561.27954" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Free element header(struct malloc_elem, state = FREE)</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="16.996887" - y="583.24792" - id="text10506-2-3-1" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34-1" - x="16.996887" - y="583.24792" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Used element header(struct malloc_elem, state = BUSY)</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="108.84206" - y="161.39597" - id="text10506-2-6-8" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-7" - x="108.84206" - y="161.39597" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">size</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="62.299515" - y="119.27286" - id="text10506-2-6-4" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-2" - x="62.299515" - y="119.27286" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Memseg 0</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="63.905106" - y="406.73242" - id="text10506-2-6-4-7" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-2-7" - x="63.905106" - y="406.73242" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Memseg 1</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="-25.028084" - y="192.57199" - id="text10506-2-9" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-31" - x="-25.028084" - y="192.57199" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="-26.795866" - y="379.95526" - id="text10506-2-98" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-6" - x="-26.795866" - y="379.95526" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="416.73682" - y="269.53305" - id="text10506-2-6-5" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-0" - x="416.73682" - y="269.53305" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">next_free</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="228.00418" - y="259.55359" - id="text10506-2-6-5-2" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-0-8" - x="228.00418" - y="259.55359" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">next_free</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="356.16727" - y="55.376503" - id="text10506-2-6-5-6" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-0-0" - x="356.16727" - y="55.376503" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">free_head</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="49.218113" - y="254.00189" - id="text10506-2-9-0" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-31-9" - x="49.218113" - y="254.00189" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">prev</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="690.51538" - y="236.82936" - id="text10506-2-6-0" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-06" - x="690.51538" - y="236.82936" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Dummy Elements:</tspan><tspan - sodipodi:role="line" - x="690.51538" - y="256.02936" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan13581">Size = 0</tspan><tspan - sodipodi:role="line" - x="690.51538" - y="275.22937" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan13583">State = BUSY</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="541.03906" - y="347.20566" - id="text10506-2-6-8-8" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-7-9" - x="541.03906" - y="347.20566" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">pad</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="16.661926" - y="605.21631" - id="text10506-2-3-1-4" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34-1-4" - x="16.661926" - y="605.21631" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Pad element header(struct malloc_elem, state = PAD)</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="17.290833" - y="627.77881" - id="text10506-2-3-1-6" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34-1-0" - x="17.290833" - y="627.77881" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Generic element pointers</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="429.11118" - y="449.84528" - id="text10506-2-6-6" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - x="429.11118" - y="449.84528" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan13711">Malloc element header:</tspan><tspan - sodipodi:role="line" - x="429.11118" - y="469.04529" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan13713">state = BUSY</tspan><tspan - sodipodi:role="line" - x="429.11118" - y="488.24527" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan13715">size = <size></tspan><tspan - sodipodi:role="line" - x="429.11118" - y="507.44528" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan13717">pad = <padsize></tspan></text> - <flowRoot - xml:space="preserve" - id="flowRoot13719" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"><flowRegion - id="flowRegion13721"><rect - id="rect13723" - width="968.73627" - height="188.26718" - x="-81.317276" - y="460.64972" /></flowRegion><flowPara - id="flowPara13725"></flowPara></flowRoot> <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="594.30859" - y="378.91797" - id="text10506-2-6-8-8-1" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-8-7-9-3" - x="594.30859" - y="378.91797" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">size</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="505.86865" - y="563.34613" - id="text10506-2-3-1-6-8" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34-1-0-4" - x="505.86865" - y="563.34613" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Free / Unallocated data space</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="660.39099" - y="449.92532" - id="text10506-2-6-6-0" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - x="660.39099" - y="449.92532" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan14527">Pad element header:</tspan><tspan - sodipodi:role="line" - x="660.39099" - y="469.12534" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan14531">state = PAD</tspan><tspan - sodipodi:role="line" - x="660.39099" - y="488.32532" - style="font-size:16px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas" - id="tspan14533">pad = padsize</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="506.5249" - y="584.28369" - id="text10506-2-3-1-6-8-7" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34-1-0-4-2" - x="506.5249" - y="584.28369" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Used / allocated data space</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:120.00000477%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;display:inline;font-family:Sans" - x="506.18994" - y="605.30322" - id="text10506-2-3-1-6-8-7-0" - sodipodi:linespacing="120%"><tspan - sodipodi:role="line" - id="tspan10508-1-34-1-0-4-2-1" - x="506.18994" - y="605.30322" - style="font-size:14px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:120.00000477%;writing-mode:lr-tb;text-anchor:start;font-family:Consolas;-inkscape-font-specification:Consolas">Padding / unavailable space</tspan></text> - </g> + <defs id="Patterns_And_Gradients"> + <pattern id="ptrn2-71" patternUnits="userSpaceOnUse" width="6" height="6" viewBox="0 0 64 64"> + <image x="0" y="0" width="64" height="64" image-rendering="optimizeSpeed" + xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAICAYAAADED76LAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAA7SURBVChTY/i3f7/Pv3//MDFIXETEhwGfJIjGVIAkCeKjKkCTRFWARRKhAIckRAEeSYgCPJL/9u/3AQC1aLsBz7wFUwAAAABJRU5ErkJggg=="/> + </pattern> + <linearGradient id="grad0-168" x1="0" y1="0" x2="1" y2="0" gradientTransform="rotate(60 0.5 0.5)"> + <stop offset="0" stop-color="#e9eff7" stop-opacity="1"/> + <stop offset="0.24" stop-color="#f4f7fb" stop-opacity="1"/> + <stop offset="0.54" stop-color="#feffff" stop-opacity="1"/> + </linearGradient> + </defs> + <defs id="Markers"> + <g id="lend10"> + <path + d="M 0 0.75 C -0.414214 0.75 -0.75 0.414214 -0.75 0 -0.75 -0.414214 -0.414214 -0.75 0 -0.75 0.414214 -0.75 0.75 -0.414214 0.75 0 0.75 0.414214 0.414214 0.75 0 0.75 Z " + style="stroke:none"/> + </g> + <marker id="mrkr10-144" class="st25" refX="2.79" orient="auto" markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend10" transform="scale(4.36) "/> + </marker> + <marker id="mrkr10-146" class="st25" refX="-2.79" orient="auto" markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend10" transform="scale(-4.36,-4.36) "/> + </marker> + <g id="lend5"> + <path d="M 2 1 L 0 0 L 1.98117 -0.993387 C 1.67173 -0.364515 1.67301 0.372641 1.98465 1.00043 " style="stroke:none"/> + </g> + <marker id="mrkr5-171" class="st32" refX="-7.15" orient="auto" markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend5" transform="scale(-4.36,-4.36) "/> + </marker> + </defs> + <defs id="Filters"> + <filter id="filter_2"> + <feGaussianBlur stdDeviation="2"/> + </filter> + </defs> + <g> + <title>Page-1</title> + <g id="group14-1" transform="translate(45,-360)"> + <title>Sheet.14</title> + <g id="shape3-2"> + <title>Sheet.3</title> + <g id="shadow3-3" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="18" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="18" height="81" class="st3"/> + </g> + <g id="shape4-7" transform="translate(18,0)"> + <title>Sheet.4</title> + <g id="shadow4-8" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="117" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="117" height="81" class="st4"/> + </g> + </g> + <g id="group15-12" transform="translate(180,-360)"> + <title>Sheet.15</title> + <g id="shape5-13"> + <title>Sheet.5</title> + <g id="shadow5-14" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="18" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="18" height="81" class="st5"/> + </g> + <g id="shape6-18" transform="translate(18,0)"> + <title>Sheet.6</title> + <g id="shadow6-19" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="117" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="117" height="81" class="st6"/> + </g> + </g> + <g id="shape7-23" transform="translate(612,-360)"> + <title>Sheet.7</title> + <g id="shadow7-24" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="18" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="18" height="81" class="st5"/> + </g> + <g id="shape10-28" transform="translate(630,-360)"> + <title>Sheet.10</title> + <g id="shadow10-29" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="51.75" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="51.75" height="81" class="st7"/> + </g> + <g id="shape12-33" transform="translate(681.75,-360)"> + <title>Sheet.12</title> + <g id="shadow12-34" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="18" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="18" height="81" class="st8"/> + </g> + <g id="shape13-38" transform="translate(699.75,-360)"> + <title>Sheet.13</title> + <g id="shadow13-39" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="47.25" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="47.25" height="81" class="st6"/> + </g> + <g id="group29-43" transform="translate(315,-360)"> + <title>Sheet.29</title> + <g id="shape23-44"> + <title>Sheet.23</title> + <g id="shadow23-45" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="18" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="18" height="81" class="st3"/> + </g> + <g id="shape24-49" transform="translate(18,0)"> + <title>Sheet.24</title> + <g id="shadow24-50" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="36" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="36" height="81" class="st4"/> + </g> + </g> + <g id="group30-54" transform="translate(477,-360)"> + <title>Sheet.30</title> + <g id="shape27-55"> + <title>Sheet.27</title> + <g id="shadow27-56" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="18" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="18" height="81" class="st3"/> + </g> + <g id="shape28-60" transform="translate(18,0)"> + <title>Sheet.28</title> + <g id="shadow28-61" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="117" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="117" height="81" class="st4"/> + </g> + </g> + <g id="shape31-65" transform="translate(369,-360)"> + <title>Sheet.31</title> + <g id="shadow31-66" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="531" width="108" height="81" class="st2"/> + </g> + <rect x="0" y="531" width="108" height="81" class="st9"/> + </g> + <g id="shape32-72" transform="translate(184.5,-260)"> + <title>Sheet.32</title> + <g id="shadow32-73" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st10"/> + </g> + <g id="shape39-77" transform="translate(252,-259)"> + <title>Sheet.39</title> + <desc>Free element header</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Free element header</text> </g> + <g id="shape43-80" transform="translate(184.5,-232)"> + <title>Sheet.43</title> + <g id="shadow43-81" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st13"/> + </g> + <g id="shape44-85" transform="translate(252,-231)"> + <title>Sheet.44</title> + <desc>Used element header</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Used element header</text> </g> + <g id="shape46-88" transform="translate(409.5,-260)"> + <title>Sheet.46</title> + <g id="shadow46-89" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st14"/> + </g> + <g id="shape47-93" transform="translate(477,-259)"> + <title>Sheet.47</title> + <desc>Free space</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Free space</text> </g> + <g id="shape49-96" transform="translate(409.5,-232)"> + <title>Sheet.49</title> + <g id="shadow49-97" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st15"/> + </g> + <g id="shape50-101" transform="translate(477,-231)"> + <title>Sheet.50</title> + <desc>Allocated data</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Allocated data</text> </g> + <g id="shape52-104" transform="translate(184.5,-204)"> + <title>Sheet.52</title> + <g id="shadow52-105" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st16"/> + </g> + <g id="shape53-109" transform="translate(252,-203)"> + <title>Sheet.53</title> + <desc>Pad element header</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Pad element header</text> </g> + <g id="shape62-112" transform="translate(409.5,-204)"> + <title>Sheet.62</title> + <g id="shadow62-113" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st17"/> + </g> + <g id="shape63-117" transform="translate(477,-203)"> + <title>Sheet.63</title> + <desc>Padding</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Padding</text> </g> + <g id="shape65-120" transform="translate(184.5,-176)"> + <title>Sheet.65</title> + <g id="shadow65-121" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <rect x="0" y="594" width="63" height="18" class="st2"/> + </g> + <rect x="0" y="594" width="63" height="18" class="st18"/> + </g> + <g id="shape66-126" transform="translate(252,-175)"> + <title>Sheet.66</title> + <desc>Unavailable space</desc> + <rect x="0" y="592" width="135" height="20" class="st11"/> + <text x="4" y="605.6" class="st12">Unavailable space</text> </g> + <g id="shape97-129" transform="translate(612,-375.75)"> + <title>Simple Double Arrow</title> + <desc>size</desc> + <path d="M0 612 L18 598.5 L18 605.25 L117 605.25 L117 598.5 L135 612 L117 625.5 L117 618.75 L18 618.75 L18 625.5 L0 612 + Z" class="st19"/> + <text x="59.93" y="615" class="st20">size</text> </g> + <g id="shape99-132" transform="translate(630,-400.5)"> + <title>Simple Double Arrow.99</title> + <desc>pad</desc> + <g id="shadow99-133" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 612 L12 600 L12 606 L57.75 606 L57.75 600 L69.75 612 L57.75 624 L57.75 618 L12 618 L12 624 L0 612 Z" + class="st21"/> + </g> + <path d="M0 612 L12 600 L12 606 L57.75 606 L57.75 600 L69.75 612 L57.75 624 L57.75 618 L12 618 L12 624 L0 612 Z" + class="st22"/> + <text x="27.23" y="615" class="st23">pad</text> </g> + <g id="shape113-138" transform="translate(54,-337.5)"> + <title>Sheet.113</title> + <desc>prev/next</desc> + <path d="M134.64 591.56 L134.58 591.92 A72 22.5 0 0 1 63 612 A63 22.5 0 0 1 0.37 591.92 L0.31 591.57" class="st24"/> + <rect x="43.4968" y="593.55" width="48.0064" height="14.4001" class="st26"/> + <text x="43.5" y="604.35" class="st27">prev/next</text> </g> + <g id="shape115-149" transform="translate(324,-337.5)"> + <title>Sheet.115</title> + <desc>prev/next</desc> + <path d="M0.44 591.55 L0.51 591.9 A90 22.5 -180 0 0 90 612 A72 22.5 -180 0 0 161.58 591.92 L161.64 591.56" class="st24"/> + <rect x="56.9968" y="593.55" width="48.0064" height="14.4001" class="st28"/> + <text x="57" y="604.35" class="st27">prev/next</text> </g> + <g id="shape118-158" transform="translate(315,-390.375)"> + <title>Simple Double Arrow.118</title> + <desc>size</desc> + <g id="shadow118-159" transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 612 L12 600 L12 606 L42 606 L42 600 L54 612 L42 624 L42 618 L12 618 L12 624 L0 612 Z" class="st21"/> + </g> + <path d="M0 612 L12 600 L12 606 L42 606 L42 600 L54 612 L42 624 L42 618 L12 618 L12 624 L0 612 Z" class="st29"/> + <text x="19.43" y="615" class="st30">size</text> </g> + <g id="shape119-164" transform="translate(54,-441)"> + <title>Sheet.119</title> + <desc>next free</desc> + <path d="M-0 612 A135 22.5 0 0 1 135 589.5 A134.606 21.534 0 0 1 266.35 606.33 L266.56 606.62" class="st31"/> + <rect x="112.807" y="593.55" width="43.9926" height="14.4001" class="st26"/> + <text x="112.81" y="604.35" class="st33">next free</text> </g> + <g id="shape120-174" transform="translate(323.739,-441.34)"> + <title>Sheet.120</title> + <desc>next free</desc> + <path d="M0.24 612 A78.4445 18.5592 178.15 0 1 72.26 589.84 A81.2523 26.5101 179.07 0 1 159.23 607.01 L159.43 607.31" + class="st31"/> + <rect x="59.193" y="593.55" width="43.9926" height="14.4001" class="st28"/> + <text x="59.19" y="604.35" class="st33">next free</text> </g> + <g id="shape122-182" transform="translate(189,-337.5)"> + <title>Sheet.122</title> + <desc>prev/next</desc> + <path d="M0.33 591.57 L0.39 591.92 A67.5 22.5 -180 0 0 67.5 612 A69.1875 22.5 -180 0 0 136.29 591.92 L136.35 591.56" + class="st24"/> + <rect x="44.3405" y="593.55" width="48.0064" height="14.4001" class="st26"/> + <text x="44.34" y="604.35" class="st27">prev/next</text> </g> + <g id="shape123-191" transform="translate(486.563,-337.5)"> + <title>Sheet.123</title> + <desc>prev/next</desc> + <path d="M0.35 591.56 L0.41 591.92 A71.4375 22.5 -180 0 0 71.44 612 A63 22.5 -180 0 0 134.07 591.92 L134.12 591.57" + class="st24"/> + <rect x="43.2155" y="593.55" width="48.0064" height="14.4001" class="st26"/> + <text x="43.22" y="604.35" class="st27">prev/next</text> </g> + </g> </svg> diff --git a/doc/guides/prog_guide/img/stateful-op.svg b/doc/guides/prog_guide/img/stateful-op.svg new file mode 100644 index 00000000..e6ef6353 --- /dev/null +++ b/doc/guides/prog_guide/img/stateful-op.svg @@ -0,0 +1,116 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export stateful-ops.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="1.49139in" height="1.35359in" + viewBox="0 0 107.38 97.4587" xml:space="preserve" color-interpolation-filters="sRGB" class="st6"> + <v:documentProperties v:langID="16393" v:metric="true" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25} + .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em} + .st5 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st6 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <defs id="Filters"> + <filter id="filter_2"> + <feGaussianBlur stdDeviation="2"/> + </filter> + </defs> + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="8.50394" + v:shadowOffsetY="-8.50394"/> + <g id="shape38-1" v:mID="38" v:groupContext="shape" transform="translate(58.305,-28.025)"> + <title>Circle</title> + <desc>stream</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="22.6772" cy="74.7815" width="39.69" height="34.0157"/> + <g id="shadow38-2" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 74.78 A22.6772 22.6772 0 0 1 45.35 74.78 A22.6772 22.6772 0 1 1 0 74.78 Z" class="st2"/> + </g> + <path d="M0 74.78 A22.6772 22.6772 0 0 1 45.35 74.78 A22.6772 22.6772 0 1 1 0 74.78 Z" class="st3"/> + <text x="8.43" y="77.78" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>stream</text> </g> + <g id="shape39-7" v:mID="39" v:groupContext="shape" transform="translate(3.0294,-73.3793)"> + <title>Circle.39</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/> + <g id="shadow39-8" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/> + </g> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/> + <text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape40-13" v:mID="40" v:groupContext="shape" transform="translate(3.0294,-50.7021)"> + <title>Circle.40</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/> + <g id="shadow40-14" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/> + </g> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/> + <text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape41-19" v:mID="41" v:groupContext="shape" transform="translate(3.0294,-28.025)"> + <title>Circle.41</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/> + <g id="shadow41-20" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/> + </g> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/> + <text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape42-25" v:mID="42" v:groupContext="shape" transform="translate(3.0294,-5.34779)"> + <title>Circle.249</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="11.3386" cy="86.1201" width="19.85" height="17.0079"/> + <g id="shadow42-26" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st2"/> + </g> + <path d="M0 86.12 A11.3386 11.3386 0 0 1 22.68 86.12 A11.3386 11.3386 0 1 1 0 86.12 Z" class="st3"/> + <text x="6.07" y="89.12" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape43-31" v:mID="43" v:groupContext="shape" transform="translate(66.3024,-75.8604) rotate(24.6166)"> + <title>Sheet.43</title> + <path d="M0 97.46 L43.16 97.46" class="st5"/> + </g> + <g id="shape44-34" v:mID="44" v:groupContext="shape" transform="translate(37.2064,-61.3598) rotate(6.77654)"> + <title>Sheet.44</title> + <path d="M0 97.46 L34.05 97.46" class="st5"/> + </g> + <g id="shape45-37" v:mID="45" v:groupContext="shape" transform="translate(-6.31062,-33.9543) rotate(-19.179)"> + <title>Sheet.45</title> + <path d="M0 97.46 L34.51 97.46" class="st5"/> + </g> + <g id="shape46-40" v:mID="46" v:groupContext="shape" transform="translate(-14.8893,-7.82888) rotate(-24.6166)"> + <title>Sheet.46</title> + <path d="M0 97.46 L43.16 97.46" class="st5"/> + </g> + </g> +</svg> diff --git a/doc/guides/prog_guide/img/stateless-op-shared.svg b/doc/guides/prog_guide/img/stateless-op-shared.svg new file mode 100644 index 00000000..257a69a5 --- /dev/null +++ b/doc/guides/prog_guide/img/stateless-op-shared.svg @@ -0,0 +1,124 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export Drawing5.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="1.89687in" height="1.60662in" + viewBox="0 0 136.575 115.676" xml:space="preserve" color-interpolation-filters="sRGB" class="st7"> + <v:documentProperties v:langID="16393" v:metric="true" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25} + .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em} + .st5 {fill:none} + .st6 {stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:0.999999} + .st7 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <defs id="Filters"> + <filter id="filter_2"> + <feGaussianBlur stdDeviation="2"/> + </filter> + </defs> + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="8.50394" + v:shadowOffsetY="-8.50394"/> + <g id="group47-1" transform="translate(3.02997,-5.34779)" v:mID="47" v:groupContext="group"> + <title>Sheet.47</title> + <g id="shape36-2" v:mID="36" v:groupContext="shape" transform="translate(66.2255,-27.0553)"> + <title>Circle</title> + <desc>priv_xform</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/> + <v:textRect cx="31.7998" cy="88.2699" width="55.66" height="40.7542"/> + <g id="shadow36-3" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 88.27 A31.7998 27.1694 0 1 1 63.6 88.27 A31.7998 27.1694 0 1 1 0 88.27 Z" class="st2"/> + </g> + <path d="M0 88.27 A31.7998 27.1694 0 1 1 63.6 88.27 A31.7998 27.1694 0 1 1 0 88.27 Z" class="st3"/> + <text x="9.47" y="91.27" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text> </g> + <g id="shape39-8" v:mID="39" v:groupContext="shape" transform="translate(-5.9952E-015,-81.5083)"> + <title>Circle.40</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/> + <v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/> + <g id="shadow39-9" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/> + </g> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/> + <text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape40-14" v:mID="40" v:groupContext="shape" transform="translate(-5.9952E-015,-54.3389)"> + <title>Circle.41</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/> + <v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/> + <g id="shadow40-15" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/> + </g> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/> + <text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape41-20" v:mID="41" v:groupContext="shape" transform="translate(-5.9952E-015,-27.1694)"> + <title>Circle.42</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/> + <v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/> + <g id="shadow41-21" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/> + </g> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/> + <text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape42-26" v:mID="42" v:groupContext="shape"> + <title>Circle.249</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(3.99999,3.99999,3.99999,3.99999)" v:tabSpace="42.5196"/> + <v:textRect cx="13.5848" cy="101.968" width="23.78" height="20.3771"/> + <g id="shadow42-27" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st2"/> + </g> + <path d="M0 101.97 A13.5848 13.5848 0 1 1 27.17 101.97 A13.5848 13.5848 0 1 1 0 101.97 Z" class="st3"/> + <text x="8.32" y="104.97" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape43-32" v:mID="43" v:groupContext="shape" transform="translate(75.3544,-84.7046) rotate(24.6166)"> + <title>Sheet.43</title> + <path d="M0 115.68 L51.71 115.68 L0 115.68 Z" class="st5"/> + <path d="M0 115.68 L51.71 115.68" class="st6"/> + </g> + <g id="shape44-35" v:mID="44" v:groupContext="shape" transform="translate(40.8189,-67.2403) rotate(6.77654)"> + <title>Sheet.44</title> + <path d="M0 115.68 L40.8 115.68 L0 115.68 Z" class="st5"/> + <path d="M0 115.68 L40.8 115.68" class="st6"/> + </g> + <g id="shape45-38" v:mID="45" v:groupContext="shape" transform="translate(-10.8336,-34.4585) rotate(-19.179)"> + <title>Sheet.45</title> + <path d="M0 115.68 L41.35 115.68 L0 115.68 Z" class="st5"/> + <path d="M0 115.68 L41.35 115.68" class="st6"/> + </g> + <g id="shape46-41" v:mID="46" v:groupContext="shape" transform="translate(-21.0159,-3.19618) rotate(-24.6166)"> + <title>Sheet.46</title> + <path d="M0 115.68 L51.71 115.68 L0 115.68 Z" class="st5"/> + <path d="M0 115.68 L51.71 115.68" class="st6"/> + </g> + </g> + </g> +</svg> diff --git a/doc/guides/prog_guide/img/stateless-op.svg b/doc/guides/prog_guide/img/stateless-op.svg new file mode 100644 index 00000000..fd951b7a --- /dev/null +++ b/doc/guides/prog_guide/img/stateless-op.svg @@ -0,0 +1,140 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Generated by Microsoft Visio, SVG Export stateless-ops.svg Page-1 --> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ev="http://www.w3.org/2001/xml-events" + xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/" width="2.24024in" height="2.70592in" + viewBox="0 0 161.298 194.826" xml:space="preserve" color-interpolation-filters="sRGB" class="st8"> + <v:documentProperties v:langID="16393" v:metric="true" v:viewMarkup="false"/> + + <style type="text/css"> + <![CDATA[ + .st1 {visibility:visible} + .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22} + .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25} + .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em} + .st5 {fill:#feffff;font-family:Calibri;font-size:0.75em} + .st6 {marker-start:url(#mrkr13-19);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1} + .st7 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409094308259} + .st8 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3} + ]]> + </style> + + <defs id="Markers"> + <g id="lend13"> + <path d="M 3 1 L 0 0 L 3 -1 L 3 1 " style="stroke:none"/> + </g> + <marker id="mrkr13-19" class="st7" v:arrowType="13" v:arrowSize="2" v:setback="0" refX="0" orient="auto" + markerUnits="strokeWidth" overflow="visible"> + <use xlink:href="#lend13" transform="scale(3.5199995788296) "/> + </marker> + </defs> + <defs id="Filters"> + <filter id="filter_2"> + <feGaussianBlur stdDeviation="2"/> + </filter> + </defs> + <g v:mID="0" v:index="1" v:groupContext="foregroundPage"> + <title>Page-1</title> + <v:pageProperties v:drawingScale="0.0393701" v:pageScale="0.0393701" v:drawingUnits="24" v:shadowOffsetX="9" + v:shadowOffsetY="-9"/> + <g id="group61-1" transform="translate(3.02943,-5.34782)" v:mID="61" v:groupContext="group"> + <title>Sheet.61</title> + <g id="shape52-2" v:mID="52" v:groupContext="shape" transform="translate(97.856,-133.39)"> + <title>Circle.40</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="28.3463" cy="167.479" width="49.62" height="41.4408"/> + <g id="shadow52-3" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st2"/> + </g> + <path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st3"/> + <text x="23.08" y="170.48" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape53-8" v:mID="53" v:groupContext="shape" transform="translate(-3.9968E-015,-133.39)"> + <title>Circle.299</title> + <desc>priv_xform</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="28.3463" cy="167.479" width="49.62" height="41.4408"/> + <g id="shadow53-9" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st2"/> + </g> + <path d="M-0 167.48 A28.3465 27.6272 180 0 1 56.69 167.48 A28.3465 27.6272 180 0 1 -0 167.48 Z" class="st3"/> + <text x="8.25" y="170.18" class="st5" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text> </g> + <g id="shape54-14" v:mID="54" v:groupContext="shape" transform="translate(56.693,-160.74)"> + <title>Sheet.54</title> + <path d="M0 194.83 L10.2 194.83 L10.56 194.83 L41.16 194.83" class="st6"/> + </g> + <g id="shape55-20" v:mID="55" v:groupContext="shape" transform="translate(97.856,-65.1969)"> + <title>Circle.479</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="28.3463" cy="166.185" width="49.62" height="42.5197"/> + <g id="shadow55-21" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st2"/> + </g> + <path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st3"/> + <text x="23.08" y="169.19" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape56-26" v:mID="56" v:groupContext="shape" transform="translate(-3.9968E-015,-65.7801)"> + <title>Circle.480</title> + <desc>priv_xform</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="28.3463" cy="166.768" width="49.62" height="42.5197"/> + <g id="shadow56-27" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st2"/> + </g> + <path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st3"/> + <text x="8.25" y="169.47" class="st5" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text> </g> + <g id="shape57-32" v:mID="57" v:groupContext="shape" transform="translate(56.693,-93.8414)"> + <title>Sheet.57</title> + <path d="M0 194.83 L10.2 194.83 L10.56 194.83 L41.16 194.83" class="st6"/> + </g> + <g id="shape58-37" v:mID="58" v:groupContext="shape" transform="translate(97.856,0)"> + <title>Circle.482</title> + <desc>op</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="28.3463" cy="166.185" width="49.62" height="42.5197"/> + <g id="shadow58-38" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st2"/> + </g> + <path d="M0 166.19 A28.3465 28.3465 0 1 1 56.69 166.19 A28.3465 28.3465 0 1 1 0 166.19 Z" class="st3"/> + <text x="23.08" y="169.19" class="st4" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>op</text> </g> + <g id="shape59-43" v:mID="59" v:groupContext="shape" transform="translate(-3.9968E-015,-0.583223)"> + <title>Circle.483</title> + <desc>priv_xform</desc> + <v:userDefs> + <v:ud v:nameU="visVersion" v:val="VT0(15):26"/> + </v:userDefs> + <v:textBlock v:margins="rect(4,4,4,4)" v:tabSpace="42.5197"/> + <v:textRect cx="28.3463" cy="166.768" width="49.62" height="42.5197"/> + <g id="shadow59-44" v:groupContext="shadow" v:shadowOffsetX="0.345598" v:shadowOffsetY="-1.97279" v:shadowType="1" + transform="matrix(1,0,0,1,0.345598,1.97279)" class="st1"> + <path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st2"/> + </g> + <path d="M0 166.77 A28.3465 28.3465 0 0 1 56.69 166.77 A28.3465 28.3465 0 0 1 0 166.77 Z" class="st3"/> + <text x="8.25" y="169.47" class="st5" v:langID="16393"><v:paragraph v:horizAlign="1"/><v:tabList/>priv_xform</text> </g> + <g id="shape60-49" v:mID="60" v:groupContext="shape" transform="translate(56.693,-28.6446)"> + <title>Sheet.60</title> + <path d="M0 194.83 L10.2 194.83 L10.56 194.83 L41.16 194.83" class="st6"/> + </g> + </g> + </g> +</svg> diff --git a/doc/guides/prog_guide/img/turbo_tb_decode.svg b/doc/guides/prog_guide/img/turbo_tb_decode.svg new file mode 100644 index 00000000..a259f458 --- /dev/null +++ b/doc/guides/prog_guide/img/turbo_tb_decode.svg @@ -0,0 +1,1471 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2018 Intel Corporation --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="194.21973mm" + height="163.25349mm" + viewBox="0 0 194.21973 163.25349" + version="1.1" + id="svg8" + inkscape:version="0.92.3 (2405546, 2018-03-11)" + sodipodi:docname="turbo_tb_decode.svg"> + <defs + id="defs2"> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8474" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path8472" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path6507" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5140" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5122" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.4,0,0,0.4,4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5134" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(1.1,0,0,1.1,1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5116" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Sstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Sstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5128" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.2,0,0,0.2,1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + id="path5143" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0" + refX="0" + id="Arrow1Send" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5131" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.2,0,0,-0.2,-1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5119" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <filter + id="filter_2"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur4" /> + </filter> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-4" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-9" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-11" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-2" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-6" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-1" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8474-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + inkscape:connector-curvature="0" + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path8472-4" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="0.98994949" + inkscape:cx="148.9027" + inkscape:cy="256.96386" + inkscape:document-units="mm" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:snap-text-baseline="true" + inkscape:window-width="1920" + inkscape:window-height="1137" + inkscape:window-x="1072" + inkscape:window-y="185" + inkscape:window-maximized="1" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0" /> + <metadata + id="metadata5"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + style="display:inline" + transform="translate(-9.7553377,-54.351435)"> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="41.159508" + y="61.13464" + id="text873"><tspan + sodipodi:role="line" + id="tspan871" + x="41.159508" + y="64.996841" + style="font-size:4.23333311px;stroke-width:0.26458332" /></text> + <g + id="g4997"> + <a + id="a990"> + <rect + style="opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.1950596;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect854" + width="44.771584" + height="14.03559" + x="18.573261" + y="54.450935" /> + </a> + <text + id="text877" + y="62.738258" + x="21.648832" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:3.52777767px;stroke-width:0.26458332" + y="62.738258" + x="21.648832" + id="tspan875" + sodipodi:role="line">w<tspan + id="tspan885" + style="font-size:64.99999762%;baseline-shift:sub">k</tspan> LLR circular buffer</tspan></text> + </g> + <rect + style="opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.1981452;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967" + width="25.196077" + height="14.03559" + x="63.344845" + y="54.450935" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="73.52343" + y="62.445942" + id="text877-8"><tspan + sodipodi:role="line" + id="tspan875-6" + x="73.52343" + y="62.445942" + style="font-size:3.52777767px;stroke-width:0.26458332">...</tspan></text> + <rect + style="opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.1950596;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect854-5" + width="44.771584" + height="14.03559" + x="88.540924" + y="54.450935" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="91.90699" + y="62.531521" + id="text877-4"><tspan + sodipodi:role="line" + id="tspan875-0" + x="91.90699" + y="62.531521" + style="font-size:3.52777767px;stroke-width:0.26458332">w<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan885-9">k</tspan> LLR circular buffer</tspan></text> + <rect + style="opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.199;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735" + width="6.6797671" + height="14.033618" + x="11.893495" + y="54.450935" /> + <g + id="g4807" + transform="translate(0,0.188984)"> + <path + inkscape:connector-curvature="0" + id="path4741" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <g + id="g5063"> + <path + inkscape:connector-curvature="0" + id="path4741-0" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <flowRoot + xml:space="preserve" + id="flowRoot4811" + style="font-style:normal;font-weight:normal;font-size:13.33333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + transform="scale(0.26458333)"><flowRegion + id="flowRegion4813"><rect + id="rect4815" + width="41.785713" + height="14.642858" + x="39.285713" + y="287.16254" /></flowRegion><flowPara + id="flowPara4817">offse</flowPara></flowRoot> <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="74.16684" + y="75.043541" + id="text4821-3" + transform="scale(0.95903923,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0" + x="74.16684" + y="75.043541" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="11.603812" + y="75.449066" + id="text4821-3-9" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0" + x="11.603812" + y="75.449066" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="150.53461" + y="58.039307" + id="text4821-3-6" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-1" + x="150.53461" + y="58.039307" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144">The encoded TB is given as a</tspan><tspan + sodipodi:role="line" + x="150.53461" + y="61.97533" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan4877">contiguous buffer</tspan></text> + <path + style="fill:none;stroke:#7f7f7f;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.1199999, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" + d="M 10.423511,83.31801 H 72.162772" + id="path4885" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#818181;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.1199999, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" + d="M 82.319012,83.31801 176.93243,83.184377" + id="path4887" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:5.39796209px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#818181;fill-opacity:1;stroke:none;stroke-width:0.40484715" + x="77.368088" + y="81.855705" + id="text4891" + transform="scale(0.96100989,1.040572)"><tspan + sodipodi:role="line" + id="tspan4889" + x="77.368088" + y="81.855705" + style="fill:#818181;fill-opacity:1;stroke-width:0.40484715">or</tspan></text> + <path + style="fill:none;stroke:#7f7f7f;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.11999992, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" + d="M 9.7553377,181.2723 H 71.494599" + id="path4885-9" + inkscape:connector-curvature="0" /> + <path + style="fill:none;stroke:#818181;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.11999992, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" + d="m 81.786039,181.24224 94.613421,-0.13363" + id="path4887-9" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:5.39796209px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#818181;fill-opacity:1;stroke:none;stroke-width:0.40484715" + x="76.813484" + y="175.96187" + id="text4891-9" + transform="scale(0.9610099,1.040572)"><tspan + sodipodi:role="line" + id="tspan4889-7" + x="76.813484" + y="175.96187" + style="fill:#818181;fill-opacity:1;stroke-width:0.40484715">or</tspan></text> + <rect + style="opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.199;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7" + width="6.6797671" + height="14.033618" + x="10.825401" + y="101.15727" /> + <g + id="g4807-3" + transform="translate(-0.98393573,46.759016)"> + <path + inkscape:connector-curvature="0" + id="path4741-2" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-81" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-3" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="10.57785" + y="120.11156" + id="text4821-3-9-7" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-3" + x="10.57785" + y="120.11156" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <g + transform="matrix(0.99106501,0,0,1.0000618,-0.90257595,46.700562)" + id="g4997-1"> + <a + id="a990-9"> + <rect + style="opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.1950596;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect854-4" + width="44.771584" + height="14.03559" + x="18.573261" + y="54.450935" /> + </a> + <text + id="text877-1" + y="62.738258" + x="21.648832" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:3.52777767px;stroke-width:0.26458332" + y="62.738258" + x="21.648832" + id="tspan875-9" + sodipodi:role="line">w<tspan + id="tspan885-5" + style="font-size:64.99999762%;baseline-shift:sub">k</tspan> LLR circular buffer</tspan></text> + </g> + <g + transform="matrix(0.98808659,0,0,1.0000825,72.450284,46.566642)" + id="g4997-1-4"> + <a + id="a990-9-5"> + <rect + style="opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.1950596;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect854-4-6" + width="44.771584" + height="14.03559" + x="18.573261" + y="54.450935" /> + </a> + <text + id="text877-1-0" + y="62.738258" + x="21.648832" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:3.52777767px;stroke-width:0.26458332" + y="62.738258" + x="21.648832" + id="tspan875-9-8" + sodipodi:role="line">w<tspan + id="tspan885-5-7" + style="font-size:64.99999762%;baseline-shift:sub">k</tspan> LLR circular buffer</tspan></text> + </g> + <g + transform="matrix(1.0292712,0,0,0.99978365,-1.5276486,46.585803)" + id="g5063-5"> + <path + inkscape:connector-curvature="0" + id="path4741-0-0" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-2" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-4" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-8" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="74.729973" + y="119.73718" + id="text4821-3-3" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7" + x="74.729973" + y="119.73718" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <path + style="fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + d="m 61.876283,101.15486 9.985811,-0.03 v 4.04245 l -2.53906,1.46999 5.57925,2.87314 -3.006781,1.60362 v 4.04245 l -10.01922,0.0348 z" + id="path5575" + inkscape:connector-curvature="0" /> + <path + style="fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + d="m 90.802274,101.02207 -9.953242,0.003 -0.03341,3.94222 -2.639287,1.63703 5.679477,2.87314 -2.939965,1.57021 v 4.00905 l 9.886426,0.003 z" + id="path5581" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="64.84626" + y="109.37679" + id="text5879"><tspan + sodipodi:role="line" + id="tspan5877" + x="64.84626" + y="109.37679" + style="stroke-width:0.26458332">..</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="85.559654" + y="109.51042" + id="text5883"><tspan + sodipodi:role="line" + id="tspan5881" + x="85.559654" + y="109.51042" + style="stroke-width:0.26458332">..</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend)" + d="m 66.783973,101.05803 c 0,0 5.144937,-13.096212 18.942727,-0.33409" + id="path5899" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.06663418px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.22999756" + x="155.38319" + y="97.845207" + id="text4821-3-6-2" + transform="scale(0.93400804,1.0706546)"><tspan + sodipodi:role="line" + id="tspan4819-0-1-0" + x="155.38319" + y="97.845207" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756">The encoded TB is given as a</tspan><tspan + sodipodi:role="line" + x="155.38319" + y="101.6785" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan4877-1">"scattered" buffer through a</tspan><tspan + sodipodi:role="line" + x="155.38319" + y="105.5118" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan5992">chained mbuf</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.06663418px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.22999756" + x="147.15628" + y="145.52435" + id="text4821-3-6-2-4" + transform="scale(0.93400803,1.0706546)"><tspan + sodipodi:role="line" + x="147.15628" + y="145.52435" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan5992-2">Result is decoded back into the given output</tspan><tspan + sodipodi:role="line" + x="147.15628" + y="149.35765" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6023">mbuf as one contiguous buffer with no </tspan><tspan + sodipodi:role="line" + x="147.15628" + y="153.19093" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6025">CRC24B retaining</tspan></text> + <g + id="g6253"> + <g + transform="translate(10.356694,1.2027129)" + id="g6079"> + <path + style="fill:#fec000;fill-opacity:1;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + d="m 64.144685,140.61392 -0.03341,-11.65963 h 4.209497 l -0.03341,11.65963 1.971111,-0.0334 -4.042449,3.90882 -4.04245,-3.90882 z" + id="path6066" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="-142.49652" + y="67.226768" + id="text6074" + transform="rotate(-90)"><tspan + sodipodi:role="line" + id="tspan6072" + x="-142.49652" + y="67.226768" + style="font-size:2.82222223px;stroke-width:0.26458332">decode</tspan></text> + </g> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.06663418px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.22999756" + x="146.80908" + y="180.29805" + id="text4821-3-6-2-4-6" + transform="scale(0.93400803,1.0706546)"><tspan + sodipodi:role="line" + x="146.80908" + y="180.29805" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6025-0">Result is decoded back into the given output</tspan><tspan + sodipodi:role="line" + x="146.80908" + y="184.13135" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6110">mbuf as one contiguous buffer with CRC24B</tspan><tspan + sodipodi:role="line" + x="146.80908" + y="187.96463" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6112">retained in place when</tspan><tspan + sodipodi:role="line" + x="146.80908" + y="191.79793" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6114">RTE_BBDEV_TURBO_DEC_TB_CRC_24B_KEEP</tspan><tspan + sodipodi:role="line" + x="146.80908" + y="195.63123" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.22999756" + id="tspan6116">is set in op_flags</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="25.9951" + y="169.83803" + id="text4821-3-9-7-1" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-3-0" + x="25.9951" + y="169.83803" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="75.794968" + y="168.951" + id="text4821-3-3-0" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8" + x="75.794968" + y="168.951" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <path + style="display:inline;fill:none;stroke:#a8d08d;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart);marker-end:url(#marker6509)" + d="m 32.606882,149.73449 c 19.777945,-0.0668 19.777945,-0.0668 19.777945,-0.0668" + id="path6255" + inkscape:connector-curvature="0" /> + <g + style="display:inline" + id="g4807-5" + transform="translate(13.985119,97.480562)"> + <path + inkscape:connector-curvature="0" + id="path4741-6" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-88" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-9" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <g + style="display:inline" + transform="matrix(0.78752716,0,0,1.0016782,17.928141,97.168708)" + id="g5063-5-2"> + <path + inkscape:connector-curvature="0" + id="path4741-0-0-0" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-2-1" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-4-5" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-8-8" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <rect + style="display:inline;opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.199;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7-3" + width="6.6797671" + height="14.033618" + x="25.781357" + y="152.21487" /> + <g + id="g8093"> + <rect + y="152.21487" + x="32.461124" + height="14.0336" + width="16.248745" + id="rect6210" + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + id="text6214" + y="158.15347" + x="40.321487" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="158.15347" + x="40.321487" + id="tspan6212" + sodipodi:role="line">CB<tspan + id="tspan6218" + style="font-size:64.99999762%;text-align:center;baseline-shift:sub;text-anchor:middle">1</tspan></tspan><tspan + id="tspan6216" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="162.5632" + x="40.321487" + sodipodi:role="line">hard</tspan></text> + </g> + <g + id="g8100"> + <rect + y="152.21487" + x="48.709869" + height="14.0336" + width="16.248745" + id="rect6210-1" + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + id="text6214-7" + y="158.15347" + x="56.570232" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="158.15347" + x="56.570232" + id="tspan6212-1" + sodipodi:role="line">CB<tspan + id="tspan6247" + style="font-size:64.99999762%;baseline-shift:sub">2</tspan></tspan><tspan + id="tspan6216-4" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="162.5632" + x="56.570232" + sodipodi:role="line">hard</tspan></text> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="39.47636" + y="140.81966" + id="text4821-3-3-0-5" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3" + x="39.47636" + y="140.81966" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">k_neg</tspan></text> + <g + id="g8252"> + <rect + y="152.21487" + x="64.958618" + height="14.033598" + width="24.977577" + id="rect6693" + style="opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26701048;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + id="text6697" + y="160.55891" + x="75.637276" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="stroke-width:0.26458332" + y="160.55891" + x="75.637276" + id="tspan6695" + sodipodi:role="line">...</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-8" + width="16.248745" + height="14.0336" + x="89.936195" + y="152.21487" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="98.231049" + y="158.29669" + id="text6214-78"><tspan + sodipodi:role="line" + id="tspan6212-2" + x="98.231049" + y="158.29669" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;text-align:center;baseline-shift:sub;text-anchor:middle;stroke-width:0.26458332" + id="tspan6218-3">c-1</tspan></tspan><tspan + sodipodi:role="line" + x="98.231049" + y="162.70642" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-2">hard</tspan></text> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.23060164;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-8-3" + width="12.2741" + height="14.067998" + x="106.16774" + y="152.19768" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="112.17137" + y="158.42459" + id="text6214-78-6"><tspan + sodipodi:role="line" + id="tspan6212-2-9" + x="112.17137" + y="158.42459" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;text-align:center;baseline-shift:sub;text-anchor:middle;stroke-width:0.26458332" + id="tspan6218-3-7">c</tspan></tspan><tspan + sodipodi:role="line" + x="112.17137" + y="162.83432" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-2-9">hard</tspan></text> + <g + id="g6838" + transform="translate(-4.1092682)"> + <rect + y="152.22496" + x="122.55111" + height="14.040706" + width="4.0010114" + id="rect6777" + style="opacity:1;fill:#375623;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781" + y="125.5383" + x="-164.82439" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="125.5383" + x="-164.82439" + id="tspan6779" + sodipodi:role="line">CRC24A</tspan></text> + </g> + <path + style="display:inline;fill:none;stroke:#a8d08d;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0);marker-end:url(#marker6509-1)" + d="m 105.83873,149.28245 c 19.77795,-0.0668 19.77795,-0.0668 19.77795,-0.0668" + id="path6255-2" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="115.83596" + y="140.38614" + id="text4821-3-3-0-5-1" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-7" + x="115.83596" + y="140.38614" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">k_pos</tspan></text> + <rect + style="display:inline;opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.199;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7-3-8" + width="6.6797671" + height="14.033618" + x="17.908045" + y="192.83072" /> + <g + style="display:inline" + id="g4807-5-7" + transform="translate(6.0254188,138.42182)"> + <path + inkscape:connector-curvature="0" + id="path4741-6-7" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-88-6" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-9-4" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="17.775927" + y="208.62222" + id="text4821-3-9-7-1-7" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-3-0-0" + x="17.775927" + y="208.62222" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <path + style="display:inline;fill:none;stroke:#a8d08d;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-4);marker-end:url(#marker6509-11)" + d="m 24.97933,190.59164 c 19.777946,-0.0668 19.777946,-0.0668 19.777946,-0.0668" + id="path6255-0" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="31.523031" + y="180.00327" + id="text4821-3-3-0-5-5" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-9" + x="31.523031" + y="180.00327" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">k_neg</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="75.525093" + y="208.04131" + id="text4821-3-3-0-9" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-8" + x="75.525093" + y="208.04131" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <g + style="display:inline" + transform="matrix(0.91917288,0,0,1.0006169,7.5499955,138.19202)" + id="g5063-5-2-9"> + <path + inkscape:connector-curvature="0" + id="path4741-0-0-0-9" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-2-1-3" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-4-5-3" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-8-8-4" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <path + style="display:inline;fill:none;stroke:#a8d08d;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0-1);marker-end:url(#marker6509-1-3)" + d="m 109.83498,190.21366 c 19.77795,-0.0668 19.77795,-0.0668 19.77795,-0.0668" + id="path6255-2-4" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="120.00289" + y="179.64078" + id="text4821-3-3-0-5-1-5" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-7-6" + x="120.00289" + y="179.64078" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">k_pos</tspan></text> + <g + transform="translate(-7.873312,40.61586)" + style="display:inline" + id="g8093-1"> + <rect + y="152.21487" + x="32.461124" + height="14.0336" + width="16.248745" + id="rect6210-9" + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + id="text6214-9" + y="158.15347" + x="40.321487" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="158.15347" + x="40.321487" + id="tspan6212-7" + sodipodi:role="line">CB<tspan + id="tspan6218-8" + style="font-size:64.99999762%;text-align:center;baseline-shift:sub;text-anchor:middle">1</tspan></tspan><tspan + id="tspan6216-0" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="162.5632" + x="40.321487" + sodipodi:role="line">hard</tspan></text> + </g> + <g + style="display:inline" + id="g6838-7" + transform="translate(-81.714552,40.598663)"> + <rect + y="152.22496" + x="122.55111" + height="14.040706" + width="4.0010114" + id="rect6777-5" + style="opacity:1;fill:#375623;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1" + y="125.5383" + x="-164.82439" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="125.5383" + x="-164.82439" + id="tspan6779-7" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <g + transform="translate(-3.8722974,40.608749)" + style="display:inline" + id="g8100-3"> + <rect + y="152.21487" + x="48.709869" + height="14.0336" + width="16.248745" + id="rect6210-1-8" + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + id="text6214-7-4" + y="158.15347" + x="56.570232" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="158.15347" + x="56.570232" + id="tspan6212-1-0" + sodipodi:role="line">CB<tspan + id="tspan6247-0" + style="font-size:64.99999762%;baseline-shift:sub">2</tspan></tspan><tspan + id="tspan6216-4-1" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + y="162.5632" + x="56.570232" + sodipodi:role="line">hard</tspan></text> + </g> + <g + style="display:inline" + id="g6838-7-2" + transform="translate(-61.464789,40.591552)"> + <rect + y="152.22496" + x="122.55111" + height="14.040706" + width="4.0010114" + id="rect6777-5-2" + style="opacity:1;fill:#375623;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7" + y="125.5383" + x="-164.82439" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="125.5383" + x="-164.82439" + id="tspan6779-7-1" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <g + transform="translate(0.12871686,40.608749)" + style="display:inline" + id="g8252-9"> + <rect + y="152.21487" + x="64.958618" + height="14.033598" + width="24.977577" + id="rect6693-0" + style="opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26701048;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + id="text6697-7" + y="160.55891" + x="75.637276" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="stroke-width:0.26458332" + y="160.55891" + x="75.637276" + id="tspan6695-1" + sodipodi:role="line">...</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-8-5" + width="16.248745" + height="14.0336" + x="90.064911" + y="192.82362" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="98.359756" + y="198.90544" + id="text6214-78-69"><tspan + sodipodi:role="line" + id="tspan6212-2-98" + x="98.359756" + y="198.90544" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;text-align:center;baseline-shift:sub;text-anchor:middle;stroke-width:0.26458332" + id="tspan6218-3-1">c-1</tspan></tspan><tspan + sodipodi:role="line" + x="98.359756" + y="203.31517" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-2-8">hard</tspan></text> + <g + style="display:inline" + id="g6838-7-2-9" + transform="translate(-16.23745,40.591553)"> + <rect + y="152.22496" + x="122.55111" + height="14.040706" + width="4.0010114" + id="rect6777-5-2-0" + style="opacity:1;fill:#375623;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-2" + y="125.5383" + x="-164.82439" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="125.5383" + x="-164.82439" + id="tspan6779-7-1-9" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.23060165;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-8-3-2" + width="12.2741" + height="14.067998" + x="110.31467" + y="192.81651" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="116.3183" + y="199.04343" + id="text6214-78-6-8"><tspan + sodipodi:role="line" + id="tspan6212-2-9-7" + x="116.3183" + y="199.04343" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;text-align:center;baseline-shift:sub;text-anchor:middle;stroke-width:0.26458332" + id="tspan6218-3-7-9">c</tspan></tspan><tspan + sodipodi:role="line" + x="116.3183" + y="203.45316" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-2-9-1">hard</tspan></text> + <g + style="display:inline" + id="g6838-8" + transform="translate(0.03765869,40.591553)"> + <rect + y="152.22496" + x="122.55111" + height="14.040706" + width="4.0010114" + id="rect6777-6" + style="opacity:1;fill:#375623;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-15" + y="125.5383" + x="-164.82439" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="125.5383" + x="-164.82439" + id="tspan6779-6" + sodipodi:role="line">CRC24A</tspan></text> + </g> + <g + style="display:inline" + id="g6838-7-2-7" + transform="translate(4.0386734,40.591553)"> + <rect + y="152.22496" + x="122.55111" + height="14.040706" + width="4.0010114" + id="rect6777-5-2-04" + style="opacity:1;fill:#375623;fill-opacity:1;stroke:#000000;stroke-width:0.26499999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5" + y="125.5383" + x="-164.82439" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="125.5383" + x="-164.82439" + id="tspan6779-7-1-0" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="27.138876" + y="85.186432" + id="text4821-3-9-0" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-1" + x="27.138876" + y="85.186432" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">mbuf seg 1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="112.34333" + y="85.141403" + id="text4821-3-9-0-8" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-1-0" + x="112.34333" + y="85.141403" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">mbuf seg 2</tspan></text> + <path + style="fill:none;stroke:#000000;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker8474)" + d="m 34.076864,90.734741 5.946746,5.746295" + id="path8464" + inkscape:connector-curvature="0" /> + <path + style="display:inline;fill:none;stroke:#000000;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker8474-2)" + d="m 115.35315,91.80841 -5.7463,5.946752" + id="path8464-8" + inkscape:connector-curvature="0" /> + </g> +</svg> diff --git a/doc/guides/prog_guide/img/turbo_tb_encode.svg b/doc/guides/prog_guide/img/turbo_tb_encode.svg new file mode 100644 index 00000000..e3708a93 --- /dev/null +++ b/doc/guides/prog_guide/img/turbo_tb_encode.svg @@ -0,0 +1,1948 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> + +<!-- SPDX-License-Identifier: BSD-3-Clause --> +<!-- Copyright(c) 2018 Intel Corporation --> + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="189.87321mm" + height="166.91023mm" + viewBox="0 0 189.87321 166.91023" + version="1.1" + id="svg8" + inkscape:version="0.92.3 (2405546, 2018-03-11)" + sodipodi:docname="turbo_tb_encode.svg"> + <defs + id="defs2"> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8474" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path8472" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path6507" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5140" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Mstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5122" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.4,0,0,0.4,4,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow2Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5134" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="matrix(1.1,0,0,1.1,1.1,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5116" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.8,0,0,0.8,10,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Sstart" + orient="auto" + refY="0" + refX="0" + id="Arrow1Sstart" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5128" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(0.2,0,0,0.2,1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0" + refX="0" + id="Arrow1Send" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5131" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.2,0,0,-0.2,-1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible" + inkscape:isstock="true"> + <path + id="path5119" + d="M 0,0 5,-5 -12.5,0 5,5 Z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <filter + id="filter_2"> + <feGaussianBlur + stdDeviation="2" + id="feGaussianBlur4" /> + </filter> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-4" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-9" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-11" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-2" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-6" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-3" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-1" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8474-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + inkscape:connector-curvature="0" + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path8472-4" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-9" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-7" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-38" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-6" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-6" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-22" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-1-2" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-6-4" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-7-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-22-7" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-9-1" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-7-6" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-38-0" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-6-0" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-1-2-5" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-6-4-6" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-7-7-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-22-7-9" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-9-1-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-7-6-8" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-38-0-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-6-0-8" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-9-1-4" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-7-6-2" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-38-0-7" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-6-0-5" + style="fill:#9cc3e5;fill-opacity:1;fill-rule:evenodd;stroke:#9cc3e5;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8474-2-2" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + inkscape:connector-curvature="0" + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#818181;fill-opacity:1;fill-rule:evenodd;stroke:#818181;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path8472-4-6" /> + </marker> + <marker + inkscape:isstock="true" + style="overflow:visible" + id="marker8474-3" + refX="0" + refY="0" + orient="auto" + inkscape:stockid="Arrow2Mend"> + <path + inkscape:connector-curvature="0" + transform="scale(-0.6)" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + style="fill:#818181;fill-opacity:1;fill-rule:evenodd;stroke:#818181;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + id="path8472-3" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mend-1" + style="overflow:visible" + inkscape:isstock="true" + inkscape:collect="always"> + <path + inkscape:connector-curvature="0" + id="path5143-7" + style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-0-6" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-5-77" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-1-39" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-0-9" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mstart" + orient="auto" + refY="0" + refX="0" + id="Arrow2Mstart-8" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path5140-99" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(0.6)" /> + </marker> + <marker + inkscape:stockid="Arrow2Mend" + orient="auto" + refY="0" + refX="0" + id="marker6509-78" + style="overflow:visible" + inkscape:isstock="true"> + <path + inkscape:connector-curvature="0" + id="path6507-02" + style="fill:#a8d08d;fill-opacity:1;fill-rule:evenodd;stroke:#a8d08d;stroke-width:0.625;stroke-linejoin:round;stroke-opacity:1" + d="M 8.7185878,4.0337352 -2.2072895,0.01601326 8.7185884,-4.0017078 c -1.7454984,2.3720609 -1.7354408,5.6174519 -6e-7,8.035443 z" + transform="scale(-0.6)" /> + </marker> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="0.98994949" + inkscape:cx="213.35065" + inkscape:cy="360.88227" + inkscape:document-units="mm" + inkscape:current-layer="layer1" + showgrid="false" + inkscape:snap-text-baseline="true" + inkscape:window-width="1920" + inkscape:window-height="1137" + inkscape:window-x="1072" + inkscape:window-y="185" + inkscape:window-maximized="1" /> + <metadata + id="metadata5"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title /> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:groupmode="layer" + id="layer3" + inkscape:label="bgImage" + style="display:inline" + transform="translate(-10.86151,-57.361626)" /> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + style="display:inline" + transform="translate(-10.86151,-57.361626)"> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="41.159508" + y="61.13464" + id="text873"><tspan + sodipodi:role="line" + id="tspan871" + x="41.159508" + y="64.996841" + style="font-size:4.23333311px;stroke-width:0.26458332" /></text> + <flowRoot + xml:space="preserve" + id="flowRoot4811" + style="font-style:normal;font-weight:normal;font-size:13.33333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none" + transform="scale(0.26458333)"><flowRegion + id="flowRegion4813"><rect + id="rect4815" + width="41.785713" + height="14.642858" + x="39.285713" + y="287.16254" /></flowRegion><flowPara + id="flowPara4817">offse</flowPara></flowRoot> <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="16.351753" + y="215.03786" + id="text4821-3-9-7-1-7" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-3-0-0" + x="16.351753" + y="215.03786" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <rect + style="display:inline;opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7-3-8-5" + width="6.6797671" + height="14.033618" + x="13.480058" + y="65.465332" /> + <g + style="display:inline" + id="g4807-4" + transform="translate(1.6626143,11.103676)"> + <path + inkscape:connector-curvature="0" + id="path4741-8" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-5" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-5" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <g + style="display:inline" + id="g5063-4" + transform="matrix(0.96955809,0,0,1.0002284,2.2280641,10.898039)"> + <path + inkscape:connector-curvature="0" + id="path4741-0-01" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-9" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-0" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-2" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="74.602684" + y="85.144012" + id="text4821-3-0" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-5" + x="74.602684" + y="85.144012" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="13.386705" + y="85.690132" + id="text4821-3-9-4" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-4" + x="13.386705" + y="85.690132" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <g + transform="translate(209.08086,-15.131588)" + style="display:inline" + id="g10789-0"> + <path + inkscape:connector-curvature="0" + id="path4885-7" + d="m -198.21935,107.88646 h 61.73926" + style="fill:none;stroke:#7f7f7f;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.1199999, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4887-5" + d="m -126.32385,107.88646 94.613422,-0.13363" + style="fill:none;stroke:#818181;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.1199999, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="scale(0.9610099,1.040572)" + id="text4891-3" + y="105.46623" + x="-139.73984" + style="font-style:normal;font-weight:normal;font-size:5.39796209px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#818181;fill-opacity:1;stroke:none;stroke-width:0.40484715" + xml:space="preserve"><tspan + style="fill:#818181;fill-opacity:1;stroke-width:0.40484715" + y="105.46623" + x="-139.73984" + id="tspan4889-9" + sodipodi:role="line">or</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15867083;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6" + width="16.144258" + height="14.033618" + x="20.159824" + y="65.465332" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="25.127815" + y="73.846748" + id="text877-8-6"><tspan + sodipodi:role="line" + id="tspan875-6-8" + x="25.127815" + y="73.846748" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:64.99999762%;baseline-shift:sub" + id="tspan10851">1</tspan></tspan></text> + <g + id="g10891"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-1" + width="16.163868" + height="14.074809" + x="40.305096" + y="65.443756" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="44.893223" + y="73.780594" + id="text877-8-6-7"><tspan + sodipodi:role="line" + id="tspan875-6-8-8" + x="44.893223" + y="73.780594" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:64.99999762%;baseline-shift:sub" + id="tspan10935">2</tspan></tspan></text> + <g + transform="translate(20.164879)" + style="display:inline" + id="g10891-1"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3-5" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6-4" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4-5" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-8" + width="30.921177" + height="14.014396" + x="60.480572" + y="65.493568" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="73.468872" + y="73.892609" + id="text877-8-4"><tspan + sodipodi:role="line" + id="tspan875-6-3" + x="73.468872" + y="73.892609" + style="font-size:3.52777767px;stroke-width:0.26458332">...</tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-0" + width="16.163868" + height="14.074809" + x="91.401749" + y="65.433159" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="95.206711" + y="73.84742" + id="text877-8-6-1"><tspan + sodipodi:role="line" + id="tspan875-6-8-9" + x="95.206711" + y="73.84742" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-4">c-1</tspan></tspan></text> + <g + transform="translate(71.261528,-0.01059723)" + style="display:inline" + id="g10891-1-1"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3-5-4" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6-4-0" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4-5-4" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-0-8" + width="11.644219" + height="14.098742" + x="111.55466" + y="65.421196" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="114.16887" + y="73.780602" + id="text877-8-6-1-0"><tspan + sodipodi:role="line" + id="tspan875-6-8-9-9" + x="114.16887" + y="73.780602" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-4-4">c</tspan></tspan></text> + <g + transform="translate(86.894791,0.00137329)" + style="display:inline" + id="g10891-1-1-1"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3-5-4-9" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6-4-0-4" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4-5-4-7" + sodipodi:role="line">CRC24A</tspan></text> + </g> + <g + transform="translate(90.895802,0.00137329)" + style="display:inline" + id="g10891-1-1-6"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3-5-4-95" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6-4-0-6" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4-5-4-2" + sodipodi:role="line">CRC24B</tspan></text> + </g> + <path + style="display:inline;fill:none;stroke:#9cc3e5;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-1);marker-end:url(#marker6509-7)" + d="m 20.359726,62.756584 c 19.77795,-0.0668 19.77795,-0.0668 19.77795,-0.0668" + id="path6255-4" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="26.706127" + y="57.404415" + id="text4821-3-3-0-5-4" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-95" + x="26.706127" + y="57.404415" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">k_neg</tspan></text> + <path + style="display:inline;fill:none;stroke:#9cc3e5;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0-9);marker-end:url(#marker6509-1-38)" + d="m 111.58503,62.795193 c 19.77795,-0.0668 19.77795,-0.0668 19.77795,-0.0668" + id="path6255-2-3" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="121.82767" + y="57.441448" + id="text4821-3-3-0-5-1-6" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-7-9" + x="121.82767" + y="57.441448" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">k_pos</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="144.51123" + y="64.972511" + id="text4821-3-6-0" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + x="144.51123" + y="64.972511" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan4877-6">- CRC24B & CRC24A were pre-calculated</tspan><tspan + sodipodi:role="line" + x="144.51123" + y="68.908539" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11806">by the application</tspan><tspan + sodipodi:role="line" + x="144.51123" + y="72.844559" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11808">- The raw TB is given as a contiguous</tspan><tspan + sodipodi:role="line" + x="144.51123" + y="76.780586" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11810">buffer</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="143.92094" + y="97.043495" + id="text4821-3-6-0-2" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + x="143.92094" + y="97.043495" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11810-4">- Only CRC24A was pre-calculated by the</tspan><tspan + sodipodi:role="line" + x="143.92094" + y="100.97952" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11848">application, therefore</tspan><tspan + sodipodi:role="line" + x="143.92094" + y="104.91554" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11850">RTE_BBDEV_TURBO_CRC_24B_ATTACH</tspan><tspan + sodipodi:role="line" + x="143.92094" + y="108.85157" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11852">is set in op_flags</tspan><tspan + sodipodi:role="line" + x="143.92094" + y="112.78759" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11854">- The raw TB is given as a contiguous</tspan><tspan + sodipodi:role="line" + x="143.92094" + y="116.72362" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11856">buffer</tspan></text> + <g + style="display:inline" + id="g4807-4-6" + transform="translate(9.4628222,49.06356)"> + <path + inkscape:connector-curvature="0" + id="path4741-8-0" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-5-5" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-5-6" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="21.520061" + y="122.09515" + id="text4821-3-9-4-8" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-4-4" + x="21.520061" + y="122.09515" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <g + style="display:inline" + id="g5063-4-7" + transform="matrix(0.83046983,0,0,1.0013214,12.615148,48.778811)"> + <path + inkscape:connector-curvature="0" + id="path4741-0-01-3" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-9-4" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-0-6" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-2-9" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="72.49247" + y="121.91208" + id="text4821-3-0-1" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-5-7" + x="72.49247" + y="121.91208" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <g + transform="translate(213.42759,24.366924)" + style="display:inline" + id="g10789-0-0"> + <path + inkscape:connector-curvature="0" + id="path4885-7-2" + d="m -198.21935,107.88646 h 61.73926" + style="fill:none;stroke:#7f7f7f;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.1199999, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4887-5-0" + d="m -126.32385,107.88646 94.613422,-0.13363" + style="fill:none;stroke:#818181;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:2.1199999, 0.26499999;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="scale(0.9610099,1.040572)" + id="text4891-3-9" + y="105.46623" + x="-139.73984" + style="font-style:normal;font-weight:normal;font-size:5.39796209px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#818181;fill-opacity:1;stroke:none;stroke-width:0.40484715" + xml:space="preserve"><tspan + style="fill:#818181;fill-opacity:1;stroke-width:0.40484715" + y="105.46623" + x="-139.73984" + id="tspan4889-9-9" + sodipodi:role="line">or</tspan></text> + </g> + <rect + style="display:inline;opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7-3-8-5-5" + width="6.6797671" + height="14.033618" + x="20.985983" + y="103.46108" /> + <path + style="display:inline;fill:none;stroke:#9cc3e5;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-1-2);marker-end:url(#marker6509-7-7)" + d="m 28.597837,100.50577 c 19.777951,-0.0668 19.777951,-0.0668 19.777951,-0.0668" + id="path6255-4-7" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="35.296089" + y="93.607361" + id="text4821-3-3-0-5-4-7" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-95-5" + x="35.296089" + y="93.607361" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">k_neg</tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-3" + width="16.163868" + height="14.074809" + x="27.665751" + y="103.46108" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="32.131104" + y="111.79969" + id="text877-8-6-9"><tspan + sodipodi:role="line" + id="tspan875-6-8-0" + x="32.131104" + y="111.79969" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-5">1</tspan></tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-1-0" + width="16.163868" + height="14.074809" + x="43.82962" + y="103.46108" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="48.417747" + y="111.79792" + id="text877-8-6-7-1"><tspan + sodipodi:role="line" + id="tspan875-6-8-8-5" + x="48.417747" + y="111.79792" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10935-8">2</tspan></tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-8-8" + width="30.921177" + height="14.014396" + x="59.993488" + y="103.46108" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="73.38308" + y="111.96056" + id="text877-8-4-6"><tspan + sodipodi:role="line" + id="tspan875-6-3-9" + x="73.38308" + y="111.96056" + style="font-size:3.52777767px;stroke-width:0.26458332">...</tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-0-7" + width="16.163868" + height="14.074809" + x="90.914665" + y="103.46108" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="94.719627" + y="111.87534" + id="text877-8-6-1-5"><tspan + sodipodi:role="line" + id="tspan875-6-8-9-4" + x="94.719627" + y="111.87534" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-4-2">c-1</tspan></tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-0-8-6" + width="12.207969" + height="14.095527" + x="107.08015" + y="103.43876" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="109.69276" + y="111.79655" + id="text877-8-6-1-0-1"><tspan + sodipodi:role="line" + id="tspan875-6-8-9-9-0" + x="109.69276" + y="111.79655" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-4-4-5">c</tspan></tspan></text> + <g + transform="translate(82.984032,37.995003)" + style="display:inline" + id="g10891-1-1-1-9"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3-5-4-9-2" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6-4-0-4-0" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4-5-4-7-9" + sodipodi:role="line">CRC24A</tspan></text> + </g> + <path + style="display:inline;fill:none;stroke:#9cc3e5;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0-9-1);marker-end:url(#marker6509-1-38-0)" + d="m 107.37813,100.63331 c 19.77795,-0.0668 19.77795,-0.0668 19.77795,-0.0668" + id="path6255-2-3-9" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="117.44109" + y="93.729691" + id="text4821-3-3-0-5-1-6-2" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-7-9-7" + x="117.44109" + y="93.729691" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">k_pos</tspan></text> + <rect + style="display:inline;opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7-3-8-5-5-2" + width="6.6797671" + height="14.033618" + x="13.230828" + y="152.20575" /> + <path + style="display:inline;fill:none;stroke:#9cc3e5;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-1-2-5);marker-end:url(#marker6509-7-7-6)" + d="m 20.312481,149.54973 c 19.777952,-0.0668 19.777952,-0.0668 19.777952,-0.0668" + id="path6255-4-7-7" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="26.656864" + y="140.64244" + id="text4821-3-3-0-5-4-7-0" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-95-5-0" + x="26.656864" + y="140.64244" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">k_neg</tspan></text> + <path + style="display:inline;fill:none;stroke:#9cc3e5;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0-9-1-4);marker-end:url(#marker6509-1-38-0-7)" + d="m 114.65863,149.41609 c 19.77795,-0.0668 19.77795,-0.0668 19.77795,-0.0668" + id="path6255-2-3-9-5" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="125.03253" + y="140.5143" + id="text4821-3-3-0-5-1-6-2-0" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-7-9-7-4" + x="125.03253" + y="140.5143" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">k_pos</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="26.321789" + y="132.12051" + id="text4821-3-9-0-4" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-1-04" + x="26.321789" + y="132.12051" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">mbuf seg 1</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="111.52624" + y="132.07547" + id="text4821-3-9-0-8-6" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-1-0-4" + x="111.52624" + y="132.07547" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">mbuf seg 2</tspan></text> + <path + style="display:inline;fill:none;stroke:#818181;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker8474-3)" + d="m 33.293244,139.67339 5.946743,5.7463" + id="path8464-83" + inkscape:connector-curvature="0" /> + <path + style="display:inline;fill:none;stroke:#818181;stroke-width:0.26499999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker8474-2-2)" + d="m 114.56953,140.74706 -5.7463,5.94675" + id="path8464-8-8" + inkscape:connector-curvature="0" /> + <g + style="display:inline" + id="g4807-4-6-1" + transform="translate(1.5234255,97.894043)"> + <path + inkscape:connector-curvature="0" + id="path4741-8-0-9" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-5-5-9" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-5-6-4" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="13.241572" + y="168.92551" + id="text4821-3-9-4-8-3" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-0-4-4-2" + x="13.241572" + y="168.92551" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">offset</tspan></text> + <g + style="display:inline" + id="g5063-4-7-3" + transform="matrix(0.96708187,0,0,1.0002483,2.1378366,97.68752)"> + <path + inkscape:connector-curvature="0" + id="path4741-0-01-3-5" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-9-4-4" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-0-6-6" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-2-9-4" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#9cc3e5;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="74.165497" + y="168.19868" + id="text4821-3-0-1-2" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-5-7-5" + x="74.165497" + y="168.19868" + style="fill:#9cc3e5;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-3-0" + width="16.163868" + height="14.074809" + x="19.910595" + y="152.20575" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="24.37595" + y="160.54436" + id="text877-8-6-9-1"><tspan + sodipodi:role="line" + id="tspan875-6-8-0-4" + x="24.37595" + y="160.54436" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-5-1">1</tspan></tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-1-0-2" + width="16.163868" + height="14.074809" + x="36.074467" + y="152.20575" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="40.662594" + y="160.54259" + id="text877-8-6-7-1-7"><tspan + sodipodi:role="line" + id="tspan875-6-8-8-5-6" + x="40.662594" + y="160.54259" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10935-8-3">2</tspan></tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-8-8-7" + width="8.9107389" + height="14.115565" + x="52.187752" + y="152.21559" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="54.513344" + y="160.1346" + id="text877-8-4-6-7"><tspan + sodipodi:role="line" + id="tspan875-6-3-9-3" + x="54.513344" + y="160.1346" + style="font-size:3.52777767px;stroke-width:0.26458332">...</tspan></text> + <path + style="display:inline;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + d="m 61.098492,152.29471 9.985814,-0.03 v 4.04245 l -2.539064,1.46999 5.579254,2.87314 -3.00678,1.60362 v 4.04245 l -10.019224,0.0348 z" + id="path5575-6" + inkscape:connector-curvature="0" /> + <path + style="display:inline;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + d="m 90.024487,152.16192 -9.953251,0.003 -0.0334,3.94222 -2.63928,1.63703 5.67947,2.87314 -2.93996,1.57021 v 4.00905 l 9.886431,0.003 z" + id="path5581-0" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="62.273087" + y="160.61116" + id="text5879-0"><tspan + sodipodi:role="line" + id="tspan5877-5" + x="62.273087" + y="160.61116" + style="stroke-width:0.26458332">CB<tspan + style="font-size:64.99999762%;baseline-shift:sub" + id="tspan15311">N</tspan></tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="83.078026" + y="160.61731" + id="text5883-5"><tspan + sodipodi:role="line" + id="tspan5881-6" + x="83.078026" + y="160.61731" + style="stroke-width:0.26458332">CB<tspan + style="font-size:64.99999762%;baseline-shift:sub" + id="tspan15313">N</tspan></tspan></text> + <path + style="display:inline;fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#Arrow2Mend-1)" + d="m 66.006182,152.19788 c 0,0 5.144934,-13.09621 18.942724,-0.3341" + id="path5899-9" + inkscape:connector-curvature="0" /> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-8-8-7-8" + width="8.9107389" + height="14.115566" + x="90.024498" + y="152.084" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="92.247215" + y="160.20142" + id="text877-8-4-6-7-9"><tspan + sodipodi:role="line" + id="tspan875-6-3-9-3-4" + x="92.247215" + y="160.20142" + style="font-size:3.52777767px;stroke-width:0.26458332">...</tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-3-0-3" + width="16.163868" + height="14.074809" + x="98.935234" + y="152.12476" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="102.77936" + y="160.27509" + id="text877-8-6-9-1-0"><tspan + sodipodi:role="line" + id="tspan875-6-8-0-4-7" + x="102.77936" + y="160.27509" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-5-1-1">c-1</tspan></tspan></text> + <rect + style="display:inline;opacity:1;fill:#9cc3e5;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect967-6-0-8-6-9" + width="12.207969" + height="14.095527" + x="115.09911" + y="152.10405" /> + <g + transform="translate(91.002987,86.660285)" + style="display:inline" + id="g10891-1-1-1-9-9"> + <rect + y="65.443756" + x="36.304085" + height="14.074809" + width="4.0010114" + id="rect6777-5-2-04-3-5-4-9-2-3" + style="opacity:1;fill:#0070c0;fill-opacity:1;stroke:#000000;stroke-width:0.15919298;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" /> + <text + transform="rotate(-90)" + id="text6781-1-7-5-6-4-0-4-0-6" + y="39.324684" + x="-78.343857" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + xml:space="preserve"><tspan + style="font-size:2.82222223px;fill:#ffffff;fill-opacity:1;stroke-width:0.26458332" + y="39.324684" + x="-78.343857" + id="tspan6779-7-1-0-4-5-4-7-9-7" + sodipodi:role="line">CRC24A</tspan></text> + </g> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:4.23333311px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="118.12826" + y="160.26109" + id="text877-8-6-1-0-1-8"><tspan + sodipodi:role="line" + id="tspan875-6-8-9-9-0-1" + x="118.12826" + y="160.26109" + style="font-size:3.52777767px;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;baseline-shift:sub;stroke-width:0.26458332" + id="tspan10851-4-4-5-0">c</tspan></tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="144.46991" + y="144.41747" + id="text4821-3-6-0-2-7" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + x="144.46991" + y="144.41747" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan11856-1">- CRC24A was pre-calculated and </tspan><tspan + sodipodi:role="line" + x="144.46991" + y="148.35349" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan15485">RTE_BBDEV_TURBO_CRC_24B_ATTACH</tspan><tspan + sodipodi:role="line" + x="144.46991" + y="152.28952" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan15487">is set in op_flags</tspan><tspan + sodipodi:role="line" + x="144.46991" + y="156.22554" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan15489">- The raw TB is given as a "scattered"</tspan><tspan + sodipodi:role="line" + x="144.46991" + y="160.16156" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan15491">buffer through a chained mbuf</tspan></text> + <path + style="fill:#fec000;fill-opacity:1;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + d="m 73.879064,190.90497 -0.03341,-11.65962 h 4.209497 l -0.03341,11.65962 1.971111,-0.0334 -4.042449,3.90882 -4.04245,-3.90882 z" + id="path6066-6" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="-190.93837" + y="76.819031" + id="text6074-7" + transform="rotate(-90)"><tspan + sodipodi:role="line" + id="tspan6072-6" + x="-190.93837" + y="76.819031" + style="font-size:2.82222223px;stroke-width:0.26458332">encode</tspan></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="74.319527" + y="213.86391" + id="text4821-3-3-0-8" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-6" + x="74.319527" + y="213.86391" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">length</tspan></text> + <path + style="display:inline;fill:none;stroke:#a8d08d;stroke-width:0.25237256;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-8);marker-end:url(#marker6509-78)" + d="m 22.905754,197.23808 c 15.904037,-0.0753 15.904037,-0.0753 15.904037,-0.0753" + id="path6255-22" + inkscape:connector-curvature="0" /> + <g + style="display:inline" + id="g4807-5-2" + transform="translate(4.1511086,145.38078)"> + <path + inkscape:connector-curvature="0" + id="path4741-6-72" + d="m 11.929873,70.237907 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-88-63" + d="m 18.686199,70.237907 c 0,0 -0.4016,2.480469 -1.370164,2.456847 -0.968564,-0.02363 -2.007999,-0.02363 -2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-9-0" + d="m 15.308036,72.671124 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <g + style="display:inline" + transform="matrix(0.9338915,0,0,1.0005032,5.402228,145.15927)" + id="g5063-5-2-4"> + <path + inkscape:connector-curvature="0" + id="path4741-0-0-0-2" + d="m 18.686199,70.426891 c 0,0 0.4016,2.480469 1.370164,2.456847 0.968564,-0.02363 2.007999,-0.02363 2.007999,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4741-5-8-2-1-6" + d="m 132.59878,70.332401 c 0,0 -0.4016,2.480469 -1.37017,2.456847 -0.96856,-0.02363 -2.008,-0.02363 -2.008,-0.02363" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4760-2-4-5-5" + d="m 75.831475,72.954606 -0.02362,2.527721" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + <path + inkscape:connector-curvature="0" + id="path4809-8-8-9" + d="M 22.064362,72.860108 129.22061,72.765618" + style="fill:none;stroke:#000000;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" /> + </g> + <rect + style="display:inline;opacity:1;fill:#d8d8d8;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" + id="rect4735-7-3-3" + width="6.6797671" + height="14.033618" + x="15.753516" + y="199.83667" /> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-97" + width="16.248745" + height="14.0336" + x="22.433283" + y="199.83669" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="30.577127" + y="208.42111" + id="text6214-8"><tspan + sodipodi:role="line" + id="tspan6212-4" + x="30.577127" + y="208.42111" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:64.99999762%;text-align:center;baseline-shift:sub;text-anchor:middle" + id="tspan6218-7">1</tspan></tspan><tspan + sodipodi:role="line" + x="30.577127" + y="212.83084" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-3" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="29.919048" + y="186.11703" + id="text4821-3-3-0-5-7" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-0" + x="29.919048" + y="186.11703" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">ea</tspan></text> + <rect + style="opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.15930426;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6693-01" + width="42.577148" + height="13.993384" + x="54.930775" + y="199.87691" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="74.028419" + y="208.31436" + id="text6697-5"><tspan + sodipodi:role="line" + id="tspan6695-3" + x="74.028419" + y="208.31436" + style="stroke-width:0.26458332">...</tspan></text> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.15922768;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-8-1" + width="16.248745" + height="14.073822" + x="97.507919" + y="199.79646" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="105.80277" + y="207.61554" + id="text6214-78-1"><tspan + sodipodi:role="line" + id="tspan6212-2-2" + x="105.80277" + y="207.61554" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;text-align:center;baseline-shift:sub;text-anchor:middle;stroke-width:0.26458332" + id="tspan6218-3-0">c-1</tspan></tspan><tspan + sodipodi:role="line" + x="105.80277" + y="212.02527" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-2-3" /></text> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.15929575;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-8-3-28" + width="15.999747" + height="14.073821" + x="113.75667" + y="199.79646" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="121.63119" + y="208.12714" + id="text6214-78-6-0"><tspan + sodipodi:role="line" + id="tspan6212-2-9-72" + x="121.63119" + y="208.12714" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:2.2930553px;text-align:center;baseline-shift:sub;text-anchor:middle;stroke-width:0.26458332" + id="tspan6218-3-7-8">c</tspan></tspan><tspan + sodipodi:role="line" + x="121.63119" + y="212.53687" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-2-9-4" /></text> + <path + style="display:inline;fill:none;stroke:#a8d08d;stroke-width:0.25187415;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#Arrow2Mstart-0-6);marker-end:url(#marker6509-1-39)" + d="m 114.04452,197.18711 c 15.77071,-0.0757 15.77071,-0.0757 15.77071,-0.0757" + id="path6255-2-9" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#a8d08d;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="125.36855" + y="186.19617" + id="text4821-3-3-0-5-1-9" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + id="tspan4819-0-7-8-3-7-5" + x="125.36855" + y="186.19617" + style="fill:#a8d08d;fill-opacity:1;stroke-width:0.23616144">eb</tspan></text> + <rect + style="display:inline;opacity:1;fill:#a8d08d;fill-opacity:1;stroke:#000000;stroke-width:0.15899999;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1" + id="rect6210-97-3" + width="16.248745" + height="14.0336" + x="38.68203" + y="199.83669" /> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.52777767px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;display:inline;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332" + x="46.825874" + y="208.42111" + id="text6214-8-2"><tspan + sodipodi:role="line" + id="tspan6212-4-7" + x="46.825874" + y="208.42111" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332">CB<tspan + style="font-size:64.99999762%;text-align:center;baseline-shift:sub;text-anchor:middle" + id="tspan6218-7-0">2</tspan></tspan><tspan + sodipodi:role="line" + x="46.825874" + y="212.83084" + style="text-align:center;text-anchor:middle;stroke-width:0.26458332" + id="tspan6216-3-5" /></text> + <text + xml:space="preserve" + style="font-style:normal;font-weight:normal;font-size:3.14881921px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;display:inline;fill:#7f8085;fill-opacity:1;stroke:none;stroke-width:0.23616144" + x="145.06511" + y="197.72243" + id="text4821-3-6-0-2-7-2" + transform="scale(0.95903924,1.0427102)"><tspan + sodipodi:role="line" + x="145.06511" + y="197.72243" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan15491-4">Result is encoded back into the given</tspan><tspan + sodipodi:role="line" + x="145.06511" + y="201.65845" + style="fill:#7f8085;fill-opacity:1;stroke-width:0.23616144" + id="tspan16671">output mbuf as one contiguous buffer</tspan></text> + </g> +</svg> diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst index bbbe7895..3b920e53 100644 --- a/doc/guides/prog_guide/index.rst +++ b/doc/guides/prog_guide/index.rst @@ -17,10 +17,12 @@ Programmer's Guide mbuf_lib poll_mode_drv rte_flow + switch_representation traffic_metering_and_policing traffic_management bbdev cryptodev_lib + compressdev rte_security rawdev link_bonding_poll_mode_drv_lib @@ -42,6 +44,8 @@ Programmer's Guide thread_safety_dpdk_functions eventdev event_ethernet_rx_adapter + event_timer_adapter + event_crypto_adapter qos_framework power_man packet_classif_access_ctrl @@ -49,6 +53,7 @@ Programmer's Guide vhost_lib metrics_lib port_hotplug_framework + bpf_lib source_org dev_kit_build_system dev_kit_root_make_help diff --git a/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst b/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst index 0da1e637..56abee54 100644 --- a/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst +++ b/doc/guides/prog_guide/link_bonding_poll_mode_drv_lib.rst @@ -148,6 +148,8 @@ the RX and TX queues are also reconfigured using ``rte_eth_tx_queue_setup`` / ``rte_eth_rx_queue_setup`` with the parameters use to configure the bonding device. If RSS is enabled for bonding device, this mode is also enabled on new slave and configured as well. +Any flow which was configured to the bond device also is configured to the added +slave. Setting up multi-queue mode for bonding device to RSS, makes it fully RSS-capable, so all slaves are synchronized with its configuration. This mode is @@ -166,6 +168,43 @@ it can be easily used as a pattern providing expected behavior, even if slave RETAs' sizes are different. If RSS Key is not set for bonded device, it's not changed on the slaves and default key for device is used. +As RSS configurations, there is flow consistency in the bonded slaves for the +next rte flow operations: + +Validate: + - Validate flow for each slave, failure at least for one slave causes to + bond validation failure. + +Create: + - Create the flow in all slaves. + - Save all the slaves created flows objects in bonding internal flow + structure. + - Failure in flow creation for existed slave rejects the flow. + - Failure in flow creation for new slaves in slave adding time rejects + the slave. + +Destroy: + - Destroy the flow in all slaves and release the bond internal flow + memory. + +Flush: + - Destroy all the bonding PMD flows in all the slaves. + +.. note:: + + Don't call slaves flush directly, It destroys all the slave flows which + may include external flows or the bond internal LACP flow. + +Query: + - Summarize flow counters from all the slaves, relevant only for + ``RTE_FLOW_ACTION_TYPE_COUNT``. + +Isolate: + - Call to flow isolate for all slaves. + - Failure in flow isolation for existed slave rejects the isolate mode. + - Failure in flow isolation for new slaves in slave adding time rejects + the slave. + All settings are managed through the bonding port API and always are propagated in one direction (from bonding to slaves). @@ -207,8 +246,8 @@ common hash function available for each of them. Changing RSS key is only possible, when all slave devices support the same key size. To prevent inconsistency on how slaves process packets, once a device is added -to a bonding device, RSS configuration should be managed through the bonding -device API, and not directly on the slave. +to a bonding device, RSS and rte flow configurations should be managed through +the bonding device API, and not directly on the slave. Like all other PMD, all functions exported by a PMD are lock-free functions that are assumed not to be invoked in parallel on different logical cores to diff --git a/doc/guides/prog_guide/mbuf_lib.rst b/doc/guides/prog_guide/mbuf_lib.rst index 210a9af9..0d3223b0 100644 --- a/doc/guides/prog_guide/mbuf_lib.rst +++ b/doc/guides/prog_guide/mbuf_lib.rst @@ -10,9 +10,8 @@ The mbuf library provides the ability to allocate and free buffers (mbufs) that may be used by the DPDK application to store message buffers. The message buffers are stored in a mempool, using the :ref:`Mempool Library <Mempool_Library>`. -A rte_mbuf struct can carry network packet buffers -or generic control buffers (indicated by the CTRL_MBUF_FLAG). -This can be extended to other types. +A rte_mbuf struct generally carries network packet buffers, but it can actually +be any data (control data, events, ...). The rte_mbuf header structure is kept as small as possible and currently uses just two cache lines, with the most frequently used fields being on the first of the two cache lines. @@ -68,13 +67,13 @@ Buffers Stored in Memory Pools The Buffer Manager uses the :ref:`Mempool Library <Mempool_Library>` to allocate buffers. Therefore, it ensures that the packet header is interleaved optimally across the channels and ranks for L3 processing. An mbuf contains a field indicating the pool that it originated from. -When calling rte_ctrlmbuf_free(m) or rte_pktmbuf_free(m), the mbuf returns to its original pool. +When calling rte_pktmbuf_free(m), the mbuf returns to its original pool. Constructors ------------ -Packet and control mbuf constructors are provided by the API. -The rte_pktmbuf_init() and rte_ctrlmbuf_init() functions initialize some fields in the mbuf structure that +Packet mbuf constructors are provided by the API. +The rte_pktmbuf_init() function initializes some fields in the mbuf structure that are not modified by the user once created (mbuf type, origin pool, buffer start address, and so on). This function is given as a callback function to the rte_mempool_create() function at pool creation time. diff --git a/doc/guides/prog_guide/multi_proc_support.rst b/doc/guides/prog_guide/multi_proc_support.rst index 90596794..1384fe33 100644 --- a/doc/guides/prog_guide/multi_proc_support.rst +++ b/doc/guides/prog_guide/multi_proc_support.rst @@ -29,6 +29,9 @@ after a primary process has already configured the hugepage shared memory for th Secondary processes should run alongside primary process with same DPDK version. + Secondary processes which requires access to physical devices in Primary process, must + be passed with the same whitelist and blacklist options. + To support these two process types, and other multi-process setups described later, two additional command-line parameters are available to the EAL: @@ -60,6 +63,10 @@ and point to the same objects, in both processes. Refer to `Multi-process Limitations`_ for details of how Linux kernel Address-Space Layout Randomization (ASLR) can affect memory sharing. + If the primary process was run with ``--legacy-mem`` or + ``--single-file-segments`` switch, secondary processes must be run with the + same switch specified. Otherwise, memory corruption may occur. + .. _figure_multi_process_memory: .. figure:: img/multi_process_memory.* @@ -113,8 +120,13 @@ The rte part of the filenames of each of the above is configurable using the fil In addition to specifying the file-prefix parameter, any DPDK applications that are to be run side-by-side must explicitly limit their memory use. -This is done by passing the -m flag to each process to specify how much hugepage memory, in megabytes, -each process can use (or passing ``--socket-mem`` to specify how much hugepage memory on each socket each process can use). +This is less of a problem on Linux, as by default, applications will not +allocate more memory than they need. However if ``--legacy-mem`` is used, DPDK +will attempt to preallocate all memory it can get to, and memory use must be +explicitly limited. This is done by passing the ``-m`` flag to each process to +specify how much hugepage memory, in megabytes, each process can use (or passing +``--socket-mem`` to specify how much hugepage memory on each socket each process +can use). .. note:: @@ -141,8 +153,10 @@ There are a number of limitations to what can be done when running DPDK multi-pr Some of these are documented below: * The multi-process feature requires that the exact same hugepage memory mappings be present in all applications. - The Linux security feature - Address-Space Layout Randomization (ASLR) can interfere with this mapping, - so it may be necessary to disable this feature in order to reliably run multi-process applications. + This makes secondary process startup process generally unreliable. Disabling + Linux security feature - Address-Space Layout Randomization (ASLR) may + help getting more consistent mappings, but not necessarily more reliable - + if the mappings are wrong, they will be consistently wrong! .. warning:: @@ -175,3 +189,153 @@ instead of the functions which do the hashing internally, such as rte_hash_add() which means that only the first, primary DPDK process instance can open and mmap /dev/hpet. If the number of required DPDK processes exceeds that of the number of available HPET comparators, the TSC (which is the default timer in this release) must be used as a time source across all processes instead of the HPET. + +Communication between multiple processes +---------------------------------------- + +While there are multiple ways one can approach inter-process communication in +DPDK, there is also a native DPDK IPC API available. It is not intended to be +performance-critical, but rather is intended to be a convenient, general +purpose API to exchange short messages between primary and secondary processes. + +DPDK IPC API supports the following communication modes: + +* Unicast message from secondary to primary +* Broadcast message from primary to all secondaries + +In other words, any IPC message sent in a primary process will be delivered to +all secondaries, while any IPC message sent in a secondary process will only be +delivered to primary process. Unicast from primary to secondary or from +secondary to secondary is not supported. + +There are three types of communications that are available within DPDK IPC API: + +* Message +* Synchronous request +* Asynchronous request + +A "message" type does not expect a response and is meant to be a best-effort +notification mechanism, while the two types of "requests" are meant to be a two +way communication mechanism, with the requester expecting a response from the +other side. + +Both messages and requests will trigger a named callback on the receiver side. +These callbacks will be called from within a dedicated IPC or interrupt thread +that are not part of EAL lcore threads. + +Registering for incoming messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before any messages can be received, a callback will need to be registered. +This is accomplished by calling ``rte_mp_action_register()`` function. This +function accepts a unique callback name, and a function pointer to a callback +that will be called when a message or a request matching this callback name +arrives. + +If the application is no longer willing to receive messages intended for a +specific callback function, ``rte_mp_action_unregister()`` function can be +called to ensure that callback will not be triggered again. + +Sending messages +~~~~~~~~~~~~~~~~ + +To send a message, a ``rte_mp_msg`` descriptor must be populated first. The list +of fields to be populated are as follows: + +* ``name`` - message name. This name must match receivers' callback name. +* ``param`` - message data (up to 256 bytes). +* ``len_param`` - length of message data. +* ``fds`` - file descriptors to pass long with the data (up to 8 fd's). +* ``num_fds`` - number of file descriptors to send. + +Once the structure is populated, calling ``rte_mp_sendmsg()`` will send the +descriptor either to all secondary processes (if sent from primary process), or +to primary process (if sent from secondary process). The function will return +a value indicating whether sending the message succeeded or not. + +Sending requests +~~~~~~~~~~~~~~~~ + +Sending requests involves waiting for the other side to reply, so they can block +for a relatively long time. + +To send a request, a message descriptor ``rte_mp_msg`` must be populated. +Additionally, a ``timespec`` value must be specified as a timeout, after which +IPC will stop waiting and return. + +For synchronous synchronous requests, the ``rte_mp_reply`` descriptor must also +be created. This is where the responses will be stored. The list of fields that +will be populated by IPC are as follows: + +* ``nb_sent`` - number indicating how many requests were sent (i.e. how many + peer processes were active at the time of the request). +* ``nb_received`` - number indicating how many responses were received (i.e. of + those peer processes that were active at the time of request, how many have + replied) +* ``msgs`` - pointer to where all of the responses are stored. The order in + which responses appear is undefined. Whendoing sycnrhonous requests, this + memory must be freed by the requestor after request completes! + +For asynchronous requests, a function pointer to the callback function must be +provided instead. This callback will be called when the request either has timed +out, or will have received a response to all the messages that were sent. + +.. warning:: + + When an asynchronous request times out, the callback will be called not by + a dedicated IPC thread, but rather from EAL interrupt thread. Because of + this, it may not be possible for DPDK to trigger another interrupt-based + event (such as an alarm) while handling asynchronous IPC callback. + +When the callback is called, the original request descriptor will be provided +(so that it would be possible to determine for which sent message this is a +callback to), along with a response descriptor like the one described above. +When doing asynchronous requests, there is no need to free the resulting +``rte_mp_reply`` descriptor. + +Receiving and responding to messages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To receive a message, a name callback must be registered using the +``rte_mp_action_register()`` function. The name of the callback must match the +``name`` field in sender's ``rte_mp_msg`` message descriptor in order for this +message to be delivered and for the callback to be trigger. + +The callback's definition is ``rte_mp_t``, and consists of the incoming message +pointer ``msg``, and an opaque pointer ``peer``. Contents of ``msg`` will be +identical to ones sent by the sender. + +If a response is required, a new ``rte_mp_msg`` message descriptor must be +constructed and sent via ``rte_mp_reply()`` function, along with ``peer`` +pointer. The resulting response will then be delivered to the correct requestor. + +Misc considerations +~~~~~~~~~~~~~~~~~~~~~~~~ + +Due to the underlying IPC implementation being single-threaded, recursive +requests (i.e. sending a request while responding to another request) is not +supported. However, since sending messages (not requests) does not involve an +IPC thread, sending messages while processing another message or request is +supported. + +Asynchronous request callbacks may be triggered either from IPC thread or from +interrupt thread, depending on whether the request has timed out. It is +therefore suggested to avoid waiting for interrupt-based events (such as alarms) +inside asynchronous IPC request callbacks. This limitation does not apply to +messages or synchronous requests. + +If callbacks spend a long time processing the incoming requests, the requestor +might time out, so setting the right timeout value on the requestor side is +imperative. + +If some of the messages timed out, ``nb_sent`` and ``nb_received`` fields in the +``rte_mp_reply`` descriptor will not have matching values. This is not treated +as error by the IPC API, and it is expected that the user will be responsible +for deciding how to handle such cases. + +If a callback has been registered, IPC will assume that it is safe to call it. +This is important when registering callbacks during DPDK initialization. +During initialization, IPC will consider the receiving side as non-existing if +the callback has not been registered yet. However, once the callback has been +registered, it is expected that IPC should be safe to trigger it, even if the +rest of the DPDK initialization hasn't finished yet. diff --git a/doc/guides/prog_guide/overview.rst b/doc/guides/prog_guide/overview.rst index 2663fe0e..c01f37e3 100644 --- a/doc/guides/prog_guide/overview.rst +++ b/doc/guides/prog_guide/overview.rst @@ -130,8 +130,8 @@ The mbuf library provides the facility to create and destroy buffers that may be used by the DPDK application to store message buffers. The message buffers are created at startup time and stored in a mempool, using the DPDK mempool library. -This library provides an API to allocate/free mbufs, manipulate control message buffers (ctrlmbuf) which are generic message buffers, -and packet buffers (pktmbuf) which are used to carry network packets. +This library provides an API to allocate/free mbufs, manipulate +packet buffers which are used to carry network packets. Network Packet Buffer Management is described in :ref:`Mbuf Library <Mbuf_Library>`. diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index e5d01874..b2cf4835 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -296,22 +296,37 @@ described in the mbuf API documentation and in the in :ref:`Mbuf Library Per-Port and Per-Queue Offloads ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -In the DPDK offload API, offloads are divided into per-port and per-queue offloads. +In the DPDK offload API, offloads are divided into per-port and per-queue offloads as follows: + +* A per-queue offloading can be enabled on a queue and disabled on another queue at the same time. +* A pure per-port offload is the one supported by device but not per-queue type. +* A pure per-port offloading can't be enabled on a queue and disabled on another queue at the same time. +* A pure per-port offloading must be enabled or disabled on all queues at the same time. +* Any offloading is per-queue or pure per-port type, but can't be both types at same devices. +* Port capabilities = per-queue capabilities + pure per-port capabilities. +* Any supported offloading can be enabled on all queues. + The different offloads capabilities can be queried using ``rte_eth_dev_info_get()``. +The ``dev_info->[rt]x_queue_offload_capa`` returned from ``rte_eth_dev_info_get()`` includes all per-queue offloading capabilities. +The ``dev_info->[rt]x_offload_capa`` returned from ``rte_eth_dev_info_get()`` includes all pure per-port and per-queue offloading capabilities. Supported offloads can be either per-port or per-queue. Offloads are enabled using the existing ``DEV_TX_OFFLOAD_*`` or ``DEV_RX_OFFLOAD_*`` flags. -Per-port offload configuration is set using ``rte_eth_dev_configure``. -Per-queue offload configuration is set using ``rte_eth_rx_queue_setup`` and ``rte_eth_tx_queue_setup``. -To enable per-port offload, the offload should be set on both device configuration and queue setup. -In case of a mixed configuration the queue setup shall return with an error. -To enable per-queue offload, the offload can be set only on the queue setup. -Offloads which are not enabled are disabled by default. - -For an application to use the Tx offloads API it should set the ``ETH_TXQ_FLAGS_IGNORE`` flag in the ``txq_flags`` field located in ``rte_eth_txconf`` struct. -In such cases it is not required to set other flags in ``txq_flags``. -For an application to use the Rx offloads API it should set the ``ignore_offload_bitfield`` bit in the ``rte_eth_rxmode`` struct. -In such cases it is not required to set other bitfield offloads in the ``rxmode`` struct. +Any requested offloading by an application must be within the device capabilities. +Any offloading is disabled by default if it is not set in the parameter +``dev_conf->[rt]xmode.offloads`` to ``rte_eth_dev_configure()`` and +``[rt]x_conf->offloads`` to ``rte_eth_[rt]x_queue_setup()``. + +If any offloading is enabled in ``rte_eth_dev_configure()`` by an application, +it is enabled on all queues no matter whether it is per-queue or +per-port type and no matter whether it is set or cleared in +``[rt]x_conf->offloads`` to ``rte_eth_[rt]x_queue_setup()``. + +If a per-queue offloading hasn't been enabled in ``rte_eth_dev_configure()``, +it can be enabled or disabled in ``rte_eth_[rt]x_queue_setup()`` for individual queue. +A newly added offloads in ``[rt]x_conf->offloads`` to ``rte_eth_[rt]x_queue_setup()`` input by application +is the one which hasn't been enabled in ``rte_eth_dev_configure()`` and is requested to be enabled +in ``rte_eth_[rt]x_queue_setup()``. It must be per-queue type, otherwise trigger an error log. Poll Mode Driver API -------------------- @@ -345,6 +360,27 @@ Ethernet Device API The Ethernet device API exported by the Ethernet PMDs is described in the *DPDK API Reference*. +.. _ethernet_device_standard_device_arguments: + +Ethernet Device Standard Device Arguments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Standard Ethernet device arguments allow for a set of commonly used arguments/ +parameters which are applicable to all Ethernet devices to be available to for +specification of specific device and for passing common configuration +parameters to those ports. + +* ``representor`` for a device which supports the creation of representor ports + this argument allows user to specify which switch ports to enable port + representors for.:: + + -w BDBF,representor=0 + -w BDBF,representor=[0,4,6,9] + -w BDBF,representor=[0-31] + +Note: PMDs are not required to support the standard device arguments and users +should consult the relevant PMD documentation to see support devargs. + Extended Statistics API ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/prog_guide/rawdev.rst b/doc/guides/prog_guide/rawdev.rst index 54bffc58..42c195ce 100644 --- a/doc/guides/prog_guide/rawdev.rst +++ b/doc/guides/prog_guide/rawdev.rst @@ -43,7 +43,7 @@ Key factors guiding design of the Rawdevice library: * Firmware Management - Firmware load/unload/status 2. Application API should be able to pass along arbitrary state information - to/fro device driver. This can be achieved by maintaining context + to/from device driver. This can be achieved by maintaining context information through opaque data or pointers. Figure below outlines the layout of the rawdevice library and device vis-a-vis diff --git a/doc/guides/prog_guide/rte_flow.rst b/doc/guides/prog_guide/rte_flow.rst index 961943dd..b305a72a 100644 --- a/doc/guides/prog_guide/rte_flow.rst +++ b/doc/guides/prog_guide/rte_flow.rst @@ -1,32 +1,6 @@ -.. BSD LICENSE +.. SPDX-License-Identifier: BSD-3-Clause Copyright 2016 6WIND S.A. - Copyright 2016 Mellanox. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of 6WIND S.A. nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + Copyright 2016 Mellanox Technologies, Ltd .. _Generic_flow_API: @@ -55,9 +29,6 @@ encompasses and supersedes (including all functions and filter types) in order to expose a single interface with an unambiguous behavior that is common to all poll-mode drivers (PMDs). -Several methods to migrate existing applications are described in `API -migration`_. - Flow rule --------- @@ -93,8 +64,12 @@ Thus predictable results for a given priority level can only be achieved with non-overlapping rules, using perfect matching on all protocol layers. Flow rules can also be grouped, the flow rule priority is specific to the -group they belong to. All flow rules in a given group are thus processed -either before or after another group. +group they belong to. All flow rules in a given group are thus processed within +the context of that group. Groups are not linked by default, so the logical +hierarchy of groups must be explicitly defined by flow rules themselves in each +group using the JUMP action to define the next group to redirect too. Only flow +rules defined in the default group 0 are guarantee to be matched against, this +makes group 0 the origin of any group hierarchy defined by an application. Support for multiple actions per rule may be implemented internally on top of non-default hardware priorities, as a result both features may not be @@ -141,29 +116,34 @@ Attributes Attribute: Group ^^^^^^^^^^^^^^^^ -Flow rules can be grouped by assigning them a common group number. Lower -values have higher priority. Group 0 has the highest priority. +Flow rules can be grouped by assigning them a common group number. Groups +allow a logical hierarchy of flow rule groups (tables) to be defined. These +groups can be supported virtually in the PMD or in the physical device. +Group 0 is the default group and this is the only group which flows are +guarantee to matched against, all subsequent groups can only be reached by +way of the JUMP action from a matched flow rule. Although optional, applications are encouraged to group similar rules as much as possible to fully take advantage of hardware capabilities (e.g. optimized matching) and work around limitations (e.g. a single pattern -type possibly allowed in a given group). +type possibly allowed in a given group), while being aware that the groups +hierarchies must be programmed explicitly. Note that support for more than a single group is not guaranteed. Attribute: Priority ^^^^^^^^^^^^^^^^^^^ -A priority level can be assigned to a flow rule. Like groups, lower values +A priority level can be assigned to a flow rule, lower values denote higher priority, with 0 as the maximum. -A rule with priority 0 in group 8 is always matched after a rule with -priority 8 in group 0. - -Group and priority levels are arbitrary and up to the application, they do +Priority levels are arbitrary and up to the application, they do not need to be contiguous nor start from 0, however the maximum number varies between devices and may be affected by existing flow rules. +A flow which matches multiple rules in the same group will always matched by +the rule with the highest priority in that group. + If a packet is matched by several rules of a given group for a given priority level, the outcome is undefined. It can take any path, may be duplicated or even cause unrecoverable errors. @@ -173,7 +153,13 @@ Note that support for more than a single priority level is not guaranteed. Attribute: Traffic direction ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Flow rules can apply to inbound and/or outbound traffic (ingress/egress). +Flow rule patterns apply to inbound and/or outbound traffic. + +In the context of this API, **ingress** and **egress** respectively stand +for **inbound** and **outbound** based on the standpoint of the application +creating a flow rule. + +There are no exceptions to this definition. Several pattern items and actions are valid and can be used in both directions. At least one direction must be specified. @@ -181,17 +167,36 @@ directions. At least one direction must be specified. Specifying both directions at once for a given rule is not recommended but may be valid in a few cases (e.g. shared counters). +Attribute: Transfer +^^^^^^^^^^^^^^^^^^^ + +Instead of simply matching the properties of traffic as it would appear on a +given DPDK port ID, enabling this attribute transfers a flow rule to the +lowest possible level of any device endpoints found in the pattern. + +When supported, this effectively enables an application to reroute traffic +not necessarily intended for it (e.g. coming from or addressed to different +physical ports, VFs or applications) at the device level. + +It complements the behavior of some pattern items such as `Item: PHY_PORT`_ +and is meaningless without them. + +When transferring flow rules, **ingress** and **egress** attributes +(`Attribute: Traffic direction`_) keep their original meaning, as if +processing traffic emitted or received by the application. + Pattern item ~~~~~~~~~~~~ Pattern items fall in two categories: -- Matching protocol headers and packet data (ANY, RAW, ETH, VLAN, IPV4, - IPV6, ICMP, UDP, TCP, SCTP, VXLAN, MPLS, GRE, ESP and so on), usually - associated with a specification structure. +- Matching protocol headers and packet data, usually associated with a + specification structure. These must be stacked in the same order as the + protocol layers to match inside packets, starting from the lowest. -- Matching meta-data or affecting pattern processing (END, VOID, INVERT, PF, - VF, PORT and so on), often without a specification structure. +- Matching meta-data or affecting pattern processing, often without a + specification structure. Since they do not match packet contents, their + position in the list is usually not relevant. Item specification structures are used to match specific values among protocol fields (or item properties). Documentation describes for each item @@ -506,15 +511,12 @@ Usage example, matching non-TCPv4 packets only: Item: ``PF`` ^^^^^^^^^^^^ -Matches packets addressed to the physical function of the device. +Matches traffic originating from (ingress) or going to (egress) the physical +function of the current device. -If the underlying device function differs from the one that would normally -receive the matched traffic, specifying this item prevents it from reaching -that device unless the flow rule contains a `Action: PF`_. Packets are not -duplicated between device instances by default. +If supported, should work even if the physical function is not managed by +the application and thus not associated with a DPDK port ID. -- Likely to return an error or never match any traffic if applied to a VF - device. - Can be combined with any number of `Item: VF`_ to match both PF and VF traffic. - ``spec``, ``last`` and ``mask`` must not be set. @@ -536,15 +538,15 @@ duplicated between device instances by default. Item: ``VF`` ^^^^^^^^^^^^ -Matches packets addressed to a virtual function ID of the device. +Matches traffic originating from (ingress) or going to (egress) a given +virtual function of the current device. + +If supported, should work even if the virtual function is not managed by the +application and thus not associated with a DPDK port ID. -If the underlying device function differs from the one that would normally -receive the matched traffic, specifying this item prevents it from reaching -that device unless the flow rule contains a `Action: VF`_. Packets are not -duplicated between device instances by default. +Note this pattern item does not match VF representors traffic which, as +separate entities, should be addressed through their own DPDK port IDs. -- Likely to return an error or never match any traffic if this causes a VF - device to match traffic addressed to a different VF. - Can be specified multiple times to match traffic addressed to several VF IDs. - Can be combined with a PF item to match both PF and VF traffic. @@ -564,15 +566,15 @@ duplicated between device instances by default. | ``mask`` | ``id`` | zeroed to match any VF ID | +----------+----------+---------------------------+ -Item: ``PORT`` -^^^^^^^^^^^^^^ +Item: ``PHY_PORT`` +^^^^^^^^^^^^^^^^^^ -Matches packets coming from the specified physical port of the underlying -device. +Matches traffic originating from (ingress) or going to (egress) a physical +port of the underlying device. -The first PORT item overrides the physical port normally associated with the -specified DPDK input port (port_id). This item can be provided several times -to match additional physical ports. +The first PHY_PORT item overrides the physical port normally associated with +the specified DPDK input port (port_id). This item can be provided several +times to match additional physical ports. Note that physical ports are not necessarily tied to DPDK input ports (port_id) when those are not under DPDK control. Possible values are @@ -584,9 +586,9 @@ associated with a port_id should be retrieved by other means. - Default ``mask`` matches any port index. -.. _table_rte_flow_item_port: +.. _table_rte_flow_item_phy_port: -.. table:: PORT +.. table:: PHY_PORT +----------+-----------+--------------------------------+ | Field | Subfield | Value | @@ -598,6 +600,66 @@ associated with a port_id should be retrieved by other means. | ``mask`` | ``index`` | zeroed to match any port index | +----------+-----------+--------------------------------+ +Item: ``PORT_ID`` +^^^^^^^^^^^^^^^^^ + +Matches traffic originating from (ingress) or going to (egress) a given DPDK +port ID. + +Normally only supported if the port ID in question is known by the +underlying PMD and related to the device the flow rule is created against. + +This must not be confused with `Item: PHY_PORT`_ which refers to the +physical port of a device, whereas `Item: PORT_ID`_ refers to a ``struct +rte_eth_dev`` object on the application side (also known as "port +representor" depending on the kind of underlying device). + +- Default ``mask`` matches the specified DPDK port ID. + +.. _table_rte_flow_item_port_id: + +.. table:: PORT_ID + + +----------+----------+-----------------------------+ + | Field | Subfield | Value | + +==========+==========+=============================+ + | ``spec`` | ``id`` | DPDK port ID | + +----------+----------+-----------------------------+ + | ``last`` | ``id`` | upper range value | + +----------+----------+-----------------------------+ + | ``mask`` | ``id`` | zeroed to match any port ID | + +----------+----------+-----------------------------+ + +Item: ``MARK`` +^^^^^^^^^^^^^^ + +Matches an arbitrary integer value which was set using the ``MARK`` action in +a previously matched rule. + +This item can only specified once as a match criteria as the ``MARK`` action can +only be specified once in a flow action. + +Note the value of MARK field is arbitrary and application defined. + +Depending on the underlying implementation the MARK item may be supported on +the physical device, with virtual groups in the PMD or not at all. + +- Default ``mask`` matches any integer value. + +.. _table_rte_flow_item_mark: + +.. table:: MARK + + +----------+----------+---------------------------+ + | Field | Subfield | Value | + +==========+==========+===========================+ + | ``spec`` | ``id`` | integer value | + +----------+--------------------------------------+ + | ``last`` | ``id`` | upper range value | + +----------+----------+---------------------------+ + | ``mask`` | ``id`` | zeroed to match any value | + +----------+----------+---------------------------+ + Data matching item types ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -786,9 +848,15 @@ Item: ``ETH`` Matches an Ethernet header. +The ``type`` field either stands for "EtherType" or "TPID" when followed by +so-called layer 2.5 pattern items such as ``RTE_FLOW_ITEM_TYPE_VLAN``. In +the latter case, ``type`` refers to that of the outer header, with the inner +EtherType/TPID provided by the subsequent pattern item. This is the same +order as on the wire. + - ``dst``: destination MAC. - ``src``: source MAC. -- ``type``: EtherType. +- ``type``: EtherType or TPID. - Default ``mask`` matches destination and source addresses only. Item: ``VLAN`` @@ -796,9 +864,13 @@ Item: ``VLAN`` Matches an 802.1Q/ad VLAN tag. -- ``tpid``: tag protocol identifier. +The corresponding standard outer EtherType (TPID) values are +``ETHER_TYPE_VLAN`` or ``ETHER_TYPE_QINQ``. It can be overridden by the +preceding pattern item. + - ``tci``: tag control information. -- Default ``mask`` matches TCI only. +- ``inner_type``: inner EtherType or TPID. +- Default ``mask`` matches the VID part of TCI only (lower 12 bits). Item: ``IPV4`` ^^^^^^^^^^^^^^ @@ -815,7 +887,8 @@ Item: ``IPV6`` Matches an IPv6 header. -Note: IPv6 options are handled by dedicated pattern items. +Note: IPv6 options are handled by dedicated pattern items, see `Item: +IPV6_EXT`_. - ``hdr``: IPv6 header definition (``rte_ip.h``). - Default ``mask`` matches source and destination addresses only. @@ -868,12 +941,15 @@ Item: ``E_TAG`` Matches an IEEE 802.1BR E-Tag header. -- ``tpid``: tag protocol identifier (0x893F) +The corresponding standard outer EtherType (TPID) value is +``ETHER_TYPE_ETAG``. It can be overridden by the preceding pattern item. + - ``epcp_edei_in_ecid_b``: E-Tag control information (E-TCI), E-PCP (3b), E-DEI (1b), ingress E-CID base (12b). - ``rsvd_grp_ecid_b``: reserved (2b), GRP (2b), E-CID base (12b). - ``in_ecid_e``: ingress E-CID ext. - ``ecid_e``: E-CID ext. +- ``inner_type``: inner EtherType or TPID. - Default ``mask`` simultaneously matches GRP and E-CID base. Item: ``NVGRE`` @@ -992,35 +1068,155 @@ Matches a GENEVE header. - ``rsvd1``: reserved, normally 0x00. - Default ``mask`` matches VNI only. +Item: ``VXLAN-GPE`` +^^^^^^^^^^^^^^^^^^^ + +Matches a VXLAN-GPE header (draft-ietf-nvo3-vxlan-gpe-05). + +- ``flags``: normally 0x0C (I and P flags). +- ``rsvd0``: reserved, normally 0x0000. +- ``protocol``: protocol type. +- ``vni``: VXLAN network identifier. +- ``rsvd1``: reserved, normally 0x00. +- Default ``mask`` matches VNI only. + +Item: ``ARP_ETH_IPV4`` +^^^^^^^^^^^^^^^^^^^^^^ + +Matches an ARP header for Ethernet/IPv4. + +- ``hdr``: hardware type, normally 1. +- ``pro``: protocol type, normally 0x0800. +- ``hln``: hardware address length, normally 6. +- ``pln``: protocol address length, normally 4. +- ``op``: opcode (1 for request, 2 for reply). +- ``sha``: sender hardware address. +- ``spa``: sender IPv4 address. +- ``tha``: target hardware address. +- ``tpa``: target IPv4 address. +- Default ``mask`` matches SHA, SPA, THA and TPA. + +Item: ``IPV6_EXT`` +^^^^^^^^^^^^^^^^^^ + +Matches the presence of any IPv6 extension header. + +- ``next_hdr``: next header. +- Default ``mask`` matches ``next_hdr``. + +Normally preceded by any of: + +- `Item: IPV6`_ +- `Item: IPV6_EXT`_ + +Item: ``ICMP6`` +^^^^^^^^^^^^^^^ + +Matches any ICMPv6 header. + +- ``type``: ICMPv6 type. +- ``code``: ICMPv6 code. +- ``checksum``: ICMPv6 checksum. +- Default ``mask`` matches ``type`` and ``code``. + +Item: ``ICMP6_ND_NS`` +^^^^^^^^^^^^^^^^^^^^^ + +Matches an ICMPv6 neighbor discovery solicitation. + +- ``type``: ICMPv6 type, normally 135. +- ``code``: ICMPv6 code, normally 0. +- ``checksum``: ICMPv6 checksum. +- ``reserved``: reserved, normally 0. +- ``target_addr``: target address. +- Default ``mask`` matches target address only. + +Item: ``ICMP6_ND_NA`` +^^^^^^^^^^^^^^^^^^^^^ + +Matches an ICMPv6 neighbor discovery advertisement. + +- ``type``: ICMPv6 type, normally 136. +- ``code``: ICMPv6 code, normally 0. +- ``checksum``: ICMPv6 checksum. +- ``rso_reserved``: route flag (1b), solicited flag (1b), override flag + (1b), reserved (29b). +- ``target_addr``: target address. +- Default ``mask`` matches target address only. + +Item: ``ICMP6_ND_OPT`` +^^^^^^^^^^^^^^^^^^^^^^ + +Matches the presence of any ICMPv6 neighbor discovery option. + +- ``type``: ND option type. +- ``length``: ND option length. +- Default ``mask`` matches type only. + +Normally preceded by any of: + +- `Item: ICMP6_ND_NA`_ +- `Item: ICMP6_ND_NS`_ +- `Item: ICMP6_ND_OPT`_ + +Item: ``ICMP6_ND_OPT_SLA_ETH`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Matches an ICMPv6 neighbor discovery source Ethernet link-layer address +option. + +- ``type``: ND option type, normally 1. +- ``length``: ND option length, normally 1. +- ``sla``: source Ethernet LLA. +- Default ``mask`` matches source link-layer address only. + +Normally preceded by any of: + +- `Item: ICMP6_ND_NA`_ +- `Item: ICMP6_ND_OPT`_ + +Item: ``ICMP6_ND_OPT_TLA_ETH`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Matches an ICMPv6 neighbor discovery target Ethernet link-layer address +option. + +- ``type``: ND option type, normally 2. +- ``length``: ND option length, normally 1. +- ``tla``: target Ethernet LLA. +- Default ``mask`` matches target link-layer address only. + +Normally preceded by any of: + +- `Item: ICMP6_ND_NS`_ +- `Item: ICMP6_ND_OPT`_ + Actions ~~~~~~~ Each possible action is represented by a type. Some have associated configuration structures. Several actions combined in a list can be assigned -to a flow rule. That list is not ordered. +to a flow rule and are performed in order. They fall in three categories: -- Terminating actions (such as QUEUE, DROP, RSS, PF, VF) that prevent - processing matched packets by subsequent flow rules, unless overridden - with PASSTHRU. +- Actions that modify the fate of matching traffic, for instance by dropping + or assigning it a specific destination. -- Non-terminating actions (PASSTHRU, DUP) that leave matched packets up for - additional processing by subsequent flow rules. +- Actions that modify matching traffic contents or its properties. This + includes adding/removing encapsulation, encryption, compression and marks. -- Other non-terminating meta actions that do not affect the fate of packets - (END, VOID, MARK, FLAG, COUNT, SECURITY). +- Actions related to the flow rule itself, such as updating counters or + making it non-terminating. -When several actions are combined in a flow rule, they should all have -different types (e.g. dropping a packet twice is not possible). +Flow rules being terminating by default, not specifying any action of the +fate kind results in undefined behavior. This applies to both ingress and +egress. -Only the last action of a given type is taken into account. PMDs still -perform error checking on the entire list. +PASSTHRU, when supported, makes a flow rule non-terminating. Like matching patterns, action lists are terminated by END items. -*Note that PASSTHRU is the only action able to override a terminating rule.* - Example of action that redirects packets to queue index 10: .. _table_rte_flow_action_example: @@ -1033,12 +1229,11 @@ Example of action that redirects packets to queue index 10: | ``index`` | 10 | +-----------+-------+ -Action lists examples, their order is not significant, applications must -consider all actions to be performed simultaneously: +Actions are performed in list order: -.. _table_rte_flow_count_and_drop: +.. _table_rte_flow_count_then_drop: -.. table:: Count and drop +.. table:: Count then drop +-------+--------+ | Index | Action | @@ -1054,19 +1249,21 @@ consider all actions to be performed simultaneously: .. _table_rte_flow_mark_count_redirect: -.. table:: Mark, count and redirect - - +-------+--------+-----------+-------+ - | Index | Action | Field | Value | - +=======+========+===========+=======+ - | 0 | MARK | ``mark`` | 0x2a | - +-------+--------+-----------+-------+ - | 1 | COUNT | - +-------+--------+-----------+-------+ - | 2 | QUEUE | ``queue`` | 10 | - +-------+--------+-----------+-------+ - | 3 | END | - +-------+----------------------------+ +.. table:: Mark, count then redirect + + +-------+--------+------------+-------+ + | Index | Action | Field | Value | + +=======+========+============+=======+ + | 0 | MARK | ``mark`` | 0x2a | + +-------+--------+------------+-------+ + | 1 | COUNT | ``shared`` | 0 | + | | +------------+-------+ + | | | ``id`` | 0 | + +-------+--------+------------+-------+ + | 2 | QUEUE | ``queue`` | 10 | + +-------+--------+------------+-------+ + | 3 | END | + +-------+-----------------------------+ | @@ -1084,12 +1281,15 @@ consider all actions to be performed simultaneously: | 2 | END | +-------+----------------------------+ -In the above example, considering both actions are performed simultaneously, -the end result is that only QUEUE has any effect. +In the above example, while DROP and QUEUE must be performed in order, both +have to happen before reaching END. Only QUEUE has a visible effect. + +Note that such a list may be thought as ambiguous and rejected on that +basis. -.. _table_rte_flow_redirect_queue_3: +.. _table_rte_flow_redirect_queue_5_3: -.. table:: Redirect to queue 3 +.. table:: Redirect to queues 5 and 3 +-------+--------+-----------+-------+ | Index | Action | Field | Value | @@ -1103,9 +1303,9 @@ the end result is that only QUEUE has any effect. | 3 | END | +-------+----------------------------+ -As previously described, only the last action of a given type found in the -list is taken into account. The above example also shows that VOID is -ignored. +As previously described, all actions must be taken into account. This +effectively duplicates traffic to both queues. The above example also shows +that VOID is ignored. Action types ~~~~~~~~~~~~ @@ -1155,9 +1355,8 @@ PMDs. Action: ``PASSTHRU`` ^^^^^^^^^^^^^^^^^^^^ -Leaves packets up for additional processing by subsequent flow rules. This -is the default when a rule does not contain a terminating action, but can be -specified to force a rule to become non-terminating. +Leaves traffic up for additional processing by subsequent flow rules; makes +a flow rule non-terminating. - No configurable properties. @@ -1188,6 +1387,38 @@ flow rules: | 2 | END | +-------+----------------------------+ +Action: ``JUMP`` +^^^^^^^^^^^^^^^^ + +Redirects packets to a group on the current device. + +In a hierarchy of groups, which can be used to represent physical or logical +flow group/tables on the device, this action redirects the matched flow to +the specified group on that device. + +If a matched flow is redirected to a table which doesn't contain a matching +rule for that flow then the behavior is undefined and the resulting behavior +is up to the specific device. Best practice when using groups would be define +a default flow rule for each group which a defines the default actions in that +group so a consistent behavior is defined. + +Defining an action for matched flow in a group to jump to a group which is +higher in the group hierarchy may not be supported by physical devices, +depending on how groups are mapped to the physical devices. In the +definitions of jump actions, applications should be aware that it may be +possible to define flow rules which trigger an undefined behavior causing +flows to loop between groups. + +.. _table_rte_flow_action_jump: + +.. table:: JUMP + + +-----------+------------------------------+ + | Field | Value | + +===========+==============================+ + | ``group`` | Group to redirect packets to | + +-----------+------------------------------+ + Action: ``MARK`` ^^^^^^^^^^^^^^^^ @@ -1231,8 +1462,6 @@ Action: ``QUEUE`` Assigns packets to a given queue index. -- Terminating by default. - .. _table_rte_flow_action_queue: .. table:: QUEUE @@ -1249,8 +1478,6 @@ Action: ``DROP`` Drop packets. - No configurable properties. -- Terminating by default. -- PASSTHRU overrides this action if both are specified. .. _table_rte_flow_action_drop: @@ -1265,23 +1492,36 @@ Drop packets. Action: ``COUNT`` ^^^^^^^^^^^^^^^^^ -Enables counters for this rule. +Adds a counter action to a matched flow. + +If more than one count action is specified in a single flow rule, then each +action must specify a unique id. -These counters can be retrieved and reset through ``rte_flow_query()``, see +Counters can be retrieved and reset through ``rte_flow_query()``, see ``struct rte_flow_query_count``. -- Counters can be retrieved with ``rte_flow_query()``. -- No configurable properties. +The shared flag indicates whether the counter is unique to the flow rule the +action is specified with, or whether it is a shared counter. + +For a count action with the shared flag set, then then a global device +namespace is assumed for the counter id, so that any matched flow rules using +a count action with the same counter id on the same port will contribute to +that counter. + +For ports within the same switch domain then the counter id namespace extends +to all ports within that switch domain. .. _table_rte_flow_action_count: .. table:: COUNT - +---------------+ - | Field | - +===============+ - | no properties | - +---------------+ + +------------+---------------------+ + | Field | Value | + +============+=====================+ + | ``shared`` | shared counter flag | + +------------+---------------------+ + | ``id`` | counter id | + +------------+---------------------+ Query structure to retrieve and reset flow rule counters: @@ -1303,59 +1543,75 @@ Query structure to retrieve and reset flow rule counters: | ``bytes`` | out | number of bytes through this rule | +---------------+-----+-----------------------------------+ -Action: ``DUP`` +Action: ``RSS`` ^^^^^^^^^^^^^^^ -Duplicates packets to a given queue index. +Similar to QUEUE, except RSS is additionally performed on packets to spread +them among several queues according to the provided parameters. -This is normally combined with QUEUE, however when used alone, it is -actually similar to QUEUE + PASSTHRU. +Unlike global RSS settings used by other DPDK APIs, unsetting the ``types`` +field does not disable RSS in a flow rule. Doing so instead requests safe +unspecified "best-effort" settings from the underlying PMD, which depending +on the flow rule, may result in anything ranging from empty (single queue) +to all-inclusive RSS. -- Non-terminating by default. +Note: RSS hash result is stored in the ``hash.rss`` mbuf field which +overlaps ``hash.fdir.lo``. Since `Action: MARK`_ sets the ``hash.fdir.hi`` +field only, both can be requested simultaneously. -.. _table_rte_flow_action_dup: +Also, regarding packet encapsulation ``level``: -.. table:: DUP +- ``0`` requests the default behavior. Depending on the packet type, it can + mean outermost, innermost, anything in between or even no RSS. - +-----------+------------------------------------+ - | Field | Value | - +===========+====================================+ - | ``index`` | queue index to duplicate packet to | - +-----------+------------------------------------+ + It basically stands for the innermost encapsulation level RSS can be + performed on according to PMD and device capabilities. -Action: ``RSS`` -^^^^^^^^^^^^^^^ +- ``1`` requests RSS to be performed on the outermost packet encapsulation + level. -Similar to QUEUE, except RSS is additionally performed on packets to spread -them among several queues according to the provided parameters. +- ``2`` and subsequent values request RSS to be performed on the specified + inner packet encapsulation level, from outermost to innermost (lower to + higher values). -Note: RSS hash result is stored in the ``hash.rss`` mbuf field which -overlaps ``hash.fdir.lo``. Since `Action: MARK`_ sets the ``hash.fdir.hi`` -field only, both can be requested simultaneously. +Values other than ``0`` are not necessarily supported. -- Terminating by default. +Requesting a specific RSS level on unrecognized traffic results in undefined +behavior. For predictable results, it is recommended to make the flow rule +pattern match packet headers up to the requested encapsulation level so that +only matching traffic goes through. .. _table_rte_flow_action_rss: .. table:: RSS - +--------------+------------------------------+ - | Field | Value | - +==============+==============================+ - | ``rss_conf`` | RSS parameters | - +--------------+------------------------------+ - | ``num`` | number of entries in queue[] | - +--------------+------------------------------+ - | ``queue[]`` | queue indices to use | - +--------------+------------------------------+ + +---------------+---------------------------------------------+ + | Field | Value | + +===============+=============================================+ + | ``func`` | RSS hash function to apply | + +---------------+---------------------------------------------+ + | ``level`` | encapsulation level for ``types`` | + +---------------+---------------------------------------------+ + | ``types`` | specific RSS hash types (see ``ETH_RSS_*``) | + +---------------+---------------------------------------------+ + | ``key_len`` | hash key length in bytes | + +---------------+---------------------------------------------+ + | ``queue_num`` | number of entries in ``queue`` | + +---------------+---------------------------------------------+ + | ``key`` | hash key | + +---------------+---------------------------------------------+ + | ``queue`` | queue indices to use | + +---------------+---------------------------------------------+ Action: ``PF`` ^^^^^^^^^^^^^^ -Redirects packets to the physical function (PF) of the current device. +Directs matching traffic to the physical function (PF) of the current +device. + +See `Item: PF`_. - No configurable properties. -- Terminating by default. .. _table_rte_flow_action_pf: @@ -1370,14 +1626,14 @@ Redirects packets to the physical function (PF) of the current device. Action: ``VF`` ^^^^^^^^^^^^^^ -Redirects packets to a virtual function (VF) of the current device. +Directs matching traffic to a given virtual function of the current device. Packets matched by a VF pattern item can be redirected to their original VF ID instead of the specified one. This parameter may not be available and is not guaranteed to work properly if the VF part is matched by a prior flow rule or if packets are not addressed to a VF in the first place. -- Terminating by default. +See `Item: VF`_. .. _table_rte_flow_action_vf: @@ -1388,9 +1644,47 @@ rule or if packets are not addressed to a VF in the first place. +==============+================================+ | ``original`` | use original VF ID if possible | +--------------+--------------------------------+ - | ``vf`` | VF ID to redirect packets to | + | ``id`` | VF ID | +--------------+--------------------------------+ +Action: ``PHY_PORT`` +^^^^^^^^^^^^^^^^^^^^ + +Directs matching traffic to a given physical port index of the underlying +device. + +See `Item: PHY_PORT`_. + +.. _table_rte_flow_action_phy_port: + +.. table:: PHY_PORT + + +--------------+-------------------------------------+ + | Field | Value | + +==============+=====================================+ + | ``original`` | use original port index if possible | + +--------------+-------------------------------------+ + | ``index`` | physical port index | + +--------------+-------------------------------------+ + +Action: ``PORT_ID`` +^^^^^^^^^^^^^^^^^^^ +Directs matching traffic to a given DPDK port ID. + +See `Item: PORT_ID`_. + +.. _table_rte_flow_action_port_id: + +.. table:: PORT_ID + + +--------------+---------------------------------------+ + | Field | Value | + +==============+=======================================+ + | ``original`` | use original DPDK port ID if possible | + +--------------+---------------------------------------+ + | ``id`` | DPDK port ID | + +--------------+---------------------------------------+ + Action: ``METER`` ^^^^^^^^^^^^^^^^^ @@ -1402,8 +1696,6 @@ action parameter. More than one flow can use the same MTR object through the meter action. The MTR object can be further updated or queried using the rte_mtr* API. -- Non-terminating by default. - .. _table_rte_flow_action_meter: .. table:: METER @@ -1439,8 +1731,6 @@ direction. Multiple flows can be configured to use the same security session. -- Non-terminating by default. - .. _table_rte_flow_action_security: .. table:: SECURITY @@ -1486,6 +1776,306 @@ fields in the pattern items. | 1 | END | +-------+----------+ +Action: ``OF_SET_MPLS_TTL`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_SET_MPLS_TTL`` ("MPLS TTL") as defined by the `OpenFlow +Switch Specification`_. + +.. _table_rte_flow_action_of_set_mpls_ttl: + +.. table:: OF_SET_MPLS_TTL + + +--------------+----------+ + | Field | Value | + +==============+==========+ + | ``mpls_ttl`` | MPLS TTL | + +--------------+----------+ + +Action: ``OF_DEC_MPLS_TTL`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_DEC_MPLS_TTL`` ("decrement MPLS TTL") as defined by the +`OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_dec_mpls_ttl: + +.. table:: OF_DEC_MPLS_TTL + + +---------------+ + | Field | + +===============+ + | no properties | + +---------------+ + +Action: ``OF_SET_NW_TTL`` +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_SET_NW_TTL`` ("IP TTL") as defined by the `OpenFlow +Switch Specification`_. + +.. _table_rte_flow_action_of_set_nw_ttl: + +.. table:: OF_SET_NW_TTL + + +------------+--------+ + | Field | Value | + +============+========+ + | ``nw_ttl`` | IP TTL | + +------------+--------+ + +Action: ``OF_DEC_NW_TTL`` +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_DEC_NW_TTL`` ("decrement IP TTL") as defined by the +`OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_dec_nw_ttl: + +.. table:: OF_DEC_NW_TTL + + +---------------+ + | Field | + +===============+ + | no properties | + +---------------+ + +Action: ``OF_COPY_TTL_OUT`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_COPY_TTL_OUT`` ("copy TTL "outwards" -- from +next-to-outermost to outermost") as defined by the `OpenFlow Switch +Specification`_. + +.. _table_rte_flow_action_of_copy_ttl_out: + +.. table:: OF_COPY_TTL_OUT + + +---------------+ + | Field | + +===============+ + | no properties | + +---------------+ + +Action: ``OF_COPY_TTL_IN`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_COPY_TTL_IN`` ("copy TTL "inwards" -- from outermost to +next-to-outermost") as defined by the `OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_copy_ttl_in: + +.. table:: OF_COPY_TTL_IN + + +---------------+ + | Field | + +===============+ + | no properties | + +---------------+ + +Action: ``OF_POP_VLAN`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_POP_VLAN`` ("pop the outer VLAN tag") as defined +by the `OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_pop_vlan: + +.. table:: OF_POP_VLAN + + +---------------+ + | Field | + +===============+ + | no properties | + +---------------+ + +Action: ``OF_PUSH_VLAN`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_PUSH_VLAN`` ("push a new VLAN tag") as defined by the +`OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_push_vlan: + +.. table:: OF_PUSH_VLAN + + +---------------+-----------+ + | Field | Value | + +===============+===========+ + | ``ethertype`` | EtherType | + +---------------+-----------+ + +Action: ``OF_SET_VLAN_VID`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_SET_VLAN_VID`` ("set the 802.1q VLAN id") as defined by +the `OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_set_vlan_vid: + +.. table:: OF_SET_VLAN_VID + + +--------------+---------+ + | Field | Value | + +==============+=========+ + | ``vlan_vid`` | VLAN id | + +--------------+---------+ + +Action: ``OF_SET_VLAN_PCP`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_SET_LAN_PCP`` ("set the 802.1q priority") as defined by +the `OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_set_vlan_pcp: + +.. table:: OF_SET_VLAN_PCP + + +--------------+---------------+ + | Field | Value | + +==============+===============+ + | ``vlan_pcp`` | VLAN priority | + +--------------+---------------+ + +Action: ``OF_POP_MPLS`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_POP_MPLS`` ("pop the outer MPLS tag") as defined by the +`OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_pop_mpls: + +.. table:: OF_POP_MPLS + + +---------------+-----------+ + | Field | Value | + +===============+===========+ + | ``ethertype`` | EtherType | + +---------------+-----------+ + +Action: ``OF_PUSH_MPLS`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +Implements ``OFPAT_PUSH_MPLS`` ("push a new MPLS tag") as defined by the +`OpenFlow Switch Specification`_. + +.. _table_rte_flow_action_of_push_mpls: + +.. table:: OF_PUSH_MPLS + + +---------------+-----------+ + | Field | Value | + +===============+===========+ + | ``ethertype`` | EtherType | + +---------------+-----------+ + +Action: ``VXLAN_ENCAP`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Performs a VXLAN encapsulation action by encapsulating the matched flow in the +VXLAN tunnel as defined in the``rte_flow_action_vxlan_encap`` flow items +definition. + +This action modifies the payload of matched flows. The flow definition specified +in the ``rte_flow_action_tunnel_encap`` action structure must define a valid +VLXAN network overlay which conforms with RFC 7348 (Virtual eXtensible Local +Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks +over Layer 3 Networks). The pattern must be terminated with the +RTE_FLOW_ITEM_TYPE_END item type. + +.. _table_rte_flow_action_vxlan_encap: + +.. table:: VXLAN_ENCAP + + +----------------+-------------------------------------+ + | Field | Value | + +================+=====================================+ + | ``definition`` | Tunnel end-point overlay definition | + +----------------+-------------------------------------+ + +.. _table_rte_flow_action_vxlan_encap_example: + +.. table:: IPv4 VxLAN flow pattern example. + + +-------+----------+ + | Index | Item | + +=======+==========+ + | 0 | Ethernet | + +-------+----------+ + | 1 | IPv4 | + +-------+----------+ + | 2 | UDP | + +-------+----------+ + | 3 | VXLAN | + +-------+----------+ + | 4 | END | + +-------+----------+ + +Action: ``VXLAN_DECAP`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Performs a decapsulation action by stripping all headers of the VXLAN tunnel +network overlay from the matched flow. + +The flow items pattern defined for the flow rule with which a ``VXLAN_DECAP`` +action is specified, must define a valid VXLAN tunnel as per RFC7348. If the +flow pattern does not specify a valid VXLAN tunnel then a +RTE_FLOW_ERROR_TYPE_ACTION error should be returned. + +This action modifies the payload of matched flows. + +Action: ``NVGRE_ENCAP`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Performs a NVGRE encapsulation action by encapsulating the matched flow in the +NVGRE tunnel as defined in the``rte_flow_action_tunnel_encap`` flow item +definition. + +This action modifies the payload of matched flows. The flow definition specified +in the ``rte_flow_action_tunnel_encap`` action structure must defined a valid +NVGRE network overlay which conforms with RFC 7637 (NVGRE: Network +Virtualization Using Generic Routing Encapsulation). The pattern must be +terminated with the RTE_FLOW_ITEM_TYPE_END item type. + +.. _table_rte_flow_action_nvgre_encap: + +.. table:: NVGRE_ENCAP + + +----------------+-------------------------------------+ + | Field | Value | + +================+=====================================+ + | ``definition`` | NVGRE end-point overlay definition | + +----------------+-------------------------------------+ + +.. _table_rte_flow_action_nvgre_encap_example: + +.. table:: IPv4 NVGRE flow pattern example. + + +-------+----------+ + | Index | Item | + +=======+==========+ + | 0 | Ethernet | + +-------+----------+ + | 1 | IPv4 | + +-------+----------+ + | 2 | NVGRE | + +-------+----------+ + | 3 | END | + +-------+----------+ + +Action: ``NVGRE_DECAP`` +^^^^^^^^^^^^^^^^^^^^^^^ + +Performs a decapsulation action by stripping all headers of the NVGRE tunnel +network overlay from the matched flow. + +The flow items pattern defined for the flow rule with which a ``NVGRE_DECAP`` +action is specified, must define a valid NVGRE tunnel as per RFC7637. If the +flow pattern does not specify a valid NVGRE tunnel then a +RTE_FLOW_ERROR_TYPE_ACTION error should be returned. + +This action modifies the payload of matched flows. + Negative types ~~~~~~~~~~~~~~ @@ -1681,7 +2271,7 @@ definition. int rte_flow_query(uint16_t port_id, struct rte_flow *flow, - enum rte_flow_action_type action, + const struct rte_flow_action *action, void *data, struct rte_flow_error *error); @@ -1689,7 +2279,7 @@ Arguments: - ``port_id``: port identifier of Ethernet device. - ``flow``: flow rule handle to query. -- ``action``: action type to query. +- ``action``: action to query, this must match prototype from flow rule. - ``data``: pointer to storage for the associated query data type. - ``error``: perform verbose error reporting if not NULL. PMDs initialize this structure in case of error only. @@ -2014,9 +2604,6 @@ Unsupported actions and tagging (`Action: MARK`_ or `Action: FLAG`_) may be implemented in software as long as the target queue is used by a single rule. -- A rule specifying both `Action: DUP`_ + `Action: QUEUE`_ may be translated - to two hidden rules combining `Action: QUEUE`_ and `Action: PASSTHRU`_. - - When a single target queue is provided, `Action: RSS`_ can also be implemented through `Action: QUEUE`_. @@ -2070,297 +2657,4 @@ Future evolutions - Optional software fallback when PMDs are unable to handle requested flow rules so applications do not have to implement their own. -API migration -------------- - -Exhaustive list of deprecated filter types (normally prefixed with -*RTE_ETH_FILTER_*) found in ``rte_eth_ctrl.h`` and methods to convert them -to *rte_flow* rules. - -``MACVLAN`` to ``ETH`` → ``VF``, ``PF`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*MACVLAN* can be translated to a basic `Item: ETH`_ flow rule with a -terminating `Action: VF`_ or `Action: PF`_. - -.. _table_rte_flow_migration_macvlan: - -.. table:: MACVLAN conversion - - +--------------------------+---------+ - | Pattern | Actions | - +===+=====+==========+=====+=========+ - | 0 | ETH | ``spec`` | any | VF, | - | | +----------+-----+ PF | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-----+----------+-----+---------+ - | 1 | END | END | - +---+----------------------+---------+ - -``ETHERTYPE`` to ``ETH`` → ``QUEUE``, ``DROP`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*ETHERTYPE* is basically an `Item: ETH`_ flow rule with a terminating -`Action: QUEUE`_ or `Action: DROP`_. - -.. _table_rte_flow_migration_ethertype: - -.. table:: ETHERTYPE conversion - - +--------------------------+---------+ - | Pattern | Actions | - +===+=====+==========+=====+=========+ - | 0 | ETH | ``spec`` | any | QUEUE, | - | | +----------+-----+ DROP | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-----+----------+-----+---------+ - | 1 | END | END | - +---+----------------------+---------+ - -``FLEXIBLE`` to ``RAW`` → ``QUEUE`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*FLEXIBLE* can be translated to one `Item: RAW`_ pattern with a terminating -`Action: QUEUE`_ and a defined priority level. - -.. _table_rte_flow_migration_flexible: - -.. table:: FLEXIBLE conversion - - +--------------------------+---------+ - | Pattern | Actions | - +===+=====+==========+=====+=========+ - | 0 | RAW | ``spec`` | any | QUEUE | - | | +----------+-----+ | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-----+----------+-----+---------+ - | 1 | END | END | - +---+----------------------+---------+ - -``SYN`` to ``TCP`` → ``QUEUE`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*SYN* is a `Item: TCP`_ rule with only the ``syn`` bit enabled and masked, -and a terminating `Action: QUEUE`_. - -Priority level can be set to simulate the high priority bit. - -.. _table_rte_flow_migration_syn: - -.. table:: SYN conversion - - +-----------------------------------+---------+ - | Pattern | Actions | - +===+======+==========+=============+=========+ - | 0 | ETH | ``spec`` | unset | QUEUE | - | | +----------+-------------+ | - | | | ``last`` | unset | | - | | +----------+-------------+ | - | | | ``mask`` | unset | | - +---+------+----------+-------------+---------+ - | 1 | IPV4 | ``spec`` | unset | END | - | | +----------+-------------+ | - | | | ``mask`` | unset | | - | | +----------+-------------+ | - | | | ``mask`` | unset | | - +---+------+----------+---------+---+ | - | 2 | TCP | ``spec`` | ``syn`` | 1 | | - | | +----------+---------+---+ | - | | | ``mask`` | ``syn`` | 1 | | - +---+------+----------+---------+---+ | - | 3 | END | | - +---+-------------------------------+---------+ - -``NTUPLE`` to ``IPV4``, ``TCP``, ``UDP`` → ``QUEUE`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*NTUPLE* is similar to specifying an empty L2, `Item: IPV4`_ as L3 with -`Item: TCP`_ or `Item: UDP`_ as L4 and a terminating `Action: QUEUE`_. - -A priority level can be specified as well. - -.. _table_rte_flow_migration_ntuple: - -.. table:: NTUPLE conversion - - +-----------------------------+---------+ - | Pattern | Actions | - +===+======+==========+=======+=========+ - | 0 | ETH | ``spec`` | unset | QUEUE | - | | +----------+-------+ | - | | | ``last`` | unset | | - | | +----------+-------+ | - | | | ``mask`` | unset | | - +---+------+----------+-------+---------+ - | 1 | IPV4 | ``spec`` | any | END | - | | +----------+-------+ | - | | | ``last`` | unset | | - | | +----------+-------+ | - | | | ``mask`` | any | | - +---+------+----------+-------+ | - | 2 | TCP, | ``spec`` | any | | - | | UDP +----------+-------+ | - | | | ``last`` | unset | | - | | +----------+-------+ | - | | | ``mask`` | any | | - +---+------+----------+-------+ | - | 3 | END | | - +---+-------------------------+---------+ - -``TUNNEL`` to ``ETH``, ``IPV4``, ``IPV6``, ``VXLAN`` (or other) → ``QUEUE`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*TUNNEL* matches common IPv4 and IPv6 L3/L4-based tunnel types. - -In the following table, `Item: ANY`_ is used to cover the optional L4. - -.. _table_rte_flow_migration_tunnel: - -.. table:: TUNNEL conversion - - +-------------------------------------------------------+---------+ - | Pattern | Actions | - +===+==========================+==========+=============+=========+ - | 0 | ETH | ``spec`` | any | QUEUE | - | | +----------+-------------+ | - | | | ``last`` | unset | | - | | +----------+-------------+ | - | | | ``mask`` | any | | - +---+--------------------------+----------+-------------+---------+ - | 1 | IPV4, IPV6 | ``spec`` | any | END | - | | +----------+-------------+ | - | | | ``last`` | unset | | - | | +----------+-------------+ | - | | | ``mask`` | any | | - +---+--------------------------+----------+-------------+ | - | 2 | ANY | ``spec`` | any | | - | | +----------+-------------+ | - | | | ``last`` | unset | | - | | +----------+---------+---+ | - | | | ``mask`` | ``num`` | 0 | | - +---+--------------------------+----------+---------+---+ | - | 3 | VXLAN, GENEVE, TEREDO, | ``spec`` | any | | - | | NVGRE, GRE, ... +----------+-------------+ | - | | | ``last`` | unset | | - | | +----------+-------------+ | - | | | ``mask`` | any | | - +---+--------------------------+----------+-------------+ | - | 4 | END | | - +---+---------------------------------------------------+---------+ - -``FDIR`` to most item types → ``QUEUE``, ``DROP``, ``PASSTHRU`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -*FDIR* is more complex than any other type, there are several methods to -emulate its functionality. It is summarized for the most part in the table -below. - -A few features are intentionally not supported: - -- The ability to configure the matching input set and masks for the entire - device, PMDs should take care of it automatically according to the - requested flow rules. - - For example if a device supports only one bit-mask per protocol type, - source/address IPv4 bit-masks can be made immutable by the first created - rule. Subsequent IPv4 or TCPv4 rules can only be created if they are - compatible. - - Note that only protocol bit-masks affected by existing flow rules are - immutable, others can be changed later. They become mutable again after - the related flow rules are destroyed. - -- Returning four or eight bytes of matched data when using flex bytes - filtering. Although a specific action could implement it, it conflicts - with the much more useful 32 bits tagging on devices that support it. - -- Side effects on RSS processing of the entire device. Flow rules that - conflict with the current device configuration should not be - allowed. Similarly, device configuration should not be allowed when it - affects existing flow rules. - -- Device modes of operation. "none" is unsupported since filtering cannot be - disabled as long as a flow rule is present. - -- "MAC VLAN" or "tunnel" perfect matching modes should be automatically set - according to the created flow rules. - -- Signature mode of operation is not defined but could be handled through - "FUZZY" item. - -.. _table_rte_flow_migration_fdir: - -.. table:: FDIR conversion - - +----------------------------------------+-----------------------+ - | Pattern | Actions | - +===+===================+==========+=====+=======================+ - | 0 | ETH, RAW | ``spec`` | any | QUEUE, DROP, PASSTHRU | - | | +----------+-----+ | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-------------------+----------+-----+-----------------------+ - | 1 | IPV4, IPv6 | ``spec`` | any | MARK | - | | +----------+-----+ | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-------------------+----------+-----+-----------------------+ - | 2 | TCP, UDP, SCTP | ``spec`` | any | END | - | | +----------+-----+ | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-------------------+----------+-----+ | - | 3 | VF, PF, FUZZY | ``spec`` | any | | - | | (optional) +----------+-----+ | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | any | | - +---+-------------------+----------+-----+ | - | 4 | END | | - +---+------------------------------------+-----------------------+ - -``HASH`` -~~~~~~~~ - -There is no counterpart to this filter type because it translates to a -global device setting instead of a pattern item. Device settings are -automatically set according to the created flow rules. - -``L2_TUNNEL`` to ``VOID`` → ``VXLAN`` (or others) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -All packets are matched. This type alters incoming packets to encapsulate -them in a chosen tunnel type, optionally redirect them to a VF as well. - -The destination pool for tag based forwarding can be emulated with other -flow rules using `Action: DUP`_. - -.. _table_rte_flow_migration_l2tunnel: - -.. table:: L2_TUNNEL conversion - - +---------------------------+--------------------+ - | Pattern | Actions | - +===+======+==========+=====+====================+ - | 0 | VOID | ``spec`` | N/A | VXLAN, GENEVE, ... | - | | | | | | - | | | | | | - | | +----------+-----+ | - | | | ``last`` | N/A | | - | | +----------+-----+ | - | | | ``mask`` | N/A | | - | | | | | | - +---+------+----------+-----+--------------------+ - | 1 | END | VF (optional) | - +---+ +--------------------+ - | 2 | | END | - +---+-----------------------+--------------------+ +.. _OpenFlow Switch Specification: https://www.opennetworking.org/software-defined-standards/specifications/ diff --git a/doc/guides/prog_guide/source_org.rst b/doc/guides/prog_guide/source_org.rst index a8f5832b..cee4ce6b 100644 --- a/doc/guides/prog_guide/source_org.rst +++ b/doc/guides/prog_guide/source_org.rst @@ -40,13 +40,13 @@ The lib directory contains:: +-- librte_cmdline # Command line interface helper +-- librte_distributor # Packet distributor +-- librte_eal # Environment abstraction layer - +-- librte_ether # Generic interface to poll mode driver + +-- librte_ethdev # Generic interface to poll mode driver +-- librte_hash # Hash library +-- librte_ip_frag # IP fragmentation library +-- librte_kni # Kernel NIC interface +-- librte_kvargs # Argument parsing library +-- librte_lpm # Longest prefix match library - +-- librte_mbuf # Packet and control mbuf manipulation + +-- librte_mbuf # Packet buffer manipulation +-- librte_mempool # Memory pool manager (fixed sized objects) +-- librte_meter # QoS metering library +-- librte_net # Various IP-related headers diff --git a/doc/guides/prog_guide/switch_representation.rst b/doc/guides/prog_guide/switch_representation.rst new file mode 100644 index 00000000..f5ee516f --- /dev/null +++ b/doc/guides/prog_guide/switch_representation.rst @@ -0,0 +1,837 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 6WIND S.A. + +.. _switch_representation: + +Switch Representation within DPDK Applications +============================================== + +.. contents:: :local: + +Introduction +------------ + +Network adapters with multiple physical ports and/or SR-IOV capabilities +usually support the offload of traffic steering rules between their virtual +functions (VFs), physical functions (PFs) and ports. + +Like for standard Ethernet switches, this involves a combination of +automatic MAC learning and manual configuration. For most purposes it is +managed by the host system and fully transparent to users and applications. + +On the other hand, applications typically found on hypervisors that process +layer 2 (L2) traffic (such as OVS) need to steer traffic themselves +according on their own criteria. + +Without a standard software interface to manage traffic steering rules +between VFs, PFs and the various physical ports of a given device, +applications cannot take advantage of these offloads; software processing is +mandatory even for traffic which ends up re-injected into the device it +originates from. + +This document describes how such steering rules can be configured through +the DPDK flow API (**rte_flow**), with emphasis on the SR-IOV use case +(PF/VF steering) using a single physical port for clarity, however the same +logic applies to any number of ports without necessarily involving SR-IOV. + +Port Representors +----------------- + +In many cases, traffic steering rules cannot be determined in advance; +applications usually have to process a bit of traffic in software before +thinking about offloading specific flows to hardware. + +Applications therefore need the ability to receive and inject traffic to +various device endpoints (other VFs, PFs or physical ports) before +connecting them together. Device drivers must provide means to hook the +"other end" of these endpoints and to refer them when configuring flow +rules. + +This role is left to so-called "port representors" (also known as "VF +representors" in the specific context of VFs), which are to DPDK what the +Ethernet switch device driver model (**switchdev**) [1]_ is to Linux, and +which can be thought as a software "patch panel" front-end for applications. + +- DPDK port representors are implemented as additional virtual Ethernet + device (**ethdev**) instances, spawned on an as needed basis through + configuration parameters passed to the driver of the underlying + device using devargs. + +:: + + -w pci:dbdf,representor=0 + -w pci:dbdf,representor=[0-3] + -w pci:dbdf,representor=[0,5-11] + +- As virtual devices, they may be more limited than their physical + counterparts, for instance by exposing only a subset of device + configuration callbacks and/or by not necessarily having Rx/Tx capability. + +- Among other things, they can be used to assign MAC addresses to the + resource they represent. + +- Applications can tell port representors apart from other physical of virtual + port by checking the dev_flags field within their device information + structure for the RTE_ETH_DEV_REPRESENTOR bit-field. + +.. code-block:: c + + struct rte_eth_dev_info { + ... + uint32_t dev_flags; /**< Device flags */ + ... + }; + +- The device or group relationship of ports can be discovered using the + switch ``domain_id`` field within the devices switch information structure. By + default the switch ``domain_id`` of a port will be + ``RTE_ETH_DEV_SWITCH_DOMAIN_ID_INVALID`` to indicate that the port doesn't + support the concept of a switch domain, but ports which do support the concept + will be allocated a unique switch ``domain_id``, ports within the same switch + domain will share the same ``domain_id``. The switch ``port_id`` is used to + specify the port_id in terms of the switch, so in the case of SR-IOV devices + the switch ``port_id`` would represent the virtual function identifier of the + port. + +.. code-block:: c + + /** + * Ethernet device associated switch information + */ + struct rte_eth_switch_info { + const char *name; /**< switch name */ + uint16_t domain_id; /**< switch domain id */ + uint16_t port_id; /**< switch port id */ + }; + + +.. [1] `Ethernet switch device driver model (switchdev) + <https://www.kernel.org/doc/Documentation/networking/switchdev.txt>`_ + +Basic SR-IOV +------------ + +"Basic" in the sense that it is not managed by applications, which +nonetheless expect traffic to flow between the various endpoints and the +outside as if everything was linked by an Ethernet hub. + +The following diagram pictures a setup involving a device with one PF, two +VFs and one shared physical port + +:: + + .-------------. .-------------. .-------------. + | hypervisor | | VM 1 | | VM 2 | + | application | | application | | application | + `--+----------' `----------+--' `--+----------' + | | | + .-----+-----. | | + | port_id 3 | | | + `-----+-----' | | + | | | + .-+--. .---+--. .--+---. + | PF | | VF 1 | | VF 2 | + `-+--' `---+--' `--+---' + | | | + `---------. .-----------------------' | + | | .-------------------------' + | | | + .--+-----+-----+--. + | interconnection | + `--------+--------' + | + .----+-----. + | physical | + | port 0 | + `----------' + +- A DPDK application running on the hypervisor owns the PF device, which is + arbitrarily assigned port index 3. + +- Both VFs are assigned to VMs and used by unknown applications; they may be + DPDK-based or anything else. + +- Interconnection is not necessarily done through a true Ethernet switch and + may not even exist as a separate entity. The role of this block is to show + that something brings PF, VFs and physical ports together and enables + communication between them, with a number of built-in restrictions. + +Subsequent sections in this document describe means for DPDK applications +running on the hypervisor to freely assign specific flows between PF, VFs +and physical ports based on traffic properties, by managing this +interconnection. + +Controlled SR-IOV +----------------- + +Initialization +~~~~~~~~~~~~~~ + +When a DPDK application gets assigned a PF device and is deliberately not +started in `basic SR-IOV`_ mode, any traffic coming from physical ports is +received by PF according to default rules, while VFs remain isolated. + +:: + + .-------------. .-------------. .-------------. + | hypervisor | | VM 1 | | VM 2 | + | application | | application | | application | + `--+----------' `----------+--' `--+----------' + | | | + .-----+-----. | | + | port_id 3 | | | + `-----+-----' | | + | | | + .-+--. .---+--. .--+---. + | PF | | VF 1 | | VF 2 | + `-+--' `------' `------' + | + `-----. + | + .--+----------------------. + | managed interconnection | + `------------+------------' + | + .----+-----. + | physical | + | port 0 | + `----------' + +In this mode, interconnection must be configured by the application to +enable VF communication, for instance by explicitly directing traffic with a +given destination MAC address to VF 1 and allowing that with the same source +MAC address to come out of it. + +For this to work, hypervisor applications need a way to refer to either VF 1 +or VF 2 in addition to the PF. This is addressed by `VF representors`_. + +VF Representors +~~~~~~~~~~~~~~~ + +VF representors are virtual but standard DPDK network devices (albeit with +limited capabilities) created by PMDs when managing a PF device. + +Since they represent VF instances used by other applications, configuring +them (e.g. assigning a MAC address or setting up promiscuous mode) affects +interconnection accordingly. If supported, they may also be used as two-way +communication ports with VFs (assuming **switchdev** topology) + + +:: + + .-------------. .-------------. .-------------. + | hypervisor | | VM 1 | | VM 2 | + | application | | application | | application | + `--+---+---+--' `----------+--' `--+----------' + | | | | | + | | `-------------------. | | + | `---------. | | | + | | | | | + .-----+-----. .-----+-----. .-----+-----. | | + | port_id 3 | | port_id 4 | | port_id 5 | | | + `-----+-----' `-----+-----' `-----+-----' | | + | | | | | + .-+--. .-----+-----. .-----+-----. .---+--. .--+---. + | PF | | VF 1 rep. | | VF 2 rep. | | VF 1 | | VF 2 | + `-+--' `-----+-----' `-----+-----' `---+--' `--+---' + | | | | | + | | .---------' | | + `-----. | | .-----------------' | + | | | | .---------------------' + | | | | | + .--+-------+---+---+---+--. + | managed interconnection | + `------------+------------' + | + .----+-----. + | physical | + | port 0 | + `----------' + +- VF representors are assigned arbitrary port indices 4 and 5 in the + hypervisor application and are respectively associated with VF 1 and VF 2. + +- They can't be dissociated; even if VF 1 and VF 2 were not connected, + representors could still be used for configuration. + +- In this context, port index 3 can be thought as a representor for physical + port 0. + +As previously described, the "interconnection" block represents a logical +concept. Interconnection occurs when hardware configuration enables traffic +flows from one place to another (e.g. physical port 0 to VF 1) according to +some criteria. + +This is discussed in more detail in `traffic steering`_. + +Traffic Steering +~~~~~~~~~~~~~~~~ + +In the following diagram, each meaningful traffic origin or endpoint as seen +by the hypervisor application is tagged with a unique letter from A to F. + +:: + + .-------------. .-------------. .-------------. + | hypervisor | | VM 1 | | VM 2 | + | application | | application | | application | + `--+---+---+--' `----------+--' `--+----------' + | | | | | + | | `-------------------. | | + | `---------. | | | + | | | | | + .----(A)----. .----(B)----. .----(C)----. | | + | port_id 3 | | port_id 4 | | port_id 5 | | | + `-----+-----' `-----+-----' `-----+-----' | | + | | | | | + .-+--. .-----+-----. .-----+-----. .---+--. .--+---. + | PF | | VF 1 rep. | | VF 2 rep. | | VF 1 | | VF 2 | + `-+--' `-----+-----' `-----+-----' `--(D)-' `-(E)--' + | | | | | + | | .---------' | | + `-----. | | .-----------------' | + | | | | .---------------------' + | | | | | + .--+-------+---+---+---+--. + | managed interconnection | + `------------+------------' + | + .---(F)----. + | physical | + | port 0 | + `----------' + +- **A**: PF device. +- **B**: port representor for VF 1. +- **C**: port representor for VF 2. +- **D**: VF 1 proper. +- **E**: VF 2 proper. +- **F**: physical port. + +Although uncommon, some devices do not enforce a one to one mapping between +PF and physical ports. For instance, by default all ports of **mlx4** +adapters are available to all their PF/VF instances, in which case +additional ports appear next to **F** in the above diagram. + +Assuming no interconnection is provided by default in this mode, setting up +a `basic SR-IOV`_ configuration involving physical port 0 could be broken +down as: + +PF: + +- **A to F**: let everything through. +- **F to A**: PF MAC as destination. + +VF 1: + +- **A to D**, **E to D** and **F to D**: VF 1 MAC as destination. +- **D to A**: VF 1 MAC as source and PF MAC as destination. +- **D to E**: VF 1 MAC as source and VF 2 MAC as destination. +- **D to F**: VF 1 MAC as source. + +VF 2: + +- **A to E**, **D to E** and **F to E**: VF 2 MAC as destination. +- **E to A**: VF 2 MAC as source and PF MAC as destination. +- **E to D**: VF 2 MAC as source and VF 1 MAC as destination. +- **E to F**: VF 2 MAC as source. + +Devices may additionally support advanced matching criteria such as +IPv4/IPv6 addresses or TCP/UDP ports. + +The combination of matching criteria with target endpoints fits well with +**rte_flow** [6]_, which expresses flow rules as combinations of patterns +and actions. + +Enhancing **rte_flow** with the ability to make flow rules match and target +these endpoints provides a standard interface to manage their +interconnection without introducing new concepts and whole new API to +implement them. This is described in `flow API (rte_flow)`_. + +.. [6] `Generic flow API (rte_flow) + <http://dpdk.org/doc/guides/prog_guide/rte_flow.html>`_ + +Flow API (rte_flow) +------------------- + +Extensions +~~~~~~~~~~ + +Compared to creating a brand new dedicated interface, **rte_flow** was +deemed flexible enough to manage representor traffic only with minor +extensions: + +- Using physical ports, PF, VF or port representors as targets. + +- Affecting traffic that is not necessarily addressed to the DPDK port ID a + flow rule is associated with (e.g. forcing VF traffic redirection to PF). + +For advanced uses: + +- Rule-based packet counters. + +- The ability to combine several identical actions for traffic duplication + (e.g. VF representor in addition to a physical port). + +- Dedicated actions for traffic encapsulation / decapsulation before + reaching an endpoint. + +Traffic Direction +~~~~~~~~~~~~~~~~~ + +From an application standpoint, "ingress" and "egress" flow rule attributes +apply to the DPDK port ID they are associated with. They select a traffic +direction for matching patterns, but have no impact on actions. + +When matching traffic coming from or going to a different place than the +immediate port ID a flow rule is associated with, these attributes keep +their meaning while applying to the chosen origin, as highlighted by the +following diagram + +:: + + .-------------. .-------------. .-------------. + | hypervisor | | VM 1 | | VM 2 | + | application | | application | | application | + `--+---+---+--' `----------+--' `--+----------' + | | | | | + | | `-------------------. | | + | `---------. | | | + | ^ | ^ | ^ | | + | | ingress | | ingress | | ingress | | + | | egress | | egress | | egress | | + | v | v | v | | + .----(A)----. .----(B)----. .----(C)----. | | + | port_id 3 | | port_id 4 | | port_id 5 | | | + `-----+-----' `-----+-----' `-----+-----' | | + | | | | | + .-+--. .-----+-----. .-----+-----. .---+--. .--+---. + | PF | | VF 1 rep. | | VF 2 rep. | | VF 1 | | VF 2 | + `-+--' `-----+-----' `-----+-----' `--(D)-' `-(E)--' + | | | ^ | | ^ + | | | egress | | | | egress + | | | ingress | | | | ingress + | | .---------' v | | v + `-----. | | .-----------------' | + | | | | .---------------------' + | | | | | + .--+-------+---+---+---+--. + | managed interconnection | + `------------+------------' + ^ | + ingress | | + egress | | + v | + .---(F)----. + | physical | + | port 0 | + `----------' + +Ingress and egress are defined as relative to the application creating the +flow rule. + +For instance, matching traffic sent by VM 2 would be done through an ingress +flow rule on VF 2 (**E**). Likewise for incoming traffic on physical port +(**F**). This also applies to **C** and **A** respectively. + +Transferring Traffic +~~~~~~~~~~~~~~~~~~~~ + +Without Port Representors +^^^^^^^^^^^^^^^^^^^^^^^^^ + +`Traffic direction`_ describes how an application could match traffic coming +from or going to a specific place reachable from a DPDK port ID. This makes +sense when the traffic in question is normally seen (i.e. sent or received) +by the application creating the flow rule (e.g. as in "redirect all traffic +coming from VF 1 to local queue 6"). + +However this does not force such traffic to take a specific route. Creating +a flow rule on **A** matching traffic coming from **D** is only meaningful +if it can be received by **A** in the first place, otherwise doing so simply +has no effect. + +A new flow rule attribute named "transfer" is necessary for that. Combining +it with "ingress" or "egress" and a specific origin requests a flow rule to +be applied at the lowest level + +:: + + ingress only : ingress + transfer + : + .-------------. .-------------. : .-------------. .-------------. + | hypervisor | | VM 1 | : | hypervisor | | VM 1 | + | application | | application | : | application | | application | + `------+------' `--+----------' : `------+------' `--+----------' + | | | traffic : | | | traffic + .----(A)----. | v : .----(A)----. | v + | port_id 3 | | : | port_id 3 | | + `-----+-----' | : `-----+-----' | + | | : | ^ | + | | : | | traffic | + .-+--. .---+--. : .-+--. .---+--. + | PF | | VF 1 | : | PF | | VF 1 | + `-+--' `--(D)-' : `-+--' `--(D)-' + | | | traffic : | ^ | | traffic + | | v : | | traffic | v + .--+-----------+--. : .--+-----------+--. + | interconnection | : | interconnection | + `--------+--------' : `--------+--------' + | | traffic : | + | v : | + .---(F)----. : .---(F)----. + | physical | : | physical | + | port 0 | : | port 0 | + `----------' : `----------' + +With "ingress" only, traffic is matched on **A** thus still goes to physical +port **F** by default + + +:: + + testpmd> flow create 3 ingress pattern vf id is 1 / end + actions queue index 6 / end + +With "ingress + transfer", traffic is matched on **D** and is therefore +successfully assigned to queue 6 on **A** + + +:: + + testpmd> flow create 3 ingress transfer pattern vf id is 1 / end + actions queue index 6 / end + + +With Port Representors +^^^^^^^^^^^^^^^^^^^^^^ + +When port representors exist, implicit flow rules with the "transfer" +attribute (described in `without port representors`_) are be assumed to +exist between them and their represented resources. These may be immutable. + +In this case, traffic is received by default through the representor and +neither the "transfer" attribute nor traffic origin in flow rule patterns +are necessary. They simply have to be created on the representor port +directly and may target a different representor as described in `PORT_ID +action`_. + +Implicit traffic flow with port representor + +:: + + .-------------. .-------------. + | hypervisor | | VM 1 | + | application | | application | + `--+-------+--' `----------+--' + | | ^ | | traffic + | | | traffic | v + | `-----. | + | | | + .----(A)----. .----(B)----. | + | port_id 3 | | port_id 4 | | + `-----+-----' `-----+-----' | + | | | + .-+--. .-----+-----. .---+--. + | PF | | VF 1 rep. | | VF 1 | + `-+--' `-----+-----' `--(D)-' + | | | + .--|-------------|-----------|--. + | | | | | + | | `-----------' | + | | <-- traffic | + `--|----------------------------' + | + .---(F)----. + | physical | + | port 0 | + `----------' + +Pattern Items And Actions +~~~~~~~~~~~~~~~~~~~~~~~~~ + +PORT Pattern Item +^^^^^^^^^^^^^^^^^ + +Matches traffic originating from (ingress) or going to (egress) a physical +port of the underlying device. + +Using this pattern item without specifying a port index matches the physical +port associated with the current DPDK port ID by default. As described in +`traffic steering`_, specifying it should be rarely needed. + +- Matches **F** in `traffic steering`_. + +PORT Action +^^^^^^^^^^^ + +Directs matching traffic to a given physical port index. + +- Targets **F** in `traffic steering`_. + +PORT_ID Pattern Item +^^^^^^^^^^^^^^^^^^^^ + +Matches traffic originating from (ingress) or going to (egress) a given DPDK +port ID. + +Normally only supported if the port ID in question is known by the +underlying PMD and related to the device the flow rule is created against. + +This must not be confused with the `PORT pattern item`_ which refers to the +physical port of a device. ``PORT_ID`` refers to a ``struct rte_eth_dev`` +object on the application side (also known as "port representor" depending +on the kind of underlying device). + +- Matches **A**, **B** or **C** in `traffic steering`_. + +PORT_ID Action +^^^^^^^^^^^^^^ + +Directs matching traffic to a given DPDK port ID. + +Same restrictions as `PORT_ID pattern item`_. + +- Targets **A**, **B** or **C** in `traffic steering`_. + +PF Pattern Item +^^^^^^^^^^^^^^^ + +Matches traffic originating from (ingress) or going to (egress) the physical +function of the current device. + +If supported, should work even if the physical function is not managed by +the application and thus not associated with a DPDK port ID. Its behavior is +otherwise similar to `PORT_ID pattern item`_ using PF port ID. + +- Matches **A** in `traffic steering`_. + +PF Action +^^^^^^^^^ + +Directs matching traffic to the physical function of the current device. + +Same restrictions as `PF pattern item`_. + +- Targets **A** in `traffic steering`_. + +VF Pattern Item +^^^^^^^^^^^^^^^ + +Matches traffic originating from (ingress) or going to (egress) a given +virtual function of the current device. + +If supported, should work even if the virtual function is not managed by +the application and thus not associated with a DPDK port ID. Its behavior is +otherwise similar to `PORT_ID pattern item`_ using VF port ID. + +Note this pattern item does not match VF representors traffic which, as +separate entities, should be addressed through their own port IDs. + +- Matches **D** or **E** in `traffic steering`_. + +VF Action +^^^^^^^^^ + +Directs matching traffic to a given virtual function of the current device. + +Same restrictions as `VF pattern item`_. + +- Targets **D** or **E** in `traffic steering`_. + +\*_ENCAP actions +^^^^^^^^^^^^^^^^ + +These actions are named according to the protocol they encapsulate traffic +with (e.g. ``VXLAN_ENCAP``) and using specific parameters (e.g. VNI for +VXLAN). + +While they modify traffic and can be used multiple times (order matters), +unlike `PORT_ID action`_ and friends, they have no impact on steering. + +As described in `actions order and repetition`_ this means they are useless +if used alone in an action list, the resulting traffic gets dropped unless +combined with either ``PASSTHRU`` or other endpoint-targeting actions. + +\*_DECAP actions +^^^^^^^^^^^^^^^^ + +They perform the reverse of `\*_ENCAP actions`_ by popping protocol headers +from traffic instead of pushing them. They can be used multiple times as +well. + +Note that using these actions on non-matching traffic results in undefined +behavior. It is recommended to match the protocol headers to decapsulate on +the pattern side of a flow rule in order to use these actions or otherwise +make sure only matching traffic goes through. + +Actions Order and Repetition +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Flow rules are currently restricted to at most a single action of each +supported type, performed in an unpredictable order (or all at once). To +repeat actions in a predictable fashion, applications have to make rules +pass-through and use priority levels. + +It's now clear that PMD support for chaining multiple non-terminating flow +rules of varying priority levels is prohibitively difficult to implement +compared to simply allowing multiple identical actions performed in a +defined order by a single flow rule. + +- This change is required to support protocol encapsulation offloads and the + ability to perform them multiple times (e.g. VLAN then VXLAN). + +- It makes the ``DUP`` action redundant since multiple ``QUEUE`` actions can + be combined for duplication. + +- The (non-)terminating property of actions must be discarded. Instead, flow + rules themselves must be considered terminating by default (i.e. dropping + traffic if there is no specific target) unless a ``PASSTHRU`` action is + also specified. + +Switching Examples +------------------ + +This section provides practical examples based on the established testpmd +flow command syntax [2]_, in the context described in `traffic steering`_ + +:: + + .-------------. .-------------. .-------------. + | hypervisor | | VM 1 | | VM 2 | + | application | | application | | application | + `--+---+---+--' `----------+--' `--+----------' + | | | | | + | | `-------------------. | | + | `---------. | | | + | | | | | + .----(A)----. .----(B)----. .----(C)----. | | + | port_id 3 | | port_id 4 | | port_id 5 | | | + `-----+-----' `-----+-----' `-----+-----' | | + | | | | | + .-+--. .-----+-----. .-----+-----. .---+--. .--+---. + | PF | | VF 1 rep. | | VF 2 rep. | | VF 1 | | VF 2 | + `-+--' `-----+-----' `-----+-----' `--(D)-' `-(E)--' + | | | | | + | | .---------' | | + `-----. | | .-----------------' | + | | | | .---------------------' + | | | | | + .--|-------|---|---|---|--. + | | | `---|---' | + | | `-------' | + | `---------. | + `------------|------------' + | + .---(F)----. + | physical | + | port 0 | + `----------' + +By default, PF (**A**) can communicate with the physical port it is +associated with (**F**), while VF 1 (**D**) and VF 2 (**E**) are isolated +and restricted to communicate with the hypervisor application through their +respective representors (**B** and **C**) if supported. + +Examples in subsequent sections apply to hypervisor applications only and +are based on port representors **A**, **B** and **C**. + +.. [2] `Flow syntax + <http://dpdk.org/doc/guides/testpmd_app_ug/testpmd_funcs.html#flow-syntax>`_ + +Associating VF 1 with Physical Port 0 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assign all port traffic (**F**) to VF 1 (**D**) indiscriminately through +their representors + +:: + + flow create 3 ingress pattern / end actions port_id id 4 / end + flow create 4 ingress pattern / end actions port_id id 3 / end + +More practical example with MAC address restrictions + +:: + + flow create 3 ingress + pattern eth dst is {VF 1 MAC} / end + actions port_id id 4 / end + +:: + + flow create 4 ingress + pattern eth src is {VF 1 MAC} / end + actions port_id id 3 / end + + +Sharing Broadcasts +~~~~~~~~~~~~~~~~~~ + +From outside to PF and VFs + +:: + + flow create 3 ingress + pattern eth dst is ff:ff:ff:ff:ff:ff / end + actions port_id id 3 / port_id id 4 / port_id id 5 / end + +Note ``port_id id 3`` is necessary otherwise only VFs would receive matching +traffic. + +From PF to outside and VFs + +:: + + flow create 3 egress + pattern eth dst is ff:ff:ff:ff:ff:ff / end + actions port / port_id id 4 / port_id id 5 / end + +From VFs to outside and PF + +:: + + flow create 4 ingress + pattern eth dst is ff:ff:ff:ff:ff:ff src is {VF 1 MAC} / end + actions port_id id 3 / port_id id 5 / end + + flow create 5 ingress + pattern eth dst is ff:ff:ff:ff:ff:ff src is {VF 2 MAC} / end + actions port_id id 4 / port_id id 4 / end + +Similar ``33:33:*`` rules based on known MAC addresses should be added for +IPv6 traffic. + +Encapsulating VF 2 Traffic in VXLAN +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assuming pass-through flow rules are supported + +:: + + flow create 5 ingress + pattern eth / end + actions vxlan_encap vni 42 / passthru / end + +:: + + flow create 5 egress + pattern vxlan vni is 42 / end + actions vxlan_decap / passthru / end + +Here ``passthru`` is needed since as described in `actions order and +repetition`_, flow rules are otherwise terminating; if supported, a rule +without a target endpoint will drop traffic. + +Without pass-through support, ingress encapsulation on the destination +endpoint might not be supported and action list must provide one + +:: + + flow create 5 ingress + pattern eth src is {VF 2 MAC} / end + actions vxlan_encap vni 42 / port_id id 3 / end + + flow create 3 ingress + pattern vxlan vni is 42 / end + actions vxlan_decap / port_id id 5 / end diff --git a/doc/guides/prog_guide/traffic_metering_and_policing.rst b/doc/guides/prog_guide/traffic_metering_and_policing.rst index dc9bdd5b..90c781eb 100644 --- a/doc/guides/prog_guide/traffic_metering_and_policing.rst +++ b/doc/guides/prog_guide/traffic_metering_and_policing.rst @@ -28,7 +28,7 @@ The metering and policing stage typically sits on top of flow classification, which is why the MTR objects are enabled through a special "meter" action. The MTR objects are created and updated in their own name space (``rte_mtr``) -within the ``librte_ether`` library. Whether an MTR object is private to a +within the ``librte_ethdev`` library. Whether an MTR object is private to a flow or potentially shared by several flows has to be specified at its creation time. diff --git a/doc/guides/prog_guide/vhost_lib.rst b/doc/guides/prog_guide/vhost_lib.rst index 18227b6a..77af4d77 100644 --- a/doc/guides/prog_guide/vhost_lib.rst +++ b/doc/guides/prog_guide/vhost_lib.rst @@ -65,14 +65,18 @@ The following is an overview of some key Vhost API functions: * zero copy is really good for VM2VM case. For iperf between two VMs, the boost could be above 70% (when TSO is enableld). - * for VM2NIC case, the ``nb_tx_desc`` has to be small enough: <= 64 if virtio - indirect feature is not enabled and <= 128 if it is enabled. + * For zero copy in VM2NIC case, guest Tx used vring may be starved if the + PMD driver consume the mbuf but not release them timely. - This is because when dequeue zero copy is enabled, guest Tx used vring will - be updated only when corresponding mbuf is freed. Thus, the nb_tx_desc - has to be small enough so that the PMD driver will run out of available - Tx descriptors and free mbufs timely. Otherwise, guest Tx vring would be - starved. + For example, i40e driver has an optimization to maximum NIC pipeline which + postpones returning transmitted mbuf until only tx_free_threshold free + descs left. The virtio TX used ring will be starved if the formula + (num_i40e_tx_desc - num_virtio_tx_desc > tx_free_threshold) is true, since + i40e will not return back mbuf. + + A performance tip for tuning zero copy in VM2NIC case is to adjust the + frequency of mbuf free (i.e. adjust tx_free_threshold of i40e driver) to + balance consumer and producer. * Guest memory should be backended with huge pages to achieve better performance. Using 1G page size is the best. @@ -83,6 +87,11 @@ The following is an overview of some key Vhost API functions: of those segments, thus the fewer the segments, the quicker we will get the mapping. NOTE: we may speed it by using tree searching in future. + * zero copy can not work when using vfio-pci with iommu mode currently, this + is because we don't setup iommu dma mapping for guest memory. If you have + to use vfio-pci driver, please insert vfio-pci kernel module in noiommu + mode. + - ``RTE_VHOST_USER_IOMMU_SUPPORT`` IOMMU support will be enabled when this flag is set. It is disabled by @@ -160,6 +169,31 @@ The following is an overview of some key Vhost API functions: Receives (dequeues) ``count`` packets from guest, and stored them at ``pkts``. +* ``rte_vhost_crypto_create(vid, cryptodev_id, sess_mempool, socket_id)`` + + As an extension of new_device(), this function adds virtio-crypto workload + acceleration capability to the device. All crypto workload is processed by + DPDK cryptodev with the device ID of ``cryptodev_id``. + +* ``rte_vhost_crypto_free(vid)`` + + Frees the memory and vhost-user message handlers created in + rte_vhost_crypto_create(). + +* ``rte_vhost_crypto_fetch_requests(vid, queue_id, ops, nb_ops)`` + + Receives (dequeues) ``nb_ops`` virtio-crypto requests from guest, parses + them to DPDK Crypto Operations, and fills the ``ops`` with parsing results. + +* ``rte_vhost_crypto_finalize_requests(queue_id, ops, nb_ops)`` + + After the ``ops`` are dequeued from Cryptodev, finalizes the jobs and + notifies the guest(s). + +* ``rte_vhost_crypto_set_zero_copy(vid, option)`` + + Enable or disable zero copy feature of the vhost crypto backend. + Vhost-user Implementations -------------------------- @@ -214,8 +248,88 @@ the vhost device from the data plane. When the socket connection is closed, vhost will destroy the device. +Guest memory requirement +------------------------ + +* Memory pre-allocation + + For non-zerocopy, guest memory pre-allocation is not a must. This can help + save of memory. If users really want the guest memory to be pre-allocated + (e.g., for performance reason), we can add option ``-mem-prealloc`` when + starting QEMU. Or, we can lock all memory at vhost side which will force + memory to be allocated when mmap at vhost side; option --mlockall in + ovs-dpdk is an example in hand. + + For zerocopy, we force the VM memory to be pre-allocated at vhost lib when + mapping the guest memory; and also we need to lock the memory to prevent + pages being swapped out to disk. + +* Memory sharing + + Make sure ``share=on`` QEMU option is given. vhost-user will not work with + a QEMU version without shared memory mapping. + Vhost supported vSwitch reference --------------------------------- For more vhost details and how to support vhost in vSwitch, please refer to the vhost example in the DPDK Sample Applications Guide. + +Vhost data path acceleration (vDPA) +----------------------------------- + +vDPA supports selective datapath in vhost-user lib by enabling virtio ring +compatible devices to serve virtio driver directly for datapath acceleration. + +``rte_vhost_driver_attach_vdpa_device`` is used to configure the vhost device +with accelerated backend. + +Also vhost device capabilities are made configurable to adopt various devices. +Such capabilities include supported features, protocol features, queue number. + +Finally, a set of device ops is defined for device specific operations: + +* ``get_queue_num`` + + Called to get supported queue number of the device. + +* ``get_features`` + + Called to get supported features of the device. + +* ``get_protocol_features`` + + Called to get supported protocol features of the device. + +* ``dev_conf`` + + Called to configure the actual device when the virtio device becomes ready. + +* ``dev_close`` + + Called to close the actual device when the virtio device is stopped. + +* ``set_vring_state`` + + Called to change the state of the vring in the actual device when vring state + changes. + +* ``set_features`` + + Called to set the negotiated features to device. + +* ``migration_done`` + + Called to allow the device to response to RARP sending. + +* ``get_vfio_group_fd`` + + Called to get the VFIO group fd of the device. + +* ``get_vfio_device_fd`` + + Called to get the VFIO device fd of the device. + +* ``get_notify_area`` + + Called to get the notify area info of the queue. diff --git a/doc/guides/rawdevs/dpaa2_cmdif.rst b/doc/guides/rawdevs/dpaa2_cmdif.rst new file mode 100644 index 00000000..20a60993 --- /dev/null +++ b/doc/guides/rawdevs/dpaa2_cmdif.rst @@ -0,0 +1,144 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 NXP + +NXP DPAA2 CMDIF Driver +====================== + +The DPAA2 CMDIF is an implementation of the rawdev API, that provides +communication between the GPP and AIOP (Firmware). This is achieved +via using the DPCI devices exposed by MC for GPP <--> AIOP interaction. + +More information can be found at `NXP Official Website +<http://www.nxp.com/products/microcontrollers-and-processors/arm-processors/qoriq-arm-processors:QORIQ-ARM>`_. + +Features +-------- + +The DPAA2 CMDIF implements following features in the rawdev API; + +- Getting the object ID of the device (DPCI) using attributes +- I/O to and from the AIOP device using DPCI + +Supported DPAA2 SoCs +-------------------- + +- LS2084A/LS2044A +- LS2088A/LS2048A +- LS1088A/LS1048A + +Prerequisites +------------- + +There are three main pre-requisities for executing DPAA2 CMDIF on a DPAA2 +compatible board: + +1. **ARM 64 Tool Chain** + + For example, the `*aarch64* Linaro Toolchain <https://releases.linaro.org/components/toolchain/binaries/6.3-2017.02/aarch64-linux-gnu>`_. + +2. **Linux Kernel** + + It can be obtained from `NXP's Github hosting <https://github.com/qoriq-open-source/linux>`_. + +3. **Rootfile system** + + Any *aarch64* supporting filesystem can be used. For example, + Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained + from `here <http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_. + +As an alternative method, DPAA2 CMDIF can also be executed using images provided +as part of SDK from NXP. The SDK includes all the above prerequisites necessary +to bring up a DPAA2 board. + +The following dependencies are not part of DPDK and must be installed +separately: + +- **NXP Linux SDK** + + NXP Linux software development kit (SDK) includes support for family + of QorIQ® ARM-Architecture-based system on chip (SoC) processors + and corresponding boards. + + It includes the Linux board support packages (BSPs) for NXP SoCs, + a fully operational tool chain, kernel and board specific modules. + + SDK and related information can be obtained from: `NXP QorIQ SDK <http://www.nxp.com/products/software-and-tools/run-time-software/linux-sdk/linux-sdk-for-qoriq-processors:SDKLINUX>`_. + +- **DPDK Extra Scripts** + + DPAA2 based resources can be configured easily with the help of ready scripts + as provided in the DPDK Extra repository. + + `DPDK Extras Scripts <https://github.com/qoriq-open-source/dpdk-extras>`_. + +Currently supported by DPDK: + +- NXP SDK **2.0+**. +- MC Firmware version **10.0.0** and higher. +- Supported architectures: **arm64 LE**. + +- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment. + +.. note:: + + Some part of fslmc bus code (mc flib - object library) routines are + dual licensed (BSD & GPLv2). + +Pre-Installation Configuration +------------------------------ + +Config File Options +~~~~~~~~~~~~~~~~~~~ + +The following options can be modified in the ``config`` file. + +- ``CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV`` (default ``y``) + + Toggle compilation of the ``lrte_pmd_dpaa2_cmdif`` driver. + +Enabling logs +------------- + +For enabling logs, use the following EAL parameter: + +.. code-block:: console + + ./your_cmdif_application <EAL args> --log-level=pmd.raw.dpaa2.cmdif,<level> + +Using ``pmd.raw.dpaa2.cmdif`` as log matching criteria, all Event PMD logs can be +enabled which are lower than logging ``level``. + +Driver Compilation +~~~~~~~~~~~~~~~~~~ + +To compile the DPAA2 CMDIF PMD for Linux arm64 gcc target, run the +following ``make`` command: + +.. code-block:: console + + cd <DPDK-source-directory> + make config T=arm64-dpaa2-linuxapp-gcc install + +Initialization +-------------- + +The DPAA2 CMDIF is exposed as a vdev device which consists of dpci devices. +On EAL initialization, dpci devices will be probed and then vdev device +can be created from the application code by + +* Invoking ``rte_vdev_init("dpaa2_dpci")`` from the application + +* Using ``--vdev="dpaa2_dpci"`` in the EAL options, which will call + rte_vdev_init() internally + +Example: + +.. code-block:: console + + ./your_cmdif_application <EAL args> --vdev="dpaa2_dpci" + +Platform Requirement +~~~~~~~~~~~~~~~~~~~~ + +DPAA2 drivers for DPDK can only work on NXP SoCs as listed in the +``Supported DPAA2 SoCs``. diff --git a/doc/guides/rawdevs/dpaa2_qdma.rst b/doc/guides/rawdevs/dpaa2_qdma.rst new file mode 100644 index 00000000..b9bc4ec6 --- /dev/null +++ b/doc/guides/rawdevs/dpaa2_qdma.rst @@ -0,0 +1,140 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 NXP + +NXP DPAA2 QDMA Driver +===================== + +The DPAA2 QDMA is an implementation of the rawdev API, that provide means +to initiate a DMA transaction from CPU. The initiated DMA is performed +without CPU being involved in the actual DMA transaction. This is achieved +via using the DPDMAI device exposed by MC. + +More information can be found at `NXP Official Website +<http://www.nxp.com/products/microcontrollers-and-processors/arm-processors/qoriq-arm-processors:QORIQ-ARM>`_. + +Features +-------- + +The DPAA2 QDMA implements following features in the rawdev API; + +- Supports issuing DMA of data within memory without hogging CPU while + performing DMA operation. +- Supports configuring to optionally get status of the DMA translation on + per DMA operation basis. + +Supported DPAA2 SoCs +-------------------- + +- LS2084A/LS2044A +- LS2088A/LS2048A +- LS1088A/LS1048A + +Prerequisites +------------- + +There are three main pre-requisities for executing DPAA2 QDMA on a DPAA2 +compatible board: + +1. **ARM 64 Tool Chain** + + For example, the `*aarch64* Linaro Toolchain <https://releases.linaro.org/components/toolchain/binaries/6.3-2017.02/aarch64-linux-gnu>`_. + +2. **Linux Kernel** + + It can be obtained from `NXP's Github hosting <https://github.com/qoriq-open-source/linux>`_. + +3. **Rootfile system** + + Any *aarch64* supporting filesystem can be used. For example, + Ubuntu 15.10 (Wily) or 16.04 LTS (Xenial) userland which can be obtained + from `here <http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.1-base-arm64.tar.gz>`_. + +As an alternative method, DPAA2 QDMA can also be executed using images provided +as part of SDK from NXP. The SDK includes all the above prerequisites necessary +to bring up a DPAA2 board. + +The following dependencies are not part of DPDK and must be installed +separately: + +- **NXP Linux SDK** + + NXP Linux software development kit (SDK) includes support for family + of QorIQ® ARM-Architecture-based system on chip (SoC) processors + and corresponding boards. + + It includes the Linux board support packages (BSPs) for NXP SoCs, + a fully operational tool chain, kernel and board specific modules. + + SDK and related information can be obtained from: `NXP QorIQ SDK <http://www.nxp.com/products/software-and-tools/run-time-software/linux-sdk/linux-sdk-for-qoriq-processors:SDKLINUX>`_. + +- **DPDK Extra Scripts** + + DPAA2 based resources can be configured easily with the help of ready scripts + as provided in the DPDK Extra repository. + + `DPDK Extras Scripts <https://github.com/qoriq-open-source/dpdk-extras>`_. + +Currently supported by DPDK: + +- NXP LSDK **17.12+**. +- MC Firmware version **10.3.0** and higher. +- Supported architectures: **arm64 LE**. + +- Follow the DPDK :ref:`Getting Started Guide for Linux <linux_gsg>` to setup the basic DPDK environment. + +.. note:: + + Some part of fslmc bus code (mc flib - object library) routines are + dual licensed (BSD & GPLv2). + +Pre-Installation Configuration +------------------------------ + +Config File Options +~~~~~~~~~~~~~~~~~~~ + +The following options can be modified in the ``config`` file. + +- ``CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV`` (default ``y``) + + Toggle compilation of the ``lrte_pmd_dpaa2_qdma`` driver. + +Enabling logs +------------- + +For enabling logs, use the following EAL parameter: + +.. code-block:: console + + ./your_qdma_application <EAL args> --log-level=pmd.raw.dpaa2.qdma,<level> + +Using ``pmd.raw.dpaa2.qdma`` as log matching criteria, all Event PMD logs can be +enabled which are lower than logging ``level``. + +Driver Compilation +~~~~~~~~~~~~~~~~~~ + +To compile the DPAA2 QDMA PMD for Linux arm64 gcc target, run the +following ``make`` command: + +.. code-block:: console + + cd <DPDK-source-directory> + make config T=arm64-dpaa2-linuxapp-gcc install + +Initialization +-------------- + +The DPAA2 QDMA is exposed as a vdev device which consists of dpdmai devices. +On EAL initialization, dpdmai devices will be probed and populated into the +rawdevices. The rawdev ID of the device can be obtained using + +* Invoking ``rte_rawdev_get_dev_id("dpdmai.x")`` from the application + where x is the object ID of the DPDMAI object created by MC. Use can + use this index for further rawdev function calls. + +Platform Requirement +~~~~~~~~~~~~~~~~~~~~ + +DPAA2 drivers for DPDK can only work on NXP SoCs as listed in the +``Supported DPAA2 SoCs``. diff --git a/doc/guides/rawdevs/ifpga_rawdev.rst b/doc/guides/rawdevs/ifpga_rawdev.rst new file mode 100644 index 00000000..d400db6e --- /dev/null +++ b/doc/guides/rawdevs/ifpga_rawdev.rst @@ -0,0 +1,112 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2018 Intel Corporation. + +IFPGA Rawdev Driver +====================== + +FPGA is used more and more widely in Cloud and NFV, one primary reason is +that FPGA not only provides ASIC performance but also it's more flexible +than ASIC. + +FPGA uses Partial Reconfigure (PR) Parts of Bit Stream to achieve its +flexibility. That means one FPGA Device Bit Stream is divided into many Parts +of Bit Stream(each Part of Bit Stream is defined as AFU-Accelerated Function +Unit), and each AFU is a hardware acceleration unit which can be dynamically +reloaded respectively. + +By PR (Partial Reconfiguration) AFUs, one FPGA resources can be time-shared by +different users. FPGA hot upgrade and fault tolerance can be provided easily. + +The SW IFPGA Rawdev Driver (**ifpga_rawdev**) provides a Rawdev driver +that utilizes Intel FPGA Software Stack OPAE(Open Programmable Acceleration +Engine) for FPGA management. + +Implementation details +---------------------- + +Each instance of IFPGA Rawdev Driver is probed by Intel FpgaDev. In coordination +with OPAE share code IFPGA Rawdev Driver provides common FPGA management ops +for FPGA operation, OPAE provides all following operations: +- FPGA PR (Partial Reconfiguration) management +- FPGA AFUs Identifying +- FPGA Thermal Management +- FPGA Power Management +- FPGA Performance reporting +- FPGA Remote Debug + +All configuration parameters are taken by vdev_ifpga_cfg driver. Besides +configuration, vdev_ifpga_cfg driver also hot plugs in IFPGA Bus. + +All of the AFUs of one FPGA may share same PCI BDF and AFUs scan depend on +IFPGA Rawdev Driver so IFPGA Bus takes AFU device scan and AFU drivers probe. +All AFU device driver bind to AFU device by its UUID (Universally Unique +Identifier). + +To avoid unnecessary code duplication and ensure maximum performance, +handling of AFU devices is left to different PMDs; all the design as +summarized by the following block diagram:: + + +---------------------------------------------------------------+ + | Application(s) | + +----------------------------.----------------------------------+ + | + | + +----------------------------'----------------------------------+ + | DPDK Framework (APIs) | + +----------|------------|--------.---------------------|--------+ + / \ | + / \ | + +-------'-------+ +-------'-------+ +--------'--------+ + | Eth PMD | | Crypto PMD | | | + +-------.-------+ +-------.-------+ | | + | | | | + | | | | + +-------'-------+ +-------'-------+ | IFPGA | + | Eth AFU Dev | |Crypto AFU Dev | | Rawdev Driver | + +-------.-------+ +-------.-------+ |(OPAE Share Code)| + | | | | + | | Rawdev | | + +-------'------------------'-------+ Ops | | + | IFPGA Bus | -------->| | + +-----------------.----------------+ +--------.--------+ + | | + Hot-plugin -->| | + | | + +-----------------'------------------+ +--------'--------+ + | vdev_ifpga_cfg driver | | Intel FpgaDev | + +------------------------------------+ +-----------------+ + +Build options +------------- + +- ``CONFIG_RTE_LIBRTE_IFPGA_BUS`` (default ``y``) + + Toggle compilation of IFPGA Bus library. + +- ``CONFIG_RTE_LIBRTE_IFPGA_RAWDEV`` (default ``y``) + + Toggle compilation of the ``ifpga_rawdev`` driver. + +Run-time parameters +------------------- + +This driver is invoked automatically in systems added with Intel FPGA, +but PR and IFPGA Bus scan is trigged by command line using +``--vdev 'ifpga_rawdev_cfg`` EAL option. + +The following device parameters are supported: + +- ``ifpga`` [string] + + Provide a specific Intel FPGA device PCI BDF. Can be provided multiple + times for additional instances. + +- ``port`` [int] + + Each FPGA can provide many channels to PR AFU by software, each channels + is identified by this parameter. + +- ``afu_bts`` [string] + + If null, the AFU Bit Stream has been PR in FPGA, if not forces PR and + identifies AFU Bit Stream file. diff --git a/doc/guides/rawdevs/index.rst b/doc/guides/rawdevs/index.rst new file mode 100644 index 00000000..7c3bd958 --- /dev/null +++ b/doc/guides/rawdevs/index.rst @@ -0,0 +1,16 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 NXP + +Rawdev Drivers +============== + +The following are a list of raw device PMDs, which can be used from an +application through rawdev API. + +.. toctree:: + :maxdepth: 2 + :numbered: + + dpaa2_cmdif + dpaa2_qdma + ifpga_rawdev diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst index 9b557615..e2dbee31 100644 --- a/doc/guides/rel_notes/deprecation.rst +++ b/doc/guides/rel_notes/deprecation.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + ABI and API Deprecation ======================= @@ -8,23 +11,31 @@ API and ABI deprecation notices are to be posted here. Deprecation Notices ------------------- -* eal: both declaring and identifying devices will be streamlined in v18.05. +* eal: certain structures will change in EAL on account of upcoming external + memory support. Aside from internal changes leading to an ABI break, the + following externally visible changes will also be implemented: + + - ``rte_memseg_list`` will change to include a boolean flag indicating + whether a particular memseg list is externally allocated. This will have + implications for any users of memseg-walk-related functions, as they will + now have to skip externally allocated segments in most cases if the intent + is to only iterate over internal DPDK memory. + - ``socket_id`` parameter across the entire DPDK will gain additional meaning, + as some socket ID's will now be representing externally allocated memory. No + changes will be required for existing code as backwards compatibility will + be kept, and those who do not use this feature will not see these extra + socket ID's. + +* eal: both declaring and identifying devices will be streamlined in v18.11. New functions will appear to query a specific port from buses, classes of device and device drivers. Device declaration will be made coherent with the new scheme of device identification. As such, ``rte_devargs`` device representation will change. - - removal of ``name`` and ``args`` fields. - The enum ``rte_devtype`` was used to identify a bus and will disappear. - - The ``rte_devargs_list`` will be made private. - Functions previously deprecated will change or disappear: - + ``rte_eal_devargs_add`` + ``rte_eal_devargs_type_count`` - + ``rte_eal_parse_devargs_str``, replaced by ``rte_eal_devargs_parse`` - + ``rte_eal_devargs_parse`` will change its format and use. - + all ``rte_devargs`` related functions will be renamed, changing the - ``rte_eal_devargs_`` prefix to ``rte_devargs_``. * pci: Several exposed functions are misnamed. The following functions are deprecated starting from v17.11 and are replaced: @@ -33,75 +44,23 @@ Deprecation Notices - ``eal_parse_pci_DomBDF`` replaced by ``rte_pci_addr_parse`` - ``rte_eal_compare_pci_addr`` replaced by ``rte_pci_addr_cmp`` -* eal: The semantics of the return value for the ``rte_lcore_has_role`` function - are planned to change in v18.05. The function currently returns 0 and <0 for - success and failure, respectively. This will change to 1 and 0 for true and - false, respectively, to make use of the function more intuitive. - -* eal: new ``numa_node_count`` member will be added to ``rte_config`` structure - in v18.05. - -* eal: due to internal data layout reorganization, there will be changes to - several structures and functions as a result of coming changes to support - memory hotplug in v18.05. - ``rte_eal_get_physmem_layout`` will be deprecated and removed in subsequent - releases. - ``rte_mem_config`` contents will change due to switch to memseg lists. - ``rte_memzone`` member ``memseg_id`` will no longer serve any useful purpose - and will be removed. - -* eal: a new set of mbuf mempool ops name APIs for user, platform and best - mempool names have been defined in ``rte_mbuf`` in v18.02. The uses of - ``rte_eal_mbuf_default_mempool_ops`` shall be replaced by - ``rte_mbuf_best_mempool_ops``. - The following function is now redundant and it is target to be deprecated - in 18.05: - - - ``rte_eal_mbuf_default_mempool_ops`` - -* mempool: several API and ABI changes are planned in v18.05. - The following functions, introduced for Xen, which is not supported - anymore since v17.11, are hard to use, not used anywhere else in DPDK. - Therefore they will be deprecated in v18.05 and removed in v18.08: - - - ``rte_mempool_xmem_create`` - - ``rte_mempool_xmem_size`` - - ``rte_mempool_xmem_usage`` - - The following changes are planned: - - - removal of ``get_capabilities`` mempool ops and related flags. - - substitute ``register_memory_area`` with ``populate`` ops. - - addition of new ops to customize required memory chunk calculation, - customize objects population and allocate contiguous - block of objects if underlying driver supports it. - -* mbuf: The control mbuf API will be removed in v18.05. The impacted - functions and macros are: - - - ``rte_ctrlmbuf_init()`` - - ``rte_ctrlmbuf_alloc()`` - - ``rte_ctrlmbuf_free()`` - - ``rte_ctrlmbuf_data()`` - - ``rte_ctrlmbuf_len()`` - - ``rte_is_ctrlmbuf()`` - - ``CTRL_MBUF_FLAG`` - - The packet mbuf API should be used as a replacement. - * mbuf: The opaque ``mbuf->hash.sched`` field will be updated to support generic definition in line with the ethdev TM and MTR APIs. Currently, this field is defined in librte_sched in a non-generic way. The new generic format will contain: queue ID, traffic class, color. Field size will not change. -* ethdev: a new Tx and Rx offload API was introduced on 17.11. - In the new API, offloads are divided into per-port and per-queue offloads. - Offloads are disabled by default and enabled per application request. - The old offloads API is target to be deprecated on 18.05. This includes: +* mbuf: the macro ``RTE_MBUF_INDIRECT()`` will be removed in v18.08 or later and + replaced with ``RTE_MBUF_CLONED()`` which is already added in v18.05. As + ``EXT_ATTACHED_MBUF`` is newly introduced in v18.05, ``RTE_MBUF_INDIRECT()`` + can no longer be mutually exclusive with ``RTE_MBUF_DIRECT()`` if the new + experimental API ``rte_pktmbuf_attach_extbuf()`` is used. Removal of the macro + is to fix this semantic inconsistency. - - removal of ``ETH_TXQ_FLAGS_NO*`` flags. - - removal of ``txq_flags`` field from ``rte_eth_txconf`` struct. - - removal of the offloads bit-field from ``rte_eth_rxmode`` struct. +* ethdev: In v18.11 ``DEV_RX_OFFLOAD_CRC_STRIP`` offload flag will be removed, default + behavior without any flag will be changed to CRC strip. + To keep CRC ``DEV_RX_OFFLOAD_KEEP_CRC`` flag is required. + ``KEEP_CRC``: Keep CRC in packet + No flag: Strip CRC from packet * ethdev: the legacy filter API, including ``rte_eth_dev_filter_supported()``, ``rte_eth_dev_filter_ctrl()`` as well @@ -111,65 +70,28 @@ Deprecation Notices Target release for removal of the legacy API will be defined once most PMDs have switched to rte_flow. -* ethdev: A new rss level field planned in 18.05. - The new API add rss_level field to ``rte_eth_rss_conf`` to enable a choice - of RSS hash calculation on outer or inner header of tunneled packet. - -* ethdev: Currently, if the rte_eth_rx_burst() function returns a value less - than *nb_pkts*, the application will assume that no more packets are present. - Some of the hw queue based hardware can only support smaller burst for RX - and TX and thus break the expectation of the rx_burst API. Similar is the - case for TX burst as well as ring sizes. ``rte_eth_dev_info`` will be added - with following new parameters so as to support semantics for drivers to - define a preferred size for Rx/Tx burst and rings. - - - Member ``struct preferred_size`` would be added to enclose all preferred - size to be fetched from driver/implementation. - - Members ``uint16_t rx_burst``, ``uint16_t tx_burst``, ``uint16_t rx_ring``, - and ``uint16_t tx_ring`` would be added to ``struct preferred_size``. - -* ethdev: A work is being planned for 18.05 to expose VF port representors - as a mean to perform control and data path operation on the different VFs. - As VF representor is an ethdev port, new fields are needed in order to map - between the VF representor and the VF or the parent PF. Those new fields - are to be included in ``rte_eth_dev_info`` struct. - -* ethdev: The prototype and the behavior of - ``dev_ops->eth_mac_addr_set()`` will change in v18.05. A return code - will be added to notify the caller if an error occurred in the PMD. In - ``rte_eth_dev_default_mac_addr_set()``, the new default MAC address - will be copied in ``dev->data->mac_addrs[0]`` only if the operation is - successful. This modification will only impact the PMDs, not the - applications. - -* ethdev: functions add rx/tx callback will return named opaque type - ``rte_eth_add_rx_callback()``, ``rte_eth_add_first_rx_callback()`` and - ``rte_eth_add_tx_callback()`` functions currently return callback object as - ``void \*`` but APIs to delete callbacks get ``struct rte_eth_rxtx_callback \*`` - as parameter. For consistency functions adding callback will return - ``struct rte_eth_rxtx_callback \*`` instead of ``void \*``. - -* ethdev: The size of variables ``flow_types_mask`` in - ``rte_eth_fdir_info structure``, ``sym_hash_enable_mask`` and - ``valid_bit_mask`` in ``rte_eth_hash_global_conf`` structure - will be increased from 32 to 64 bits to fulfill hardware requirements. - This change will break existing ABI as size of the structures will increase. - -* ethdev: ``rte_eth_dev_get_sec_ctx()`` fix port id storage - ``rte_eth_dev_get_sec_ctx()`` is using ``uint8_t`` for ``port_id``, - which should be ``uint16_t``. - -* i40e: The default flexible payload configuration which extracts the first 16 - bytes of the payload for RSS will be deprecated starting from 18.02. If - required the previous behavior can be configured using existing flow - director APIs. There is no ABI/API break. This change will just remove a - global configuration setting and require explicit configuration. - -* librte_meter: The API will change to accommodate configuration profiles. - Most of the API functions will have an additional opaque parameter. - -* ring: The alignment constraints on the ring structure will be relaxed - to one cache line instead of two, and an empty cache line padding will - be added between the producer and consumer structures. The size of the - structure and the offset of the fields will remain the same on - platforms with 64B cache line, but will change on other platforms. +* ethdev: In v18.11 ``rte_eth_dev_attach()`` and ``rte_eth_dev_detach()`` + will be removed. + Hotplug functions ``rte_eal_hotplug_add()`` and ``rte_eal_hotplug_remove()`` + should be used instread. + Function ``rte_eth_dev_get_port_by_name()`` may be used to find + identifier of the added port. + +* eal: In v18.11 ``rte_eal_dev_attach()`` and ``rte_eal_dev_detach()`` + will be removed. + Hotplug functions ``rte_eal_hotplug_add()`` and ``rte_eal_hotplug_remove()`` + should be used directly. + +* pdump: As we changed to use generic IPC, some changes in APIs and structure + are expected in subsequent release. + + - ``rte_pdump_set_socket_dir`` will be removed; + - The parameter, ``path``, of ``rte_pdump_init`` will be removed; + - The enum ``rte_pdump_socktype`` will be removed. + +* ethdev: flow API function ``rte_flow_copy()`` will be deprecated in v18.11 + in favor of ``rte_flow_conv()`` (which will appear in that version) and + subsequently removed for v19.02. + + This is due to a lack of flexibility and reliance on a type unusable with + C++ programs (struct rte_flow_desc). diff --git a/doc/guides/rel_notes/index.rst b/doc/guides/rel_notes/index.rst index d8b5dfeb..d125342c 100644 --- a/doc/guides/rel_notes/index.rst +++ b/doc/guides/rel_notes/index.rst @@ -9,6 +9,8 @@ Release Notes :numbered: rel_description + release_18_08 + release_18_05 release_18_02 release_17_11 release_17_08 diff --git a/doc/guides/rel_notes/known_issues.rst b/doc/guides/rel_notes/known_issues.rst index afcc2c44..95e4ce69 100644 --- a/doc/guides/rel_notes/known_issues.rst +++ b/doc/guides/rel_notes/known_issues.rst @@ -714,3 +714,48 @@ igb_uio can not be used when running l3fwd-power **Driver/Module**: ``igb_uio`` module. + + +Linux kernel 4.10.0 iommu attribute read error +---------------------------------------------- + +**Description**: + When VT-d is enabled (``iommu=pt intel_iommu=on``), reading IOMMU attributes from + /sys/devices/virtual/iommu/dmarXXX/intel-iommu/cap on Linux kernel 4.10.0 error. + This bug is fixed in `Linux commmit a7fdb6e648fb + <https://patchwork.kernel.org/patch/9595727/>`_, + This bug is introduced in `Linux commmit 39ab9555c241 + <https://patchwork.kernel.org/patch/9554403/>`_, + +**Implication**: + When binding devices to VFIO and attempting to run testpmd application, + testpmd (and other DPDK applications) will not initialize. + +**Resolution/Workaround**: + Use other linux kernel version. It only happens in linux kernel 4.10.0. + +**Affected Environment/Platform**: + ALL OS of linux kernel 4.10.0. + +**Driver/Module**: + ``vfio-pci`` module. + +Netvsc driver and application restart +------------------------------------- + +**Description**: + The Linux kernel uio_hv_generic driver does not completely shutdown and clean up + resources properly if application using Netvsc PMD exits. + +**Implication**: + When application using Netvsc PMD is restarted it can not complete initialization + handshake sequence with the host. + +**Resolution/Workaround**: + Either reboot the guest or remove and reinsert the hv_uio_generic module. + +**Affected Environment/Platform**: + Linux Hyper-V. + +**Driver/Module**: + ``uio_hv_generic`` module. diff --git a/doc/guides/rel_notes/release_16_04.rst b/doc/guides/rel_notes/release_16_04.rst index d0a09eff..e9f1e6ff 100644 --- a/doc/guides/rel_notes/release_16_04.rst +++ b/doc/guides/rel_notes/release_16_04.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2016 The DPDK contributors + DPDK Release 16.04 ================== diff --git a/doc/guides/rel_notes/release_16_07.rst b/doc/guides/rel_notes/release_16_07.rst index a8a3fc11..2904aacf 100644 --- a/doc/guides/rel_notes/release_16_07.rst +++ b/doc/guides/rel_notes/release_16_07.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2016 The DPDK contributors + DPDK Release 16.07 ================== diff --git a/doc/guides/rel_notes/release_16_11.rst b/doc/guides/rel_notes/release_16_11.rst index 8c9ec65c..92e0ec69 100644 --- a/doc/guides/rel_notes/release_16_11.rst +++ b/doc/guides/rel_notes/release_16_11.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2016 The DPDK contributors + DPDK Release 16.11 ================== diff --git a/doc/guides/rel_notes/release_17_02.rst b/doc/guides/rel_notes/release_17_02.rst index 357965ac..d6c1c567 100644 --- a/doc/guides/rel_notes/release_17_02.rst +++ b/doc/guides/rel_notes/release_17_02.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 The DPDK contributors + DPDK Release 17.02 ================== diff --git a/doc/guides/rel_notes/release_17_05.rst b/doc/guides/rel_notes/release_17_05.rst index 68922840..64182406 100644 --- a/doc/guides/rel_notes/release_17_05.rst +++ b/doc/guides/rel_notes/release_17_05.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 The DPDK contributors + DPDK Release 17.05 ================== diff --git a/doc/guides/rel_notes/release_17_08.rst b/doc/guides/rel_notes/release_17_08.rst index 0bcdfb7b..dc622409 100644 --- a/doc/guides/rel_notes/release_17_08.rst +++ b/doc/guides/rel_notes/release_17_08.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 The DPDK contributors + DPDK Release 17.08 ================== diff --git a/doc/guides/rel_notes/release_17_11.rst b/doc/guides/rel_notes/release_17_11.rst index 088778bf..2a93af32 100644 --- a/doc/guides/rel_notes/release_17_11.rst +++ b/doc/guides/rel_notes/release_17_11.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 The DPDK contributors + DPDK Release 17.11 ================== @@ -56,7 +59,7 @@ New Features * **Added a new driver for Marvell Armada 7k/8k devices.** Added the new ``mrvl`` net driver for Marvell Armada 7k/8k devices. See the - :doc:`../nics/mrvl` NIC guide for more details on this new driver. + :doc:`../nics/mvpp2` NIC guide for more details on this new driver. * **Updated mlx4 driver.** @@ -184,7 +187,7 @@ New Features A new crypto PMD has been added, which provides several ciphering and hashing algorithms. All cryptography operations use the MUSDK library crypto API. - See the :doc:`../cryptodevs/mrvl` document for more details. + See the :doc:`../cryptodevs/mvsam` document for more details. * **Add new benchmarking mode to dpdk-test-crypto-perf application.** diff --git a/doc/guides/rel_notes/release_18_02.rst b/doc/guides/rel_notes/release_18_02.rst index 44b7de59..8e403118 100644 --- a/doc/guides/rel_notes/release_18_02.rst +++ b/doc/guides/rel_notes/release_18_02.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + DPDK Release 18.02 ================== diff --git a/doc/guides/rel_notes/release_18_05.rst b/doc/guides/rel_notes/release_18_05.rst new file mode 100644 index 00000000..8dc22b01 --- /dev/null +++ b/doc/guides/rel_notes/release_18_05.rst @@ -0,0 +1,983 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + +DPDK Release 18.05 +================== + +.. **Read this first.** + + The text in the sections below explains how to update the release notes. + + Use proper spelling, capitalization and punctuation in all sections. + + Variable and config names should be quoted as fixed width text: + ``LIKE_THIS``. + + Build the docs and view the output file to ensure the changes are correct:: + + make doc-guides-html + + xdg-open build/doc/html/guides/rel_notes/release_18_05.html + + +New Features +------------ + +.. This section should contain new features added in this release. Sample + format: + + * **Add a title in the past tense with a full stop.** + + Add a short 1-2 sentence description in the past tense. The description + should be enough to allow someone scanning the release notes to + understand the new feature. + + If the feature adds a lot of sub-features you can use a bullet list like + this: + + * Added feature foo to do something. + * Enhanced feature bar to do something else. + + Refer to the previous release notes for examples. + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* **Reworked memory subsystem.** + + Memory subsystem has been reworked to support new functionality. + + On Linux, support for reserving/unreserving hugepage memory at runtime has been + added, so applications no longer need to pre-reserve memory at startup. Due to + reorganized internal workings of memory subsystem, any memory allocated + through ``rte_malloc()`` or ``rte_memzone_reserve()`` is no longer guaranteed + to be IOVA-contiguous. + + This functionality has introduced the following changes: + + * ``rte_eal_get_physmem_layout()`` was removed. + * A new flag for memzone reservation (``RTE_MEMZONE_IOVA_CONTIG``) was added + to ensure reserved memory will be IOVA-contiguous, for use with device + drivers and other cases requiring such memory. + * New callbacks for memory allocation/deallocation events, allowing users (or + drivers) to be notified of new memory being allocated or deallocated + * New callbacks for validating memory allocations above a specified limit, + allowing user to permit or deny memory allocations. + * A new command-line switch ``--legacy-mem`` to enable EAL behavior similar to + how older versions of DPDK worked (memory segments that are IOVA-contiguous, + but hugepages are reserved at startup only, and can never be released). + * A new command-line switch ``--single-file-segments`` to put all memory + segments within a segment list in a single file. + * A set of convenience function calls to look up and iterate over allocated + memory segments. + * ``-m`` and ``--socket-mem`` command-line arguments now carry an additional + meaning and mark pre-reserved hugepages as "unfree-able", thereby acting as + a mechanism guaranteeing minimum availability of hugepage memory to the + application. + + Reserving/unreserving memory at runtime is not currently supported on FreeBSD. + +* **Added bucket mempool driver.** + + Added a bucket mempool driver which provides a way to allocate contiguous + block of objects. + The number of objects in the block depends on how many objects fit in the + ``RTE_DRIVER_MEMPOOL_BUCKET_SIZE_KB`` memory chunk which is a build time option. + The number may be obtained using ``rte_mempool_ops_get_info()`` API. + Contiguous blocks may be allocated using ``rte_mempool_get_contig_blocks()`` API. + +* **Added support for port representors.** + + Added DPDK port representors (also known as "VF representors" in the specific + context of VFs), which are to DPDK what the Ethernet switch device driver + model (**switchdev**) is to Linux, and which can be thought as a software + "patch panel" front-end for applications. DPDK port representors are + implemented as additional virtual Ethernet device (**ethdev**) instances, + spawned on an as-needed basis through configuration parameters passed to the + driver of the underlying device using devargs. + +* **Added support for VXLAN and NVGRE tunnel endpoint.** + + New actions types have been added to support encapsulation and decapsulation + operations for a tunnel endpoint. The new action types are + ``RTE_FLOW_ACTION_TYPE_[VXLAN/NVGRE]_ENCAP``, ``RTE_FLOW_ACTION_TYPE_[VXLAN/NVGRE]_DECAP``, + ``RTE_FLOW_ACTION_TYPE_JUMP``. A new item type ``RTE_FLOW_ACTION_TYPE_MARK`` has been + added to match a flow against a previously marked flow. A shared counter has also been + introduced to the flow API to count a group of flows. + +* **Added PMD-recommended Tx and Rx parameters.** + + Applications can now query drivers for device-tuned values of + ring sizes, burst sizes, and number of queues. + +* **Added RSS hash and key update to CXGBE PMD.** + + Added support for updating the RSS hash and key to the CXGBE PMD. + +* **Added CXGBE VF PMD.** + + CXGBE VF Poll Mode Driver has been added to run DPDK over Chelsio + T5/T6 NIC VF instances. + +* **Updated mlx5 driver.** + + Updated the mlx5 driver including the following changes: + + * Introduced Multi-packet Rx to enable 100Gb/sec with 64B frames. + * Support for being run by non-root users given a reduced set of capabilities + ``CAP_NET_ADMIN``, ``CAP_NET_RAW`` and ``CAP_IPC_LOCK``. + * Support for TSO and checksum for generic UDP and IP tunnels. + * Support for inner checksum and RSS for GRE, VXLAN-GPE, MPLSoGRE + and MPLSoUDP tunnels. + * Accommodate the new memory hotplug model. + * Support for non virtually contiguous mempools. + * Support for MAC adding along with allmulti and promiscuous modes from VF. + * Support for Mellanox BlueField SoC device. + * Support for PMD defaults for queue number and depth to improve the out + of the box performance. + +* **Updated mlx4 driver.** + + Updated the mlx4 driver including the following changes: + + * Support for to being run by non-root users given a reduced set of capabilities + ``CAP_NET_ADMIN``, ``CAP_NET_RAW`` and ``CAP_IPC_LOCK``. + * Supported CRC strip toggling. + * Accommodate the new memory hotplug model. + * Support non virtually contiguous mempools. + * Dropped support for Mellanox OFED 4.2. + +* **Updated Solarflare network PMD.** + + Updated the sfc_efx driver including the following changes: + + * Added support for Solarflare XtremeScale X2xxx family adapters. + * Added support for NVGRE, VXLAN and GENEVE filters in flow API. + * Added support for DROP action in flow API. + * Added support for equal stride super-buffer Rx mode (X2xxx only). + * Added support for MARK and FLAG actions in flow API (X2xxx only). + +* **Added Ethernet poll mode driver for AMD XGBE devices.** + + Added the new ``axgbe`` ethernet poll mode driver for AMD XGBE devices. + See the :doc:`../nics/axgbe` nic driver guide for more details on this + new driver. + +* **Updated szedata2 PMD.** + + Added support for new NFB-200G2QL card. + A new API was introduced in the libsze2 library which the szedata2 PMD depends + on, thus the new version of the library was needed. + New versions of the packages are available and the minimum required version + is 4.4.1. + +* **Added support for Broadcom NetXtreme-S (BCM58800) family of controllers (aka Stingray).** + + Added support for the Broadcom NetXtreme-S (BCM58800) family of controllers + (aka Stingray). The BCM58800 devices feature a NetXtreme E-Series advanced + network controller, a high-performance ARM CPU block, PCI Express (PCIe) + Gen3 interfaces, key accelerators for compute offload and a high-speed + memory subsystem including L3 cache and DDR4 interfaces, all interconnected + by a coherent Network-on-chip (NOC) fabric. + + The ARM CPU subsystem features eight ARMv8 Cortex-A72 CPUs at 3.0 GHz, + arranged in a multi-cluster configuration. + +* **Added vDPA in vhost-user lib.** + + Added support for selective datapath in the vhost-user lib. vDPA stands for vhost + Data Path Acceleration. It supports virtio ring compatible devices to serve + the virtio driver directly to enable datapath acceleration. + +* **Added IFCVF vDPA driver.** + + Added IFCVF vDPA driver to support Intel FPGA 100G VF devices. IFCVF works + as a HW vhost data path accelerator, it supports live migration and is + compatible with virtio 0.95 and 1.0. This driver registers the ifcvf vDPA driver + to vhost lib, when virtio connects. With the help of the registered vDPA + driver the assigned VF gets configured to Rx/Tx directly to VM's virtio + vrings. + +* **Added support for vhost dequeue interrupt mode.** + + Added support for vhost dequeue interrupt mode to release CPUs to others + when there is no data to transmit. Applications can register an epoll event + file descriptor to associate Rx queues with interrupt vectors. + +* **Added support for virtio-user server mode.** + + In a container environment if the vhost-user backend restarts, there's no way + for it to reconnect to virtio-user. To address this, support for server mode + has been added. In this mode the socket file is created by virtio-user, which the + backend connects to. This means that if the backend restarts, it can reconnect + to virtio-user and continue communications. + +* **Added crypto workload support to vhost library.** + + New APIs have been introduced in the vhost library to enable virtio crypto support + including session creation/deletion handling and translating virtio-crypto + requests into DPDK crypto operations. A sample application has also been introduced. + +* **Added virtio crypto PMD.** + + Added a new Poll Mode Driver for virtio crypto devices, which provides + AES-CBC ciphering and AES-CBC with HMAC-SHA1 algorithm-chaining. See the + :doc:`../cryptodevs/virtio` crypto driver guide for more details on + this new driver. + +* **Added AMD CCP Crypto PMD.** + + Added the new ``ccp`` crypto driver for AMD CCP devices. See the + :doc:`../cryptodevs/ccp` crypto driver guide for more details on + this new driver. + +* **Updated AESNI MB PMD.** + + The AESNI MB PMD has been updated with additional support for: + + * AES-CMAC (128-bit key). + +* **Added the Compressdev Library, a generic compression service library.** + + Added the Compressdev library which provides an API for offload of compression and + decompression operations to hardware or software accelerator devices. + +* **Added a new compression poll mode driver using Intels ISA-L.** + + Added the new ``ISA-L`` compression driver, for compression and decompression + operations in software. See the :doc:`../compressdevs/isal` compression driver + guide for details on this new driver. + +* **Added the Event Timer Adapter Library.** + + The Event Timer Adapter Library extends the event-based model by introducing + APIs that allow applications to arm/cancel event timers that generate + timer expiry events. This new type of event is scheduled by an event device + along with existing types of events. + +* **Added OcteonTx TIM Driver (Event timer adapter).** + + The OcteonTx Timer block enables software to schedule events for a future + time, it is exposed to an application via the Event timer adapter library. + + See the :doc:`../eventdevs/octeontx` guide for more details + +* **Added Event Crypto Adapter Library.** + + Added the Event Crypto Adapter Library. This library extends the + event-based model by introducing APIs that allow applications to + enqueue/dequeue crypto operations to/from cryptodev as events scheduled + by an event device. + +* **Added Ifpga Bus, a generic Intel FPGA Bus library.** + + Added the Ifpga Bus library which provides support for integrating any Intel + FPGA device with the DPDK framework. It provides Intel FPGA Partial Bit + Stream AFU (Accelerated Function Unit) scan and drivers probe. + +* **Added IFPGA (Intel FPGA) Rawdev Driver.** + + Added a new Rawdev driver called IFPGA (Intel FPGA) Rawdev Driver, which cooperates + with OPAE (Open Programmable Acceleration Engine) shared code to provide common FPGA + management ops for FPGA operation. + + See the :doc:`../rawdevs/ifpga_rawdev` programmer's guide for more details. + +* **Added DPAA2 QDMA Driver (in rawdev).** + + The DPAA2 QDMA is an implementation of the rawdev API, that provide a means + of initiating a DMA transaction from CPU. The initiated DMA is performed + without the CPU being involved in the actual DMA transaction. + + See the :doc:`../rawdevs/dpaa2_qdma` guide for more details. + +* **Added DPAA2 Command Interface Driver (in rawdev).** + + The DPAA2 CMDIF is an implementation of the rawdev API, that provides + communication between the GPP and NXP's QorIQ based AIOP Block (Firmware). + Advanced IO Processor i.e. AIOP are clusters of programmable RISC engines + optimized for flexible networking and I/O operations. The communication + between GPP and AIOP is achieved via using DPCI devices exposed by MC for + GPP <--> AIOP interaction. + + See the :doc:`../rawdevs/dpaa2_cmdif` guide for more details. + +* **Added device event monitor framework.** + + Added a general device event monitor framework to EAL, for device dynamic + management to facilitate device hotplug awareness and associated + actions. The list of new APIs is: + + * ``rte_dev_event_monitor_start`` and ``rte_dev_event_monitor_stop`` for + the event monitor enabling and disabling. + * ``rte_dev_event_callback_register`` and ``rte_dev_event_callback_unregister`` + for registering and un-registering user callbacks. + + Linux uevent is supported as a backend of this device event notification framework. + +* **Added support for procinfo and pdump on eth vdev.** + + For ethernet virtual devices (like TAP, PCAP, etc.), with this feature, we can get + stats/xstats on shared memory from a secondary process, and also pdump packets on + those virtual devices. + +* **Enhancements to the Packet Framework Library.** + + Design and development of new API functions for Packet Framework library that + implement a common set of actions such as traffic metering, packet + encapsulation, network address translation, TTL update, etc., for pipeline + table and input ports to speed up application development. The API functions + includes creating action profiles, registering actions to the profiles, + instantiating action profiles for pipeline table and input ports, etc. + +* **Added the BPF Library.** + + The BPF Library provides the ability to load and execute + Enhanced Berkeley Packet Filters (eBPF) within user-space DPDK applications. + It also introduces a basic framework to load/unload BPF-based filters + on Eth devices (right now only via SW RX/TX callbacks). + It also adds a dependency on libelf. + + +API Changes +----------- + +.. This section should contain API changes. Sample format: + + * Add a short 1-2 sentence description of the API change. Use fixed width + quotes for ``rte_function_names`` or ``rte_struct_names``. Use the past + tense. + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* service cores: No longer marked as experimental. + + The service cores functions are no longer marked as experimental, and have + become part of the normal DPDK API and ABI. Any future ABI changes will be + announced at least one release before the ABI change is made. There are no + ABI breaking changes planned. + +* eal: The ``rte_lcore_has_role()`` return value changed. + + This function now returns true or false, respectively, + rather than 0 or < 0 for success or failure. + It makes use of the function more intuitive. + +* mempool: The capability flags and related functions have been removed. + + Flags ``MEMPOOL_F_CAPA_PHYS_CONTIG`` and + ``MEMPOOL_F_CAPA_BLK_ALIGNED_OBJECTS`` were used by octeontx mempool + driver to customize generic mempool library behavior. + Now the new driver callbacks ``calc_mem_size`` and ``populate`` may be + used to achieve it without specific knowledge in the generic code. + +* mempool: The following xmem functions have been deprecated: + + - ``rte_mempool_xmem_create`` + - ``rte_mempool_xmem_size`` + - ``rte_mempool_xmem_usage`` + - ``rte_mempool_populate_iova_tab`` + +* mbuf: The control mbuf API has been removed in v18.05. The impacted + functions and macros are: + + - ``rte_ctrlmbuf_init()`` + - ``rte_ctrlmbuf_alloc()`` + - ``rte_ctrlmbuf_free()`` + - ``rte_ctrlmbuf_data()`` + - ``rte_ctrlmbuf_len()`` + - ``rte_is_ctrlmbuf()`` + - ``CTRL_MBUF_FLAG`` + + The packet mbuf API should be used as a replacement. + +* meter: API updated to accommodate configuration profiles. + + The meter API has been changed to support meter configuration profiles. The + configuration profile represents the set of configuration parameters + for a given meter object, such as the rates and sizes for the token + buckets. These configuration parameters were previously part of the meter + object internal data structure. The separation of the configuration + parameters from the meter object data structure results in reducing its + memory footprint which helps in better cache utilization when a large number + of meter objects are used. + +* ethdev: The function ``rte_eth_dev_count()``, often mis-used to iterate + over ports, is deprecated and replaced by ``rte_eth_dev_count_avail()``. + There is also a new function ``rte_eth_dev_count_total()`` to get the + total number of allocated ports, available or not. + The hotplug-proof applications should use ``RTE_ETH_FOREACH_DEV`` or + ``RTE_ETH_FOREACH_DEV_OWNED_BY`` as port iterators. + +* ethdev: In struct ``struct rte_eth_dev_info``, field ``rte_pci_device *pci_dev`` + has been replaced with field ``struct rte_device *device``. + +* ethdev: Changes to the semantics of ``rte_eth_dev_configure()`` parameters. + + If both the ``nb_rx_q`` and ``nb_tx_q`` parameters are zero, + ``rte_eth_dev_configure()`` will now use PMD-recommended queue sizes, or if + recommendations are not provided by the PMD the function will use ethdev + fall-back values. Previously setting both of the parameters to zero would + have resulted in ``-EINVAL`` being returned. + +* ethdev: Changes to the semantics of ``rte_eth_rx_queue_setup()`` parameters. + + If the ``nb_rx_desc`` parameter is zero, ``rte_eth_rx_queue_setup`` will + now use the PMD-recommended Rx ring size, or in the case where the PMD + does not provide a recommendation, will use an ethdev-provided + fall-back value. Previously, setting ``nb_rx_desc`` to zero would have + resulted in an error. + +* ethdev: Changes to the semantics of ``rte_eth_tx_queue_setup()`` parameters. + + If the ``nb_tx_desc`` parameter is zero, ``rte_eth_tx_queue_setup`` will + now use the PMD-recommended Tx ring size, or in the case where the PMD + does not provide a recommendation, will use an ethdev-provided + fall-back value. Previously, setting ``nb_tx_desc`` to zero would have + resulted in an error. + +* ethdev: Several changes were made to the flow API. + + * The unused DUP action was removed. + * Actions semantics in flow rules: list order now matters ("first + to last" instead of "all simultaneously"), repeated actions are now + all performed, and they do not individually have (non-)terminating + properties anymore. + * Flow rules are now always terminating unless a ``PASSTHRU`` action is + present. + * C99-style flexible arrays were replaced with standard pointers in RSS + action and in RAW pattern item structures due to compatibility issues. + * The RSS action was modified to not rely on external + ``struct rte_eth_rss_conf`` anymore to instead expose its own and more + appropriately named configuration fields directly + (``rss_conf->rss_key`` => ``key``, + ``rss_conf->rss_key_len`` => ``key_len``, + ``rss_conf->rss_hf`` => ``types``, + ``num`` => ``queue_num``), and the addition of missing RSS parameters + (``func`` for RSS hash function to apply and ``level`` for the + encapsulation level). + * The VLAN pattern item (``struct rte_flow_item_vlan``) was modified to + include inner EtherType instead of outer TPID. Its default mask was also + modified to cover the VID part (lower 12 bits) of TCI only. + * A new transfer attribute was added to ``struct rte_flow_attr`` in order + to clarify the behavior of some pattern items. + * PF and VF pattern items are now only accepted by PMDs that implement + them (bnxt and i40e) when the transfer attribute is also present, for + consistency. + * Pattern item PORT was renamed PHY_PORT to avoid confusion with DPDK port + IDs. + * An action counterpart to the PHY_PORT pattern item was added in order to + redirect matching traffic to a specific physical port. + * PORT_ID pattern item and actions were added to match and target DPDK + port IDs at a higher level than PHY_PORT. + * ``RTE_FLOW_ACTION_TYPE_[VXLAN/NVGRE]_ENCAP`` action items were added to support + tunnel encapsulation operation for VXLAN and NVGRE type tunnel endpoint. + * ``RTE_FLOW_ACTION_TYPE_[VXLAN/NVGRE]_DECAP`` action items were added to support + tunnel decapsulation operation for VXLAN and NVGRE type tunnel endpoint. + * ``RTE_FLOW_ACTION_TYPE_JUMP`` action item was added to support a matched flow + to be redirected to the specific group. + * ``RTE_FLOW_ACTION_TYPE_MARK`` item type has been added to match a flow against + a previously marked flow. + +* ethdev: Change flow APIs regarding count action: + + * ``rte_flow_create()`` API count action now requires the ``struct rte_flow_action_count``. + * ``rte_flow_query()`` API parameter changed from action type to action structure. + +* ethdev: Changes to offload API + + A pure per-port offloading isn't requested to be repeated in [rt]x_conf->offloads to + ``rte_eth_[rt]x_queue_setup()``. Now any offloading enabled in ``rte_eth_dev_configure()`` + can't be disabled by ``rte_eth_[rt]x_queue_setup()``. Any new added offloading which has + not been enabled in ``rte_eth_dev_configure()`` and is requested to be enabled in + ``rte_eth_[rt]x_queue_setup()`` must be per-queue type, or otherwise trigger an error log. + +* ethdev: Runtime queue setup + + ``rte_eth_rx_queue_setup`` and ``rte_eth_tx_queue_setup`` can be called after + ``rte_eth_dev_start`` if the device supports runtime queue setup. The device driver can + expose this capability through ``rte_eth_dev_info_get``. A Rx or Tx queue + set up at runtime need to be started explicitly by ``rte_eth_dev_rx_queue_start`` + or ``rte_eth_dev_tx_queue_start``. + + +ABI Changes +----------- + +.. This section should contain ABI changes. Sample format: + + * Add a short 1-2 sentence description of the ABI change that was announced + in the previous releases and made in this release. Use fixed width quotes + for ``rte_function_names`` or ``rte_struct_names``. Use the past tense. + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* ring: The alignment constraints on the ring structure has been relaxed + to one cache line instead of two, and an empty cache line padding is + added between the producer and consumer structures. The size of the + structure and the offset of the fields remains the same on platforms + with 64B cache line, but changes on other platforms. + +* mempool: Some ops have changed. + + A new callback ``calc_mem_size`` has been added to ``rte_mempool_ops`` + to allow customization of the required memory size calculation. + A new callback ``populate`` has been added to ``rte_mempool_ops`` + to allow customized object population. + Callback ``get_capabilities`` has been removed from ``rte_mempool_ops`` + since its features are covered by ``calc_mem_size`` and ``populate`` + callbacks. + Callback ``register_memory_area`` has been removed from ``rte_mempool_ops`` + since the new callback ``populate`` may be used instead of it. + +* ethdev: Additional fields in rte_eth_dev_info. + + The ``rte_eth_dev_info`` structure has had two extra entries appended to the + end of it: ``default_rxportconf`` and ``default_txportconf``. Each of these + in turn are ``rte_eth_dev_portconf`` structures containing three fields of + type ``uint16_t``: ``burst_size``, ``ring_size``, and ``nb_queues``. These + are parameter values recommended for use by the PMD. + +* ethdev: ABI for all flow API functions was updated. + + This includes functions ``rte_flow_copy``, ``rte_flow_create``, + ``rte_flow_destroy``, ``rte_flow_error_set``, ``rte_flow_flush``, + ``rte_flow_isolate``, ``rte_flow_query`` and ``rte_flow_validate``, due to + changes in error type definitions (``enum rte_flow_error_type``), removal + of the unused DUP action (``enum rte_flow_action_type``), modified + behavior for flow rule actions (see API changes), removal of C99 flexible + array from RAW pattern item (``struct rte_flow_item_raw``), complete + rework of the RSS action definition (``struct rte_flow_action_rss``), + sanity fix in the VLAN pattern item (``struct rte_flow_item_vlan``) and + new transfer attribute (``struct rte_flow_attr``). + +* bbdev: New parameter added to rte_bbdev_op_cap_turbo_dec. + + A new parameter ``max_llr_modulus`` has been added to + ``rte_bbdev_op_cap_turbo_dec`` structure to specify maximal LLR (likelihood + ratio) absolute value. + +* bbdev: Queue Groups split into UL/DL Groups. + + Queue Groups have been split into UL/DL Groups in the Turbo Software Driver. + They are independent for Decode/Encode. ``rte_bbdev_driver_info`` reflects + introduced changes. + + +Known Issues +------------ + +.. This section should contain new known issues in this release. Sample format: + + * **Add title in present tense with full stop.** + + Add a short 1-2 sentence description of the known issue in the present + tense. Add information on any known workarounds. + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* **Secondary process launch is not reliable.** + + Recent memory hotplug patches have made multiprocess startup less reliable + than it was in past releases. A number of workarounds are known to work depending + on the circumstances. As such it isn't recommended to use the secondary + process mechanism for critical systems. The underlying issues will be + addressed in upcoming releases. + + The issue is explained in more detail, including potential workarounds, + in the Bugzilla entry referenced below. + + Bugzilla entry: https://dpdk.org/tracker/show_bug.cgi?id=50 + +* **pdump is not compatible with old applications.** + + As we changed to use generic multi-process communication for pdump + negotiation instead of previous dedicated unix socket way, pdump + applications, including the dpdk-pdump example and any other applications + using ``librte_pdump``, will not work with older version DPDK primary + applications. + +* **rte_abort takes a long time on FreeBSD.** + + DPDK processes now allocates a large area of virtual memory address space. + As a result ``rte_abort`` on FreeBSD now dumps the contents of the + whole reserved memory range, not just the used portion, to a core dump file. + Writing this large core file can take a significant amount of time, causing + processes to appear to hang on the system. + + The work around for the issue is to set the system resource limits for core + dumps before running any tests, e.g. ``limit coredumpsize 0``. This will + effectively disable core dumps on FreeBSD. If they are not to be completely + disabled, a suitable limit, e.g. 1G might be specified instead of 0. This + needs to be run per-shell session, or before every test run. This change + can also be made persistent by adding ``kern.coredump=0`` to ``/etc/sysctl.conf``. + + Bugzilla entry: https://dpdk.org/tracker/show_bug.cgi?id=53 + +* **ixgbe PMD crash on hotplug detach when no VF created.** + + ixgbe PMD uninit path cause null pointer dereference because of port representor + cleanup when number of VF is zero. + + Bugzilla entry: https://dpdk.org/tracker/show_bug.cgi?id=57 + +* **Bonding PMD may fail to accept new slave ports in certain conditions.** + + In certain conditions when using testpmd, + bonding may fail to register new slave ports. + + Bugzilla entry: https://dpdk.org/tracker/show_bug.cgi?id=52. + +* **Unexpected performance regression in Vhost library.** + + Patches fixing CVE-2018-1059 were expected to introduce a small performance + drop. However, in some setups, bigger performance drops have been measured + when running micro-benchmarks. + + Bugzilla entry: https://dpdk.org/tracker/show_bug.cgi?id=48 + + +Shared Library Versions +----------------------- + +.. Update any library version updated in this release and prepend with a ``+`` + sign, like this: + + librte_acl.so.2 + + librte_cfgfile.so.2 + librte_cmdline.so.2 + + This section is a comment. Do not overwrite or remove it. + ========================================================= + + +The libraries prepended with a plus sign were incremented in this version. + +.. code-block:: diff + + librte_acl.so.2 + librte_bbdev.so.1 + librte_bitratestats.so.2 + + librte_bpf.so.1 + librte_bus_dpaa.so.1 + librte_bus_fslmc.so.1 + librte_bus_pci.so.1 + librte_bus_vdev.so.1 + librte_cfgfile.so.2 + librte_cmdline.so.2 + + librte_common_octeontx.so.1 + + librte_compressdev.so.1 + librte_cryptodev.so.4 + librte_distributor.so.1 + + librte_eal.so.7 + + librte_ethdev.so.9 + + librte_eventdev.so.4 + librte_flow_classify.so.1 + librte_gro.so.1 + librte_gso.so.1 + librte_hash.so.2 + librte_ip_frag.so.1 + librte_jobstats.so.1 + librte_kni.so.2 + librte_kvargs.so.1 + librte_latencystats.so.1 + librte_lpm.so.2 + + librte_mbuf.so.4 + + librte_mempool.so.4 + + librte_meter.so.2 + librte_metrics.so.1 + librte_net.so.1 + librte_pci.so.1 + librte_pdump.so.2 + librte_pipeline.so.3 + librte_pmd_bnxt.so.2 + librte_pmd_bond.so.2 + librte_pmd_i40e.so.2 + librte_pmd_ixgbe.so.2 + + librte_pmd_dpaa2_cmdif.so.1 + + librte_pmd_dpaa2_qdma.so.1 + librte_pmd_ring.so.2 + librte_pmd_softnic.so.1 + librte_pmd_vhost.so.2 + librte_port.so.3 + librte_power.so.1 + librte_rawdev.so.1 + librte_reorder.so.1 + + librte_ring.so.2 + librte_sched.so.1 + librte_security.so.1 + librte_table.so.3 + librte_timer.so.1 + librte_vhost.so.3 + + +Tested Platforms +---------------- + +.. This section should contain a list of platforms that were tested with this + release. + + The format is: + + * <vendor> platform with <vendor> <type of devices> combinations + + * List of CPU + * List of OS + * List of devices + * Other relevant details... + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* Intel(R) platforms with Intel(R) NICs combinations + + * CPU + + * Intel(R) Atom(TM) CPU C2758 @ 2.40GHz + * Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz + * Intel(R) Xeon(R) CPU E5-4667 v3 @ 2.00GHz + * Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz + * Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz + * Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz + * Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz + * Intel(R) Xeon(R) CPU E5-2658 v3 @ 2.20GHz + * Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz + + * OS: + + * CentOS 7.4 + * Fedora 25 + * Fedora 27 + * Fedora 28 + * FreeBSD 11.1 + * Red Hat Enterprise Linux Server release 7.3 + * SUSE Enterprise Linux 12 + * Wind River Linux 8 + * Ubuntu 14.04 + * Ubuntu 16.04 + * Ubuntu 16.10 + * Ubuntu 17.10 + + * NICs: + + * Intel(R) 82599ES 10 Gigabit Ethernet Controller + + * Firmware version: 0x61bf0001 + * Device id (pf/vf): 8086:10fb / 8086:10ed + * Driver version: 5.2.3 (ixgbe) + + * Intel(R) Corporation Ethernet Connection X552/X557-AT 10GBASE-T + + * Firmware version: 0x800003e7 + * Device id (pf/vf): 8086:15ad / 8086:15a8 + * Driver version: 4.4.6 (ixgbe) + + * Intel(R) Ethernet Converged Network Adapter X710-DA4 (4x10G) + + * Firmware version: 6.01 0x80003221 + * Device id (pf/vf): 8086:1572 / 8086:154c + * Driver version: 2.4.6 (i40e) + + * Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (4x10G) + + * Firmware version: 3.33 0x80000fd5 0.0.0 + * Device id (pf/vf): 8086:37d0 / 8086:37cd + * Driver version: 2.4.3 (i40e) + + * Intel(R) Ethernet Converged Network Adapter XXV710-DA2 (2x25G) + + * Firmware version: 6.01 0x80003221 + * Device id (pf/vf): 8086:158b / 8086:154c + * Driver version: 2.4.6 (i40e) + + * Intel(R) Ethernet Converged Network Adapter XL710-QDA2 (2X40G) + + * Firmware version: 6.01 0x8000321c + * Device id (pf/vf): 8086:1583 / 8086:154c + * Driver version: 2.4.6 (i40e) + + * Intel(R) Corporation I350 Gigabit Network Connection + + * Firmware version: 1.63, 0x80000dda + * Device id (pf/vf): 8086:1521 / 8086:1520 + * Driver version: 5.4.0-k (igb) + +* Intel(R) platforms with Mellanox(R) NICs combinations + + * CPU: + + * Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz + * Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz + * Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz + * Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz + * Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz + * Intel(R) Xeon(R) CPU E5-2640 @ 2.50GHz + * Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz + + * OS: + + * Red Hat Enterprise Linux Server release 7.5 (Maipo) + * Red Hat Enterprise Linux Server release 7.4 (Maipo) + * Red Hat Enterprise Linux Server release 7.3 (Maipo) + * Red Hat Enterprise Linux Server release 7.2 (Maipo) + * Ubuntu 18.04 + * Ubuntu 17.10 + * Ubuntu 16.10 + * Ubuntu 16.04 + * SUSE Linux Enterprise Server 15 + + * MLNX_OFED: 4.2-1.0.0.0 + * MLNX_OFED: 4.3-2.0.2.0 + + * NICs: + + * Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2x40G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1007 + * Firmware version: 2.42.5000 + + * Mellanox(R) ConnectX(R)-4 10G MCX4111A-XCAT (1x10G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 10G MCX4121A-XCAT (2x10G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 25G MCX4111A-ACAT (1x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 25G MCX4121A-ACAT (2x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 40G MCX4131A-BCAT/MCX413A-BCAT (1x40G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 40G MCX415A-BCAT (1x40G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX4131A-GCAT/MCX413A-GCAT (1x50G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX414A-BCAT (2x50G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX415A-GCAT/MCX416A-BCAT/MCX416A-GCAT (2x50G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX415A-CCAT (1x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 100G MCX416A-CCAT (2x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 Lx 10G MCX4121A-XCAT (2x10G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1015 + * Firmware version: 14.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 Lx 25G MCX4121A-ACAT (2x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1015 + * Firmware version: 14.21.1000 and above + + * Mellanox(R) ConnectX(R)-5 100G MCX556A-ECAT (2x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1017 + * Firmware version: 16.21.1000 and above + + * Mellanox(R) ConnectX-5 Ex EN 100G MCX516A-CDAT (2x100G) + + * Host interface: PCI Express 4.0 x16 + * Device ID: 15b3:1019 + * Firmware version: 16.21.1000 and above + +* ARM platforms with Mellanox(R) NICs combinations + + * CPU: + + * Qualcomm ARM 1.1 2500MHz + + * OS: + + * Red Hat Enterprise Linux Server release 7.5 (Maipo) + + * NICs: + + * Mellanox(R) ConnectX(R)-4 Lx 25G MCX4121A-ACAT (2x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1015 + * Firmware version: 14.22.0428 + + * Mellanox(R) ConnectX(R)-5 100G MCX556A-ECAT (2x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1017 + * Firmware version: 16.22.0428 + +* ARM SoC combinations from Cavium (with integrated NICs) + + * SoC: + + * Cavium CN81xx + * Cavium CN83xx + + * OS: + + * Ubuntu 16.04.2 LTS with Cavium SDK-6.2.0-Patch2 release support package. + +* ARM SoC combinations from NXP (with integrated NICs) + + * SoC: + + * NXP/Freescale QorIQ LS1046A with ARM Cortex A72 + * NXP/Freescale QorIQ LS2088A with ARM Cortex A72 + + * OS: + + * Ubuntu 16.04.3 LTS with NXP QorIQ LSDK 1803 support packages diff --git a/doc/guides/rel_notes/release_18_08.rst b/doc/guides/rel_notes/release_18_08.rst new file mode 100644 index 00000000..321fa845 --- /dev/null +++ b/doc/guides/rel_notes/release_18_08.rst @@ -0,0 +1,549 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2018 The DPDK contributors + +DPDK Release 18.08 +================== + +.. **Read this first.** + + The text in the sections below explains how to update the release notes. + + Use proper spelling, capitalization and punctuation in all sections. + + Variable and config names should be quoted as fixed width text: + ``LIKE_THIS``. + + Build the docs and view the output file to ensure the changes are correct:: + + make doc-guides-html + + xdg-open build/doc/html/guides/rel_notes/release_18_08.html + + +New Features +------------ + +.. This section should contain new features added in this release. + Sample format: + + * **Add a title in the past tense with a full stop.** + + Add a short 1-2 sentence description in the past tense. + The description should be enough to allow someone scanning + the release notes to understand the new feature. + + If the feature adds a lot of sub-features you can use a bullet list + like this: + + * Added feature foo to do something. + * Enhanced feature bar to do something else. + + Refer to the previous release notes for examples. + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* **Added support for Hyper-V netvsc PMD.** + + The new ``netvsc`` poll mode driver provides native support for + networking on Hyper-V. See the :doc:`../nics/netvsc` NIC driver guide + for more details on this new driver. + +* **Added Flow API support for CXGBE PMD.** + + Flow API support has been added to CXGBE Poll Mode Driver to offload + flows to Chelsio T5/T6 NICs. Support added for: + + * Wildcard (LE-TCAM) and Exact (HASH) match filters. + * Match items: physical ingress port, IPv4, IPv6, TCP and UDP. + * Action items: queue, drop, count, and physical egress port redirect. + +* **Added ixgbe preferred Rx/Tx parameters.** + + Rather than applications providing explicit Rx and Tx parameters such as + queue and burst sizes, they can request that the EAL instead uses preferred + values provided by the PMD, falling back to defaults within the EAL if the + PMD does not provide any. The provision of such tuned values now includes + the ixgbe PMD. + +* **Added descriptor status check support for fm10k.** + + The ``rte_eth_rx_descriptor_status`` and ``rte_eth_tx_descriptor_status`` + APIs are now supported by fm10K. + +* **Updated the enic driver.** + + * Add low cycle count Tx handler for no-offload Tx. + * Add low cycle count Rx handler for non-scattered Rx. + * Minor performance improvements to scattered Rx handler. + * Add handlers to add/delete VxLAN port number. + * Add devarg to specify ingress VLAN rewrite mode. + +* **Updated mlx5 driver.** + + Updated the mlx5 driver including the following changes: + + * Added port representors support. + * Added Flow API support for e-switch rules. + Added support for ACTION_PORT_ID, ACTION_DROP, ACTION_OF_POP_VLAN, + ACTION_OF_PUSH_VLAN, ACTION_OF_SET_VLAN_VID, ACTION_OF_SET_VLAN_PCP + and ITEM_PORT_ID. + * Added support for 32-bit compilation. + +* **Added TSO support for the mlx4 driver.** + + Added TSO support for the mlx4 drivers from MLNX_OFED_4.4 and above. + +* **SoftNIC PMD rework.** + + The SoftNIC PMD infrastructure has been restructured to use the Packet + Framework, which makes it more flexible, modular and easier to add new + functionality in the future. + +* **Updated the AESNI MB PMD.** + + The AESNI MB PMD has been updated with additional support for: + + * 3DES for 8, 16 and 24 byte keys. + +* **Added a new compression PMD using Intel's QuickAssist (QAT) device family.** + + Added the new ``QAT`` compression driver, for compression and decompression + operations in software. See the :doc:`../compressdevs/qat_comp` compression + driver guide for details on this new driver. + +* **Updated the ISA-L PMD.** + + Added support for chained mbufs (input and output). + + +API Changes +----------- + +.. This section should contain API changes. Sample format: + + * Add a short 1-2 sentence description of the API change. + Use fixed width quotes for ``function_names`` or ``struct_names``. + Use the past tense. + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* The path to the runtime config file has changed. The new path is determined + as follows: + + - If DPDK is running as root, ``/var/run/dpdk/<prefix>/config`` + - If DPDK is not running as root: + + * If ``$XDG_RUNTIME_DIR`` is set, ``${XDG_RUNTIME_DIR}/dpdk/<prefix>/config`` + * Otherwise, ``/tmp/dpdk/<prefix>/config`` + +* eal: The function ``rte_eal_mbuf_default_mempool_ops`` was deprecated + and is removed in 18.08. It shall be replaced by + ``rte_mbuf_best_mempool_ops``. + +* mempool: Following functions were deprecated and are removed in 18.08: + + - ``rte_mempool_populate_iova_tab`` + - ``rte_mempool_populate_phys_tab`` + - ``rte_mempool_populate_phys`` (``rte_mempool_populate_iova`` should be used) + - ``rte_mempool_virt2phy`` (``rte_mempool_virt2iova`` should be used) + - ``rte_mempool_xmem_create`` + - ``rte_mempool_xmem_size`` + - ``rte_mempool_xmem_usage`` + +* ethdev: The old offload API is removed: + + - Rx per-port ``rte_eth_conf.rxmode.[bit-fields]`` + - Tx per-queue ``rte_eth_txconf.txq_flags`` + - ``ETH_TXQ_FLAGS_NO*`` + + The transition bits are removed: + + - ``rte_eth_conf.rxmode.ignore_offload_bitfield`` + - ``ETH_TXQ_FLAGS_IGNORE`` + +* cryptodev: The following API changes have been made in 18.08: + + - In struct ``struct rte_cryptodev_info``, field ``rte_pci_device *pci_dev`` + has been replaced with field ``struct rte_device *device``. + - Value 0 is accepted in ``sym.max_nb_sessions``, meaning that a device + supports an unlimited number of sessions. + - Two new fields of type ``uint16_t`` have been added: + ``min_mbuf_headroom_req`` and ``min_mbuf_tailroom_req``. These parameters + specify the recommended headroom and tailroom for mbufs to be processed by + the PMD. + +* cryptodev: The following functions were deprecated and are removed in 18.08: + + - ``rte_cryptodev_queue_pair_start`` + - ``rte_cryptodev_queue_pair_stop`` + - ``rte_cryptodev_queue_pair_attach_sym_session`` + - ``rte_cryptodev_queue_pair_detach_sym_session`` + +* cryptodev: The following functions were deprecated and are replaced by other + functions in 18.08: + + - ``rte_cryptodev_get_header_session_size`` is replaced with + ``rte_cryptodev_sym_get_header_session_size`` + - ``rte_cryptodev_get_private_session_size`` is replaced with + ``rte_cryptodev_sym_get_private_session_size`` + +* cryptodev: Feature flag ``RTE_CRYPTODEV_FF_MBUF_SCATTER_GATHER`` is + replaced with the following more explicit flags: + + - ``RTE_CRYPTODEV_FF_IN_PLACE_SGL`` + - ``RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT`` + - ``RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT`` + - ``RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT`` + - ``RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT`` + +* cryptodev: Renamed cryptodev experimental APIs: + + Used user_data instead of private_data in following APIs to avoid confusion + with the existing session parameter ``sess_private_data[]`` and related APIs. + + - ``rte_cryptodev_sym_session_set_private_data()`` changed to + ``rte_cryptodev_sym_session_set_user_data()`` + - ``rte_cryptodev_sym_session_get_private_data()`` changed to + ``rte_cryptodev_sym_session_get_user_data()`` + +* compressdev: Feature flag ``RTE_COMP_FF_MBUF_SCATTER_GATHER`` is + replaced with the following more explicit flags: + + - ``RTE_COMP_FF_OOP_SGL_IN_SGL_OUT`` + - ``RTE_COMP_FF_OOP_SGL_IN_LB_OUT`` + - ``RTE_COMP_FF_OOP_LB_IN_SGL_OUT`` + + +Shared Library Versions +----------------------- + +.. Update any library version updated in this release + and prepend with a ``+`` sign, like this: + + librte_acl.so.2 + + librte_cfgfile.so.2 + librte_cmdline.so.2 + + This section is a comment. Do not overwrite or remove it. + ========================================================= + +The libraries prepended with a plus sign were incremented in this version. + +.. code-block:: diff + + librte_acl.so.2 + librte_bbdev.so.1 + librte_bitratestats.so.2 + librte_bpf.so.1 + librte_bus_dpaa.so.1 + librte_bus_fslmc.so.1 + librte_bus_pci.so.1 + librte_bus_vdev.so.1 + + librte_bus_vmbus.so.1 + librte_cfgfile.so.2 + librte_cmdline.so.2 + librte_common_octeontx.so.1 + librte_compressdev.so.1 + + librte_cryptodev.so.5 + librte_distributor.so.1 + + librte_eal.so.8 + + librte_ethdev.so.10 + librte_eventdev.so.4 + librte_flow_classify.so.1 + librte_gro.so.1 + librte_gso.so.1 + librte_hash.so.2 + librte_ip_frag.so.1 + librte_jobstats.so.1 + librte_kni.so.2 + librte_kvargs.so.1 + librte_latencystats.so.1 + librte_lpm.so.2 + librte_mbuf.so.4 + + librte_mempool.so.5 + librte_meter.so.2 + librte_metrics.so.1 + librte_net.so.1 + librte_pci.so.1 + librte_pdump.so.2 + librte_pipeline.so.3 + librte_pmd_bnxt.so.2 + librte_pmd_bond.so.2 + librte_pmd_i40e.so.2 + librte_pmd_ixgbe.so.2 + librte_pmd_dpaa2_cmdif.so.1 + librte_pmd_dpaa2_qdma.so.1 + librte_pmd_ring.so.2 + librte_pmd_softnic.so.1 + librte_pmd_vhost.so.2 + librte_port.so.3 + librte_power.so.1 + librte_rawdev.so.1 + librte_reorder.so.1 + librte_ring.so.2 + librte_sched.so.1 + librte_security.so.1 + librte_table.so.3 + librte_timer.so.1 + librte_vhost.so.3 + + +Tested Platforms +---------------- + +.. This section should contain a list of platforms that were tested + with this release. + + The format is: + + * <vendor> platform with <vendor> <type of devices> combinations + + * List of CPU + * List of OS + * List of devices + * Other relevant details... + + This section is a comment. Do not overwrite or remove it. + Also, make sure to start the actual text at the margin. + ========================================================= + +* Intel(R) platforms with Intel(R) NICs combinations + + * CPU + + * Intel(R) Atom(TM) CPU C3858 @ 2.00GHz + * Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz + * Intel(R) Xeon(R) CPU E5-4667 v3 @ 2.00GHz + * Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz + * Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz + * Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz + * Intel(R) Xeon(R) CPU E5-2658 v2 @ 2.40GHz + * Intel(R) Xeon(R) CPU E5-2658 v3 @ 2.20GHz + * Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz + + * OS: + + * CentOS 7.4 + * Fedora 25 + * Fedora 27 + * Fedora 28 + * FreeBSD 11.1 + * Red Hat Enterprise Linux Server release 7.5 + * SUSE Enterprise Linux 12 + * Wind River Linux 8 + * Ubuntu 14.04 + * Ubuntu 16.04 + * Ubuntu 16.10 + * Ubuntu 17.10 + * Ubuntu 18.04 + + * NICs: + + * Intel(R) 82599ES 10 Gigabit Ethernet Controller + + * Firmware version: 0x61bf0001 + * Device id (pf/vf): 8086:10fb / 8086:10ed + * Driver version: 5.2.3 (ixgbe) + + * Intel(R) Corporation Ethernet Connection X552/X557-AT 10GBASE-T + + * Firmware version: 0x800003e7 + * Device id (pf/vf): 8086:15ad / 8086:15a8 + * Driver version: 4.4.6 (ixgbe) + + * Intel(R) Ethernet Converged Network Adapter X710-DA4 (4x10G) + + * Firmware version: 6.01 0x80003221 + * Device id (pf/vf): 8086:1572 / 8086:154c + * Driver version: 2.4.6 (i40e) + + * Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (4x10G) + + * Firmware version: 3.33 0x80000fd5 0.0.0 + * Device id (pf/vf): 8086:37d0 / 8086:37cd + * Driver version: 2.4.3 (i40e) + + * Intel(R) Ethernet Converged Network Adapter XXV710-DA2 (2x25G) + + * Firmware version: 6.01 0x80003221 + * Device id (pf/vf): 8086:158b / 8086:154c + * Driver version: 2.4.6 (i40e) + + * Intel(R) Ethernet Converged Network Adapter XL710-QDA2 (2X40G) + + * Firmware version: 6.01 0x8000321c + * Device id (pf/vf): 8086:1583 / 8086:154c + * Driver version: 2.4.6 (i40e) + + * Intel(R) Corporation I350 Gigabit Network Connection + + * Firmware version: 1.63, 0x80000dda + * Device id (pf/vf): 8086:1521 / 8086:1520 + * Driver version: 5.4.0-k (igb) + +* Intel(R) platforms with Mellanox(R) NICs combinations + + * CPU: + + * Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz + * Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz + * Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz + * Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz + * Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz + * Intel(R) Xeon(R) CPU E5-2640 @ 2.50GHz + * Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz + + * OS: + + * Red Hat Enterprise Linux Server release 7.5 (Maipo) + * Red Hat Enterprise Linux Server release 7.4 (Maipo) + * Red Hat Enterprise Linux Server release 7.3 (Maipo) + * Red Hat Enterprise Linux Server release 7.2 (Maipo) + * Ubuntu 18.04 + * Ubuntu 17.10 + * Ubuntu 16.04 + * SUSE Linux Enterprise Server 15 + + * MLNX_OFED: 4.3-2.0.2.0 + * MLNX_OFED: 4.4-2.0.1.0 + + * NICs: + + * Mellanox(R) ConnectX(R)-3 Pro 40G MCX354A-FCC_Ax (2x40G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1007 + * Firmware version: 2.42.5000 + + * Mellanox(R) ConnectX(R)-4 10G MCX4111A-XCAT (1x10G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 10G MCX4121A-XCAT (2x10G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 25G MCX4111A-ACAT (1x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 25G MCX4121A-ACAT (2x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 40G MCX4131A-BCAT/MCX413A-BCAT (1x40G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 40G MCX415A-BCAT (1x40G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX4131A-GCAT/MCX413A-GCAT (1x50G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX414A-BCAT (2x50G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX415A-GCAT/MCX416A-BCAT/MCX416A-GCAT (2x50G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 50G MCX415A-CCAT (1x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 100G MCX416A-CCAT (2x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1013 + * Firmware version: 12.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 Lx 10G MCX4121A-XCAT (2x10G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1015 + * Firmware version: 14.21.1000 and above + + * Mellanox(R) ConnectX(R)-4 Lx 25G MCX4121A-ACAT (2x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1015 + * Firmware version: 14.21.1000 and above + + * Mellanox(R) ConnectX(R)-5 100G MCX556A-ECAT (2x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1017 + * Firmware version: 16.21.1000 and above + + * Mellanox(R) ConnectX-5 Ex EN 100G MCX516A-CDAT (2x100G) + + * Host interface: PCI Express 4.0 x16 + * Device ID: 15b3:1019 + * Firmware version: 16.21.1000 and above + +* ARM platforms with Mellanox(R) NICs combinations + + * CPU: + + * Qualcomm ARM 1.1 2500MHz + + * OS: + + * Red Hat Enterprise Linux Server release 7.5 (Maipo) + + * NICs: + + * Mellanox(R) ConnectX(R)-4 Lx 25G MCX4121A-ACAT (2x25G) + + * Host interface: PCI Express 3.0 x8 + * Device ID: 15b3:1015 + * Firmware version: 14.23.1000 + + * Mellanox(R) ConnectX(R)-5 100G MCX556A-ECAT (2x100G) + + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:1017 + * Firmware version: 16.23.1000 + +* Mellanox BlueField SmartNIC + + * Mellanox(R) BlueField SmartNIC MT416842 (2x25G) + * Host interface: PCI Express 3.0 x16 + * Device ID: 15b3:a2d2 + * Firmware version: 18.99.3950 + + * SoC ARM cores running OS: + * CentOS Linux release 7.4.1708 (AltArch) + * Mellanox MLNX_OFED 4.2-1.4.21.0 + + * DPDK application running on ARM cores inside SmartNIC + * Bluefield representors support planned for next release. diff --git a/doc/guides/rel_notes/release_2_2.rst b/doc/guides/rel_notes/release_2_2.rst index bb7d15a6..cea5c874 100644 --- a/doc/guides/rel_notes/release_2_2.rst +++ b/doc/guides/rel_notes/release_2_2.rst @@ -1,3 +1,6 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2016 The DPDK contributors + DPDK Release 2.2 ================ diff --git a/doc/guides/sample_app_ug/bbdev_app.rst b/doc/guides/sample_app_ug/bbdev_app.rst index f17125da..653f972f 100644 --- a/doc/guides/sample_app_ug/bbdev_app.rst +++ b/doc/guides/sample_app_ug/bbdev_app.rst @@ -31,8 +31,8 @@ Limitations Compiling the Application ------------------------- -#. DPDK needs to be built with ``turbo_sw`` PMD driver enabled along with - ``FLEXRAN SDK`` Libraries. Refer to *SW Turbo Poll Mode Driver* +#. DPDK needs to be built with ``baseband_turbo_sw`` PMD driver enabled along + with ``FLEXRAN SDK`` Libraries. Refer to *SW Turbo Poll Mode Driver* documentation for more details on this. #. Go to the example directory: @@ -84,13 +84,14 @@ issue the command: .. code-block:: console - $ ./build/bbdev --vdev='turbo_sw' -w <NIC0PCIADDR> -c 0x38 --socket-mem=2,2 \ + $ ./build/bbdev --vdev='baseband_turbo_sw' -w <NIC0PCIADDR> -c 0x38 --socket-mem=2,2 \ --file-prefix=bbdev -- -e 0x10 -d 0x20 where, NIC0PCIADDR is the PCI addresse of the Rx port -This command creates one virtual bbdev devices ``turbo_sw`` where the device -gets linked to a corresponding ethernet port as whitelisted by the parameter -w. +This command creates one virtual bbdev devices ``baseband_turbo_sw`` where the +device gets linked to a corresponding ethernet port as whitelisted by +the parameter -w. 3 cores are allocated to the application, and assigned as: - core 3 is the master and used to print the stats live on screen, diff --git a/doc/guides/sample_app_ug/ethtool.rst b/doc/guides/sample_app_ug/ethtool.rst index 1b79aca1..47e09f6e 100644 --- a/doc/guides/sample_app_ug/ethtool.rst +++ b/doc/guides/sample_app_ug/ethtool.rst @@ -44,6 +44,7 @@ they do as as follows: * ``drvinfo``: Print driver info * ``eeprom``: Dump EEPROM to file +* ``module-eeprom``: Dump plugin module EEPROM to file * ``link``: Print port link states * ``macaddr``: Gets/sets MAC address * ``mtu``: Set NIC MTU @@ -97,6 +98,8 @@ the following functions: - ``rte_ethtool_get_eeprom_len()`` - ``rte_ethtool_get_eeprom()`` - ``rte_ethtool_set_eeprom()`` +- ``rte_ethtool_get_module_info()`` +- ``rte_ethtool_get_module_eeprom()`` - ``rte_ethtool_get_pauseparam()`` - ``rte_ethtool_set_pauseparam()`` - ``rte_ethtool_net_open()`` diff --git a/doc/guides/sample_app_ug/flow_classify.rst b/doc/guides/sample_app_ug/flow_classify.rst index 52474774..003ed035 100644 --- a/doc/guides/sample_app_ug/flow_classify.rst +++ b/doc/guides/sample_app_ug/flow_classify.rst @@ -187,7 +187,7 @@ The ``main()`` function also initializes all the ports using the user defined .. code-block:: c - for (portid = 0; portid < nb_ports; portid++) { + RTE_ETH_FOREACH_DEV(portid) { if (port_init(portid, mbuf_pool) != 0) { rte_exit(EXIT_FAILURE, "Cannot init port %" PRIu8 "\n", portid); @@ -279,9 +279,6 @@ Forwarding application is shown below: int retval; uint16_t q; - if (port >= rte_eth_dev_count()) - return -1; - /* Configure the Ethernet device. */ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf); if (retval != 0) @@ -424,14 +421,13 @@ following: static __attribute__((noreturn)) void lcore_main(cls_app) { - const uint8_t nb_ports = rte_eth_dev_count(); - uint8_t port; + uint16_t port; /* * Check that the port is on the same NUMA node as the polling thread * for best performance. */ - for (port = 0; port < nb_ports; port++) + RTE_ETH_FOREACH_DEV(port) if (rte_eth_dev_socket_id(port) > 0 && rte_eth_dev_socket_id(port) != (int)rte_socket_id()) { printf("\n\n"); @@ -451,7 +447,7 @@ following: * Receive packets on a port and forward them on the paired * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc. */ - for (port = 0; port < nb_ports; port++) { + RTE_ETH_FOREACH_DEV(port) { /* Get burst of RX packets, from first port of pair. */ struct rte_mbuf *bufs[BURST_SIZE]; @@ -501,7 +497,7 @@ The main work of the application is done within the loop: .. code-block:: c for (;;) { - for (port = 0; port < nb_ports; port++) { + RTE_ETH_FOREACH_DEV(port) { /* Get burst of RX packets, from first port of pair. */ struct rte_mbuf *bufs[BURST_SIZE]; diff --git a/doc/guides/sample_app_ug/flow_filtering.rst b/doc/guides/sample_app_ug/flow_filtering.rst index 06230c07..bd0ae1e2 100644 --- a/doc/guides/sample_app_ug/flow_filtering.rst +++ b/doc/guides/sample_app_ug/flow_filtering.rst @@ -1,33 +1,5 @@ -.. BSD LICENSE - Copyright(c) 2017 Mellanox Corporation. All rights reserved. - All rights reserved. - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in - the documentation and/or other materials provided with the - distribution. - * Neither the name of Mellanox Corporation nor the names of its - contributors may be used to endorse or promote products derived - from this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - +.. SPDX-License-Identifier: BSD-3-Clause + Copyright 2017 Mellanox Technologies, Ltd Basic RTE Flow Filtering Sample Application =========================================== @@ -167,7 +139,6 @@ application is shown below: struct rte_eth_conf port_conf = { .rxmode = { .split_hdr_size = 0, - .ignore_offload_bitfield = 1, .offloads = DEV_RX_OFFLOAD_CRC_STRIP, }, .txmode = { @@ -244,7 +215,6 @@ The Ethernet port is configured with default settings using the struct rte_eth_conf port_conf = { .rxmode = { .split_hdr_size = 0, - .ignore_offload_bitfield = 1, .offloads = DEV_RX_OFFLOAD_CRC_STRIP, }, .txmode = { diff --git a/doc/guides/sample_app_ug/img/ip_pipelines_1.svg b/doc/guides/sample_app_ug/img/ip_pipelines_1.svg deleted file mode 100644 index a114ed82..00000000 --- a/doc/guides/sample_app_ug/img/ip_pipelines_1.svg +++ /dev/null @@ -1,738 +0,0 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Created with Inkscape (http://www.inkscape.org/) --> - -<svg - xmlns:dc="http://purl.org/dc/elements/1.1/" - xmlns:cc="http://creativecommons.org/ns#" - xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" - xmlns:svg="http://www.w3.org/2000/svg" - xmlns="http://www.w3.org/2000/svg" - xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" - xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" - width="638.18219" - height="273.16391" - id="svg2" - version="1.1" - inkscape:version="0.48.4 r9939" - sodipodi:docname="ipPipelines_1_update.svg"> - <defs - id="defs4"> - <marker - inkscape:stockid="Arrow1Mend" - orient="auto" - refY="0.0" - refX="0.0" - id="Arrow1Mend" - style="overflow:visible;"> - <path - id="path3887" - d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " - style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt;" - transform="scale(0.4) rotate(180) translate(10,0)" /> - </marker> - <marker - inkscape:stockid="Arrow1Lstart" - orient="auto" - refY="0.0" - refX="0.0" - id="Arrow1Lstart" - style="overflow:visible"> - <path - id="path3878" - d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " - style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" - transform="scale(0.8) translate(12.5,0)" /> - </marker> - <clipPath - id="clipEmfPath1" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect2990" - height="213.83858" - width="585" - y="0" - x="0" /> - </clipPath> - <clipPath - id="clipEmfPath2" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect2993" - height="71.379494" - width="66.300003" - y="118.0161" - x="132.75" /> - </clipPath> - <clipPath - id="clipEmfPath3" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect2996" - height="72.879066" - width="66.300003" - y="19.794313" - x="132.3" /> - </clipPath> - <clipPath - id="clipEmfPath4" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect2999" - height="75.428337" - width="65.699997" - y="56.983631" - x="31.200001" /> - </clipPath> - <clipPath - id="clipEmfPath5" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect3002" - height="157.60474" - width="168" - y="16.795176" - x="331.95001" /> - </clipPath> - </defs> - <sodipodi:namedview - id="base" - pagecolor="#ffffff" - bordercolor="#666666" - borderopacity="1.0" - inkscape:pageopacity="0.0" - inkscape:pageshadow="2" - inkscape:zoom="3.959798" - inkscape:cx="287.6677" - inkscape:cy="153.39982" - inkscape:document-units="px" - inkscape:current-layer="layer1" - showgrid="false" - inkscape:window-width="1200" - inkscape:window-height="1857" - inkscape:window-x="-8" - inkscape:window-y="-8" - inkscape:window-maximized="1" - fit-margin-top="0" - fit-margin-left="0" - fit-margin-right="0" - fit-margin-bottom="0" /> - <metadata - id="metadata7"> - <rdf:RDF> - <cc:Work - rdf:about=""> - <dc:format>image/svg+xml</dc:format> - <dc:type - rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> - <dc:title></dc:title> - </cc:Work> - </rdf:RDF> - </metadata> - <g - inkscape:label="Layer 1" - inkscape:groupmode="layer" - id="layer1" - transform="translate(-82.82706,-337.43917)"> - <text - xml:space="preserve" - x="797.93707" - y="551.73584" - style="font-size:17.86045074px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Calibri" - id="text3006" - transform="scale(0.90359163,1.1066946)"> </text> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath1)" - d="m 132.375,113.1425 0,80.97674 66.9,0 0,-80.97674 -66.9,0 z" - id="path3008" - transform="matrix(1.1694604,0,0,1.4323235,83.010165,320.85569)" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.03640038,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 237.46665,598.89724 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0,-3.43658 0.70167,0 0,3.43658 z m 0,-6.01404 0,-3.43658 0.70167,0 0,3.43658 z m 0,-6.01402 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0,-3.4366 0.70167,0 0,3.4366 z m 0,-6.01404 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01402 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01402 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0,-3.43658 0.70167,0 0,3.43658 z m 0,-6.01402 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0,-3.43658 0.70167,0 0,3.43658 z m 0,-6.01404 0,-3.43658 0.70167,0 0,3.43658 z m 0,-6.01402 0,-3.43659 0.70167,0 0,3.43659 z m 0,-6.01403 0, --1.7183 c 0,-0.22821 0.16445,-0.42957 0.35084,-0.42957 l 1.40335,0 0,0.85914 -1.40335,0 0.35083,-0.42957 0,1.7183 z m 3.85922,-2.14787 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 1.05252,0 c 0.19734,0 0.35083,0.20136 0.35083, -0.42957 l 0,2.14787 -0.70167,0 0,-2.14787 0.35084,0.42957 -1.05252,0 z m 1.40335,5.15488 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01402 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43658 -0.70167,0 0,-3.43658 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43658 -0.70167,0 0,-3.43658 z m 0,6.01402 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01402 0,3.43661 -0.70167,0 0,-3.43661 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167, -0 0,-3.43659 z m 0,6.01402 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.00702 c 0,0.24163 -0.15349,0.42958 -0.35083,0.42958 l -0.35084,0 0,-0.85915 0.35084,0 -0.35084,0.42957 0,-3.00702 z m -2.8067,3.4366 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.10502,0 0,-0.85915 2.10502,0 z" - id="path3010" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath2)" - d="m 131.925,14.920714 0,82.626267 66.9,0 0,-82.626267 -66.9,0 z" - id="path3012" - transform="matrix(1.1694604,0,0,1.4323235,83.010165,320.85569)" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.04853384,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 236.94039,460.57452 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01404 0,-3.43658 0.70168,0 0,3.43658 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168, -0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 1.92961,-4.51052 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85914 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85914 -2.8067,0 z m 3.33296,2.79222 0, -3.43659 -0.70168,0 0,-3.43659 z m 0,6.01404 0,3.43658 -0.70168,0 0,-3.43658 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 z m 0,6.01404 0,3.43658 -0.70168,0 0,-3.43658 z m 0,6.01403 0,3.43659 -0.70168, -0 0,-3.43659 z m 0,6.01403 0,1.71829 c 0,0.24164 -0.15349,0.42958 -0.35084,0.42958 l -1.40335,0 0,-0.85915 1.40335,0 -0.35084,0.42957 0,-1.71829 z m -3.85922,2.14787 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 z m -4.91174,0 -1.05251,0 0,-0.85915 1.05251,0 z" - id="path3014" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath3)" - d="m 30.825,52.110032 0,85.025578 66.3,0 0,-85.025578 -66.3,0 z" - id="path3016" - transform="matrix(1.1694604,0,0,1.4323235,83.010165,320.85569)" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.03640038,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 118.70794,517.27825 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.4366 0.70168,0 0,3.4366 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01402 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.4366 0.70168,0 0,3.4366 z m 0,-6.01404 0,-3.43658 0.70168,0 0,3.43658 z m 0,-6.01402 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01404 0,-3.43659 0.70168, -0 0,3.43659 z m 0,-6.01403 0,-3.43658 0.70168,0 0,3.43658 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 z m 0,-6.01403 0,-1.50351 c 0,-0.22821 0.16446,-0.42957 0.35084,-0.42957 l 1.57877,0 0,0.85915 -1.57877,0 0.35084,-0.42958 0,1.50351 z m 4.03464,-1.93308 2.80671,0 0,0.85915 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 z m 4.91173,0 2.8067, -0 0,0.85915 -2.8067,0 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 z m 4.91174,0 0.17542,0 c 0.19734,0 0.35083,0.20136 0.35083,0.42957 l 0,3.22181 -0.70167,0 0,-3.22181 0.35084,0.42958 -0.17542,0 z m 0.52625,6.22882 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43658 -0.70167,0 0,-3.43658 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01402 0,3.43659 -0.70167, -0 0,-3.43659 z m 0,6.01403 0,3.4366 -0.70167,0 0,-3.4366 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01402 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01404 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,3.43659 -0.70167,0 0,-3.43659 z m 0,6.01403 0,1.7183 c 0,0.24163 -0.15349,0.42957 -0.35083,0.42957 l -1.40336,0 0,-0.85914 1.40336,0 -0.35084,0.42957 0,-1.7183 z m -3.85921,2.14787 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91173,0 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85914 2.8067, -0 z m -4.91173,0 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -2.80671,0 0,-0.85914 2.80671,0 z m -4.91174,0 -2.8067,0 0,-0.85914 2.8067,0 z m -4.91173,0 -0.35084,0 0,-0.85914 0.35084,0 z" - id="path3018" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - clip-path="url(#clipEmfPath4)" - d="m 15.225,83.001159 3.253125,0 c 0.35625,0 0.646875,0.299914 0.646875,0.656061 l 0,16.73894 -3.9,0 z" - id="path3020" - transform="matrix(1.1694604,0,0,1.4323235,83.010165,320.85569)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 105.3761,439.52541 3.8044,0 c 0.41662,0 0.75649,0.42958 0.75649,0.9397 l 0,24.19036 -4.56089,0 z" - id="path3022" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 109.93699,439.7402 3.66187,0 c 0.40566,0 0.72361,0.40273 0.72361,0.89942 l 0,24.01585 -4.38548,0 z" - id="path3024" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 130.46102,435.44446 0,75.17539 60.695,0 0,-75.17539 -60.695,0 z" - id="path3026" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.77654135px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 130.46102,435.44446 60.695,0 0,75.17539 -60.695,0 z" - id="path3028" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 100.8152,483.12714 3.8044,0 c 0.41662,0 0.7565,0.42957 0.7565,0.93969 l 0,23.97558 -4.5609,0 z" - id="path3030" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 105.3761,482.91235 3.8044,0 c 0.41662,0 0.75649,0.42957 0.75649,0.93969 l 0,24.19037 -4.56089,0 z" - id="path3032" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 109.93699,483.12714 3.66187,0 c 0.40566,0 0.72361,0.40273 0.72361,0.89942 l 0,24.01585 -4.38548,0 z" - id="path3034" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 216.06552,512.55294 3.66187,0 c 0.40566,0 0.72361,0.40273 0.72361,0.89941 l 0,24.23065 -4.38548,0 z" - id="path3036" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 220.451,512.55294 3.80439,0 c 0.41662,0 0.7565,0.42957 0.7565,0.93969 l 0,23.97558 -4.56089,0 z" - id="path3038" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 225.01189,512.55294 3.8044,0 c 0.41662,0 0.7565,0.42957 0.7565,0.93969 l 0,24.19037 -4.5609,0 z" - id="path3040" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 248.86889,377.88159 0,75.17539 60.51957,0 0,-75.17539 -60.51957,0 z" - id="path3042" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.77654135px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 248.86889,377.88159 60.51957,0 0,75.17539 -60.51957,0 z" - id="path3044" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 249.57056,488.06724 0,75.17539 60.695,0 0,-75.17539 -60.695,0 z" - id="path3046" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.77654135px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 249.57056,488.06724 60.695,0 0,75.17539 -60.695,0 z" - id="path3048" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 216.06552,404.30037 3.66187,0 c 0.40566,0 0.72361,0.40273 0.72361,0.89942 l 0,24.01586 -4.38548,0 z" - id="path3050" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 220.451,404.30037 3.80439,0 c 0.41662,0 0.7565,0.42958 0.7565,0.9397 l 0,23.97558 -4.56089,0 z" - id="path3052" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 225.01189,404.30037 3.8044,0 c 0.41662,0 0.7565,0.42958 0.7565,0.9397 l 0,23.97558 -4.5609,0 z" - id="path3054" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 114.36632,451.31184 9.51649,0.57724 -0.0768,1.78542 -9.51648,-0.57724 0.0768,-1.78542 z m 9.48359,1.47666 -2.70803,-5.54418 8.54072,5.89321 -8.97926,4.8327 3.14657,-5.18173 z" - id="path3056" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 115.24342,495.34314 9.51648,0.56382 -0.0768,1.78541 -9.51648,-0.56381 0.0768,-1.78542 z m 9.48359,1.46324 -2.70803,-5.54418 8.54071,5.89321 -8.9683,4.8327 3.13562,-5.18173 z" - id="path3058" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 191.15602,445.78108 12.2574,0 0,-28.37871 -0.73457,0.89942 5.66823,0 0,-1.78542 -6.40279,0 0,28.36529 0.73456,-0.886 -11.52283,0 0,1.78542 z m 17.19106,-28.37871 -2.91634,5.36967 8.77095,-5.36967 -8.77095,-5.36967 2.91634,5.36967 z" - id="path3060" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 192.03311,499.85366 11.87367,0 0,23.8145 -0.73456,-0.89943 5.2845,0 0,1.78542 -6.01906,0 0,-23.81448 0.73456,0.89941 -11.13911,0 0,-1.78542 z m 16.42361,23.8145 -2.91634,-5.36968 8.77095,5.36968 -8.77095,5.36966 2.91634,-5.36966 z" - id="path3062" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 82.96631,451.31184 9.516484,0.57724 -0.07675,1.78542 -9.516484,-0.57724 0.07675,-1.78542 z m 9.483593,1.47666 -2.708032,-5.54418 8.540716,5.89321 -8.979263,4.8327 3.146579,-5.18173 z" - id="path3064" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 83.667986,494.48399 9.516484,0.57724 -0.07675,1.78542 -9.516484,-0.57724 0.07675,-1.78542 z m 9.483593,1.47666 -2.708031,-5.54418 8.540715,5.89321 -8.979263,4.83271 3.146579,-5.18174 z" - id="path3066" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 230.28542,414.58329 12.57536,0.0268 0,1.78542 -12.58631,-0.0268 0.0105,-1.78542 z m 12.57536,0.92627 -2.91634,-5.38309 8.75999,5.39651 -8.77096,5.34283 2.92731,-5.35625 z" - id="path3068" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 229.59472,524.12458 14.12123,0.36245 -0.0329,1.78542 -14.12123,-0.36246 0.0329,-1.78541 z m 14.0993,1.24844 -2.82863,-5.43678 8.67228,5.58446 -8.85866,5.15487 3.01501,-5.30255 z" - id="path3070" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 310.27652,412.865 12.57535,0.0268 0,1.78541 -12.58631,-0.0269 0.0105,-1.78542 z m 12.57535,0.92627 -2.91634,-5.3831 8.75999,5.39652 -8.77096,5.34282 2.92731,-5.35624 z" - id="path3072" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 310.97819,524.55415 12.57536,0.0268 0,1.78542 -12.58632,-0.0268 0.0105,-1.78542 z m 12.57536,0.92627 -2.91634,-5.3831 8.75998,5.39652 -8.77095,5.34282 2.92731,-5.35624 z" - id="path3074" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 329.38623,401.50814 3.8044,0 c 0.41662,0 0.7565,0.42958 0.7565,0.9397 l 0,24.19036 -4.5609,0 z" - id="path3076" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 333.94713,401.50814 3.8044,0 c 0.41662,0 0.7565,0.42958 0.7565,0.9397 l 0,23.97558 -4.5609,0 z" - id="path3078" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 338.50803,401.50814 3.8044,0 c 0.41661,0 0.75649,0.42958 0.75649,0.9397 l 0,24.19036 -4.56089,0 z" - id="path3080" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 330.78959,512.55294 3.66187,0 c 0.40566,0 0.7236,0.40273 0.7236,0.89941 l 0,24.23065 -4.38547,0 z" - id="path3082" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 335.17506,512.55294 3.8044,0 c 0.41663,0 0.7565,0.42957 0.7565,0.93969 l 0,24.19037 -4.5609,0 z" - id="path3084" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 339.73596,512.55294 3.80439,0 c 0.41664,0 0.7565,0.42957 0.7565,0.93969 l 0,24.19037 -4.56089,0 z" - id="path3086" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.012;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.012,0.012;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 343.95699,412.865 12.57534,0.0268 0,1.78541 -12.58631,-0.0269 0.0105,-1.78542 z m 12.57534,0.92627 -2.91634,-5.3831 8.75999,5.39652 -8.77096,5.34282 2.92731,-5.35624 z" - id="path3088" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01213346;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.01213346,0.01213346;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 344.30781,524.76894 12.57536,0.0268 0,1.78542 -12.58632,-0.0268 0.0105,-1.7854 z m 12.57536,0.92627 -2.91634,-5.3831 8.75998,5.39652 -8.77095,5.34283 2.92731,-5.35625 z" - id="path3090" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.09706768,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 470.42316,577.41856 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43658 0.70168,0 0,3.43658 -0.70168,0 z m 0,-6.01402 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43658 0.70168,0 0,3.43658 -0.70168,0 z m 0,-6.01402 0,-3.4366 0.70168,0 0,3.4366 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01402 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.4366 0.70168,0 0,3.4366 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01402 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168, -0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.4366 0.70168,0 0,3.4366 -0.70168,0 z m 0,-6.01404 0,-3.43658 0.70168,0 0,3.43658 -0.70168,0 z m 0,-6.01402 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43658 0.70168,0 0,3.43658 -0.70168,0 z m 0,-6.01403 0, --3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01404 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 0,-6.01403 0,-3.43659 0.70168,0 0,3.43659 -0.70168,0 z m 1.22793,-5.36967 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174, -0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0, -0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067, -0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 4.91173,0 2.80671,0 0,0.85915 -2.80671,0 0,-0.85915 z m 4.91174,0 2.8067,0 0,0.85915 -2.8067,0 0,-0.85915 z m 5.08715,0.64436 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0, --3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01402 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168, -0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01402 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43658 -0.70168,0 0,-3.43658 0.70168,0 z m 0,6.01402 0,3.4366 -0.70168,0 0,-3.4366 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.43658 -0.70168,0 0,-3.43658 0.70168,0 z m 0,6.01404 0,3.43658 -0.70168,0 0,-3.43658 0.70168,0 z m 0,6.01402 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01403 0,3.4366 -0.70168,0 0,-3.4366 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01402 0,3.43659 -0.70168,0 0,-3.43659 0.70168,0 z m 0,6.01404 0,3.43659 -0.70168, -0 0,-3.43659 0.70168,0 z m -1.40335,5.15488 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067, -0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671, -0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z m -4.91173,0 -2.80671,0 0,-0.85915 2.80671,0 0,0.85915 z m -4.91174,0 -2.8067,0 0,-0.85915 2.8067,0 0,0.85915 z" - id="path3094" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - clip-path="url(#clipEmfPath5)" - d="m 315.975,85.250513 3.2625,0 c 0.35625,0 0.6375,0.299914 0.6375,0.656061 l 0,16.738946 -3.9,0 z" - id="path3096" - transform="matrix(1.1694604,0,0,1.4323235,83.010165,320.85569)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 457.09131,442.74722 3.81537,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,24.19037 -4.5609,0 z" - id="path3098" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 461.65221,442.962 3.66187,0 c 0.41662,0 0.7236,0.40273 0.7236,0.91285 l 0,24.00243 -4.38547,0 z" - id="path3100" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 482.17624,438.66627 0,75.17539 60.69499,0 0,-75.17539 -60.69499,0 z" - id="path3102" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.77654135px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 482.17624,438.66627 60.69499,0 0,75.17539 -60.69499,0 z" - id="path3104" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 452.53041,486.34894 3.81537,0 c 0.41662,0 0.74553,0.42958 0.74553,0.93969 l 0,23.97559 -4.5609,0 z" - id="path3106" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 457.09131,486.13415 3.81537,0 c 0.41662,0 0.74553,0.42958 0.74553,0.9397 l 0,24.19037 -4.5609,0 z" - id="path3108" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 461.65221,486.34894 3.66187,0 c 0.41662,0 0.7236,0.40273 0.7236,0.91285 l 0,24.00243 -4.38547,0 z" - id="path3110" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 567.78074,515.77474 3.66187,0 c 0.41662,0 0.7236,0.40272 0.7236,0.91285 l 0,24.21721 -4.38547,0 z" - id="path3112" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 572.16621,515.77474 3.81537,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,23.97557 -4.5609,0 z" - id="path3114" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 576.72711,515.77474 3.81536,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,24.19037 -4.56089,0 z" - id="path3116" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 600.5841,381.1034 0,75.17539 60.51958,0 0,-75.17539 -60.51958,0 z" - id="path3118" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.77654135px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 600.5841,381.1034 60.51958,0 0,75.17539 -60.51958,0 z" - id="path3120" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 601.28578,491.28904 0,75.1754 60.69499,0 0,-75.1754 -60.69499,0 z" - id="path3122" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.77654135px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 601.28578,491.28904 60.69499,0 0,75.1754 -60.69499,0 z" - id="path3124" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 567.78074,407.52218 3.66187,0 c 0.41662,0 0.7236,0.40272 0.7236,0.91284 l 0,24.00243 -4.38547,0 z" - id="path3126" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 572.16621,407.52218 3.81537,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,23.97558 -4.5609,0 z" - id="path3128" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 576.72711,407.52218 3.81536,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,23.97558 -4.56089,0 z" - id="path3130" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 466.08154,454.53364 9.51648,0.59067 -0.0657,1.77199 -9.51649,-0.56382 0.0657,-1.79884 z m 9.49455,1.47666 -2.71899,-5.53076 8.55168,5.87979 -8.99023,4.83271 3.15754,-5.18174 z" - id="path3132" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 466.95863,498.56494 9.51649,0.56382 -0.0657,1.79884 -9.51648,-0.56382 0.0657,-1.79884 z m 9.47263,1.47667 -2.69707,-5.55762 8.55168,5.90665 -8.9683,4.8327 3.11369,-5.18173 z" - id="path3134" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 542.87123,449.00288 12.25741,0 0,-28.37871 -0.72361,0.91285 5.65727,0 0,-1.79884 -6.4028,0 0,28.37871 0.74553,-0.886 -11.5338,0 0,1.77199 z m 17.19107,-28.37871 -2.91634,5.36967 8.77095,-5.36967 -8.77095,-5.36967 2.91634,5.36967 z" - id="path3136" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 543.74833,503.07547 11.88464,0 0,23.81448 -0.74553,-0.88599 5.2845,0 0,1.77199 -6.00811,0 0,-23.81449 0.72361,0.91285 -11.13911,0 0,-1.79884 z m 16.42361,23.81448 -2.91635,-5.36966 8.77096,5.36966 -8.77096,5.36967 2.91635,-5.36967 z" - id="path3138" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 434.68153,454.53364 9.51648,0.59067 -0.0658,1.77199 -9.51649,-0.56382 0.0657,-1.79884 z m 9.49455,1.47666 -2.71899,-5.53076 8.55168,5.87979 -8.99023,4.83271 3.15754,-5.18174 z" - id="path3140" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 435.3832,497.70579 9.51649,0.59068 -0.0657,1.77198 -9.51648,-0.56382 0.0657,-1.79884 z m 9.49456,1.47667 -2.719,-5.53076 8.55168,5.87979 -8.99022,4.8327 3.15754,-5.18173 z" - id="path3142" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 582.01161,417.8051 12.56439,0.0268 0,1.79884 -12.58632,-0.0269 0.022,-1.79884 z m 12.56439,0.93969 -2.91634,-5.39652 8.77095,5.39652 -8.77095,5.34282 2.91634,-5.34282 z" - id="path3144" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 581.30993,527.34638 14.12124,0.37587 -0.022,1.772 -14.12123,-0.34903 0.022,-1.79884 z m 14.09931,1.26188 -2.82863,-5.45022 8.68324,5.58445 -8.85866,5.15489 3.00405,-5.28912 z" - id="path3146" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 662.0027,416.0868 12.56439,0.0269 0,1.79884 -12.58632,-0.0269 0.022,-1.79884 z m 12.56439,0.93969 -2.91634,-5.39652 8.77095,5.39652 -8.77095,5.34282 2.91634,-5.34282 z" - id="path3148" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;stroke-dasharray:0.02426692,0.02426692;stroke-miterlimit:4;stroke-dashoffset:0;marker-mid:none" - d="m 662.70438,527.77595 12.56439,0.0268 0,1.79884 -12.58632,-0.0268 0.022,-1.79884 z m 12.56439,0.9397 -2.91635,-5.39652 8.77096,5.39652 -8.77096,5.34281 2.91635,-5.34281 z" - id="path3150" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 681.10145,404.72995 3.81536,0 c 0.41662,0 0.74554,0.42957 0.74554,0.93969 l 0,24.19037 -4.5609,0 z" - id="path3152" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 685.66235,404.72995 3.81536,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,23.97558 -4.56089,0 z" - id="path3154" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 690.22324,404.72995 3.81537,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,24.19037 -4.5609,0 z" - id="path3156" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 682.5048,515.77474 3.66188,0 c 0.41662,0 0.7236,0.40272 0.7236,0.91285 l 0,24.21721 -4.38548,0 z" - id="path3158" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 686.89028,515.77474 3.81536,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,24.19037 -4.56089,0 z" - id="path3160" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 691.45117,515.77474 3.81537,0 c 0.41662,0 0.74553,0.42957 0.74553,0.93969 l 0,24.19037 -4.5609,0 z" - id="path3162" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 695.68316,416.0868 12.56439,0.0269 0,1.79884 -12.58632,-0.0269 0.0219,-1.79884 z m 12.56439,0.93969 -2.91634,-5.39652 8.77095,5.39652 -8.77095,5.34282 2.91634,-5.34282 z" - id="path3164" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.02426692px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 696.034,527.99074 12.56439,0.0269 0,1.79882 -12.58632,-0.0268 0.022,-1.79884 z m 12.56439,0.93969 -2.91634,-5.39652 8.77095,5.39652 -8.77095,5.34282 2.91634,-5.34282 z" - id="path3166" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 374.1181,472.44149 42.53912,0 0,-3.49028 5.70112,6.98057 -5.70112,6.98057 0,-3.49028 -42.53912,0 z" - id="path3168" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.5530827px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 374.1181,472.44149 42.53912,0 0,-3.49028 5.70112,6.98057 -5.70112,6.98057 0,-3.49028 -42.53912,0 z" - id="path3170" /> - </g> - <g - inkscape:groupmode="layer" - id="layer2" - inkscape:label="blocks" - transform="translate(-82.82706,-337.43917)"> - <rect - style="fill:none;stroke:#000000;stroke-width:0;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:0,0;stroke-dashoffset:0;stroke-linejoin:miter" - id="rect4431" - width="61.07143" - height="75.714287" - x="247.14285" - y="377.00504" /> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="129.47055" - y="417.48993" - id="text4458-5-6" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - id="tspan4460-7-1" - x="129.47055" - y="417.48993" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans">CPU Core 1</tspan><tspan - sodipodi:role="line" - x="129.47055" - y="431.23993" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668" /></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="248.86055" - y="358.53632" - id="text4458-5-6-8-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - id="tspan4460-7-1-9-9" - x="248.86055" - y="358.53632" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans">CPU Core 2</tspan><tspan - sodipodi:role="line" - x="248.86055" - y="372.28632" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5" /></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="249.61816" - y="589.6087" - id="text4458-5-6-8-4" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="249.61816" - y="589.6087" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-1">CPU Core 3</tspan></text> - <text - xml:space="preserve" - style="font-size:40px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" - x="145.96704" - y="472.28207" - id="text4731" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - id="tspan4733" - x="145.96704" - y="472.28207" /></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="154.6613" - y="476.98529" - id="text4458-5-6-4" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="154.6613" - y="476.98529" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-1">P1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="272.26172" - y="419.42242" - id="text4458-5-6-4-3" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="272.26172" - y="419.42242" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-1-8">P2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="273.77084" - y="529.60803" - id="text4458-5-6-4-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="273.77084" - y="529.60803" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-1-4">P3</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="547.64105" - y="361.29749" - id="text4458-5-6-3" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - id="tspan4460-7-1-1" - x="547.64105" - y="361.29749" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans">CPU Core 1</tspan><tspan - sodipodi:role="line" - x="547.64105" - y="375.04749" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-9" /></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="506.37653" - y="480.20709" - id="text4458-5-6-4-8" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="506.37653" - y="480.20709" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-1-6">P1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="623.97693" - y="422.64423" - id="text4458-5-6-4-3-0" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="623.97693" - y="422.64423" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-1-8-2">P2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="624.72607" - y="532.76007" - id="text4458-5-6-4-7-4" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="624.72607" - y="532.76007" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-1-4-8">P3</tspan></text> - </g> -</svg> diff --git a/doc/guides/sample_app_ug/img/ip_pipelines_2.svg b/doc/guides/sample_app_ug/img/ip_pipelines_2.svg deleted file mode 100644 index 5252b174..00000000 --- a/doc/guides/sample_app_ug/img/ip_pipelines_2.svg +++ /dev/null @@ -1,997 +0,0 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Created with Inkscape (http://www.inkscape.org/) --> - -<svg - xmlns:dc="http://purl.org/dc/elements/1.1/" - xmlns:cc="http://creativecommons.org/ns#" - xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" - xmlns:svg="http://www.w3.org/2000/svg" - xmlns="http://www.w3.org/2000/svg" - xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" - xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" - width="520.43439" - height="438.61716" - id="svg5382" - version="1.1" - inkscape:version="0.48.4 r9939" - sodipodi:docname="ipPipelines_2_update.svg"> - <defs - id="defs5384"> - <marker - inkscape:stockid="Arrow1Lstart" - orient="auto" - refY="0.0" - refX="0.0" - id="Arrow1Lstart" - style="overflow:visible"> - <path - id="path4018" - d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z " - style="fill-rule:evenodd;stroke:#000000;stroke-width:1.0pt" - transform="scale(0.8) translate(12.5,0)" /> - </marker> - <clipPath - id="clipEmfPath1" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5395" - height="465.83859" - width="585" - y="0" - x="0" /> - </clipPath> - <clipPath - id="clipEmfPath2" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5398" - height="0" - width="37.650002" - y="202.92325" - x="120.9" /> - </clipPath> - <clipPath - id="clipEmfPath3" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5401" - height="186.87537" - width="391.64999" - y="247.76733" - x="49.650002" /> - </clipPath> - <clipPath - id="clipEmfPath4" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5404" - height="93.137718" - width="142.64999" - y="280.01309" - x="72.150002" /> - </clipPath> - <clipPath - id="clipEmfPath5" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5407" - height="13.648201" - width="45.900002" - y="308.50934" - x="76.650002" /> - </clipPath> - <clipPath - id="clipEmfPath6" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5410" - height="13.648201" - width="83.400002" - y="308.50934" - x="126.15" /> - </clipPath> - <clipPath - id="clipEmfPath7" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5413" - height="90.888016" - width="161.39999" - y="333.25607" - x="257.39999" /> - </clipPath> - <clipPath - id="clipEmfPath8" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5416" - height="13.648201" - width="54.599998" - y="364.75192" - x="262.64999" /> - </clipPath> - <clipPath - id="clipEmfPath9" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5419" - height="13.648201" - width="90.150002" - y="364.75192" - x="321.60001" /> - </clipPath> - <clipPath - id="clipEmfPath10" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5422" - height="133.63239" - width="240" - y="75.590034" - x="240" /> - </clipPath> - <clipPath - id="clipEmfPath11" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5425" - height="137.3819" - width="191.85001" - y="20.847252" - x="39.75" /> - </clipPath> - <clipPath - id="clipEmfPath12" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5428" - height="93.137718" - width="142.64999" - y="54.592804" - x="66.449997" /> - </clipPath> - <clipPath - id="clipEmfPath13" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5431" - height="13.648201" - width="45.900002" - y="83.08905" - x="70.949997" /> - </clipPath> - <clipPath - id="clipEmfPath14" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5434" - height="13.648201" - width="83.400002" - y="83.08905" - x="120.45" /> - </clipPath> - <clipPath - id="clipEmfPath15" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5437" - height="90.888016" - width="161.39999" - y="107.83578" - x="298.95001" /> - </clipPath> - <clipPath - id="clipEmfPath16" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5440" - height="13.648201" - width="54.599998" - y="139.33163" - x="304.20001" /> - </clipPath> - <clipPath - id="clipEmfPath17" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect5443" - height="13.648201" - width="90.150002" - y="139.33163" - x="363.29999" /> - </clipPath> - </defs> - <sodipodi:namedview - id="base" - pagecolor="#ffffff" - bordercolor="#666666" - borderopacity="1.0" - inkscape:pageopacity="0.0" - inkscape:pageshadow="2" - inkscape:zoom="2.8" - inkscape:cx="223.36548" - inkscape:cy="227.70504" - inkscape:document-units="px" - inkscape:current-layer="layer1" - showgrid="false" - inkscape:window-width="1083" - inkscape:window-height="851" - inkscape:window-x="40" - inkscape:window-y="767" - inkscape:window-maximized="0" - fit-margin-top="0" - fit-margin-left="0" - fit-margin-right="0" - fit-margin-bottom="0" /> - <metadata - id="metadata5387"> - <rdf:RDF> - <cc:Work - rdf:about=""> - <dc:format>image/svg+xml</dc:format> - <dc:type - rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> - <dc:title></dc:title> - </cc:Work> - </rdf:RDF> - </metadata> - <g - inkscape:label="Layer 1" - inkscape:groupmode="layer" - id="layer1" - transform="translate(-132.40505,-338.57503)"> - <text - xml:space="preserve" - x="652.83942" - y="777.1922" - style="font-size:13.80000019px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Calibri" - id="text5447"> </text> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath2)" - d="m 49.275,242.89298 0,196.4741 392.25,0 0,-196.4741 -392.25,0 z" - id="path5449" - transform="translate(112.83943,322.9021)" /> - <rect - style="fill:none;stroke:#000000;stroke-width:1.002;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.006,1.002;stroke-dashoffset:0" - id="rect4458-1-7" - width="393.01892" - height="197.3044" - x="161.22458" - y="565.61206" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-9" - width="15.714286" - height="18.571428" - x="153.47649" - y="664.6922" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-4" - width="15.714286" - height="18.571428" - x="547.40503" - y="689.6922" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath3)" - d="m 71.775,275.13873 0,102.73646 143.25,0 0,-102.73646 -143.25,0 z" - id="path5453" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 184.61443,598.04083 143.25,0 0,102.73646 -143.25,0 z" - id="path5455" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath4)" - d="m 76.275,303.63497 0,23.24694 46.5,0 0,-23.24694 -46.5,0 z" - id="path5457" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 189.11443,626.53707 46.5,0 0,23.24694 -46.5,0 z" - id="path5459" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath5)" - d="m 125.775,303.63497 0,23.24694 84,0 0,-23.24694 -84,0 z" - id="path5461" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 238.61443,626.53707 84,0 0,23.24694 -84,0 z" - id="path5463" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath6)" - d="m 76.275,329.88151 0,41.99447 46.5,0 0,-41.99447 -46.5,0 z" - id="path5465" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 189.11443,652.78361 46.5,0 0,41.99447 -46.5,0 z" - id="path5467" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 238.61443,652.78361 0,42.74437 84,0 0,-42.74437 -84,0 z" - id="path5469" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 238.61443,652.78361 84,0 0,42.74437 -84,0 z" - id="path5471" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 369.86443,651.28381 0,100.48676 162,0 0,-100.48676 -162,0 z" - id="path5473" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 369.86443,651.28381 162,0 0,100.48676 -162,0 z" - id="path5475" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath7)" - d="m 262.275,359.87756 0,23.24694 55.2,0 0,-23.24694 -55.2,0 z" - id="path5477" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 375.11443,682.77966 55.2,0 0,23.24694 -55.2,0 z" - id="path5479" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath8)" - d="m 321.225,359.87756 0,23.24694 90.9,0 0,-23.24694 -90.9,0 z" - id="path5481" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 434.06443,682.77966 90.9,0 0,23.24694 -90.9,0 z" - id="path5483" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath9)" - d="m 261.525,386.874 0,36.74516 55.95,0 0,-36.74516 -55.95,0 z" - id="path5485" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 374.36443,709.7761 55.95,0 0,36.74516 -55.95,0 z" - id="path5487" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 434.81443,709.7761 0,36.74516 90.15,0 0,-36.74516 -90.15,0 z" - id="path5489" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 434.81443,709.7761 90.15,0 0,36.74516 -90.15,0 z" - id="path5491" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-0" - width="15.714286" - height="18.571428" - x="152.76219" - y="617.90649" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 155.21443,618.13818 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.74154 -3.9,0 z" - id="path5493" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 159.11443,617.9882 3.13125,0 c 0.35625,0 0.61875,0.28121 0.61875,0.63742 l 0,16.91027 -3.75,0 z" - id="path5495" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 162.86443,618.13818 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.74154 -3.9,0 z" - id="path5497" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 159.11443,665.23197 3.13125,0 c 0.35625,0 0.61875,0.28122 0.61875,0.63742 l 0,16.91027 -3.75,0 z" - id="path5501" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 162.86443,665.38195 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.74154 -3.9,0 z" - id="path5503" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.05625,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 138.13318,627.94314 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33745 0.0188,1.25609 z m 12.09375,-0.95612 -2.4,3.80574 7.3875,-3.95573 -7.59375,-3.52453 2.60625,3.67452 z" - id="path5505" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.05625,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 167.53318,627.94314 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33745 0.0187,1.25609 z m 12.09375,-0.95612 -2.4,3.80574 7.3875,-3.95573 -7.59375,-3.52453 2.60625,3.67452 z" - id="path5507" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.05625,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 166.78318,674.43701 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33746 0.0187,1.25608 z m 12.09375,-0.95612 -2.4,3.80574 7.3875,-3.95572 -7.59375,-3.52454 2.60625,3.67452 z" - id="path5509" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.05625,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 138.13318,674.43701 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33746 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80574 7.3875,-3.95572 -7.59375,-3.52454 2.60625,3.67452 z" - id="path5511" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 155.21443,665.38195 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.74154 -3.9,0 z" - id="path5499" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945" - width="15.714286" - height="18.571428" - x="547.40509" - y="612.90656" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 548.96443,613.33881 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.89152 -3.9,0 z" - id="path5513" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 552.86443,613.33881 3.13125,0 c 0.35625,0 0.61875,0.28122 0.61875,0.63742 l 0,16.76029 -3.75,0 z" - id="path5515" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 556.61443,613.33881 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.89152 -3.9,0 z" - id="path5517" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 328.76443,621.419 215.04375,0 0,1.25608 -215.04375,0 0,-1.25608 z m 215.04375,0.61867 -2.49375,-3.74951 7.5,3.74951 -7.5,3.7495 2.49375,-3.7495 z" - id="path5519" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.05625,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 560.38318,623.14377 12.1125,-0.33745 -0.0375,-1.25609 -12.09375,0.33746 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80575 7.3875,-3.95573 -7.59375,-3.52454 2.60625,3.67452 z" - id="path5521" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 329.36443,682.16099 20.86875,0 0,29.93981 -0.61875,-0.63742 15.225,0 0,1.25608 -15.8625,0 0,-29.9398 0.6375,0.63742 -20.25,0 0,-1.25609 z m 35.475,29.93981 -2.49375,-3.74951 7.5,3.74951 -7.5,3.7495 2.49375,-3.7495 z" - id="path5523" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 548.81443,690.57863 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.89152 -3.9,0 z" - id="path5525" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 552.71443,690.57863 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65617 l 0,16.74154 -3.9,0 z" - id="path5527" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 556.61443,690.57863 3.13125,0 c 0.35625,0 0.61875,0.28122 0.61875,0.63742 l 0,16.91027 -3.75,0 z" - id="path5529" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 530.98318,699.18375 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33745 0.0188,1.25609 z m 12.09375,-0.95613 -2.4,3.80575 7.3875,-3.95573 -7.59375,-3.52453 2.60625,3.67451 z" - id="path5531" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 561.88318,698.73381 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33745 0.0188,1.25609 z m 12.09375,-0.95613 -2.4,3.80575 7.3875,-3.95573 -7.59375,-3.52453 2.60625,3.67451 z" - id="path5533" /> - <rect - style="fill:none;stroke:#000000;stroke-width:1.03885794;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.11657381,1.03885794;stroke-dashoffset:0" - id="rect4458-1" - width="212.04707" - height="146.38971" - x="384.98563" - y="391.8602" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-48" - width="15.714286" - height="18.571428" - x="586.69073" - y="463.62076" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath10)" - d="m 39.225,15.972894 0,146.980626 192.6,0 0,-146.980626 -192.6,0 z" - id="path5539" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath11)" - d="m 66.075,49.718446 0,102.736454 143.25,0 0,-102.736454 -143.25,0 z" - id="path5543" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 178.91443,372.62055 143.25,0 0,102.73645 -143.25,0 z" - id="path5545" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath12)" - d="m 70.575,78.21469 0,23.24694 46.5,0 0,-23.24694 -46.5,0 z" - id="path5547" - transform="translate(112.83943,322.9021)" /> - <rect - style="fill:none;stroke:#000000;stroke-width:0.99358916;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:2.98076741, 0.99358914;stroke-dashoffset:0" - id="rect4458" - width="193.90961" - height="146.43498" - x="155.25899" - y="342.90326" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 183.41443,401.11679 46.5,0 0,23.24694 -46.5,0 z" - id="path5549" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath13)" - d="m 120.075,78.21469 0,23.24694 84,0 0,-23.24694 -84,0 z" - id="path5551" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 232.91443,401.11679 84,0 0,23.24694 -84,0 z" - id="path5553" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath14)" - d="m 70.575,104.46123 0,41.99446 46.5,0 0,-41.99446 -46.5,0 z" - id="path5555" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 183.41443,427.36333 46.5,0 0,41.99446 -46.5,0 z" - id="path5557" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 232.91443,427.36333 0,42.74437 84,0 0,-42.74437 -84,0 z" - id="path5559" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 232.91443,427.36333 84,0 0,42.74437 -84,0 z" - id="path5561" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 408.38397,422.83307 0,100.48675 162,0 0,-100.48675 -162,0 z" - id="path5563" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 408.38397,422.83307 162,0 0,100.48675 -162,0 z" - id="path5565" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath15)" - d="m 303.825,134.45728 0,23.24693 55.2,0 0,-23.24693 -55.2,0 z" - id="path5567" - transform="translate(109.80897,319.87164)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 413.63397,454.32892 55.2,0 0,23.24693 -55.2,0 z" - id="path5569" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath16)" - d="m 362.775,134.45728 0,23.24693 90.9,0 0,-23.24693 -90.9,0 z" - id="path5571" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 475.58964,454.32892 90.9,0 0,23.24693 -90.9,0 z" - id="path5573" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath17)" - d="m 303.075,161.45372 0,36.74515 55.95,0 0,-36.74515 -55.95,0 z" - id="path5575" - transform="translate(112.83943,322.9021)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 412.88397,481.32536 55.95,0 0,36.74515 -55.95,0 z" - id="path5577" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 473.33397,481.32536 0,36.74515 90.15,0 0,-36.74515 -90.15,0 z" - id="path5579" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 473.33397,481.32536 90.15,0 0,36.74515 -90.15,0 z" - id="path5581" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-45" - width="15.714286" - height="18.571428" - x="147.94077" - y="392.37076" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 149.51443,392.7179 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74154 -3.9,0 z" - id="path5583" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 153.41443,392.56792 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.89152 -3.9,0 z" - id="path5585" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 157.31443,392.7179 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76966 -3.75,0 z" - id="path5587" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-5" - width="15.714286" - height="18.571428" - x="147.76219" - y="439.6922" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 149.51443,439.96167 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74155 -3.9,0 z" - id="path5589" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 153.41443,439.81169 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.89153 -3.9,0 z" - id="path5591" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 157.31443,439.96167 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76967 -3.75,0 z" - id="path5593" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009,0.009;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 132.43318,402.51348 12.10312,-0.32808 -0.0375,-1.24671 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95612 -2.39062,3.81512 7.3875,-3.95573 -7.59375,-3.53391 2.59687,3.67452 z" - id="path5595" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009,0.009;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 161.83318,402.51348 12.10312,-0.32808 -0.0375,-1.24671 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95612 -2.39062,3.81512 7.3875,-3.95573 -7.59375,-3.53391 2.59687,3.67452 z" - id="path5597" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 161.08318,449.00735 12.10312,-0.32808 -0.0375,-1.24671 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95612 -2.39062,3.81512 7.3875,-3.95573 -7.59375,-3.53391 2.59687,3.67452 z" - id="path5599" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009,0.009;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 132.43318,449.00735 12.10312,-0.32808 -0.0375,-1.24671 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95612 -2.39062,3.81512 7.3875,-3.95573 -7.59375,-3.53391 2.59687,3.67452 z" - id="path5601" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-2" - width="15.714286" - height="18.571428" - x="341.36935" - y="371.26364" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 343.61443,372.17061 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76966 -3.75,0 z" - id="path5603" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 347.36443,372.17061 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74154 -3.9,0 z" - id="path5605" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 351.26443,372.17061 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74154 -3.9,0 z" - id="path5607" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 323.21443,378.75099 13.7625,0 0,1.24671 -13.7625,0 0,-1.24671 z m 13.7625,0.61867 -2.50312,-3.74951 7.5,3.74951 -7.5,3.7495 2.50312,-3.7495 z" - id="path5609" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:0.009375, 0.009375;stroke-dashoffset:0" - d="m 356.53318,380.16643 259.24599,-0.32809 -0.0375,-1.24671 -259.23661,0.32809 0.0281,1.24671 z m 259.22723,-0.95613 -2.39062,3.81512 7.3875,-3.95572 -7.59375,-3.53391 z" - id="path5611" - sodipodi:nodetypes="ccccccccccc" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 590.51443,465.15835 3.13125,0 c 0.35625,0 0.61875,0.28121 0.61875,0.63741 l 0,16.91027 -3.75,0 z" - id="path5613" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 594.26443,465.15835 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.74154 -3.9,0 z" - id="path5615" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 598.16443,465.15835 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.89152 -3.9,0 z" - id="path5617" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 572.68318,473.76346 12.1125,-0.33745 -0.0375,-1.25609 -12.09375,0.33746 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80575 7.3875,-3.95573 -7.59375,-3.52454 2.60625,3.67452 z" - id="path5619" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 603.43318,473.31352 12.1125,-0.33745 -0.0375,-1.25609 -12.09375,0.33746 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80575 7.3875,-3.95573 -7.59375,-3.52454 2.60625,3.67452 z" - id="path5621" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3945-8" - width="15.714286" - height="18.571428" - x="342.40506" - y="438.62076" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 343.61443,439.96167 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76967 -3.75,0 z" - id="path5623" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 347.36443,439.96167 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74155 -3.9,0 z" - id="path5625" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 351.26443,439.96167 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74155 -3.9,0 z" - id="path5627" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 323.21443,446.54205 13.7625,0 0,1.24671 -13.7625,0 0,-1.24671 z m 13.7625,0.61867 -2.50312,-3.74951 7.5,3.74951 -7.5,3.74951 2.50312,-3.74951 z" - id="path5629" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01777934;stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:0.01777934, 0.01777934;stroke-dashoffset:0" - d="m 356.50859,445.94165 25.2708,0 0,30.18438 -0.55633,-0.61869 20.19642,0 0,1.23737 -20.7696,0 0,-30.18438 0.57318,0.63744 -24.71447,0 0,-1.25612 z m 44.91089,30.18438 -2.24217,-3.74961 6.74338,3.74961 -6.74338,3.74961 2.24217,-3.74961 z" - id="path5631" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 256.12693,502.50342 0,36.37021 2.4375,0 -4.875,4.87436 -4.875,-4.87436 2.4375,0 0,-36.37021 z" - id="path5633" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 256.12693,502.50342 0,36.37021 2.4375,0 -4.875,4.87436 -4.875,-4.87436 2.4375,0 0,-36.37021 z" - id="path5635" /> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="224.7719" - y="353.7381" - id="text4458-5-6-8-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="224.7719" - y="353.7381" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5">Pipeline 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="233.06458" - y="387.51147" - id="text4458-5-6-8-7-0" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="233.06458" - y="387.51147" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6">Table 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="191.77844" - y="416.61282" - id="text4458-5-6-8-7-0-1" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="191.77844" - y="416.61282" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-3">Route</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="251.78381" - y="415.58426" - id="text4458-5-6-8-7-0-1-8" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="251.78381" - y="415.58426" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-3-9">Next Hop</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="462.5943" - y="411.55435" - id="text4458-5-6-8-7-3" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="462.5943" - y="411.55435" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-4">Pipeline 2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="471.90912" - y="442.29727" - id="text4458-5-6-8-7-0-4" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="471.90912" - y="442.29727" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6">Table 2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="431.39413" - y="468.73193" - id="text4458-5-6-8-7-0-4-0" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="431.39413" - y="468.73193" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6-6">Key</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="486.88754" - y="469.8894" - id="text4458-5-6-8-7-0-4-0-6" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="486.88754" - y="469.8894" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6-6-1">MAC Address</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="334.64691" - y="584.99622" - id="text4458-5-6-8-7-8" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="334.64691" - y="584.99622" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-49">Pipeline 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="197.47844" - y="642.03308" - id="text4458-5-6-8-7-0-1-6" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="197.47844" - y="642.03308" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-3-3">Route</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="257.48383" - y="641.00452" - id="text4458-5-6-8-7-0-1-8-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="257.48383" - y="641.00452" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-3-9-8">Next Hop</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="392.87457" - y="697.18268" - id="text4458-5-6-8-7-0-4-0-8" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="392.87457" - y="697.18268" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6-6-2">Key</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="445.36234" - y="698.34015" - id="text4458-5-6-8-7-0-4-0-6-9" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="445.36234" - y="698.34015" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6-6-1-1">MAC Address</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="238.76457" - y="611.66492" - id="text4458-5-6-8-7-0-3" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="238.76457" - y="611.66492" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-5">Table 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="432.66986" - y="666.95581" - id="text4458-5-6-8-7-0-4-9" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="432.66986" - y="666.95581" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6-8">Table 2</tspan></text> - </g> -</svg> diff --git a/doc/guides/sample_app_ug/img/ip_pipelines_3.svg b/doc/guides/sample_app_ug/img/ip_pipelines_3.svg deleted file mode 100644 index 1cf54136..00000000 --- a/doc/guides/sample_app_ug/img/ip_pipelines_3.svg +++ /dev/null @@ -1,826 +0,0 @@ -<?xml version="1.0" encoding="UTF-8" standalone="no"?> -<!-- Created with Inkscape (http://www.inkscape.org/) --> - -<svg - xmlns:dc="http://purl.org/dc/elements/1.1/" - xmlns:cc="http://creativecommons.org/ns#" - xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" - xmlns:svg="http://www.w3.org/2000/svg" - xmlns="http://www.w3.org/2000/svg" - xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" - xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" - width="469.492" - height="382.11536" - id="svg6126" - version="1.1" - inkscape:version="0.48.4 r9939" - sodipodi:docname="ipPipelines_3_update.svg"> - <defs - id="defs6128"> - <clipPath - id="clipEmfPath1" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6139" - height="406.59448" - width="585" - y="0" - x="0" /> - </clipPath> - <clipPath - id="clipEmfPath2" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6142" - height="134.38165" - width="277.79999" - y="247.76617" - x="49.650002" /> - </clipPath> - <clipPath - id="clipEmfPath3" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6145" - height="93.137283" - width="230.55" - y="280.01178" - x="72.150002" /> - </clipPath> - <clipPath - id="clipEmfPath4" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6148" - height="13.648136" - width="45.900002" - y="308.50787" - x="76.650002" /> - </clipPath> - <clipPath - id="clipEmfPath5" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6151" - height="13.648136" - width="83.400002" - y="308.50787" - x="126.15" /> - </clipPath> - <clipPath - id="clipEmfPath6" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6154" - height="137.38124" - width="208.2" - y="18.147522" - x="240.75" /> - </clipPath> - <clipPath - id="clipEmfPath7" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6157" - height="137.38124" - width="190.2" - y="18.147522" - x="42.75" /> - </clipPath> - <clipPath - id="clipEmfPath8" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6160" - height="93.137283" - width="142.64999" - y="54.592545" - x="66.449997" /> - </clipPath> - <clipPath - id="clipEmfPath9" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6163" - height="13.648136" - width="45.900002" - y="83.088654" - x="70.949997" /> - </clipPath> - <clipPath - id="clipEmfPath10" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6166" - height="13.648136" - width="83.400002" - y="83.088654" - x="120.45" /> - </clipPath> - <clipPath - id="clipEmfPath11" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6169" - height="90.887589" - width="161.39999" - y="53.092751" - x="265.20001" /> - </clipPath> - <clipPath - id="clipEmfPath12" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6172" - height="13.648136" - width="54.599998" - y="84.588448" - x="270.45001" /> - </clipPath> - <clipPath - id="clipEmfPath13" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6175" - height="13.648136" - width="90.150002" - y="84.588448" - x="329.54999" /> - </clipPath> - <clipPath - id="clipEmfPath14" - clipPathUnits="userSpaceOnUse"> - <rect - id="rect6178" - height="13.648136" - width="83.400002" - y="308.50787" - x="213.89999" /> - </clipPath> - </defs> - <sodipodi:namedview - id="base" - pagecolor="#ffffff" - bordercolor="#666666" - borderopacity="1.0" - inkscape:pageopacity="0.0" - inkscape:pageshadow="2" - inkscape:zoom="1.979899" - inkscape:cx="228.24994" - inkscape:cy="175.22545" - inkscape:document-units="px" - inkscape:current-layer="layer1" - showgrid="false" - inkscape:window-width="1033" - inkscape:window-height="1284" - inkscape:window-x="86" - inkscape:window-y="249" - inkscape:window-maximized="0" - fit-margin-top="0" - fit-margin-left="0" - fit-margin-right="0" - fit-margin-bottom="0" - showborder="true" - borderlayer="false" /> - <metadata - id="metadata6131"> - <rdf:RDF> - <cc:Work - rdf:about=""> - <dc:format>image/svg+xml</dc:format> - <dc:type - rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> - <dc:title></dc:title> - </cc:Work> - </rdf:RDF> - </metadata> - <g - inkscape:label="Layer 1" - inkscape:groupmode="layer" - id="layer1" - transform="translate(-156.65375,-367.02875)"> - <text - xml:space="preserve" - x="626.14575" - y="749.1441" - style="font-size:13.80000019px;font-style:normal;font-weight:normal;text-align:start;text-anchor:start;fill:#000000;font-family:Calibri" - id="text6182"> </text> - <rect - style="fill:none;stroke:#000000;stroke-width:1.1530993;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.45929811, 1.15309937;stroke-dashoffset:0" - id="rect3114" - width="279.14426" - height="142.7157" - x="186.01018" - y="597.78625" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath2)" - d="m 71.775,275.13743 0,102.73597 231.15,0 0,-102.73597 -231.15,0 z" - id="path6188" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 208.92075,629.23551 231.15,0 0,102.73597 -231.15,0 z" - id="path6190" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath3)" - d="m 76.275,303.63354 0,23.24683 46.5,0 0,-23.24683 -46.5,0 z" - id="path6192" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 213.42075,657.73162 46.5,0 0,23.24683 -46.5,0 z" - id="path6194" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath4)" - d="m 125.775,303.63354 0,23.24683 84,0 0,-23.24683 -84,0 z" - id="path6196" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 262.92075,657.73162 84,0 0,23.24683 -84,0 z" - id="path6198" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath5)" - d="m 76.275,329.87996 0,41.99426 46.5,0 0,-41.99426 -46.5,0 z" - id="path6200" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 213.42075,683.97804 46.5,0 0,41.99426 -46.5,0 z" - id="path6202" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 262.92075,683.97804 0,42.74416 84,0 0,-42.74416 -84,0 z" - id="path6204" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 262.92075,683.97804 84,0 0,42.74416 -84,0 z" - id="path6206" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920-48" - width="16.515228" - height="21.21936" - x="177.68185" - y="647.82019" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 179.52075,649.33277 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.74146 -3.9,0 z" - id="path6208" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 183.42075,649.18279 3.13125,0 c 0.35625,0 0.61875,0.28121 0.61875,0.63741 l 0,16.91019 -3.75,0 z" - id="path6210" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 187.17075,649.33277 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.74146 -3.9,0 z" - id="path6212" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920-48-8" - width="16.515228" - height="21.21936" - x="176.61043" - y="694.24872" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 179.52075,696.57632 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.74146 -3.9,0 z" - id="path6214" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 183.42075,696.42634 3.13125,0 c 0.35625,0 0.61875,0.28121 0.61875,0.63741 l 0,16.91019 -3.75,0 z" - id="path6216" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 187.17075,696.57632 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.74146 -3.9,0 z" - id="path6218" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 162.4395,659.13768 12.1125,-0.33745 -0.0375,-1.25608 -12.09375,0.33745 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80573 7.3875,-3.95571 -7.59375,-3.52452 2.60625,3.6745 z" - id="path6220" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 191.8395,659.13768 12.1125,-0.33745 -0.0375,-1.25608 -12.09375,0.33745 0.0187,1.25608 z m 12.09375,-0.95612 -2.4,3.80573 7.3875,-3.95571 -7.59375,-3.52452 2.60625,3.6745 z" - id="path6222" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.0375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 191.0895,705.63133 12.1125,-0.33745 -0.0375,-1.25608 -12.09375,0.33745 0.0187,1.25608 z m 12.09375,-0.95612 -2.4,3.80573 7.3875,-3.95571 -7.59375,-3.52452 2.60625,3.6745 z" - id="path6224" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 162.4395,705.63133 12.1125,-0.33745 -0.0375,-1.25608 -12.09375,0.33745 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80573 7.3875,-3.95571 -7.59375,-3.52452 2.60625,3.6745 z" - id="path6226" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 377.52075,367.37127 0,146.97993 208.8,0 0,-146.97993 -208.8,0 z" - id="path6228" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath6)" - d="m 42.375,13.273188 0,146.979932 190.8,0 0,-146.979932 -190.8,0 z" - id="path6232" - transform="translate(137.14575,354.09808)" /> - <rect - style="fill:none;stroke:#000000;stroke-width:1.1530993;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.45929801, 1.15309934;stroke-dashoffset:0" - id="rect3114-1" - width="191.96552" - height="148.42998" - x="180.02544" - y="368.29129" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath7)" - d="m 66.075,49.718211 0,102.735969 143.25,0 0,-102.735969 -143.25,0 z" - id="path6236" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 203.22075,403.81629 143.25,0 0,102.73597 -143.25,0 z" - id="path6238" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath8)" - d="m 70.575,78.21432 0,23.24683 46.5,0 0,-23.24683 -46.5,0 z" - id="path6240" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 207.72075,432.3124 46.5,0 0,23.24683 -46.5,0 z" - id="path6242" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath9)" - d="m 120.075,78.21432 0,23.24683 84,0 0,-23.24683 -84,0 z" - id="path6244" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 257.22075,432.3124 84,0 0,23.24683 -84,0 z" - id="path6246" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath10)" - d="m 70.575,104.46074 0,41.99426 46.5,0 0,-41.99426 -46.5,0 z" - id="path6248" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 207.72075,458.55882 46.5,0 0,41.99426 -46.5,0 z" - id="path6250" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 257.22075,458.55882 0,42.74416 84,0 0,-42.74416 -84,0 z" - id="path6252" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 257.22075,458.55882 84,0 0,42.74416 -84,0 z" - id="path6254" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 401.97075,402.3165 0,100.48628 162,0 0,-100.48628 -162,0 z" - id="path6256" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 401.97075,402.3165 162,0 0,100.48628 -162,0 z" - id="path6258" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath11)" - d="m 270.075,79.714116 0,23.246824 55.2,0 0,-23.246824 -55.2,0 z" - id="path6260" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 407.22075,433.8122 55.2,0 0,23.24682 -55.2,0 z" - id="path6262" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath12)" - d="m 329.025,79.714116 0,23.246824 90.9,0 0,-23.246824 -90.9,0 z" - id="path6264" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 466.17075,433.8122 90.9,0 0,23.24682 -90.9,0 z" - id="path6266" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath13)" - d="m 269.325,106.71043 0,36.74498 55.95,0 0,-36.74498 -55.95,0 z" - id="path6268" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 406.47075,460.80851 55.95,0 0,36.74498 -55.95,0 z" - id="path6270" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 466.92075,460.80851 0,36.74498 90.15,0 0,-36.74498 -90.15,0 z" - id="path6272" /> - <rect - style="fill:none;stroke:#000000;stroke-width:1.1530993;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:3.45929818, 1.15309941;stroke-dashoffset:0" - id="rect3114-1-7" - width="208.7556" - height="148.37921" - x="377.85892" - y="367.84274" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 466.92075,460.80851 90.15,0 0,36.74498 -90.15,0 z" - id="path6274" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920-9" - width="16.515228" - height="21.21936" - x="171.55568" - y="422.16541" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 173.82075,423.91355 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74146 -3.9,0 z" - id="path6276" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 177.72075,423.76357 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.89144 -3.9,0 z" - id="path6278" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 181.62075,423.91355 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76958 -3.75,0 z" - id="path6280" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920-0" - width="16.515228" - height="21.21936" - x="171.50339" - y="469.72925" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 173.82075,471.1571 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74146 -3.9,0 z" - id="path6282" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 177.72075,471.00712 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.89144 -3.9,0 z" - id="path6284" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 181.62075,471.1571 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76958 -3.75,0 z" - id="path6286" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 156.7395,433.70908 12.10312,-0.32808 -0.0375,-1.2467 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95611 -2.39062,3.8151 7.3875,-3.95571 -7.59375,-3.53389 2.59687,3.6745 z" - id="path6288" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 186.1395,433.70908 12.10312,-0.32808 -0.0375,-1.2467 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95611 -2.39062,3.8151 7.3875,-3.95571 -7.59375,-3.53389 2.59687,3.6745 z" - id="path6290" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 185.3895,480.20274 12.10312,-0.32808 -0.0375,-1.24671 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95612 -2.39062,3.8151 7.3875,-3.95571 -7.59375,-3.53389 2.59687,3.6745 z" - id="path6292" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 156.7395,480.20274 12.10312,-0.32808 -0.0375,-1.24671 -12.09375,0.32808 0.0281,1.24671 z m 12.08437,-0.95612 -2.39062,3.8151 7.3875,-3.95571 -7.59375,-3.53389 2.59687,3.6745 z" - id="path6294" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920-4" - width="16.515228" - height="21.21936" - x="578.84741" - y="448.11557" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 581.07075,450.6099 3.13125,0 c 0.35625,0 0.61875,0.28121 0.61875,0.63741 l 0,16.9102 -3.75,0 z" - id="path6296" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 584.82075,450.6099 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.74147 -3.9,0 z" - id="path6298" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 588.72075,450.6099 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.89145 -3.9,0 z" - id="path6300" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 563.2395,459.21498 12.1125,-0.33746 -0.0375,-1.25608 -12.09375,0.33746 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80573 7.3875,-3.95571 -7.59375,-3.52452 2.60625,3.6745 z" - id="path6302" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 593.9895,458.76504 12.1125,-0.33746 -0.0375,-1.25607 -12.09375,0.33745 0.0188,1.25608 z m 12.09375,-0.95612 -2.4,3.80573 7.3875,-3.95571 -7.59375,-3.52452 2.60625,3.6745 z" - id="path6304" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920" - width="16.515228" - height="21.21936" - x="366.35788" - y="448.70401" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 368.67075,450.90986 3.13125,0 c 0.34687,0 0.61875,0.28121 0.61875,0.62804 l 0,16.76959 -3.75,0 z" - id="path6306" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 372.42075,450.90986 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74147 -3.9,0 z" - id="path6308" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 376.32075,450.90986 3.25312,0 c 0.35625,0 0.64688,0.29996 0.64688,0.65616 l 0,16.74147 -3.9,0 z" - id="path6310" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 347.52075,457.79017 13.7625,0 0,1.24671 -13.7625,0 0,-1.24671 z m 13.7625,0.61867 -2.50312,-3.74949 7.5,3.74949 -7.5,3.74949 2.50312,-3.74949 z" - id="path6312" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.009375;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.009375,0.009375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 381.27075,457.79017 13.7625,0 0,1.24671 -13.7625,0 0,-1.24671 z m 13.7625,0.61867 -2.50313,-3.74949 7.5,3.74949 -7.5,3.74949 2.50313,-3.74949 z" - id="path6314" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 350.67075,657.73162 0,23.24683 84,0 0,-23.24683 -84,0 z" - id="path6316" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 350.67075,657.73162 84,0 0,23.24683 -84,0 z" - id="path6318" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - clip-path="url(#clipEmfPath14)" - d="m 213.675,329.87996 0,42.74416 84,0 0,-42.74416 -84,0 z" - id="path6320" - transform="translate(137.14575,354.09808)" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:0.60000002px;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:none" - d="m 350.82075,683.97804 84,0 0,42.74416 -84,0 z" - id="path6322" /> - <rect - style="fill:#ffffff;fill-opacity:1;stroke:none" - id="rect3920-48-2" - width="16.515228" - height="21.21936" - x="456.96756" - y="669.96301" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 459.57075,671.67972 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.89144 -3.9,0 z" - id="path6324" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 463.47075,671.67972 3.13125,0 c 0.35625,0 0.61875,0.28121 0.61875,0.63741 l 0,16.76021 -3.75,0 z" - id="path6326" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 467.22075,671.67972 3.2625,0 c 0.35625,0 0.6375,0.29996 0.6375,0.65616 l 0,16.89144 -3.9,0 z" - id="path6328" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.01875;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 440.0895,681.16592 14.475,-0.11248 -0.0188,-1.25608 -14.475,0.11248 0.0188,1.25608 z m 14.45625,-0.73115 -2.45625,3.76824 7.4625,-3.82448 -7.5375,-3.6745 2.53125,3.73074 z" - id="path6330" /> - <path - inkscape:connector-curvature="0" - style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.01875;stroke-linecap:butt;stroke-linejoin:round;stroke-opacity:1;stroke-dasharray:0.01875,0.0375;stroke-miterlimit:4;stroke-dashoffset:0" - d="m 471.12075,679.4599 13.48125,0 0,1.25608 -13.48125,0 0,-1.25608 z m 13.48125,0.61867 -2.5125,-3.74949 7.5,3.74949 -7.5,3.74949 2.5125,-3.74949 z" - id="path6332" /> - <path - inkscape:connector-curvature="0" - style="fill:#ffffff;fill-opacity:1;fill-rule:evenodd;stroke:none" - d="m 316.58325,533.69856 0,36.37003 2.4375,0 -4.875,4.87434 -4.875,-4.87434 2.4375,0 0,-36.37003 z" - id="path6334" /> - <path - inkscape:connector-curvature="0" - style="fill:none;stroke:#000000;stroke-width:1.20000005px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" - d="m 316.58325,533.69856 0,36.37003 2.4375,0 -4.875,4.87434 -4.875,-4.87434 2.4375,0 0,-36.37003 z" - id="path6336" /> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="251.32822" - y="385.15088" - id="text4458-5-6-8-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="251.32822" - y="385.15088" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5">Pipeline 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="458.32822" - y="385.15088" - id="text4458-5-6-8-7-4" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="458.32822" - y="385.15088" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan6590">Pipeline 2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="252.12177" - y="418.59369" - id="text4458-5-6-8-7-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="252.12177" - y="418.59369" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6">Flow Table 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="452.02521" - y="418.59369" - id="text4458-5-6-8-7-7-3" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="452.02521" - y="418.59369" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-6">Flow Table 2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="214.38213" - y="447.87283" - id="text4458-5-6-8-7-7-1" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="214.38213" - y="447.87283" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-5">Flow #</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="418.23212" - y="449.37262" - id="text4458-5-6-8-7-7-1-4" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="418.23212" - y="449.37262" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-5-2">Flow #</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="489.28775" - y="449.32428" - id="text4458-5-6-8-7-7-1-4-0" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="489.28775" - y="449.32428" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan6678">Actions 2</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="277.60748" - y="447.82449" - id="text4458-5-6-8-7-7-1-4-0-7" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="277.60748" - y="447.82449" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan6678-3">Actions 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="302.02823" - y="613.45862" - id="text4458-5-6-8-7-72" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="302.02823" - y="613.45862" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-60">Pipeline 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="293.5502" - y="644.90143" - id="text4458-5-6-8-7-7-16" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="293.5502" - y="644.90143" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-57">Flow Table 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="220.08212" - y="673.29205" - id="text4458-5-6-8-7-7-1-5" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="220.08212" - y="673.29205" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan4668-2-5-6-5-4">Flow #</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="283.30746" - y="673.24371" - id="text4458-5-6-8-7-7-1-4-0-7-1" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="283.30746" - y="673.24371" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan6678-3-2">Actions 1</tspan></text> - <text - xml:space="preserve" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans;-inkscape-font-specification:Sans" - x="370.33774" - y="673.24371" - id="text4458-5-6-8-7-7-1-4-0-7-1-0" - sodipodi:linespacing="125%"><tspan - sodipodi:role="line" - x="370.33774" - y="673.24371" - style="font-size:11px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:125%;writing-mode:lr-tb;text-anchor:start;font-family:Sans;-inkscape-font-specification:Sans" - id="tspan6678-3-2-0">Actions 2</tspan></text> - </g> -</svg> diff --git a/doc/guides/sample_app_ug/img/master_slave_proc.png b/doc/guides/sample_app_ug/img/master_slave_proc.png Binary files differdeleted file mode 100644 index a0065e85..00000000 --- a/doc/guides/sample_app_ug/img/master_slave_proc.png +++ /dev/null diff --git a/doc/guides/sample_app_ug/img/slave_proc_recov.png b/doc/guides/sample_app_ug/img/slave_proc_recov.png Binary files differdeleted file mode 100644 index 62f1f088..00000000 --- a/doc/guides/sample_app_ug/img/slave_proc_recov.png +++ /dev/null diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst index e87afda8..5bedf4f6 100644 --- a/doc/guides/sample_app_ug/index.rst +++ b/doc/guides/sample_app_ug/index.rst @@ -44,6 +44,7 @@ Sample Applications User Guides vmdq_dcb_forwarding vhost vhost_scsi + vhost_crypto netmap_compatibility ip_pipeline test_pipeline @@ -82,10 +83,6 @@ Sample Applications User Guides :numref:`figure_client_svr_sym_multi_proc_app` :ref:`figure_client_svr_sym_multi_proc_app` -:numref:`figure_master_slave_proc` :ref:`figure_master_slave_proc` - -:numref:`figure_slave_proc_recov` :ref:`figure_slave_proc_recov` - :numref:`figure_qos_sched_app_arch` :ref:`figure_qos_sched_app_arch` :numref:`figure_pipeline_overview` :ref:`figure_pipeline_overview` diff --git a/doc/guides/sample_app_ug/ip_pipeline.rst b/doc/guides/sample_app_ug/ip_pipeline.rst index 69af6b8d..b75509a0 100644 --- a/doc/guides/sample_app_ug/ip_pipeline.rst +++ b/doc/guides/sample_app_ug/ip_pipeline.rst @@ -1,5 +1,5 @@ .. SPDX-License-Identifier: BSD-3-Clause - Copyright(c) 2015-2016 Intel Corporation. + Copyright(c) 2015-2018 Intel Corporation. Internet Protocol (IP) Pipeline Application =========================================== @@ -8,1163 +8,523 @@ Application overview -------------------- The *Internet Protocol (IP) Pipeline* application is intended to be a vehicle for rapid development of packet processing -applications running on multi-core CPUs. +applications on multi-core CPUs. -The application provides a library of reusable functional blocks called pipelines. -These pipelines can be seen as prefabricated blocks that can be instantiated and inter-connected through packet queues +Following OpenFlow and P4 design principles, the application can be used to create functional blocks called pipelines out +of input/output ports, tables and actions in a modular way. Multiple pipelines can be inter-connected through packet queues to create complete applications (super-pipelines). -Pipelines are created and inter-connected through the application configuration file. -By using different configuration files, different applications are effectively created, therefore this application -can be seen as an application generator. -The configuration of each pipeline can be updated at run-time through the application Command Line Interface (CLI). +The pipelines are mapped to application threads, with each pipeline executed by a single thread and each thread able to run +one or several pipelines. The possibilities of creating pipelines out of ports, tables and actions, connecting multiple +pipelines together and mapping the pipelines to execution threads are endless, therefore this application can be seen as +a true application generator. -Main application components are: +Pipelines are created and managed through Command Line Interface (CLI): -**A Library of reusable pipelines** + * Any standard TCP client (e.g. telnet, netcat, custom script, etc) is typically able to connect to the application, send + commands through the network and wait for the response before pushing the next command. - * Each pipeline represents a functional block, e.g. flow classification, firewall, routing, master, etc. - - * Each pipeline type can be instantiated several times in the same application, which each instance configured - separately and mapped to a single CPU core. - Each CPU core can run one or several pipeline instances, which can be of same or different type. - - * Pipeline instances are inter-connected through packet queues (for packet processing) and message queues - (for run-time configuration). - - * Pipelines are implemented using DPDK Packet Framework. - - * More pipeline types can always be built and added to the existing pipeline types. - -**The Configuration file** - - * The configuration file defines the application structure. - By using different configuration files, different applications are created. - - * All the application resources are created and configured through the application configuration file: - pipeline instances, buffer pools, links (i.e. network interfaces), hardware device RX/TX queues, - software queues, traffic manager devices, EAL startup arguments, etc. - - * The configuration file syntax is “define by reference”, meaning that resources are defined as they are referenced. - First time a resource name is detected, it is registered with default parameters. - Optionally, the resource parameters can be further refined through a configuration file section dedicated to - that resource. - - * Command Line Interface (CLI) - -**Global CLI commands: link configuration, etc.** - - * Common pipeline CLI commands: ping (keep-alive), statistics, etc. - - * Pipeline type specific CLI commands: used to configure instances of specific pipeline type. - These commands are registered with the application when the pipeline type is registered. - For example, the commands for routing pipeline instances include: route add, route delete, route list, etc. - - * CLI commands can be grouped into scripts that can be invoked at initialization and at runtime. - - -Design goals ------------- - - -Rapid development -~~~~~~~~~~~~~~~~~ - -This application enables rapid development through quick connectivity of standard components called pipelines. -These components are built using DPDK Packet Framework and encapsulate packet processing features at different levels: -ports, tables, actions, pipelines and complete applications. - -Pipeline instances are instantiated, configured and inter-connected through low complexity configuration files loaded -during application initialization. -Each pipeline instance is mapped to a single CPU core, with each CPU core able to run one or multiple pipeline -instances of same or different types. By loading a different configuration file, a different application is -effectively started. - - -Flexibility -~~~~~~~~~~~ - -Each packet processing application is typically represented as a chain of functional stages which is often called -the functional pipeline of the application. -These stages are mapped to CPU cores to create chains of CPU cores (pipeline model), clusters of CPU cores -(run-to-completion model) or chains of clusters of CPU cores (hybrid model). - -This application allows all the above programming models. -By applying changes to the configuration file, the application provides the flexibility to reshuffle its -building blocks in different ways until the configuration providing the best performance is identified. - - -Move pipelines around -^^^^^^^^^^^^^^^^^^^^^ - -The mapping of pipeline instances to CPU cores can be reshuffled through the configuration file. -One or several pipeline instances can be mapped to the same CPU core. - -.. _figure_ip_pipelines_1: - -.. figure:: img/ip_pipelines_1.* - - Example of moving pipeline instances across different CPU cores - - -Move tables around -^^^^^^^^^^^^^^^^^^ - -There is some degree of flexibility for moving tables from one pipeline instance to another. -Based on the configuration arguments passed to each pipeline instance in the configuration file, specific tables -can be enabled or disabled. -This way, a specific table can be “moved” from pipeline instance A to pipeline instance B by simply disabling its -associated functionality for pipeline instance A while enabling it for pipeline instance B. - -Due to requirement to have simple syntax for the configuration file, moving tables across different pipeline -instances is not as flexible as the mapping of pipeline instances to CPU cores, or mapping actions to pipeline tables. -Complete flexibility in moving tables from one pipeline to another could be achieved through a complex pipeline -description language that would detail the structural elements of the pipeline (ports, tables and actions) and -their connectivity, resulting in complex syntax for the configuration file, which is not acceptable. -Good configuration file readability through simple syntax is preferred. - -*Example*: the IP routing pipeline can run the routing function only (with ARP function run by a different -pipeline instance), or it can run both the routing and ARP functions as part of the same pipeline instance. - - -.. _figure_ip_pipelines_2: - -.. figure:: img/ip_pipelines_2.* - - Example of moving tables across different pipeline instances - - -Move actions around -^^^^^^^^^^^^^^^^^^^ - -When it makes sense, packet processing actions can be moved from one pipeline instance to another. -Based on the configuration arguments passed to each pipeline instance in the configuration file, specific actions -can be enabled or disabled. -This way, a specific action can be "moved" from pipeline instance A to pipeline instance B by simply disabling its -associated functionality for pipeline instance A while enabling it for pipeline instance B. - -*Example*: The flow actions of accounting, traffic metering, application identification, NAT, etc can be run as part -of the flow classification pipeline instance or split across several flow actions pipeline instances, depending on -the number of flow instances and their compute requirements. - - -.. _figure_ip_pipelines_3: - -.. figure:: img/ip_pipelines_3.* - - Example of moving actions across different tables and pipeline instances - - -Performance -~~~~~~~~~~~ - -Performance of the application is the highest priority requirement. -Flexibility is not provided at the expense of performance. - -The purpose of flexibility is to provide an incremental development methodology that allows monitoring the -performance evolution: - -* Apply incremental changes in the configuration (e.g. mapping on pipeline instances to CPU cores) - in order to identify the configuration providing the best performance for a given application; - -* Add more processing incrementally (e.g. by enabling more actions for specific pipeline instances) until - the application is feature complete while checking the performance impact at each step. - - -Debug capabilities -~~~~~~~~~~~~~~~~~~ - -The application provides a significant set of debug capabilities: - -* Command Line Interface (CLI) support for statistics polling: pipeline instance ping (keep-alive checks), - pipeline instance statistics per input port/output port/table, link statistics, etc; - -* Logging: Turn on/off application log messages based on priority level; + * All the application objects are created and managed through CLI commands: + * 'Primitive' objects used to create pipeline ports: memory pools, links (i.e. network interfaces), SW queues, traffic managers, etc. + * Action profiles: used to define the actions to be executed by pipeline input/output ports and tables. + * Pipeline components: input/output ports, tables, pipelines, mapping of pipelines to execution threads. Running the application ----------------------- The application startup command line is:: - ip_pipeline [-f CONFIG_FILE] [-s SCRIPT_FILE] -p PORT_MASK [-l LOG_LEVEL] + ip_pipeline [EAL_ARGS] -- [-s SCRIPT_FILE] [-h HOST] [-p PORT] The application startup arguments are: -``-f CONFIG_FILE`` - - * Optional: Yes - - * Default: ``./config/ip_pipeline.cfg`` - - * Argument: Path to the configuration file to be loaded by the application. - Please refer to the :ref:`ip_pipeline_configuration_file` for details on how to write the configuration file. - ``-s SCRIPT_FILE`` * Optional: Yes * Default: Not present - * Argument: Path to the CLI script file to be run by the master pipeline at application startup. - No CLI script file will be run at startup if this argument is not present. - -``-p PORT_MASK`` - - * Optional: No + * Argument: Path to the CLI script file to be run at application startup. + No CLI script file will run at startup if this argument is not present. - * Default: N/A - - * Argument: Hexadecimal mask of NIC port IDs to be used by the application. - First port enabled in this mask will be referenced as LINK0 as part of the application configuration file, - next port as LINK1, etc. - -``-l LOG_LEVEL`` +``-h HOST`` * Optional: Yes - * Default: 1 (High priority) - - * Argument: Log level to determine which application messages are to be printed to standard output. - Available log levels are: 0 (None), 1 (High priority), 2 (Low priority). - Only application messages whose priority is higher than or equal to the application log level will be printed. - - -Application stages ------------------- - - -Configuration -~~~~~~~~~~~~~ - -During this stage, the application configuration file is parsed and its content is loaded into the application data -structures. -In case of any configuration file parse error, an error message is displayed and the application is terminated. -Please refer to the :ref:`ip_pipeline_configuration_file` for a description of the application configuration file format. - - -Configuration checking -~~~~~~~~~~~~~~~~~~~~~~ - -In the absence of any parse errors, the loaded content of application data structures is checked for overall consistency. -In case of any configuration check error, an error message is displayed and the application is terminated. - - -Initialization -~~~~~~~~~~~~~~ - -During this stage, the application resources are initialized and the handles to access them are saved into the -application data structures. -In case of any initialization error, an error message is displayed and the application is terminated. - -The typical resources to be initialized are: pipeline instances, buffer pools, links (i.e. network interfaces), -hardware device RX/TX queues, software queues, traffic management devices, etc. - - -.. _ip_pipeline_runtime: + * Default: ``0.0.0.0`` -Run-time -~~~~~~~~ + * Argument: IP Address of the host running ip pipeline application to be used by + remote TCP based client (telnet, netcat, etc.) for connection. -Each CPU core runs the pipeline instances assigned to it in time sharing mode and in round robin order: +``-p PORT`` -1. *Packet processing task*: The pipeline run-time code is typically a packet *processing* task built on top of - DPDK Packet Framework rte_pipeline library, which reads bursts of packets from the pipeline input ports, - performs table lookups and executes the identified actions for all tables in the pipeline, with packet - eventually written to pipeline output ports or dropped. + * Optional: Yes -2. *Message handling task*: Each CPU core will also periodically execute the *message handling* code of each - of the pipelines mapped to it. - The pipeline message handling code is processing the messages that are pending in the pipeline input message - queues, which are typically sent by the master CPU core for the on-the-fly pipeline configuration: check - that pipeline is still alive (ping), add/delete entries in the pipeline tables, get statistics, etc. - The frequency of executing the message handling code is usually much smaller than the frequency of executing - the packet processing work. + * Default: ``8086`` -Please refer to the :ref:`ip_pipeline_pipeline_section` for more details about the application pipeline module encapsulation. + * Argument: TCP port number at which the ip pipeline is running. + This port number should be used by remote TCP client (such as telnet, netcat, etc.) to connect to host application. -.. _ip_pipeline_configuration_file: +Refer to *DPDK Getting Started Guide* for general information on running applications and the Environment Abstraction Layer (EAL) options. -Configuration file syntax -------------------------- +The following is an example command to run ip pipeline application configured for layer 2 forwarding: -Syntax overview -~~~~~~~~~~~~~~~ +.. code-block:: console -The syntax of the configuration file is designed to be simple, which favors readability. -The configuration file is parsed using the DPDK library librte_cfgfile, which supports simple -`INI file format <http://en.wikipedia.org/wiki/INI_file>`__ for configuration files. + $ ./build/ip_pipeline -c 0x3 -- -s examples/route_ecmp.cli -As result, the configuration file is split into several sections, with each section containing one or more entries. -The scope of each entry is its section, and each entry specifies a variable that is assigned a specific value. -Any text after the ``;`` character is considered a comment and is therefore ignored. +The application should start successfully and display as follows: -The following are application specific: number of sections, name of each section, number of entries of each section, -name of the variables used for each section entry, the value format (e.g. signed/unsigned integer, string, etc) -and range of each section entry variable. +.. code-block:: console -Generic example of configuration file section: + EAL: Detected 40 lcore(s) + EAL: Detected 2 NUMA nodes + EAL: Multi-process socket /var/run/.rte_unix + EAL: Probing VFIO support... + EAL: PCI device 0000:02:00.0 on NUMA socket 0 + EAL: probe driver: 8086:10fb net_ixgbe + ... -.. code-block:: ini +To run remote client (e.g. telnet) to communicate with the ip pipeline application: - [<section_name>] +.. code-block:: console - <variable_name_1> = <value_1> + $ telnet 127.0.0.1 8086 - ; ... +When running a telnet client as above, command prompt is displayed: - <variable_name_N> = <value_N> +.. code-block:: console + Trying 127.0.0.1... + Connected to 127.0.0.1. + Escape character is '^]'. -Application resources present in the configuration file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Welcome to IP Pipeline! -.. _table_ip_pipelines_resource_name: + pipeline> -.. tabularcolumns:: |p{4cm}|p{6cm}|p{6cm}| +Once application and telnet client start running, messages can be sent from client to application. +At any stage, telnet client can be terminated using the quit command. -.. table:: Application resource names in the configuration file - +----------------------------+-----------------------------+-------------------------------------------------+ - | Resource type | Format | Examples | - +============================+=============================+=================================================+ - | Pipeline | ``PIPELINE<ID>`` | ``PIPELINE0``, ``PIPELINE1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Mempool | ``MEMPOOL<ID>`` | ``MEMPOOL0``, ``MEMPOOL1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Link (network interface) | ``LINK<ID>`` | ``LINK0``, ``LINK1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Link RX queue | ``RXQ<LINK_ID>.<QUEUE_ID>`` | ``RXQ0.0``, ``RXQ1.5`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Link TX queue | ``TXQ<LINK_ID>.<QUEUE_ID>`` | ``TXQ0.0``, ``TXQ1.5`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Software queue | ``SWQ<ID>`` | ``SWQ0``, ``SWQ1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Traffic Manager | ``TM<LINK_ID>`` | ``TM0``, ``TM1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | KNI (kernel NIC interface) | ``KNI<LINK_ID>`` | ``KNI0``, ``KNI1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Source | ``SOURCE<ID>`` | ``SOURCE0``, ``SOURCE1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Sink | ``SINK<ID>`` | ``SINK0``, ``SINK1`` | - +----------------------------+-----------------------------+-------------------------------------------------+ - | Message queue | ``MSGQ<ID>`` | ``MSGQ0``, ``MSGQ1``, | - | | ``MSGQ-REQ-PIPELINE<ID>`` | ``MSGQ-REQ-PIPELINE2``, ``MSGQ-RSP-PIPELINE2,`` | - | | ``MSGQ-RSP-PIPELINE<ID>`` | ``MSGQ-REQ-CORE-s0c1``, ``MSGQ-RSP-CORE-s0c1`` | - | | ``MSGQ-REQ-CORE-<CORE_ID>`` | | - | | ``MSGQ-RSP-CORE-<CORE_ID>`` | | - +----------------------------+-----------------------------+-------------------------------------------------+ - -``LINK`` instances are created implicitly based on the ``PORT_MASK`` application startup argument. -``LINK0`` is the first port enabled in the ``PORT_MASK``, port 1 is the next one, etc. -The LINK ID is different than the DPDK PMD-level NIC port ID, which is the actual position in the bitmask mentioned above. -For example, if bit 5 is the first bit set in the bitmask, then ``LINK0`` is having the PMD ID of 5. -This mechanism creates a contiguous LINK ID space and isolates the configuration file against changes in the board -PCIe slots where NICs are plugged in. - -``RXQ``, ``TXQ``, ``TM`` and ``KNI`` instances have the LINK ID as part of their name. -For example, ``RXQ2.1``, ``TXQ2.1`` and ``TM2`` are all associated with ``LINK2``. - - -Rules to parse the configuration file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The main rules used to parse the configuration file are: - -1. Application resource name determines the type of resource based on the name prefix. - - *Example*: all software queues need to start with ``SWQ`` prefix, so ``SWQ0`` and ``SWQ5`` are valid software - queue names. - -2. An application resource is defined by creating a configuration file section with its name. - The configuration file section allows fine tuning on any of the resource parameters. - Some resource parameters are mandatory, in which case it is required to have them specified as part of the - section, while some others are optional, in which case they get assigned their default value when not present. - - *Example*: section ``SWQ0`` defines a software queue named SWQ0, whose parameters are detailed as part of this section. - -3. An application resource can also be defined by referencing it. - Referencing a resource takes place by simply using its name as part of the value assigned to a variable in any - configuration file section. - In this case, the resource is registered with all its parameters having their default values. - Optionally, a section with the resource name can be added to the configuration file to fine tune some or all - of the resource parameters. - - *Example*: in section ``PIPELINE3``, variable ``pktq_in`` includes ``SWQ5`` as part of its list, which results - in defining a software queue named ``SWQ5``; when there is no ``SWQ5`` section present in the configuration file, - ``SWQ5`` gets registered with default parameters. - - -.. _ip_pipeline_pipeline_section: - -PIPELINE section -~~~~~~~~~~~~~~~~ - -.. _table_ip_pipelines_pipeline_section_1: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}| - -.. table:: Configuration file PIPELINE section (1/2) - - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | Section | Description | Optional | Range | Default value | - +===============+===========================================================+===============+========================+================+ - | type | Pipeline type. Defines the functionality to be | NO | See "List | N/A | - | | executed. | | of pipeline types" | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | core | CPU core to run the current pipeline. | YES | See "CPU Core | CPU socket 0, | - | | | | notation" | core 0, | - | | | | | hyper-thread 0 | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | pktq_in | Packet queues to serve as input ports for the | YES | List of input | Empty list | - | | current pipeline instance. The acceptable packet | | packet queue IDs | | - | | queue types are: ``RXQ``, ``SWQ``, ``TM`` and ``SOURCE``. | | | | - | | First device in this list is used as pipeline input port | | | | - | | 0, second as pipeline input port 1, etc. | | | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | pktq_out | Packet queues to serve as output ports for the | YES | List of output | Empty list | - | | current pipeline instance. The acceptable packet | | packet queue IDs. | | - | | queue types are: ``TXQ``, ``SWQ``, ``TM`` and ``SINK``. | | | | - | | First device in this list is used as pipeline output | | | | - | | port 0, second as pipeline output port 1, etc. | | | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - -.. _table_ip_pipelines_pipeline_section_2: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}| - -.. table:: Configuration file PIPELINE section (2/2) - - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | Section | Description | Optional | Range | Default value | - +===============+===========================================================+===============+========================+================+ - | msgq_in | Input message queues. These queues contain | YES | List of message | Empty list | - | | request messages that need to be handled by the | | queue IDs | | - | | current pipeline instance. The type and format of | | | | - | | request messages is defined by the pipeline type. | | | | - | | For each pipeline instance, there is an input | | | | - | | message queue defined implicitly, whose name is: | | | | - | | ``MSGQ-REQ-<PIPELINE_ID>``. This message queue | | | | - | | should not be mentioned as part of msgq_in list. | | | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | msgq_out | Output message queues. These queues are used by | YES | List of message | Empty list | - | | the current pipeline instance to write response | | queue IDs | | - | | messages as result of request messages being | | | | - | | handled. The type and format of response | | | | - | | messages is defined by the pipeline type. | | | | - | | For each pipeline instance, there is an output | | | | - | | message queue defined implicitly, whose name is: | | | | - | | ``MSGQ-RSP-<PIPELINE_ID>``. This message queue | | | | - | | should not be mentioned as part of msgq_out list. | | | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | timer_period | Time period, measured in milliseconds, | YES | milliseconds | 1 ms | - | | for handling the input message queues. | | | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - | <any other> | Arguments to be passed to the current pipeline | Depends on | Depends on | Depends on | - | | instance. Format of the arguments, their type, | pipeline type | pipeline type | pipeline type | - | | whether each argument is optional or mandatory | | | | - | | and its default value (when optional) are defined | | | | - | | by the pipeline type. | | | | - | | The value of the arguments is applicable to the | | | | - | | current pipeline instance only. | | | | - +---------------+-----------------------------------------------------------+---------------+------------------------+----------------+ - - -CPU core notation -^^^^^^^^^^^^^^^^^ - -The CPU Core notation is:: - - <CPU core> ::= [s|S<CPU socket ID>][c|C]<CPU core ID>[h|H] - -For example:: - - CPU socket 0, core 0, hyper-thread 0: 0, c0, s0c0 - - CPU socket 0, core 0, hyper-thread 1: 0h, c0h, s0c0h - - CPU socket 3, core 9, hyper-thread 1: s3c9h - - -MEMPOOL section -~~~~~~~~~~~~~~~ +Application stages +------------------ -.. _table_ip_pipelines_mempool_section: - -.. tabularcolumns:: |p{2.5cm}|p{6cm}|p{1.5cm}|p{1.5cm}|p{3cm}| - -.. table:: Configuration file MEMPOOL section - - +---------------+-----------------------------------------------+----------+----------+---------------------------+ - | Section | Description | Optional | Type | Default value | - +===============+===============================================+==========+==========+===========================+ - | buffer_size | Buffer size (in bytes) for the current | YES | uint32_t | 2048 | - | | buffer pool. | | | + sizeof(struct rte_mbuf) | - | | | | | + HEADROOM | - +---------------+-----------------------------------------------+----------+----------+---------------------------+ - | pool_size | Number of buffers in the current buffer pool. | YES | uint32_t | 32K | - +---------------+-----------------------------------------------+----------+----------+---------------------------+ - | cache_size | Per CPU thread cache size (in number of | YES | uint32_t | 256 | - | | buffers) for the current buffer pool. | | | | - +---------------+-----------------------------------------------+----------+----------+---------------------------+ - | cpu | CPU socket ID where to allocate memory for | YES | uint32_t | 0 | - | | the current buffer pool. | | | | - +---------------+-----------------------------------------------+----------+----------+---------------------------+ - - -LINK section -~~~~~~~~~~~~ - -.. _table_ip_pipelines_link_section: - -.. tabularcolumns:: |p{3cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{2cm}| - -.. table:: Configuration file LINK section - - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | Section entry | Description | Optional | Type | Default value | - +=================+==============================================+==========+==========+===================+ - | arp_q | NIC RX queue where ARP packets should | YES | 0 .. 127 | 0 (default queue) | - | | be filtered. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | tcp_syn_local_q | NIC RX queue where TCP packets with SYN | YES | 0 .. 127 | 0 (default queue) | - | | flag should be filtered. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | ip_local_q | NIC RX queue where IP packets with local | YES | 0 .. 127 | 0 (default queue) | - | | destination should be filtered. | | | | - | | When TCP, UDP and SCTP local queues are | | | | - | | defined, they take higher priority than this | | | | - | | queue. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | tcp_local_q | NIC RX queue where TCP packets with local | YES | 0 .. 127 | 0 (default queue) | - | | destination should be filtered. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | udp_local_q | NIC RX queue where TCP packets with local | YES | 0 .. 127 | 0 (default queue) | - | | destination should be filtered. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | sctp_local_q | NIC RX queue where TCP packets with local | YES | 0 .. 127 | 0 (default queue) | - | | destination should be filtered. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - | promisc | Indicates whether current link should be | YES | YES/NO | YES | - | | started in promiscuous mode. | | | | - +-----------------+----------------------------------------------+----------+----------+-------------------+ - - -RXQ section -~~~~~~~~~~~ - -.. _table_ip_pipelines_rxq_section: - -.. tabularcolumns:: |p{3cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{2cm}| - -.. table:: Configuration file RXQ section - - +---------------+--------------------------------------------+----------+----------+---------------+ - | Section | Description | Optional | Type | Default value | - +===============+============================================+==========+==========+===============+ - | mempool | Mempool to use for buffer allocation for | YES | uint32_t | MEMPOOL0 | - | | current NIC RX queue. The mempool ID has | | | | - | | to be associated with a valid instance | | | | - | | defined in the mempool entry of the global | | | | - | | section. | | | | - +---------------+--------------------------------------------+----------+----------+---------------+ - | Size | NIC RX queue size (number of descriptors) | YES | uint32_t | 128 | - +---------------+--------------------------------------------+----------+----------+---------------+ - | burst | Read burst size (number of descriptors) | YES | uint32_t | 32 | - +---------------+--------------------------------------------+----------+----------+---------------+ - - -TXQ section -~~~~~~~~~~~ - -.. _table_ip_pipelines_txq_section: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{2cm}|p{1.5cm}| - -.. table:: Configuration file TXQ section - - +---------------+----------------------------------------------+----------+------------------+---------------+ - | Section | Description | Optional | Type | Default value | - +===============+==============================================+==========+==================+===============+ - | size | NIC TX queue size (number of descriptors) | YES | uint32_t | 512 | - | | | | power of 2 | | - | | | | > 0 | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | burst | Write burst size (number of descriptors) | YES | uint32_t | 32 | - | | | | power of 2 | | - | | | | 0 < burst < size | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | dropless | When dropless is set to NO, packets can be | YES | YES/NO | NO | - | | dropped if not enough free slots are | | | | - | | currently available in the queue, so the | | | | - | | write operation to the queue is non- | | | | - | | blocking. | | | | - | | When dropless is set to YES, packets cannot | | | | - | | be dropped if not enough free slots are | | | | - | | currently available in the queue, so the | | | | - | | write operation to the queue is blocking, as | | | | - | | the write operation is retried until enough | | | | - | | free slots become available and all the | | | | - | | packets are successfully written to the | | | | - | | queue. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | n_retries | Number of retries. Valid only when dropless | YES | uint32_t | 0 | - | | is set to YES. When set to 0, it indicates | | | | - | | unlimited number of retries. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - - -SWQ section -~~~~~~~~~~~ - -.. _table_ip_pipelines_swq_section: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}| - -.. table:: Configuration file SWQ section - - +---------------+----------------------------------------------+----------+------------------+---------------+ - | Section | Description | Optional | Type | Default value | - +===============+==============================================+==========+==================+===============+ - | size | Queue size (number of packets) | YES | uint32_t | 256 | - | | | | power of 2 | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | burst_read | Read burst size (number of packets) | YES | uint32_t | 32 | - | | | | power of 2 | | - | | | | 0 < burst < size | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | burst_write | Write burst size (number of packets) | YES | uint32_t | 32 | - | | | | power of 2 | | - | | | | 0 < burst < size | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | dropless | When dropless is set to NO, packets can be | YES | YES/NO | NO | - | | dropped if not enough free slots are | | | | - | | currently available in the queue, so the | | | | - | | write operation to the queue is non- | | | | - | | blocking. | | | | - | | When dropless is set to YES, packets cannot | | | | - | | be dropped if not enough free slots are | | | | - | | currently available in the queue, so the | | | | - | | write operation to the queue is blocking, as | | | | - | | the write operation is retried until enough | | | | - | | free slots become available and all the | | | | - | | packets are successfully written to the | | | | - | | queue. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | n_retries | Number of retries. Valid only when dropless | YES | uint32_t | 0 | - | | is set to YES. When set to 0, it indicates | | | | - | | unlimited number of retries. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | cpu | CPU socket ID where to allocate memory | YES | uint32_t | 0 | - | | for this SWQ. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - - -TM section -~~~~~~~~~~ - -.. _table_ip_pipelines_tm_section: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}| - -.. table:: Configuration file TM section - - +---------------+---------------------------------------------+----------+----------+---------------+ - | Section | Description | Optional | Type | Default value | - +===============+=============================================+==========+==========+===============+ - | Cfg | File name to parse for the TM configuration | YES | string | tm_profile | - | | to be applied. The syntax of this file is | | | | - | | described in the examples/qos_sched DPDK | | | | - | | application documentation. | | | | - +---------------+---------------------------------------------+----------+----------+---------------+ - | burst_read | Read burst size (number of packets) | YES | uint32_t | 64 | - +---------------+---------------------------------------------+----------+----------+---------------+ - | burst_write | Write burst size (number of packets) | YES | uint32_t | 32 | - +---------------+---------------------------------------------+----------+----------+---------------+ - - -KNI section -~~~~~~~~~~~ - -.. _table_ip_pipelines_kni_section: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}| - -.. table:: Configuration file KNI section - - +---------------+----------------------------------------------+----------+------------------+---------------+ - | Section | Description | Optional | Type | Default value | - +===============+==============================================+==========+==================+===============+ - | core | CPU core to run the KNI kernel thread. | YES | See "CPU Core | Not set | - | | When core config is set, the KNI kernel | | notation" | | - | | thread will be bound to the particular core. | | | | - | | When core config is not set, the KNI kernel | | | | - | | thread will be scheduled by the OS. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | mempool | Mempool to use for buffer allocation for | YES | uint32_t | MEMPOOL0 | - | | current KNI port. The mempool ID has | | | | - | | to be associated with a valid instance | | | | - | | defined in the mempool entry of the global | | | | - | | section. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | burst_read | Read burst size (number of packets) | YES | uint32_t | 32 | - | | | | power of 2 | | - | | | | 0 < burst < size | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | burst_write | Write burst size (number of packets) | YES | uint32_t | 32 | - | | | | power of 2 | | - | | | | 0 < burst < size | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | dropless | When dropless is set to NO, packets can be | YES | YES/NO | NO | - | | dropped if not enough free slots are | | | | - | | currently available in the queue, so the | | | | - | | write operation to the queue is non- | | | | - | | blocking. | | | | - | | When dropless is set to YES, packets cannot | | | | - | | be dropped if not enough free slots are | | | | - | | currently available in the queue, so the | | | | - | | write operation to the queue is blocking, as | | | | - | | the write operation is retried until enough | | | | - | | free slots become available and all the | | | | - | | packets are successfully written to the | | | | - | | queue. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - | n_retries | Number of retries. Valid only when dropless | YES | uint64_t | 0 | - | | is set to YES. When set to 0, it indicates | | | | - | | unlimited number of retries. | | | | - +---------------+----------------------------------------------+----------+------------------+---------------+ - - -SOURCE section +Initialization ~~~~~~~~~~~~~~ -.. _table_ip_pipelines_source_section: - -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{2cm}| - -.. table:: Configuration file SOURCE section - - +---------------+---------------------------------------+----------+----------+---------------+ - | Section | Description | Optional | Type | Default value | - +===============+=======================================+==========+==========+===============+ - | Mempool | Mempool to use for buffer allocation. | YES | uint32_t | MEMPOOL0 | - +---------------+---------------------------------------+----------+----------+---------------+ - | Burst | Read burst size (number of packets) | | uint32_t | 32 | - +---------------+---------------------------------------+----------+----------+---------------+ - - -SINK section -~~~~~~~~~~~~ +During this stage, EAL layer is initialised and application specific arguments are parsed. Furthermore, the data strcutures +(i.e. linked lists) for application objects are initialized. In case of any initialization error, an error message +is displayed and the application is terminated. -Currently, there are no parameters to be passed to a sink device, so -SINK section is not allowed. +.. _ip_pipeline_runtime: -MSGQ section -~~~~~~~~~~~~ +Run-time +~~~~~~~~ -.. _table_ip_pipelines_msgq_section: +The master thread is creating and managing all the application objects based on CLI input. -.. tabularcolumns:: |p{2.5cm}|p{7cm}|p{1.5cm}|p{1.5cm}|p{1.5cm}| +Each data plane thread runs one or several pipelines previously assigned to it in round-robin order. Each data plane thread +executes two tasks in time-sharing mode: -.. table:: Configuration file MSGQ section +1. *Packet processing task*: Process bursts of input packets read from the pipeline input ports. - +---------+--------------------------------------------+----------+------------+---------------+ - | Section | Description | Optional | Type | Default value | - +=========+============================================+==========+============+===============+ - | size | Queue size (number of packets) | YES | uint32_t | 64 | - | | | | != 0 | | - | | | | power of 2 | | - +---------+--------------------------------------------+----------+------------+---------------+ - | cpu | CPU socket ID where to allocate memory for | YES | uint32_t | 0 | - | | the current queue. | | | | - +---------+--------------------------------------------+----------+------------+---------------+ +2. *Message handling task*: Periodically, the data plane thread pauses the packet processing task and polls for request + messages send by the master thread. Examples: add/remove pipeline to/from current data plane thread, add/delete rules + to/from given table of a specific pipeline owned by the current data plane thread, read statistics, etc. +Examples +-------- -EAL section -~~~~~~~~~~~ +.. _table_examples: -The application generates the EAL parameters rather than reading them from the command line. +.. tabularcolumns:: |p{3cm}|p{5cm}|p{4cm}|p{4cm}| -The CPU core mask parameter is generated based on the core entry of all PIPELINE sections. -All the other EAL parameters can be set from this section of the application configuration file. +.. table:: Pipeline examples provided with the application + + +-----------------------+----------------------+----------------+------------------------------------+ + | Name | Table(s) | Actions | Messages | + +=======================+======================+================+====================================+ + | L2fwd | Stub | Forward | 1. Mempool create | + | | | | 2. Link create | + | Note: Implemented | | | 3. Pipeline create | + | using pipeline with | | | 4. Pipeline port in/out | + | a simple pass-through | | | 5. Pipeline table | + | connection between | | | 6. Pipeline port in table | + | input and output | | | 7. Pipeline enable | + | ports. | | | 8. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | Flow classification | Exact match | Forward | 1. Mempool create | + | | | | 2. Link create | + | | * Key = byte array | | 3. Pipeline create | + | | (16 bytes) | | 4. Pipeline port in/out | + | | * Offset = 278 | | 5. Pipeline table | + | | * Table size = 64K | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add default | + | | | | 9. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | KNI | Stub | Forward | 1. Mempool create | + | | | | 2. Link create | + | | | | 3. Pipeline create | + | | | | 4. Pipeline port in/out | + | | | | 5. Pipeline table | + | | | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | Firewall | ACL | Allow/Drop | 1. Mempool create | + | | | | 2. Link create | + | | * Key = n-tuple | | 3. Pipeline create | + | | * Offset = 270 | | 4. Pipeline port in/out | + | | * Table size = 4K | | 5. Pipeline table | + | | | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add default | + | | | | 9. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | IP routing | LPM (IPv4) | Forward | 1. Mempool Create | + | | | | 2. Link create | + | | * Key = IP dest addr | | 3. Pipeline creat | + | | * Offset = 286 | | 4. Pipeline port in/out | + | | * Table size = 4K | | 5. Pipeline table | + | | | | 6. Pipeline port in table | + | | | | 7. Pipeline enable | + | | | | 8. Pipeline table rule add default | + | | | | 9. Pipeline table rule add | + +-----------------------+----------------------+----------------+------------------------------------+ + | Equal-cost multi-path | LPM (IPv4) | Forward, | 1. Mempool Create | + | routing (ECMP) | | load balance, | 2. Link create | + | | * Key = IP dest addr | encap ether | 3. Pipeline create | + | | * Offset = 286 | | 4. Pipeline port in/out | + | | * Table size = 4K | | 5. Pipeline table (LPM) | + | | | | 6. Pipeline table (Array) | + | | | | 7. Pipeline port in table (LPM) | + | | Array | | 8. Pipeline enable | + | | | | 9. Pipeline table rule add default | + | | * Key = Array index | | 10. Pipeline table rule add(LPM) | + | | * Offset = 256 | | 11. Pipeline table rule add(Array) | + | | * Size = 64K | | | + | | | | | + +-----------------------+----------------------+----------------+------------------------------------+ +Command Line Interface (CLI) +---------------------------- -Library of pipeline types -------------------------- +Link +~~~~ -Pipeline module -~~~~~~~~~~~~~~~ - -A pipeline is a self-contained module that implements a packet processing function and is typically implemented on -top of the DPDK Packet Framework *librte_pipeline* library. -The application provides a run-time mechanism to register different pipeline types. - -Depending on the required configuration, each registered pipeline type (pipeline class) is instantiated one or -several times, with each pipeline instance (pipeline object) assigned to one of the available CPU cores. -Each CPU core can run one or more pipeline instances, which might be of same or different types. -For more information of the CPU core threading model, please refer to the :ref:`ip_pipeline_runtime` section. - - -Pipeline type -^^^^^^^^^^^^^ - -Each pipeline type is made up of a back-end and a front-end. The back-end represents the packet processing engine -of the pipeline, typically implemented using the DPDK Packet Framework libraries, which reads packets from the -input packet queues, handles them and eventually writes them to the output packet queues or drops them. -The front-end represents the run-time configuration interface of the pipeline, which is exposed as CLI commands. -The front-end communicates with the back-end through message queues. - -.. _table_ip_pipelines_back_end: - -.. tabularcolumns:: |p{1cm}|p{2cm}|p{12cm}| - -.. table:: Pipeline back-end - - +------------+------------------+--------------------------------------------------------------------+ - | Field name | Field type | Description | - +============+==================+====================================================================+ - | f_init | Function pointer | Function to initialize the back-end of the current pipeline | - | | | instance. Typical work implemented by this function for the | - | | | current pipeline instance: | - | | | Memory allocation; | - | | | Parse the pipeline type specific arguments; | - | | | Initialize the pipeline input ports, output ports and tables, | - | | | interconnect input ports to tables; | - | | | Set the message handlers. | - +------------+------------------+--------------------------------------------------------------------+ - | f_free | Function pointer | Function to free the resources allocated by the back-end of the | - | | | current pipeline instance. | - +------------+------------------+--------------------------------------------------------------------+ - | f_run | Function pointer | Set to NULL for pipelines implemented using the DPDK library | - | | | librte_pipeline (typical case), and to non-NULL otherwise. This | - | | | mechanism is made available to support quick integration of | - | | | legacy code. | - | | | This function is expected to provide the packet processing | - | | | related code to be called as part of the CPU thread dispatch | - | | | loop, so this function is not allowed to contain an infinite loop. | - +------------+------------------+--------------------------------------------------------------------+ - | f_timer | Function pointer | Function to read the pipeline input message queues, handle | - | | | the request messages, create response messages and write | - | | | the response queues. The format of request and response | - | | | messages is defined by each pipeline type, with the exception | - | | | of some requests which are mandatory for all pipelines (e.g. | - | | | ping, statistics). | - +------------+------------------+--------------------------------------------------------------------+ + Link configuration :: + link <link_name> + dev <device_name>|port <port_id> + rxq <n_queues> <queue_size> <mempool_name> + txq <n_queues> <queue_size> promiscuous on | off + [rss <qid_0> ... <qid_n>] -.. _table_ip_pipelines_front_end: - -.. tabularcolumns:: |p{1cm}|p{2cm}|p{12cm}| + Note: The PCI device name must be specified in the Domain:Bus:Device.Function format. -.. table:: Pipeline front-end - - +------------+-----------------------+-------------------------------------------------------------------+ - | Field name | Field type | Description | - +============+=======================+===================================================================+ - | f_init | Function pointer | Function to initialize the front-end of the current pipeline | - | | | instance. | - +------------+-----------------------+-------------------------------------------------------------------+ - | f_post_init| Function pointer | Function to run once after f_init. | - +------------+-----------------------+-------------------------------------------------------------------+ - | f_track | Function pointer | See section Tracking pipeline output port to physical link. | - +------------+-----------------------+-------------------------------------------------------------------+ - | f_free | Function pointer | Function to free the resources allocated by the front-end of | - | | | the current pipeline instance. | - +------------+-----------------------+-------------------------------------------------------------------+ - | cmds | Array of CLI commands | Array of CLI commands to be registered to the application CLI | - | | | for the current pipeline type. Even though the CLI is executed | - | | | by a different pipeline (typically, this is the master pipeline), | - | | | from modularity perspective is more efficient to keep the | - | | | message client side (part of the front-end) together with the | - | | | message server side (part of the back-end). | - +------------+-----------------------+-------------------------------------------------------------------+ +Mempool +~~~~~~~ -Tracking pipeline output port to physical link -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Each pipeline instance is a standalone block that does not have visibility into the other pipeline instances or -the application-level pipeline inter-connectivity. -In some cases, it is useful for a pipeline instance to get application level information related to pipeline -connectivity, such as to identify the output link (e.g. physical NIC port) where one of its output ports connected, -either directly or indirectly by traversing other pipeline instances. + Mempool create :: -Tracking can be successful or unsuccessful. -Typically, tracking for a specific pipeline instance is successful when each one of its input ports can be mapped -to a single output port, meaning that all packets read from the current input port can only go out on a single -output port. -Depending on the pipeline type, some exceptions may be allowed: a small portion of the packets, considered exception -packets, are sent out on an output port that is pre-configured for this purpose. - -For pass-through pipeline type, the tracking is always successful. -For pipeline types as flow classification, firewall or routing, the tracking is only successful when the number of -output ports for the current pipeline instance is 1. - -This feature is used by the IP routing pipeline for adding/removing implicit routes every time a link is brought -up/down. + mempool <mempool_name> buffer <buffer_size> + pool <pool_size> cache <cache_size> cpu <cpu_id> -Table copies -^^^^^^^^^^^^ - -Fast table copy: pipeline table used by pipeline for the packet processing task, updated through messages, table -data structures are optimized for lookup operation. - -Slow table copy: used by the configuration layer, typically updated through CLI commands, kept in sync with the fast -copy (its update triggers the fast copy update). -Required for executing advanced table queries without impacting the packet processing task, therefore the slow copy -is typically organized using different criteria than the fast copy. - -Examples: - -* Flow classification: Search through current set of flows (e.g. list all flows with a specific source IP address); - -* Firewall: List rules in descending order of priority; - -* Routing table: List routes sorted by prefix depth and their type (local, remote, default); - -* ARP: List entries sorted per output interface. - - -Packet meta-data -^^^^^^^^^^^^^^^^ - -Packet meta-data field offsets provided as argument to pipeline instances are essentially defining the data structure -for the packet meta-data used by the current application use-case. -It is very useful to put it in the configuration file as a comment in order to facilitate the readability of the -configuration file. - -The reason to use field offsets for defining the data structure for the packet meta-data is due to the C language -limitation of not being able to define data structures at run-time. -Feature to consider: have the configuration file parser automatically generate and print the data structure defining -the packet meta-data for the current application use-case. - -Packet meta-data typically contains: +Software queue +~~~~~~~~~~~~~~ -1. Pure meta-data: intermediate data per packet that is computed internally, passed between different tables of - the same pipeline instance (e.g. lookup key for the ARP table is obtained from the routing table), or between - different pipeline instances (e.g. flow ID, traffic metering color, etc); + Create software queue :: -2. Packet fields: typically, packet header fields that are read directly from the packet, or read from the packet - and saved (duplicated) as a working copy at a different location within the packet meta-data (e.g. Diffserv - 5-tuple, IP destination address, etc). + swq <swq_name> size <size> cpu <cpu_id> -Several strategies are used to design the packet meta-data, as described in the next subsections. +Traffic manager +~~~~~~~~~~~~~~~ -Store packet meta-data in a different cache line as the packet headers -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Add traffic manager subport profile :: -This approach is able to support protocols with variable header length, like MPLS, where the offset of IP header -from the start of the packet (and, implicitly, the offset of the IP header in the packet buffer) is not fixed. -Since the pipelines typically require the specification of a fixed offset to the packet fields (e.g. Diffserv -5-tuple, used by the flow classification pipeline, or the IP destination address, used by the IP routing pipeline), -the workaround is to have the packet RX pipeline copy these fields at fixed offsets within the packet meta-data. + tmgr subport profile + <tb_rate> <tb_size> + <tc0_rate> <tc1_rate> <tc2_rate> <tc3_rate> + <tc_period> -As this approach duplicates some of the packet fields, it requires accessing more cache lines per packet for filling -in selected packet meta-data fields (on RX), as well as flushing selected packet meta-data fields into the -packet (on TX). -Example: + Add traffic manager pipe profile :: -.. code-block:: ini + tmgr pipe profile + <tb_rate> <tb_size> + <tc0_rate> <tc1_rate> <tc2_rate> <tc3_rate> + <tc_period> + <tc_ov_weight> <wrr_weight0..15> + Create traffic manager port :: - ; struct app_pkt_metadata { - ; uint32_t ip_da; - ; uint32_t hash; - ; uint32_t flow_id; - ; uint32_t color; - ; } __attribute__((__packed__)); - ; + tmgr <tmgr_name> + rate <rate> + spp <n_subports_per_port> + pps <n_pipes_per_subport> + qsize <qsize_tc0> + <qsize_tc1> <qsize_tc2> <qsize_tc3> + fo <frame_overhead> mtu <mtu> cpu <cpu_id> - [PIPELINE1] - ; Packet meta-data offsets - ip_da_offset = 0; Used by: routing - hash_offset = 4; Used by: RX, flow classification - flow_id_offset = 8; Used by: flow classification, flow actions - color_offset = 12; Used by: flow actions, routing + Configure traffic manager subport :: + tmgr <tmgr_name> + subport <subport_id> + profile <subport_profile_id> -Overlay the packet meta-data in the same cache line with the packet headers -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Configure traffic manager pipe :: -This approach is minimizing the number of cache line accessed per packet by storing the packet metadata in the -same cache line with the packet headers. -To enable this strategy, either some headroom is reserved for meta-data at the beginning of the packet headers -cache line (e.g. if 16 bytes are needed for meta-data, then the packet headroom can be set to 128+16 bytes, so -that NIC writes the first byte of the packet at offset 16 from the start of the first packet cache line), -or meta-data is reusing the space of some packet headers that are discarded from the packet (e.g. input Ethernet -header). + tmgr <tmgr_name> + subport <subport_id> + pipe from <pipe_id_first> to <pipe_id_last> + profile <pipe_profile_id> -Example: -.. code-block:: ini +Tap +~~~ - ; struct app_pkt_metadata { - ; uint8_t headroom[RTE_PKTMBUF_HEADROOM]; /* 128 bytes (default) */ - ; union { - ; struct { - ; struct ether_hdr ether; /* 14 bytes */ - ; struct qinq_hdr qinq; /* 8 bytes */ - ; }; - ; struct { - ; uint32_t hash; - ; uint32_t flow_id; - ; uint32_t color; - ; }; - ; }; - ; struct ipv4_hdr ip; /* 20 bytes */ - ; } __attribute__((__packed__)); - ; - [PIPELINE2] - ; Packet meta-data offsets - qinq_offset = 142; Used by: RX, flow classification - ip_da_offset = 166; Used by: routing - hash_offset = 128; Used by: RX, flow classification - flow_id_offset = 132; Used by: flow classification, flow actions - color_offset = 136; Used by: flow actions, routing + Create tap port :: + tap <name> -List of pipeline types -~~~~~~~~~~~~~~~~~~~~~~ -.. _table_ip_pipelines_types: +Kni +~~~ -.. tabularcolumns:: |p{3cm}|p{5cm}|p{4cm}|p{4cm}| + Create kni port :: -.. table:: List of pipeline types provided with the application - - +-----------------------+-----------------------------+-----------------------+------------------------------------------+ - | Name | Table(s) | Actions | Messages | - +=======================+=============================+=======================+==========================================+ - | Pass-through | Passthrough | 1. Pkt metadata build | 1. Ping | - | | | 2. Flow hash | 2. Stats | - | Note: depending on | | 3. Pkt checks | | - | port type, can be | | 4. Load balancing | | - | used for RX, TX, IP | | | | - | fragmentation, IP | | | | - | reassembly or Traffic | | | | - | Management | | | | - +-----------------------+-----------------------------+-----------------------+------------------------------------------+ - | Flow classification | Exact match | 1. Flow ID | 1. Ping | - | | | | | - | | * Key = byte array | 2. Flow stats | 2. Stats | - | | (source: pkt metadata) | 3. Metering | 3. Flow stats | - | | * Data = action dependent | 4. Network Address | 4. Action stats | - | | | 5. Translation (NAT) | 5. Flow add/ update/ delete | - | | | | 6. Default flow add/ update/ delete | - | | | | 7. Action update | - +-----------------------+-----------------------------+-----------------------+------------------------------------------+ - | Flow actions | Array | 1. Flow stats | 1. Ping | - | | | | | - | | * Key = Flow ID | 2. Metering | 2. Stats | - | | (source: pkt metadata) | 3. Network Address | 3. Action stats | - | | * Data = action dependent | 4. Translation (NAT) | 4. Action update | - +-----------------------+-----------------------------+-----------------------+------------------------------------------+ - | Firewall | ACL | 1. Allow/Drop | 1. Ping | - | | | | | - | | * Key = n-tuple | | 2. Stats | - | | (source: pkt headers) | | 3. Rule add/ update/ delete | - | | * Data = none | | 4. Default rule add/ update/ delete | - +-----------------------+-----------------------------+-----------------------+------------------------------------------+ - | IP routing | LPM (IPv4 or IPv6, | 1. TTL decrement and | 1. Ping | - | | depending on pipeline type) | 2. IPv4 checksum | 2. Stats | - | | | | | - | | * Key = IP destination | 3. update | 3. Route add/ update/ delete | - | | (source: pkt metadata) | 4. Header | 4. Default route add/ update/ delete | - | | * Data = Dependent on | 5. encapsulation | 5. ARP entry add/ update/ delete | - | | actions and next hop | 6. (based on next hop | 6. Default ARP entry add/ update/ delete | - | | type | 7. type) | | - | | | | | - | | Hash table (for ARP, only | | | - | | | | | - | | when ARP is enabled) | | | - | | | | | - | | * Key = (Port ID, | | | - | | next hop IP address) | | | - | | (source: pkt meta-data) | | | - | | * Data: MAC address | | | - +-----------------------+-----------------------------+-----------------------+------------------------------------------+ + kni <kni_name> + link <link_name> + mempool <mempool_name> + [thread <thread_id>] +Action profile +~~~~~~~~~~~~~~ -Command Line Interface (CLI) ----------------------------- + Create action profile for pipeline input port :: + + port in action profile <profile_name> + [filter match | mismatch offset <key_offset> mask <key_mask> key <key_value> port <port_id>] + [balance offset <key_offset> mask <key_mask> port <port_id0> ... <port_id15>] + + Create action profile for the pipeline table :: + + table action profile <profile_name> + ipv4 | ipv6 + offset <ip_offset> + fwd + [balance offset <key_offset> mask <key_mask> outoffset <out_offset>] + [meter srtcm | trtcm + tc <n_tc> + stats none | pkts | bytes | both] + [tm spp <n_subports_per_port> pps <n_pipes_per_subport>] + [encap ether | vlan | qinq | mpls | pppoe] + [nat src | dst + proto udp | tcp] + [ttl drop | fwd + stats none | pkts] + [stats pkts | bytes | both] + [time] + + +Pipeline +~~~~~~~~ -Global CLI commands -~~~~~~~~~~~~~~~~~~~ +Create pipeline :: + + pipeline <pipeline_name> + period <timer_period_ms> + offset_port_id <offset_port_id> + cpu <cpu_id> + +Create pipeline input port :: + + pipeline <pipeline_name> port in + bsz <burst_size> + link <link_name> rxq <queue_id> + | swq <swq_name> + | tmgr <tmgr_name> + | tap <tap_name> mempool <mempool_name> mtu <mtu> + | kni <kni_name> + | source mempool <mempool_name> file <file_name> bpp <n_bytes_per_pkt> + [action <port_in_action_profile_name>] + [disabled] + +Create pipeline output port :: + + pipeline <pipeline_name> port out + bsz <burst_size> + link <link_name> txq <txq_id> + | swq <swq_name> + | tmgr <tmgr_name> + | tap <tap_name> + | kni <kni_name> + | sink [file <file_name> pkts <max_n_pkts>] + +Create pipeline table :: + + pipeline <pipeline_name> table + match + acl + ipv4 | ipv6 + offset <ip_header_offset> + size <n_rules> + | array + offset <key_offset> + size <n_keys> + | hash + ext | lru + key <key_size> + mask <key_mask> + offset <key_offset> + buckets <n_buckets> + size <n_keys> + | lpm + ipv4 | ipv6 + offset <ip_header_offset> + size <n_rules> + | stub + [action <table_action_profile_name>] + +Connect pipeline input port to table :: + + pipeline <pipeline_name> port in <port_id> table <table_id> + +Display statistics for specific pipeline input port, output port +or table :: + + pipeline <pipeline_name> port in <port_id> stats read [clear] + pipeline <pipeline_name> port out <port_id> stats read [clear] + pipeline <pipeline_name> table <table_id> stats read [clear] + +Enable given input port for specific pipeline instance :: + + pipeline <pipeline_name> port out <port_id> disable + +Disable given input port for specific pipeline instance :: + + pipeline <pipeline_name> port out <port_id> disable + +Add default rule to table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule add + match + default + action + fwd + drop + | port <port_id> + | meta + | table <table_id> + +Add rule to table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule add + + match + acl + priority <priority> + ipv4 | ipv6 <sa> <sa_depth> <da> <da_depth> + <sp0> <sp1> <dp0> <dp1> <proto> + | array <pos> + | hash + raw <key> + | ipv4_5tuple <sa> <da> <sp> <dp> <proto> + | ipv6_5tuple <sa> <da> <sp> <dp> <proto> + | ipv4_addr <addr> + | ipv6_addr <addr> + | qinq <svlan> <cvlan> + | lpm + ipv4 | ipv6 <addr> <depth> + + action + fwd + drop + | port <port_id> + | meta + | table <table_id> + [balance <out0> ... <out7>] + [meter + tc0 meter <meter_profile_id> policer g <pa> y <pa> r <pa> + [tc1 meter <meter_profile_id> policer g <pa> y <pa> r <pa> + tc2 meter <meter_profile_id> policer g <pa> y <pa> r <pa> + tc3 meter <meter_profile_id> policer g <pa> y <pa> r <pa>]] + [tm subport <subport_id> pipe <pipe_id>] + [encap + ether <da> <sa> + | vlan <da> <sa> <pcp> <dei> <vid> + | qinq <da> <sa> <pcp> <dei> <vid> <pcp> <dei> <vid> + | mpls unicast | multicast + <da> <sa> + label0 <label> <tc> <ttl> + [label1 <label> <tc> <ttl> + [label2 <label> <tc> <ttl> + [label3 <label> <tc> <ttl>]]] + | pppoe <da> <sa> <session_id>] + [nat ipv4 | ipv6 <addr> <port>] + [ttl dec | keep] + [stats] + [time] + + where: + <pa> ::= g | y | r | drop + +Add bulk rules to table for specific pipeline instance :: + + pipeline <pipeline_name> table <table_id> rule add bulk <file_name> <n_rules> -.. _table_ip_pipelines_cli_commands: + Where: + - file_name = path to file + - File line format = match <match> action <action> -.. tabularcolumns:: |p{3cm}|p{6cm}|p{6cm}| +Delete table rule for specific pipeline instance :: -.. table:: Global CLI commands + pipeline <pipeline_name> table <table_id> rule delete + match <match> - +---------+---------------------------------------+--------------------------------------------+ - | Command | Description | Syntax | - +=========+=======================================+============================================+ - | run | Run CLI commands script file. | run <file> | - | | | <file> = path to file with CLI commands to | - | | | execute | - +---------+---------------------------------------+--------------------------------------------+ - | quit | Gracefully terminate the application. | quit | - +---------+---------------------------------------+--------------------------------------------+ +Delete default table rule for specific pipeline instance :: + pipeline <pipeline_name> table <table_id> rule delete + match + default -CLI commands for link configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Add meter profile to the table for specific pipeline instance :: -.. _table_ip_pipelines_runtime_config: + pipeline <pipeline_name> table <table_id> meter profile <meter_profile_id> + add srtcm cir <cir> cbs <cbs> ebs <ebs> + | trtcm cir <cir> pir <pir> cbs <cbs> pbs <pbs> -.. tabularcolumns:: |p{3cm}|p{6cm}|p{6cm}| +Delete meter profile from the table for specific pipeline instance :: -.. table:: List of run-time configuration commands for link configuration + pipeline <pipeline_name> table <table_id> + meter profile <meter_profile_id> delete - +-------------+--------------------+--------------------------------------------+ - | Command | Description | Syntax | - +=============+====================+============================================+ - | link config | Link configuration | link <link ID> config <IP address> <depth> | - +-------------+--------------------+--------------------------------------------+ - | link up | Link up | link <link ID> up | - +-------------+--------------------+--------------------------------------------+ - | link down | Link down | link <link ID> down | - +-------------+--------------------+--------------------------------------------+ - | link ls | Link list | link ls | - +-------------+--------------------+--------------------------------------------+ +Update the dscp table for meter or traffic manager action for specific +pipeline instance :: -CLI commands common for all pipeline types -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + pipeline <pipeline_name> table <table_id> dscp <file_name> -.. _table_ip_pipelines_mandatory: + Where: + - file_name = path to file + - exactly 64 lines + - File line format = <tc_id> <tc_queue_id> <color>, with <color> as: g | y | r + + +Pipeline enable/disable +~~~~~~~~~~~~~~~~~~~~~~~ -.. tabularcolumns:: |p{3cm}|p{6cm}|p{6cm}| + Enable given pipeline instance for specific data plane thread :: -.. table:: CLI commands mandatory for all pipelines + thread <thread_id> pipeline <pipeline_name> enable - +--------------------+------------------------------------------------------+----------------------------------------------+ - | Command | Description | Syntax | - +====================+======================================================+==============================================+ - | ping | Check whether specific pipeline instance is alive. | p <pipeline ID> ping | - | | The master pipeline sends a ping request | | - | | message to given pipeline instance and waits for | | - | | a response message back. | | - | | Timeout message is displayed when the response | | - | | message is not received before the timer | | - | | expires. | | - +--------------------+------------------------------------------------------+----------------------------------------------+ - | stats | Display statistics for specific pipeline input port, | p <pipeline ID> stats port in <port in ID> | - | | output port or table. | p <pipeline ID> stats port out <port out ID> | - | | | p <pipeline ID> stats table <table ID> | - +--------------------+------------------------------------------------------+----------------------------------------------+ - | input port enable | Enable given input port for specific pipeline | p <pipeline ID> port in <port ID> enable | - | | instance. | | - +--------------------+------------------------------------------------------+----------------------------------------------+ - | input port disable | Disable given input port for specific pipeline | p <pipeline ID> port in <port ID> disable | - | | instance. | | - +--------------------+------------------------------------------------------+----------------------------------------------+ -Pipeline type specific CLI commands -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Disable given pipeline instance for specific data plane thread :: -The pipeline specific CLI commands are part of the pipeline type front-end. + thread <thread_id> pipeline <pipeline_name> disable diff --git a/doc/guides/sample_app_ug/kernel_nic_interface.rst b/doc/guides/sample_app_ug/kernel_nic_interface.rst index 580d06b5..1b3ee9a5 100644 --- a/doc/guides/sample_app_ug/kernel_nic_interface.rst +++ b/doc/guides/sample_app_ug/kernel_nic_interface.rst @@ -202,74 +202,8 @@ Setup of mbuf pool, driver and queues is similar to the setup done in the :doc:` In addition, one or more kernel NIC interfaces are allocated for each of the configured ports according to the command line parameters. -The code for allocating the kernel NIC interfaces for a specific port is as follows: - -.. code-block:: c - - static int - kni_alloc(uint16_t port_id) - { - uint8_t i; - struct rte_kni *kni; - struct rte_kni_conf conf; - struct kni_port_params **params = kni_port_params_array; - - if (port_id >= RTE_MAX_ETHPORTS || !params[port_id]) - return -1; - - params[port_id]->nb_kni = params[port_id]->nb_lcore_k ? params[port_id]->nb_lcore_k : 1; - - for (i = 0; i < params[port_id]->nb_kni; i++) { - - /* Clear conf at first */ - - memset(&conf, 0, sizeof(conf)); - if (params[port_id]->nb_lcore_k) { - snprintf(conf.name, RTE_KNI_NAMESIZE, "vEth%u_%u", port_id, i); - conf.core_id = params[port_id]->lcore_k[i]; - conf.force_bind = 1; - } else - snprintf(conf.name, RTE_KNI_NAMESIZE, "vEth%u", port_id); - conf.group_id = (uint16_t)port_id; - conf.mbuf_size = MAX_PACKET_SZ; - - /* - * The first KNI device associated to a port - * is the master, for multiple kernel thread - * environment. - */ - - if (i == 0) { - struct rte_kni_ops ops; - struct rte_eth_dev_info dev_info; - - memset(&dev_info, 0, sizeof(dev_info)); rte_eth_dev_info_get(port_id, &dev_info); - - conf.addr = dev_info.pci_dev->addr; - conf.id = dev_info.pci_dev->id; - - /* Get the interface default mac address */ - rte_eth_macaddr_get(port_id, (struct ether_addr *)&conf.mac_addr); - - memset(&ops, 0, sizeof(ops)); - - ops.port_id = port_id; - ops.change_mtu = kni_change_mtu; - ops.config_network_if = kni_config_network_interface; - ops.config_mac_address = kni_config_mac_address; - - kni = rte_kni_alloc(pktmbuf_pool, &conf, &ops); - } else - kni = rte_kni_alloc(pktmbuf_pool, &conf, NULL); - - if (!kni) - rte_exit(EXIT_FAILURE, "Fail to create kni for " - "port: %d\n", port_id); - - params[port_id]->kni[i] = kni; - } - return 0; - } +The code for allocating the kernel NIC interfaces for a specific port is +in the function ``kni_alloc``. The other step in the initialization process that is unique to this sample application is the association of each port with lcores for RX, TX and kernel threads. @@ -280,105 +214,8 @@ is the association of each port with lcores for RX, TX and kernel threads. * Other lcores for pinning the kernel threads on one by one -This is done by using the`kni_port_params_array[]` array, which is indexed by the port ID. -The code is as follows: - -.. code-block:: console - - static int - parse_config(const char *arg) - { - const char *p, *p0 = arg; - char s[256], *end; - unsigned size; - enum fieldnames { - FLD_PORT = 0, - FLD_LCORE_RX, - FLD_LCORE_TX, - _NUM_FLD = KNI_MAX_KTHREAD + 3, - }; - int i, j, nb_token; - char *str_fld[_NUM_FLD]; - unsigned long int_fld[_NUM_FLD]; - uint16_t port_id, nb_kni_port_params = 0; - - memset(&kni_port_params_array, 0, sizeof(kni_port_params_array)); - - while (((p = strchr(p0, '(')) != NULL) && nb_kni_port_params < RTE_MAX_ETHPORTS) { - p++; - if ((p0 = strchr(p, ')')) == NULL) - goto fail; - - size = p0 - p; - - if (size >= sizeof(s)) { - printf("Invalid config parameters\n"); - goto fail; - } - - snprintf(s, sizeof(s), "%.*s", size, p); - nb_token = rte_strsplit(s, sizeof(s), str_fld, _NUM_FLD, ','); - - if (nb_token <= FLD_LCORE_TX) { - printf("Invalid config parameters\n"); - goto fail; - } - - for (i = 0; i < nb_token; i++) { - errno = 0; - int_fld[i] = strtoul(str_fld[i], &end, 0); - if (errno != 0 || end == str_fld[i]) { - printf("Invalid config parameters\n"); - goto fail; - } - } - - i = 0; - port_id = (uint8_t)int_fld[i++]; - - if (port_id >= RTE_MAX_ETHPORTS) { - printf("Port ID %u could not exceed the maximum %u\n", port_id, RTE_MAX_ETHPORTS); - goto fail; - } - - if (kni_port_params_array[port_id]) { - printf("Port %u has been configured\n", port_id); - goto fail; - } - - kni_port_params_array[port_id] = (struct kni_port_params*)rte_zmalloc("KNI_port_params", sizeof(struct kni_port_params), RTE_CACHE_LINE_SIZE); - kni_port_params_array[port_id]->port_id = port_id; - kni_port_params_array[port_id]->lcore_rx = (uint8_t)int_fld[i++]; - kni_port_params_array[port_id]->lcore_tx = (uint8_t)int_fld[i++]; - - if (kni_port_params_array[port_id]->lcore_rx >= RTE_MAX_LCORE || kni_port_params_array[port_id]->lcore_tx >= RTE_MAX_LCORE) { - printf("lcore_rx %u or lcore_tx %u ID could not " - "exceed the maximum %u\n", - kni_port_params_array[port_id]->lcore_rx, kni_port_params_array[port_id]->lcore_tx, RTE_MAX_LCORE); - goto fail; - } - - for (j = 0; i < nb_token && j < KNI_MAX_KTHREAD; i++, j++) - kni_port_params_array[port_id]->lcore_k[j] = (uint8_t)int_fld[i]; - kni_port_params_array[port_id]->nb_lcore_k = j; - } - - print_config(); - - return 0; - - fail: - - for (i = 0; i < RTE_MAX_ETHPORTS; i++) { - if (kni_port_params_array[i]) { - rte_free(kni_port_params_array[i]); - kni_port_params_array[i] = NULL; - } - } - - return -1; - - } +This is done by using the ``kni_port_params_array[]`` array, which is indexed by the port ID. +The code is in the function ``parse_config``. Packet Forwarding ~~~~~~~~~~~~~~~~~ @@ -387,99 +224,18 @@ After the initialization steps are completed, the main_loop() function is run on This function first checks the lcore_id against the user provided lcore_rx and lcore_tx to see if this lcore is reading from or writing to kernel NIC interfaces. -For the case that reads from a NIC port and writes to the kernel NIC interfaces, +For the case that reads from a NIC port and writes to the kernel NIC interfaces (``kni_ingress``), the packet reception is the same as in L2 Forwarding sample application (see :ref:`l2_fwd_app_rx_tx_packets`). The packet transmission is done by sending mbufs into the kernel NIC interfaces by rte_kni_tx_burst(). The KNI library automatically frees the mbufs after the kernel successfully copied the mbufs. -.. code-block:: c - - /** - * Interface to burst rx and enqueue mbufs into rx_q - */ - - static void - kni_ingress(struct kni_port_params *p) - { - uint8_t i, nb_kni, port_id; - unsigned nb_rx, num; - struct rte_mbuf *pkts_burst[PKT_BURST_SZ]; - - if (p == NULL) - return; - - nb_kni = p->nb_kni; - port_id = p->port_id; - - for (i = 0; i < nb_kni; i++) { - /* Burst rx from eth */ - nb_rx = rte_eth_rx_burst(port_id, 0, pkts_burst, PKT_BURST_SZ); - if (unlikely(nb_rx > PKT_BURST_SZ)) { - RTE_LOG(ERR, APP, "Error receiving from eth\n"); - return; - } - - /* Burst tx to kni */ - num = rte_kni_tx_burst(p->kni[i], pkts_burst, nb_rx); - kni_stats[port_id].rx_packets += num; - rte_kni_handle_request(p->kni[i]); - - if (unlikely(num < nb_rx)) { - /* Free mbufs not tx to kni interface */ - kni_burst_free_mbufs(&pkts_burst[num], nb_rx - num); - kni_stats[port_id].rx_dropped += nb_rx - num; - } - } - } - -For the other case that reads from kernel NIC interfaces and writes to a physical NIC port, packets are retrieved by reading -mbufs from kernel NIC interfaces by `rte_kni_rx_burst()`. +For the other case that reads from kernel NIC interfaces +and writes to a physical NIC port (``kni_egress``), +packets are retrieved by reading mbufs from kernel NIC interfaces by ``rte_kni_rx_burst()``. The packet transmission is the same as in the L2 Forwarding sample application (see :ref:`l2_fwd_app_rx_tx_packets`). -.. code-block:: c - - /** - * Interface to dequeue mbufs from tx_q and burst tx - */ - - static void - - kni_egress(struct kni_port_params *p) - { - uint8_t i, nb_kni, port_id; - unsigned nb_tx, num; - struct rte_mbuf *pkts_burst[PKT_BURST_SZ]; - - if (p == NULL) - return; - - nb_kni = p->nb_kni; - port_id = p->port_id; - - for (i = 0; i < nb_kni; i++) { - /* Burst rx from kni */ - num = rte_kni_rx_burst(p->kni[i], pkts_burst, PKT_BURST_SZ); - if (unlikely(num > PKT_BURST_SZ)) { - RTE_LOG(ERR, APP, "Error receiving from KNI\n"); - return; - } - - /* Burst tx to eth */ - - nb_tx = rte_eth_tx_burst(port_id, 0, pkts_burst, (uint16_t)num); - - kni_stats[port_id].tx_packets += nb_tx; - - if (unlikely(nb_tx < num)) { - /* Free mbufs not tx to NIC */ - kni_burst_free_mbufs(&pkts_burst[nb_tx], num - nb_tx); - kni_stats[port_id].tx_dropped += num - nb_tx; - } - } - } - Callbacks for Kernel Requests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -492,106 +248,3 @@ Application may choose to not implement following callbacks: - ``config_mac_address`` - ``config_promiscusity`` - - -.. code-block:: c - - static struct rte_kni_ops kni_ops = { - .change_mtu = kni_change_mtu, - .config_network_if = kni_config_network_interface, - .config_mac_address = kni_config_mac_address, - .config_promiscusity = kni_config_promiscusity, - }; - - /* Callback for request of changing MTU */ - - static int - kni_change_mtu(uint16_t port_id, unsigned new_mtu) - { - int ret; - struct rte_eth_conf conf; - - if (port_id >= rte_eth_dev_count()) { - RTE_LOG(ERR, APP, "Invalid port id %d\n", port_id); - return -EINVAL; - } - - RTE_LOG(INFO, APP, "Change MTU of port %d to %u\n", port_id, new_mtu); - - /* Stop specific port */ - - rte_eth_dev_stop(port_id); - - memcpy(&conf, &port_conf, sizeof(conf)); - - /* Set new MTU */ - - if (new_mtu > ETHER_MAX_LEN) - conf.rxmode.jumbo_frame = 1; - else - conf.rxmode.jumbo_frame = 0; - - /* mtu + length of header + length of FCS = max pkt length */ - - conf.rxmode.max_rx_pkt_len = new_mtu + KNI_ENET_HEADER_SIZE + KNI_ENET_FCS_SIZE; - - ret = rte_eth_dev_configure(port_id, 1, 1, &conf); - if (ret < 0) { - RTE_LOG(ERR, APP, "Fail to reconfigure port %d\n", port_id); - return ret; - } - - /* Restart specific port */ - - ret = rte_eth_dev_start(port_id); - if (ret < 0) { - RTE_LOG(ERR, APP, "Fail to restart port %d\n", port_id); - return ret; - } - - return 0; - } - - /* Callback for request of configuring network interface up/down */ - - static int - kni_config_network_interface(uint16_t port_id, uint8_t if_up) - { - int ret = 0; - - if (port_id >= rte_eth_dev_count() || port_id >= RTE_MAX_ETHPORTS) { - RTE_LOG(ERR, APP, "Invalid port id %d\n", port_id); - return -EINVAL; - } - - RTE_LOG(INFO, APP, "Configure network interface of %d %s\n", - - port_id, if_up ? "up" : "down"); - - if (if_up != 0) { - /* Configure network interface up */ - rte_eth_dev_stop(port_id); - ret = rte_eth_dev_start(port_id); - } else /* Configure network interface down */ - rte_eth_dev_stop(port_id); - - if (ret < 0) - RTE_LOG(ERR, APP, "Failed to start port %d\n", port_id); - return ret; - } - - /* Callback for request of configuring device mac address */ - - static int - kni_config_mac_address(uint16_t port_id, uint8_t mac_addr[]) - { - ..... - } - - /* Callback for request of configuring promiscuous mode */ - - static int - kni_config_promiscusity(uint16_t port_id, uint8_t to_on) - { - ..... - } diff --git a/doc/guides/sample_app_ug/l2_forward_job_stats.rst b/doc/guides/sample_app_ug/l2_forward_job_stats.rst index bfdf9c8f..ba73d855 100644 --- a/doc/guides/sample_app_ug/l2_forward_job_stats.rst +++ b/doc/guides/sample_app_ug/l2_forward_job_stats.rst @@ -178,11 +178,6 @@ in the *DPDK Programmer's Guide* and the *DPDK API Reference*. .. code-block:: c - nb_ports = rte_eth_dev_count(); - - if (nb_ports == 0) - rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n"); - /* reset l2fwd_dst_ports */ for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) @@ -193,7 +188,7 @@ in the *DPDK Programmer's Guide* and the *DPDK API Reference*. /* * Each logical core is assigned a dedicated TX queue on each port. */ - for (portid = 0; portid < nb_ports; portid++) { + RTE_ETH_FOREACH_DEV(portid) { /* skip ports that are not enabled */ if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) continue; @@ -223,25 +218,6 @@ The rte_eth_dev_configure() function is used to configure the number of queues f "err=%d, port=%u\n", ret, portid); -The global configuration is stored in a static structure: - -.. code-block:: c - - static const struct rte_eth_conf port_conf = { - .rxmode = { - .split_hdr_size = 0, - .header_split = 0, /**< Header Split disabled */ - .hw_ip_checksum = 0, /**< IP checksum offload disabled */ - .hw_vlan_filter = 0, /**< VLAN filtering disabled */ - .jumbo_frame = 0, /**< Jumbo Frame Support disabled */ - .hw_strip_crc= 0, /**< CRC stripped by hardware */ - }, - - .txmode = { - .mq_mode = ETH_DCB_NONE - }, - }; - RX Queue Initialization ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst index f02be05c..2b2d5afa 100644 --- a/doc/guides/sample_app_ug/l2_forward_real_virtual.rst +++ b/doc/guides/sample_app_ug/l2_forward_real_virtual.rst @@ -197,11 +197,6 @@ in the *DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*. if (rte_pci_probe() < 0) rte_exit(EXIT_FAILURE, "Cannot probe PCI\n"); - nb_ports = rte_eth_dev_count(); - - if (nb_ports == 0) - rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n"); - /* reset l2fwd_dst_ports */ for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) @@ -213,7 +208,7 @@ in the *DPDK Programmer's Guide* - Rel 1.4 EAR and the *DPDK API Reference*. * Each logical core is assigned a dedicated TX queue on each port. */ - for (portid = 0; portid < nb_ports; portid++) { + RTE_ETH_FOREACH_DEV(portid) { /* skip ports that are not enabled */ if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) @@ -250,25 +245,6 @@ The rte_eth_dev_configure() function is used to configure the number of queues f "err=%d, port=%u\n", ret, portid); -The global configuration is stored in a static structure: - -.. code-block:: c - - static const struct rte_eth_conf port_conf = { - .rxmode = { - .split_hdr_size = 0, - .header_split = 0, /**< Header Split disabled */ - .hw_ip_checksum = 0, /**< IP checksum offload disabled */ - .hw_vlan_filter = 0, /**< VLAN filtering disabled */ - .jumbo_frame = 0, /**< Jumbo Frame Support disabled */ - .hw_strip_crc= 0, /**< CRC stripped by hardware */ - }, - - .txmode = { - .mq_mode = ETH_DCB_NONE - }, - }; - .. _l2_fwd_app_rx_init: RX Queue Initialization diff --git a/doc/guides/sample_app_ug/link_status_intr.rst b/doc/guides/sample_app_ug/link_status_intr.rst index 8fa6d76c..c7665fe5 100644 --- a/doc/guides/sample_app_ug/link_status_intr.rst +++ b/doc/guides/sample_app_ug/link_status_intr.rst @@ -91,15 +91,11 @@ To fully understand this code, it is recommended to study the chapters that rela if (rte_pci_probe() < 0) rte_exit(EXIT_FAILURE, "Cannot probe PCI\n"); - nb_ports = rte_eth_dev_count(); - if (nb_ports == 0) - rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n"); - /* * Each logical core is assigned a dedicated TX queue on each port. */ - for (portid = 0; portid < nb_ports; portid++) { + RTE_ETH_FOREACH_DEV(portid) { /* skip ports that are not enabled */ if ((lsi_enabled_port_mask & (1 << portid)) == 0) @@ -141,10 +137,7 @@ The global configuration is stored in a static structure: static const struct rte_eth_conf port_conf = { .rxmode = { .split_hdr_size = 0, - .header_split = 0, /**< Header Split disabled */ - .hw_ip_checksum = 0, /**< IP checksum offload disabled */ - .hw_vlan_filter = 0, /**< VLAN filtering disabled */ - .hw_strip_crc= 0, /**< CRC stripped by hardware */ + .offloads = DEV_RX_OFFLOAD_CRC_STRIP, }, .txmode = {}, .intr_conf = { diff --git a/doc/guides/sample_app_ug/multi_process.rst b/doc/guides/sample_app_ug/multi_process.rst index b84e9a74..9c374da6 100644 --- a/doc/guides/sample_app_ug/multi_process.rst +++ b/doc/guides/sample_app_ug/multi_process.rst @@ -321,409 +321,3 @@ In both the server and the client processes, outgoing packets are buffered befor so as to allow the sending of multiple packets in a single burst to improve efficiency. For example, the client process will buffer packets to send, until either the buffer is full or until we receive no further packets from the server. - -Master-slave Multi-process Example -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The fourth example of DPDK multi-process support demonstrates a master-slave model that -provide the capability of application recovery if a slave process crashes or meets unexpected conditions. -In addition, it also demonstrates the floating process, -which can run among different cores in contrast to the traditional way of binding a process/thread to a specific CPU core, -using the local cache mechanism of mempool structures. - -This application performs the same functionality as the L2 Forwarding sample application, -therefore this chapter does not cover that part but describes functionality that is introduced in this multi-process example only. -Please refer to :doc:`l2_forward_real_virtual` for more information. - -Unlike previous examples where all processes are started from the command line with input arguments, in this example, -only one process is spawned from the command line and that process creates other processes. -The following section describes this in more detail. - -Master-slave Process Models -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The process spawned from the command line is called the *master process* in this document. -A process created by the master is called a *slave process*. -The application has only one master process, but could have multiple slave processes. - -Once the master process begins to run, it tries to initialize all the resources such as -memory, CPU cores, driver, ports, and so on, as the other examples do. -Thereafter, it creates slave processes, as shown in the following figure. - -.. _figure_master_slave_proc: - -.. figure:: img/master_slave_proc.* - - Master-slave Process Workflow - - -The master process calls the rte_eal_mp_remote_launch() EAL function to launch an application function for each pinned thread through the pipe. -Then, it waits to check if any slave processes have exited. -If so, the process tries to re-initialize the resources that belong to that slave and launch them in the pinned thread entry again. -The following section describes the recovery procedures in more detail. - -For each pinned thread in EAL, after reading any data from the pipe, it tries to call the function that the application specified. -In this master specified function, a fork() call creates a slave process that performs the L2 forwarding task. -Then, the function waits until the slave exits, is killed or crashes. Thereafter, it notifies the master of this event and returns. -Finally, the EAL pinned thread waits until the new function is launched. - -After discussing the master-slave model, it is necessary to mention another issue, global and static variables. - -For multiple-thread cases, all global and static variables have only one copy and they can be accessed by any thread if applicable. -So, they can be used to sync or share data among threads. - -In the previous examples, each process has separate global and static variables in memory and are independent of each other. -If it is necessary to share the knowledge, some communication mechanism should be deployed, such as, memzone, ring, shared memory, and so on. -The global or static variables are not a valid approach to share data among processes. -For variables in this example, on the one hand, the slave process inherits all the knowledge of these variables after being created by the master. -On the other hand, other processes cannot know if one or more processes modifies them after slave creation since that -is the nature of a multiple process address space. -But this does not mean that these variables cannot be used to share or sync data; it depends on the use case. -The following are the possible use cases: - -#. The master process starts and initializes a variable and it will never be changed after slave processes created. This case is OK. - -#. After the slave processes are created, the master or slave cores need to change a variable, but other processes do not need to know the change. - This case is also OK. - -#. After the slave processes are created, the master or a slave needs to change a variable. - In the meantime, one or more other process needs to be aware of the change. - In this case, global and static variables cannot be used to share knowledge. Another communication mechanism is needed. - A simple approach without lock protection can be a heap buffer allocated by rte_malloc or mem zone. - -Slave Process Recovery Mechanism -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Before talking about the recovery mechanism, it is necessary to know what is needed before a new slave instance can run if a previous one exited. - -When a slave process exits, the system returns all the resources allocated for this process automatically. -However, this does not include the resources that were allocated by the DPDK. All the hardware resources are shared among the processes, -which include memzone, mempool, ring, a heap buffer allocated by the rte_malloc library, and so on. -If the new instance runs and the allocated resource is not returned, either resource allocation failed or the hardware resource is lost forever. - -When a slave process runs, it may have dependencies on other processes. -They could have execution sequence orders; they could share the ring to communicate; they could share the same port for reception and forwarding; -they could use lock structures to do exclusive access in some critical path. -What happens to the dependent process(es) if the peer leaves? -The consequence are varied since the dependency cases are complex. -It depends on what the processed had shared. -However, it is necessary to notify the peer(s) if one slave exited. -Then, the peer(s) will be aware of that and wait until the new instance begins to run. - -Therefore, to provide the capability to resume the new slave instance if the previous one exited, it is necessary to provide several mechanisms: - -#. Keep a resource list for each slave process. - Before a slave process run, the master should prepare a resource list. - After it exits, the master could either delete the allocated resources and create new ones, - or re-initialize those for use by the new instance. - -#. Set up a notification mechanism for slave process exit cases. After the specific slave leaves, - the master should be notified and then help to create a new instance. - This mechanism is provided in Section `Master-slave Process Models`_. - -#. Use a synchronization mechanism among dependent processes. - The master should have the capability to stop or kill slave processes that have a dependency on the one that has exited. - Then, after the new instance of exited slave process begins to run, the dependency ones could resume or run from the start. - The example sends a STOP command to slave processes dependent on the exited one, then they will exit. - Thereafter, the master creates new instances for the exited slave processes. - -The following diagram describes slave process recovery. - -.. _figure_slave_proc_recov: - -.. figure:: img/slave_proc_recov.* - - Slave Process Recovery Process Flow - - -Floating Process Support -^^^^^^^^^^^^^^^^^^^^^^^^ - -When the DPDK application runs, there is always a -c option passed in to indicate the cores that are enabled. -Then, the DPDK creates a thread for each enabled core. -By doing so, it creates a 1:1 mapping between the enabled core and each thread. -The enabled core always has an ID, therefore, each thread has a unique core ID in the DPDK execution environment. -With the ID, each thread can easily access the structures or resources exclusively belonging to it without using function parameter passing. -It can easily use the rte_lcore_id() function to get the value in every function that is called. - -For threads/processes not created in that way, either pinned to a core or not, they will not own a unique ID and the -rte_lcore_id() function will not work in the correct way. -However, sometimes these threads/processes still need the unique ID mechanism to do easy access on structures or resources. -For example, the DPDK mempool library provides a local cache mechanism -(refer to :ref:`mempool_local_cache`) -for fast element allocation and freeing. -If using a non-unique ID or a fake one, -a race condition occurs if two or more threads/ processes with the same core ID try to use the local cache. - -Therefore, unused core IDs from the passing of parameters with the -c option are used to organize the core ID allocation array. -Once the floating process is spawned, it tries to allocate a unique core ID from the array and release it on exit. - -A natural way to spawn a floating process is to use the fork() function and allocate a unique core ID from the unused core ID array. -However, it is necessary to write new code to provide a notification mechanism for slave exit -and make sure the process recovery mechanism can work with it. - -To avoid producing redundant code, the Master-Slave process model is still used to spawn floating processes, -then cancel the affinity to specific cores. -Besides that, clear the core ID assigned to the DPDK spawning a thread that has a 1:1 mapping with the core mask. -Thereafter, get a new core ID from the unused core ID allocation array. - -Run the Application -^^^^^^^^^^^^^^^^^^^ - -This example has a command line similar to the L2 Forwarding sample application with a few differences. - -To run the application, start one copy of the l2fwd_fork binary in one terminal. -Unlike the L2 Forwarding example, -this example requires at least three cores since the master process will wait and be accountable for slave process recovery. -The command is as follows: - -.. code-block:: console - - #./build/l2fwd_fork -l 2-4 -n 4 -- -p 3 -f - -This example provides another -f option to specify the use of floating process. -If not specified, the example will use a pinned process to perform the L2 forwarding task. - -To verify the recovery mechanism, proceed as follows: First, check the PID of the slave processes: - -.. code-block:: console - - #ps -fe | grep l2fwd_fork - root 5136 4843 29 11:11 pts/1 00:00:05 ./build/l2fwd_fork - root 5145 5136 98 11:11 pts/1 00:00:11 ./build/l2fwd_fork - root 5146 5136 98 11:11 pts/1 00:00:11 ./build/l2fwd_fork - -Then, kill one of the slaves: - -.. code-block:: console - - #kill -9 5145 - -After 1 or 2 seconds, check whether the slave has resumed: - -.. code-block:: console - - #ps -fe | grep l2fwd_fork - root 5136 4843 3 11:11 pts/1 00:00:06 ./build/l2fwd_fork - root 5247 5136 99 11:14 pts/1 00:00:01 ./build/l2fwd_fork - root 5248 5136 99 11:14 pts/1 00:00:01 ./build/l2fwd_fork - -It can also monitor the traffic generator statics to see whether slave processes have resumed. - -Explanation -^^^^^^^^^^^ - -As described in previous sections, -not all global and static variables need to change to be accessible in multiple processes; -it depends on how they are used. -In this example, -the statics info on packets dropped/forwarded/received count needs to be updated by the slave process, -and the master needs to see the update and print them out. -So, it needs to allocate a heap buffer using rte_zmalloc. -In addition, if the -f option is specified, -an array is needed to store the allocated core ID for the floating process so that the master can return it -after a slave has exited accidentally. - -.. code-block:: c - - static int - l2fwd_malloc_shared_struct(void) - { - port_statistics = rte_zmalloc("port_stat", sizeof(struct l2fwd_port_statistics) * RTE_MAX_ETHPORTS, 0); - - if (port_statistics == NULL) - return -1; - - /* allocate mapping_id array */ - - if (float_proc) { - int i; - - mapping_id = rte_malloc("mapping_id", sizeof(unsigned) * RTE_MAX_LCORE, 0); - if (mapping_id == NULL) - return -1; - - for (i = 0 ;i < RTE_MAX_LCORE; i++) - mapping_id[i] = INVALID_MAPPING_ID; - - } - return 0; - } - -For each slave process, packets are received from one port and forwarded to another port that another slave is operating on. -If the other slave exits accidentally, the port it is operating on may not work normally, -so the first slave cannot forward packets to that port. -There is a dependency on the port in this case. So, the master should recognize the dependency. -The following is the code to detect this dependency: - -.. code-block:: c - - for (portid = 0; portid < nb_ports; portid++) { - /* skip ports that are not enabled */ - - if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) - continue; - - /* Find pair ports' lcores */ - - find_lcore = find_pair_lcore = 0; - pair_port = l2fwd_dst_ports[portid]; - - for (i = 0; i < RTE_MAX_LCORE; i++) { - if (!rte_lcore_is_enabled(i)) - continue; - - for (j = 0; j < lcore_queue_conf[i].n_rx_port;j++) { - if (lcore_queue_conf[i].rx_port_list[j] == portid) { - lcore = i; - find_lcore = 1; - break; - } - - if (lcore_queue_conf[i].rx_port_list[j] == pair_port) { - pair_lcore = i; - find_pair_lcore = 1; - break; - } - } - - if (find_lcore && find_pair_lcore) - break; - } - - if (!find_lcore || !find_pair_lcore) - rte_exit(EXIT_FAILURE, "Not find port=%d pair\\n", portid); - - printf("lcore %u and %u paired\\n", lcore, pair_lcore); - - lcore_resource[lcore].pair_id = pair_lcore; - lcore_resource[pair_lcore].pair_id = lcore; - } - -Before launching the slave process, -it is necessary to set up the communication channel between the master and slave so that -the master can notify the slave if its peer process with the dependency exited. -In addition, the master needs to register a callback function in the case where a specific slave exited. - -.. code-block:: c - - for (i = 0; i < RTE_MAX_LCORE; i++) { - if (lcore_resource[i].enabled) { - /* Create ring for master and slave communication */ - - ret = create_ms_ring(i); - if (ret != 0) - rte_exit(EXIT_FAILURE, "Create ring for lcore=%u failed",i); - - if (flib_register_slave_exit_notify(i,slave_exit_cb) != 0) - rte_exit(EXIT_FAILURE, "Register master_trace_slave_exit failed"); - } - } - -After launching the slave process, the master waits and prints out the port statics periodically. -If an event indicating that a slave process exited is detected, -it sends the STOP command to the peer and waits until it has also exited. -Then, it tries to clean up the execution environment and prepare new resources. -Finally, the new slave instance is launched. - -.. code-block:: c - - while (1) { - sleep(1); - cur_tsc = rte_rdtsc(); - diff_tsc = cur_tsc - prev_tsc; - - /* if timer is enabled */ - - if (timer_period > 0) { - /* advance the timer */ - timer_tsc += diff_tsc; - - /* if timer has reached its timeout */ - if (unlikely(timer_tsc >= (uint64_t) timer_period)) { - print_stats(); - - /* reset the timer */ - timer_tsc = 0; - } - } - - prev_tsc = cur_tsc; - - /* Check any slave need restart or recreate */ - - rte_spinlock_lock(&res_lock); - - for (i = 0; i < RTE_MAX_LCORE; i++) { - struct lcore_resource_struct *res = &lcore_resource[i]; - struct lcore_resource_struct *pair = &lcore_resource[res->pair_id]; - - /* If find slave exited, try to reset pair */ - - if (res->enabled && res->flags && pair->enabled) { - if (!pair->flags) { - master_sendcmd_with_ack(pair->lcore_id, CMD_STOP); - rte_spinlock_unlock(&res_lock); - sleep(1); - rte_spinlock_lock(&res_lock); - if (pair->flags) - continue; - } - - if (reset_pair(res->lcore_id, pair->lcore_id) != 0) - rte_exit(EXIT_FAILURE, "failed to reset slave"); - - res->flags = 0; - pair->flags = 0; - } - } - rte_spinlock_unlock(&res_lock); - } - -When the slave process is spawned and starts to run, it checks whether the floating process option is applied. -If so, it clears the affinity to a specific core and also sets the unique core ID to 0. -Then, it tries to allocate a new core ID. -Since the core ID has changed, the resource allocated by the master cannot work, -so it remaps the resource to the new core ID slot. - -.. code-block:: c - - static int - l2fwd_launch_one_lcore( attribute ((unused)) void *dummy) - { - unsigned lcore_id = rte_lcore_id(); - - if (float_proc) { - unsigned flcore_id; - - /* Change it to floating process, also change it's lcore_id */ - - clear_cpu_affinity(); - - RTE_PER_LCORE(_lcore_id) = 0; - - /* Get a lcore_id */ - - if (flib_assign_lcore_id() < 0 ) { - printf("flib_assign_lcore_id failed\n"); - return -1; - } - - flcore_id = rte_lcore_id(); - - /* Set mapping id, so master can return it after slave exited */ - - mapping_id[lcore_id] = flcore_id; - printf("Org lcore_id = %u, cur lcore_id = %u\n",lcore_id, flcore_id); - remapping_slave_resource(lcore_id, flcore_id); - } - - l2fwd_main_loop(); - - /* return lcore_id before return */ - if (float_proc) { - flib_free_lcore_id(rte_lcore_id()); - mapping_id[lcore_id] = INVALID_MAPPING_ID; - } - return 0; - } diff --git a/doc/guides/sample_app_ug/quota_watermark.rst b/doc/guides/sample_app_ug/quota_watermark.rst index 8baec4df..67200e15 100644 --- a/doc/guides/sample_app_ug/quota_watermark.rst +++ b/doc/guides/sample_app_ug/quota_watermark.rst @@ -163,7 +163,7 @@ Then, a call to init_dpdk(), defined in init.c, is made to initialize the poll m if (ret < 0) rte_exit(EXIT_FAILURE, "rte_pci_probe(): error %d\n", ret); - if (rte_eth_dev_count() < 2) + if (rte_eth_dev_count_avail() < 2) rte_exit(EXIT_FAILURE, "Not enough Ethernet port available\n"); } diff --git a/doc/guides/sample_app_ug/rxtx_callbacks.rst b/doc/guides/sample_app_ug/rxtx_callbacks.rst index 0bb0d3e0..85d96d8a 100644 --- a/doc/guides/sample_app_ug/rxtx_callbacks.rst +++ b/doc/guides/sample_app_ug/rxtx_callbacks.rst @@ -83,9 +83,6 @@ comments: int retval; uint16_t q; - if (port >= rte_eth_dev_count()) - return -1; - /* Configure the Ethernet device. */ retval = rte_eth_dev_configure(port, rx_rings, tx_rings, &port_conf); if (retval != 0) diff --git a/doc/guides/sample_app_ug/skeleton.rst b/doc/guides/sample_app_ug/skeleton.rst index 0503584d..95956171 100644 --- a/doc/guides/sample_app_ug/skeleton.rst +++ b/doc/guides/sample_app_ug/skeleton.rst @@ -81,7 +81,7 @@ The ``main()`` function also initializes all the ports using the user defined .. code-block:: c - for (portid = 0; portid < nb_ports; portid++) { + RTE_ETH_FOREACH_DEV(portid) { if (port_init(portid, mbuf_pool) != 0) { rte_exit(EXIT_FAILURE, "Cannot init port %" PRIu8 "\n", portid); @@ -119,7 +119,7 @@ Forwarding application is shown below: int retval; uint16_t q; - if (port >= rte_eth_dev_count()) + if (!rte_eth_dev_is_valid_port(port)) return -1; /* Configure the Ethernet device. */ @@ -192,14 +192,13 @@ looks like the following: static __attribute__((noreturn)) void lcore_main(void) { - const uint16_t nb_ports = rte_eth_dev_count(); uint16_t port; /* * Check that the port is on the same NUMA node as the polling thread * for best performance. */ - for (port = 0; port < nb_ports; port++) + RTE_ETH_FOREACH_DEV(port) if (rte_eth_dev_socket_id(port) > 0 && rte_eth_dev_socket_id(port) != (int)rte_socket_id()) @@ -216,7 +215,7 @@ looks like the following: * Receive packets on a port and forward them on the paired * port. The mapping is 0 -> 1, 1 -> 0, 2 -> 3, 3 -> 2, etc. */ - for (port = 0; port < nb_ports; port++) { + RTE_ETH_FOREACH_DEV(port) { /* Get burst of RX packets, from first port of pair. */ struct rte_mbuf *bufs[BURST_SIZE]; @@ -246,7 +245,7 @@ The main work of the application is done within the loop: .. code-block:: c for (;;) { - for (port = 0; port < nb_ports; port++) { + RTE_ETH_FOREACH_DEV(port) { /* Get burst of RX packets, from first port of pair. */ struct rte_mbuf *bufs[BURST_SIZE]; diff --git a/doc/guides/sample_app_ug/vhost.rst b/doc/guides/sample_app_ug/vhost.rst index a4bdc6a4..fd42cb3f 100644 --- a/doc/guides/sample_app_ug/vhost.rst +++ b/doc/guides/sample_app_ug/vhost.rst @@ -147,7 +147,10 @@ retries on an RX burst, it takes effect only when rx retry is enabled. The default value is 15. **--dequeue-zero-copy** -Dequeue zero copy will be enabled when this option is given. +Dequeue zero copy will be enabled when this option is given. it is worth to +note that if NIC is binded to driver with iommu enabled, dequeue zero copy +cannot work at VM2NIC mode (vm2vm=0) due to currently we don't setup iommu +dma mapping for guest memory. **--vlan-strip 0|1** VLAN strip option is removed, because different NICs have different behaviors @@ -155,6 +158,10 @@ when disabling VLAN strip. Such feature, which heavily depends on hardware, should be removed from this example to reduce confusion. Now, VLAN strip is enabled and cannot be disabled. +**--builtin-net-driver** +A very simple vhost-user net driver which demonstrates how to use the generic +vhost APIs will be used when this option is given. It is disabled by default. + Common Issues ------------- @@ -175,15 +182,22 @@ Common Issues The command above indicates how many hugepages are free to support QEMU's allocation request. -* vhost-user will not work with QEMU without the ``-mem-prealloc`` option +* Failed to build DPDK in VM - The current implementation works properly only when the guest memory is - pre-allocated. + Make sure "-cpu host" QEMU option is given. -* vhost-user will not work with a QEMU version without shared memory mapping: +* Device start fails if NIC's max queues > the default number of 128 - Make sure ``share=on`` QEMU option is given. + mbuf pool size is dependent on the MAX_QUEUES configuration, if NIC's + max queue number is larger than 128, device start will fail due to + insufficient mbuf. -* Failed to build DPDK in VM + Change the default number to make it work as below, just set the number + according to the NIC's property. :: - Make sure "-cpu host" QEMU option is given. + make EXTRA_CFLAGS="-DMAX_QUEUES=320" + +* Option "builtin-net-driver" is incompatible with QEMU + + QEMU vhost net device start will fail if protocol feature is not negotiated. + DPDK virtio-user pmd can be the replacement of QEMU. diff --git a/doc/guides/sample_app_ug/vhost_crypto.rst b/doc/guides/sample_app_ug/vhost_crypto.rst new file mode 100644 index 00000000..65c86a53 --- /dev/null +++ b/doc/guides/sample_app_ug/vhost_crypto.rst @@ -0,0 +1,82 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2017-2018 Intel Corporation. + +Vhost_Crypto Sample Application +=============================== + +The vhost_crypto sample application implemented a simple Crypto device, +which used as the backend of Qemu vhost-user-crypto device. Similar with +vhost-user-net and vhost-user-scsi device, the sample application used +domain socket to communicate with Qemu, and the virtio ring was processed +by vhost_crypto sample application. + +Testing steps +------------- + +This section shows the steps how to start a VM with the crypto device as +fast data path for critical application. + +Compiling the Application +------------------------- + +To compile the sample application see :doc:`compiling`. + +The application is located in the ``examples`` sub-directory. + +Start the vhost_crypto example +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: console + + ./vhost_crypto [EAL options] -- [--socket-file PATH] + [--cdev-id ID] [--cdev-queue-id ID] [--zero-copy] [--guest-polling] + +where, + +* socket-file PATH: the path of UNIX socket file to be created, multiple + instances of this config item is supported. Upon absence of this item, + the default socket-file `/tmp/vhost_crypto1.socket` is used. + +* cdev-id ID: the target DPDK Cryptodev's ID to process the actual crypto + workload. Upon absence of this item the default value of `0` will be used. + For details of DPDK Cryptodev, please refer to DPDK Cryptodev Library + Programmers' Guide. + +* cdev-queue-id ID: the target DPDK Cryptodev's queue ID to process the + actual crypto workload. Upon absence of this item the default value of `0` + will be used. For details of DPDK Cryptodev, please refer to DPDK Cryptodev + Library Programmers' Guide. + +* zero-copy: the presence of this item means the ZERO-COPY feature will be + enabled. Otherwise it is disabled. PLEASE NOTE the ZERO-COPY feature is still + in experimental stage and may cause the problem like segmentation fault. If + the user wants to use LKCF in the guest, this feature shall be turned off. + +* guest-polling: the presence of this item means the application assumes the + guest works in polling mode, thus will NOT notify the guest completion of + processing. + +The application requires that crypto devices capable of performing +the specified crypto operation are available on application initialization. +This means that HW crypto device/s must be bound to a DPDK driver or +a SW crypto device/s (virtual crypto PMD) must be created (using --vdev). + +.. _vhost_crypto_app_run_vm: + +Start the VM +~~~~~~~~~~~~ + +.. code-block:: console + + qemu-system-x86_64 -machine accel=kvm \ + -m $mem -object memory-backend-file,id=mem,size=$mem,\ + mem-path=/dev/hugepages,share=on -numa node,memdev=mem \ + -drive file=os.img,if=none,id=disk \ + -device ide-hd,drive=disk,bootindex=0 \ + -chardev socket,id={chardev_id},path={PATH} \ + -object cryptodev-vhost-user,id={obj_id},chardev={chardev_id} \ + -device virtio-crypto-pci,id={dev_id},cryptodev={obj_id} \ + ... + +.. note:: + You must check whether your Qemu can support "vhost-user-crypto" or not. diff --git a/doc/guides/sample_app_ug/vm_power_management.rst b/doc/guides/sample_app_ug/vm_power_management.rst index cd7c7f34..855570d6 100644 --- a/doc/guides/sample_app_ug/vm_power_management.rst +++ b/doc/guides/sample_app_ug/vm_power_management.rst @@ -20,7 +20,7 @@ running on Virtual Machines(VMs). The Virtual Machine Power Management solution shows an example of how a DPDK application can indicate its processing requirements using VM local -only information(vCPU/lcore) to a Host based Monitor which is responsible +only information(vCPU/lcore, etc.) to a Host based Monitor which is responsible for accepting requests for frequency changes for a vCPU, translating the vCPU to a pCPU via libvirt and affecting the change in frequency. @@ -38,6 +38,26 @@ The solution is comprised of two high-level components: to the librte_power ACPI cpufreq sysfs based library. The Host Application relies on both qemu-kvm and libvirt to function. + This monitoring application is responsible for: + + - Accepting requests from client applications: Client applications can + request frequency changes for a vCPU, translating + the vCPU to a pCPU via libvirt and affecting the change in frequency. + + - Accepting policies from client applications: Client application can + send a policy to the host application. The + host application will then apply the rules of the policy independent + of the application. For example, the policy can contain time-of-day + information for busy/quiet periods, and the host application can scale + up/down the relevant cores when required. See the details of the guest + application below for more information on setting the policy values. + + - Out-of-band monitoring of workloads via cores hardware event counters: + The host application can manage power for an application in a virtualised + OR non-virtualised environment by looking at the event counters of the + cores and taking action based on the branch hit/miss ratio. See the host + application '--core-list' command line parameter below. + #. librte_power for Virtual Machines Using an alternate implementation for the librte_power API, requests for @@ -174,13 +194,20 @@ Compiling and Running the Host Application Compiling ~~~~~~~~~ -Compiling the Application -------------------------- - -To compile the sample application see :doc:`compiling`. +For information on compiling DPDK and the sample applications +see :doc:`compiling`. The application is located in the ``vm_power_manager`` sub-directory. +To build just the ``vm_power_manager`` application: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + export RTE_TARGET=build + cd ${RTE_SDK}/examples/vm_power_manager/ + make + Running ~~~~~~~ @@ -287,38 +314,129 @@ Manual control and inspection can also be carried in relation CPU frequency scal set_cpu_freq {core_num} up|down|min|max +There are also some command line parameters for enabling the out-of-band +monitoring of branch ratio on cores doing busy polling via PMDs. + + .. code-block:: console + + --core-list {list of cores} + + When this parameter is used, the list of cores specified will monitor the ratio + between branch hits and branch misses. A tightly polling PMD thread will have a + very low branch ratio, so the core frequency will be scaled down to the minimim + allowed value. When packets are received, the code path will alter, causing the + branch ratio to increase. When the ratio goes above the ratio threshold, the + core frequency will be scaled up to the maximum allowed value. + + .. code-block:: console + + --branch-ratio {ratio} + + The branch ratio is a floating point number that specifies the threshold at which + to scale up or down for the given workload. The default branch ratio is 0.01, + and will need to be adjusted for different workloads. + + Compiling and Running the Guest Applications -------------------------------------------- -For compiling and running l3fwd-power, see :doc:`l3_forward_power_man`. +l3fwd-power is one sample application that can be used with vm_power_manager. A guest CLI is also provided for validating the setup. For both l3fwd-power and guest CLI, the channels for the VM must be monitored by the -host application using the *add_channels* command on the host. +host application using the *add_channels* command on the host. This typically uses +the following commands in the host application: + +.. code-block:: console + + vm_power> add_vm vmname + vm_power> add_channels vmname all + vm_power> set_channel_status vmname all enabled + vm_power> show_vm vmname + Compiling ~~~~~~~~~ -#. export RTE_SDK=/path/to/rte_sdk -#. cd ${RTE_SDK}/examples/vm_power_manager/guest_cli -#. make +For information on compiling DPDK and the sample applications +see :doc:`compiling`. + +For compiling and running l3fwd-power, see :doc:`l3_forward_power_man`. + +The application is located in the ``guest_cli`` sub-directory under ``vm_power_manager``. + +To build just the ``guest_vm_power_manager`` application: + +.. code-block:: console + + export RTE_SDK=/path/to/rte_sdk + export RTE_TARGET=build + cd ${RTE_SDK}/examples/vm_power_manager/guest_cli/ + make Running ~~~~~~~ -The application does not have any specific command line options other than *EAL*: +The standard *EAL* command line parameters are required: .. code-block:: console - ./build/vm_power_mgr [EAL options] + ./build/guest_vm_power_mgr [EAL options] -- [guest options] -The application for example purposes uses a channel for each lcore enabled, -for example to run on cores 0,1,2,3 on a system with 4 memory channels: +The guest example uses a channel for each lcore enabled. For example, +to run on cores 0,1,2,3: .. code-block:: console - ./build/guest_vm_power_mgr -l 0-3 -n 4 + ./build/guest_vm_power_mgr -l 0-3 + +Optionally, there is a list of command line parameter should the user wish to send a power +policy down to the host application. These parameters are as follows: + + .. code-block:: console + + --vm-name {name of guest vm} + + This parameter allows the user to change the Virtual Machine name passed down to the + host application via the power policy. The default is "ubuntu2" + + .. code-block:: console + + --vcpu-list {list vm cores} + + A comma-separated list of cores in the VM that the user wants the host application to + monitor. The list of cores in any vm starts at zero, and these are mapped to the + physical cores by the host application once the policy is passed down. + Valid syntax includes individial cores '2,3,4', or a range of cores '2-4', or a + combination of both '1,3,5-7' + + .. code-block:: console + + --busy-hours {list of busy hours} + + A comma-separated list of hours within which to set the core frequency to maximum. + Valid syntax includes individial hours '2,3,4', or a range of hours '2-4', or a + combination of both '1,3,5-7'. Valid hours are 0 to 23. + + .. code-block:: console + + --quiet-hours {list of quiet hours} + + A comma-separated list of hours within which to set the core frequency to minimum. + Valid syntax includes individial hours '2,3,4', or a range of hours '2-4', or a + combination of both '1,3,5-7'. Valid hours are 0 to 23. + + .. code-block:: console + + --policy {policy type} + + The type of policy. This can be one of the following values: + TRAFFIC - based on incoming traffic rates on the NIC. + TIME - busy/quiet hours policy. + BRANCH_RATIO - uses branch ratio counters to determine core busyness. + Not all parameters are needed for all policy types. For example, BRANCH_RATIO + only needs the vcpu-list parameter, not any of the hours. After successful initialization the user is presented with VM Power Manager Guest CLI: @@ -333,3 +451,20 @@ Where {core_num} is the lcore and channel to change frequency by scaling up/down .. code-block:: console set_cpu_freq {core_num} up|down|min|max + +To start the application and configure the power policy, and send it to the host: + +.. code-block:: console + + ./build/guest_vm_power_mgr -l 0-3 -n 4 -- --vm-name=ubuntu --policy=BRANCH_RATIO --vcpu-list=2-4 + +Once the VM Power Manager Guest CLI appears, issuing the 'send_policy now' command +will send the policy to the host: + +.. code-block:: console + + send_policy now + +Once the policy is sent to the host, the host application takes over the power monitoring +of the specified cores in the policy. + diff --git a/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst b/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst index 83fcdf63..0e9da970 100644 --- a/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst +++ b/doc/guides/sample_app_ug/vmdq_dcb_forwarding.rst @@ -102,10 +102,6 @@ a default structure is provided for VMDQ and DCB configuration to be filled in l .rxmode = { .mq_mode = ETH_MQ_RX_VMDQ_DCB, .split_hdr_size = 0, - .header_split = 0, /**< Header Split disabled */ - .hw_ip_checksum = 0, /**< IP checksum offload disabled */ - .hw_vlan_filter = 0, /**< VLAN filtering disabled */ - .jumbo_frame = 0, /**< Jumbo Frame Support disabled */ }, .txmode = { .mq_mode = ETH_MQ_TX_VMDQ_DCB, diff --git a/doc/guides/testpmd_app_ug/run_app.rst b/doc/guides/testpmd_app_ug/run_app.rst index 1fd53958..f301c2b6 100644 --- a/doc/guides/testpmd_app_ug/run_app.rst +++ b/doc/guides/testpmd_app_ug/run_app.rst @@ -372,7 +372,9 @@ The commandline options are: * ``--burst=N`` Set the number of packets per burst to N, where 1 <= N <= 512. - The default value is 16. + The default value is 32. + If set to 0, driver default is used if defined. Else, if driver + default is not defined, default of 32 is used. * ``--mbcache=N`` @@ -479,3 +481,20 @@ The commandline options are: Set the hexadecimal bitmask of TX queue offloads. The default value is 0. + +* ``--hot-plug`` + + Enable device event monitor machenism for hotplug. + +* ``--vxlan-gpe-port=N`` + + Set the UDP port number of tunnel VXLAN-GPE to N. + The default value is 4790. + +* ``--mlockall`` + + Enable locking all memory. + +* ``--no-mlockall`` + + Disable locking all memory. diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst index a766ac79..dde205a2 100644 --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst @@ -320,12 +320,8 @@ The available information categories are: * ``ieee1588``: Demonstrate L2 IEEE1588 V2 PTP timestamping for RX and TX. Requires ``CONFIG_RTE_LIBRTE_IEEE1588=y``. -* ``tm``: Traffic Management forwarding mode - Demonstrates the use of ethdev traffic management APIs and softnic PMD for - QoS traffic management. In this mode, 5-level hierarchical QoS scheduler is - available as an default option that can be enabled through CLI. The user can - also modify the default hierarchy or specify the new hierarchy through CLI for - implementing QoS scheduler. Requires ``CONFIG_RTE_LIBRTE_PMD_SOFTNIC=y`` ``CONFIG_RTE_LIBRTE_SCHED=y``. +* ``softnic``: Demonstrates the softnic forwarding operation. In this mode, packet forwarding is + similar to I/O mode except for the fact that packets are loopback to the softnic ports only. Therefore, portmask parameter should be set to softnic port only. The various software based custom NIC pipelines specified through the softnic firmware (DPDK packet framework script) can be tested in this mode. Furthermore, it allows to build 5-level hierarchical QoS scheduler as a default option that can be enabled through CLI once testpmd application is initialised. The user can modify the default scheduler hierarchy or can specify the new QoS Scheduler hierarchy through CLI. Requires ``CONFIG_RTE_LIBRTE_PMD_SOFTNIC=y``. Example:: @@ -393,6 +389,34 @@ List all items from the pctype mapping table:: testpmd> show port (port_id) pctype mapping +show rx offloading capabilities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +List all per queue and per port Rx offloading capabilities of a port:: + + testpmd> show port (port_id) rx_offload capabilities + +show rx offloading configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +List port level and all queue level Rx offloading configuration:: + + testpmd> show port (port_id) rx_offload configuration + +show tx offloading capabilities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +List all per queue and per port Tx offloading capabilities of a port:: + + testpmd> show port (port_id) tx_offload capabilities + +show tx offloading configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +List port level and all queue level Tx offloading configuration:: + + testpmd> show port (port_id) tx_offload configuration + Configuration Functions ----------------------- @@ -1031,6 +1055,13 @@ By default, GSO is disabled for all ports. testpmd> csum set tcp hw <port_id> + UDP GSO is the same as IP fragmentation, which treats the UDP header + as the payload and does not modify it during segmentation. That is, + after UDP GSO, only the first output fragment has the original UDP + header. Therefore, users need to enable HW IP checksum calculation + and SW UDP checksum calculation for GSO-enabled ports, if they want + correct checksums for UDP/IPv4 packets. + set gso segsz ~~~~~~~~~~~~~ @@ -1444,6 +1475,101 @@ Reset ptype mapping table:: testpmd> ptype mapping reset (port_id) +config per port Rx offloading +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Enable or disable a per port Rx offloading on all Rx queues of a port:: + + testpmd> port config (port_id) rx_offload (offloading) on|off + +* ``offloading``: can be any of these offloading capability: + vlan_strip, ipv4_cksum, udp_cksum, tcp_cksum, tcp_lro, + qinq_strip, outer_ipv4_cksum, macsec_strip, + header_split, vlan_filter, vlan_extend, jumbo_frame, + crc_strip, scatter, timestamp, security, keep_crc + +This command should be run when the port is stopped, or else it will fail. + +config per queue Rx offloading +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Enable or disable a per queue Rx offloading only on a specific Rx queue:: + + testpmd> port (port_id) rxq (queue_id) rx_offload (offloading) on|off + +* ``offloading``: can be any of these offloading capability: + vlan_strip, ipv4_cksum, udp_cksum, tcp_cksum, tcp_lro, + qinq_strip, outer_ipv4_cksum, macsec_strip, + header_split, vlan_filter, vlan_extend, jumbo_frame, + crc_strip, scatter, timestamp, security, keep_crc + +This command should be run when the port is stopped, or else it will fail. + +config per port Tx offloading +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Enable or disable a per port Tx offloading on all Tx queues of a port:: + + testpmd> port config (port_id) tx_offload (offloading) on|off + +* ``offloading``: can be any of these offloading capability: + vlan_insert, ipv4_cksum, udp_cksum, tcp_cksum, + sctp_cksum, tcp_tso, udp_tso, outer_ipv4_cksum, + qinq_insert, vxlan_tnl_tso, gre_tnl_tso, + ipip_tnl_tso, geneve_tnl_tso, macsec_insert, + mt_lockfree, multi_segs, mbuf_fast_free, security + +This command should be run when the port is stopped, or else it will fail. + +config per queue Tx offloading +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Enable or disable a per queue Tx offloading only on a specific Tx queue:: + + testpmd> port (port_id) txq (queue_id) tx_offload (offloading) on|off + +* ``offloading``: can be any of these offloading capability: + vlan_insert, ipv4_cksum, udp_cksum, tcp_cksum, + sctp_cksum, tcp_tso, udp_tso, outer_ipv4_cksum, + qinq_insert, vxlan_tnl_tso, gre_tnl_tso, + ipip_tnl_tso, geneve_tnl_tso, macsec_insert, + mt_lockfree, multi_segs, mbuf_fast_free, security + +This command should be run when the port is stopped, or else it will fail. + +Config VXLAN Encap outer layers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configure the outer layer to encapsulate a packet inside a VXLAN tunnel:: + + set vxlan ip-version (ipv4|ipv6) vni (vni) udp-src (udp-src) \ + udp-dst (udp-dst) ip-src (ip-src) ip-dst (ip-dst) eth-src (eth-src) \ + eth-dst (eth-dst) + + set vxlan-with-vlan ip-version (ipv4|ipv6) vni (vni) udp-src (udp-src) \ + udp-dst (udp-dst) ip-src (ip-src) ip-dst (ip-dst) vlan-tci (vlan-tci) \ + eth-src (eth-src) eth-dst (eth-dst) + +Those command will set an internal configuration inside testpmd, any following +flow rule using the action vxlan_encap will use the last configuration set. +To have a different encapsulation header, one of those commands must be called +before the flow rule creation. + +Config NVGRE Encap outer layers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configure the outer layer to encapsulate a packet inside a NVGRE tunnel:: + + set nvgre ip-version (ipv4|ipv6) tni (tni) ip-src (ip-src) ip-dst (ip-dst) \ + eth-src (eth-src) eth-dst (eth-dst) + set nvgre-with-vlan ip-version (ipv4|ipv6) tni (tni) ip-src (ip-src) \ + ip-dst (ip-dst) vlan-tci (vlan-tci) eth-src (eth-src) eth-dst (eth-dst) + +Those command will set an internal configuration inside testpmd, any following +flow rule using the action nvgre_encap will use the last configuration set. +To have a different encapsulation header, one of those commands must be called +before the flow rule creation. + Port Functions -------------- @@ -1623,6 +1749,15 @@ Close all ports or a specific port:: testpmd> port close (port_id|all) +port config - queue ring size +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configure a rx/tx queue ring size:: + + testpmd> port (port_id) (rxq|txq) (queue_id) ring_size (value) + +Only take effect after command that (re-)start the port or command that setup specific queue. + port start/stop queue ~~~~~~~~~~~~~~~~~~~~~ @@ -1630,6 +1765,13 @@ Start/stop a rx/tx queue on a specific port:: testpmd> port (port_id) (rxq|txq) (queue_id) (start|stop) +port setup queue +~~~~~~~~~~~~~~~~~~~~~ + +Setup a rx/tx queue on a specific port:: + + testpmd> port (port_id) (rxq|txq) (queue_id) setup + Only take effect when port is started. port config - speed @@ -1751,10 +1893,12 @@ port config - RSS Set the RSS (Receive Side Scaling) mode on or off:: - testpmd> port config all rss (all|ip|tcp|udp|sctp|ether|port|vxlan|geneve|nvgre|none) + testpmd> port config all rss (all|default|ip|tcp|udp|sctp|ether|port|vxlan|geneve|nvgre|none) RSS is on by default. +The ``all`` option is equivalent to ip|tcp|udp|sctp|ether. +The ``default`` option enables all supported RSS types reported by device info. The ``none`` option is equivalent to the ``--disable-rss`` command-line option. port config - RSS Reta @@ -1856,6 +2000,12 @@ where: * ``pctype_id``: hardware packet classification types. * ``field_idx``: hardware field index. +port config udp_tunnel_port +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add/remove UDP tunnel port for VXLAN/GENEVE tunneling protocols:: + testpmd> port config (port_id) udp_tunnel_port add|rm vxlan|geneve (udp_port) + Link Bonding Functions ---------------------- @@ -1871,7 +2021,7 @@ Create a new bonding device:: For example, to create a bonded device in mode 1 on socket 0:: - testpmd> create bonded 1 0 + testpmd> create bonded device 1 0 created new bonded device (port X) add bonding slave @@ -2483,6 +2633,16 @@ success depends on the port support for this operation, as advertised through the port capability set. This function is valid for all nodes of the traffic management hierarchy except root node. +Suspend port traffic management hierarchy node +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + testpmd> suspend port tm node (port_id) (node_id) + +Resume port traffic management hierarchy node +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + + testpmd> resume port tm node (port_id) (node_id) + Commit port traffic management hierarchy ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2496,8 +2656,8 @@ where: call failure. On the other hand, hierarchy is preserved when this parameter is equal to zero. -Set port traffic management default hierarchy (tm forwarding mode) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Set port traffic management default hierarchy (softnic forwarding mode) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ set the traffic management default hierarchy on the port:: @@ -2970,14 +3130,14 @@ following sections. - Check whether a flow rule can be created:: flow validate {port_id} - [group {group_id}] [priority {level}] [ingress] [egress] + [group {group_id}] [priority {level}] [ingress] [egress] [transfer] pattern {item} [/ {item} [...]] / end actions {action} [/ {action} [...]] / end - Create a flow rule:: flow create {port_id} - [group {group_id}] [priority {level}] [ingress] [egress] + [group {group_id}] [priority {level}] [ingress] [egress] [transfer] pattern {item} [/ {item} [...]] / end actions {action} [/ {action} [...]] / end @@ -3010,7 +3170,7 @@ underlying device in its current state but stops short of creating it. It is bound to ``rte_flow_validate()``:: flow validate {port_id} - [group {group_id}] [priority {level}] [ingress] [egress] + [group {group_id}] [priority {level}] [ingress] [egress] [transfer] pattern {item} [/ {item} [...]] / end actions {action} [/ {action} [...]] / end @@ -3047,7 +3207,7 @@ Creating flow rules to ``rte_flow_create()``:: flow create {port_id} - [group {group_id}] [priority {level}] [ingress] [egress] + [group {group_id}] [priority {level}] [ingress] [egress] [transfer] pattern {item} [/ {item} [...]] / end actions {action} [/ {action} [...]] / end @@ -3061,7 +3221,7 @@ Otherwise it will show an error message of the form:: Parameters describe in the following order: -- Attributes (*group*, *priority*, *ingress*, *egress* tokens). +- Attributes (*group*, *priority*, *ingress*, *egress*, *transfer* tokens). - A matching pattern, starting with the *pattern* token and terminated by an *end* pattern item. - Actions, starting with the *actions* token and terminated by an *end* @@ -3089,6 +3249,7 @@ specified before the ``pattern`` token. - ``priority {level}``: priority level within group. - ``ingress``: rule applies to ingress traffic. - ``egress``: rule applies to egress traffic. +- ``transfer``: apply rule directly to endpoints found in pattern. Each instance of an attribute specified several times overrides the previous value as shown below (group 4 is used):: @@ -3201,16 +3362,24 @@ This section lists supported pattern items and their attributes, if any. - ``num {unsigned}``: number of layers covered. -- ``pf``: match packets addressed to the physical function. +- ``pf``: match traffic from/to the physical function. -- ``vf``: match packets addressed to a virtual function ID. +- ``vf``: match traffic from/to a virtual function ID. - - ``id {unsigned}``: destination VF ID. + - ``id {unsigned}``: VF ID. -- ``port``: device-specific physical port index to use. +- ``phy_port``: match traffic from/to a specific physical port. - ``index {unsigned}``: physical port index. +- ``port_id``: match traffic from/to a given DPDK port ID. + + - ``id {unsigned}``: DPDK port ID. + +- ``mark``: match value set in previously matched flow rule using the mark action. + + - ``id {unsigned}``: arbitrary integer value. + - ``raw``: match an arbitrary byte string. - ``relative {boolean}``: look for pattern after the previous item. @@ -3223,15 +3392,15 @@ This section lists supported pattern items and their attributes, if any. - ``dst {MAC-48}``: destination MAC. - ``src {MAC-48}``: source MAC. - - ``type {unsigned}``: EtherType. + - ``type {unsigned}``: EtherType or TPID. - ``vlan``: match 802.1Q/ad VLAN tag. - - ``tpid {unsigned}``: tag protocol identifier. - ``tci {unsigned}``: tag control information. - ``pcp {unsigned}``: priority code point. - ``dei {unsigned}``: drop eligible indicator. - ``vid {unsigned}``: VLAN identifier. + - ``inner_type {unsigned}``: inner EtherType or TPID. - ``ipv4``: match IPv4 header. @@ -3305,6 +3474,48 @@ This section lists supported pattern items and their attributes, if any. - ``vni {unsigned}``: virtual network identifier. - ``protocol {unsigned}``: protocol type. +- ``vxlan-gpe``: match VXLAN-GPE header. + + - ``vni {unsigned}``: VXLAN-GPE identifier. + +- ``arp_eth_ipv4``: match ARP header for Ethernet/IPv4. + + - ``sha {MAC-48}``: sender hardware address. + - ``spa {ipv4 address}``: sender IPv4 address. + - ``tha {MAC-48}``: target hardware address. + - ``tpa {ipv4 address}``: target IPv4 address. + +- ``ipv6_ext``: match presence of any IPv6 extension header. + + - ``next_hdr {unsigned}``: next header. + +- ``icmp6``: match any ICMPv6 header. + + - ``type {unsigned}``: ICMPv6 type. + - ``code {unsigned}``: ICMPv6 code. + +- ``icmp6_nd_ns``: match ICMPv6 neighbor discovery solicitation. + + - ``target_addr {ipv6 address}``: target address. + +- ``icmp6_nd_na``: match ICMPv6 neighbor discovery advertisement. + + - ``target_addr {ipv6 address}``: target address. + +- ``icmp6_nd_opt``: match presence of any ICMPv6 neighbor discovery option. + + - ``type {unsigned}``: ND option type. + +- ``icmp6_nd_opt_sla_eth``: match ICMPv6 neighbor discovery source Ethernet + link-layer address option. + + - ``sla {MAC-48}``: source Ethernet LLA. + +- ``icmp6_nd_opt_sla_eth``: match ICMPv6 neighbor discovery target Ethernet + link-layer address option. + + - ``tla {MAC-48}``: target Ethernet LLA. + Actions list ^^^^^^^^^^^^ @@ -3363,10 +3574,6 @@ actions can sometimes be combined when the end result is unambiguous:: :: - drop / dup index 6 / end # same as above - -:: - queue index 6 / rss queues 6 7 8 / end # queue has no effect :: @@ -3386,6 +3593,10 @@ This section lists supported actions and their attributes, if any. - ``passthru``: let subsequent rule process matched packets. +- ``jump``: redirect traffic to group on device. + + - ``group {unsigned}``: group to redirect to. + - ``mark``: attach 32 bit value to packets. - ``id {unsigned}``: 32 bit value to return with packets. @@ -3400,20 +3611,91 @@ This section lists supported actions and their attributes, if any. - ``count``: enable counters for this rule. -- ``dup``: duplicate packets to a given queue index. +- ``rss``: spread packets among several queues. - - ``index {unsigned}``: queue index to duplicate packets to. + - ``func {hash function}``: RSS hash function to apply, allowed tokens are + the same as `set_hash_global_config`_. -- ``rss``: spread packets among several queues. + - ``level {unsigned}``: encapsulation level for ``types``. + + - ``types [{RSS hash type} [...]] end``: specific RSS hash types, allowed + tokens are the same as `set_hash_input_set`_, except that an empty list + does not disable RSS but instead requests unspecified "best-effort" + settings. + + - ``key {string}``: RSS hash key, overrides ``key_len``. + + - ``key_len {unsigned}``: RSS hash key length in bytes, can be used in + conjunction with ``key`` to pad or truncate it. - ``queues [{unsigned} [...]] end``: queue indices to use. -- ``pf``: redirect packets to physical device function. +- ``pf``: direct traffic to physical function. -- ``vf``: redirect packets to virtual device function. +- ``vf``: direct traffic to a virtual function ID. - ``original {boolean}``: use original VF ID if possible. - - ``id {unsigned}``: VF ID to redirect packets to. + - ``id {unsigned}``: VF ID. + +- ``phy_port``: direct packets to physical port index. + + - ``original {boolean}``: use original port index if possible. + - ``index {unsigned}``: physical port index. + +- ``port_id``: direct matching traffic to a given DPDK port ID. + + - ``original {boolean}``: use original DPDK port ID if possible. + - ``id {unsigned}``: DPDK port ID. + +- ``of_set_mpls_ttl``: OpenFlow's ``OFPAT_SET_MPLS_TTL``. + + - ``mpls_ttl``: MPLS TTL. + +- ``of_dec_mpls_ttl``: OpenFlow's ``OFPAT_DEC_MPLS_TTL``. + +- ``of_set_nw_ttl``: OpenFlow's ``OFPAT_SET_NW_TTL``. + + - ``nw_ttl``: IP TTL. + +- ``of_dec_nw_ttl``: OpenFlow's ``OFPAT_DEC_NW_TTL``. + +- ``of_copy_ttl_out``: OpenFlow's ``OFPAT_COPY_TTL_OUT``. + +- ``of_copy_ttl_in``: OpenFlow's ``OFPAT_COPY_TTL_IN``. + +- ``of_pop_vlan``: OpenFlow's ``OFPAT_POP_VLAN``. + +- ``of_push_vlan``: OpenFlow's ``OFPAT_PUSH_VLAN``. + + - ``ethertype``: Ethertype. + +- ``of_set_vlan_vid``: OpenFlow's ``OFPAT_SET_VLAN_VID``. + + - ``vlan_vid``: VLAN id. + +- ``of_set_vlan_pcp``: OpenFlow's ``OFPAT_SET_VLAN_PCP``. + + - ``vlan_pcp``: VLAN priority. + +- ``of_pop_mpls``: OpenFlow's ``OFPAT_POP_MPLS``. + + - ``ethertype``: Ethertype. + +- ``of_push_mpls``: OpenFlow's ``OFPAT_PUSH_MPLS``. + + - ``ethertype``: Ethertype. + +- ``vxlan_encap``: Performs a VXLAN encapsulation, outer layer configuration + is done through `Config VXLAN Encap outer layers`_. + +- ``vxlan_decap``: Performs a decapsulation action by stripping all headers of + the VXLAN tunnel network overlay from the matched flow. + +- ``nvgre_encap``: Performs a NVGRE encapsulation, outer layer configuration + is done through `Config NVGRE Encap outer layers`_. + +- ``nvgre_decap``: Performs a decapsulation action by stripping all headers of + the NVGRE tunnel network overlay from the matched flow. Destroying flow rules ~~~~~~~~~~~~~~~~~~~~~ @@ -3679,3 +3961,122 @@ Validate and create a QinQ rule on port 0 to steer traffic to a queue on the hos ID Group Prio Attr Rule 0 0 0 i- ETH VLAN VLAN=>VF QUEUE 1 0 0 i- ETH VLAN VLAN=>PF QUEUE + +Sample VXLAN encapsulation rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +VXLAN encapsulation outer layer has default value pre-configured in testpmd +source code, those can be changed by using the following commands + +IPv4 VXLAN outer header:: + + testpmd> set vxlan ip-version ipv4 vni 4 udp-src 4 udp-dst 4 ip-src 127.0.0.1 + ip-dst 128.0.0.1 eth-src 11:11:11:11:11:11 eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions vxlan_encap / + queue index 0 / end + + testpmd> set vxlan-with-vlan ip-version ipv4 vni 4 udp-src 4 udp-dst 4 ip-src + 127.0.0.1 ip-dst 128.0.0.1 vlan-tci 34 eth-src 11:11:11:11:11:11 + eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions vxlan_encap / + queue index 0 / end + +IPv6 VXLAN outer header:: + + testpmd> set vxlan ip-version ipv6 vni 4 udp-src 4 udp-dst 4 ip-src ::1 + ip-dst ::2222 eth-src 11:11:11:11:11:11 eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions vxlan_encap / + queue index 0 / end + + testpmd> set vxlan-with-vlan ip-version ipv6 vni 4 udp-src 4 udp-dst 4 + ip-src ::1 ip-dst ::2222 vlan-tci 34 eth-src 11:11:11:11:11:11 + eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions vxlan_encap / + queue index 0 / end + +Sample NVGRE encapsulation rule +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +NVGRE encapsulation outer layer has default value pre-configured in testpmd +source code, those can be changed by using the following commands + +IPv4 NVGRE outer header:: + + testpmd> set nvgre ip-version ipv4 tni 4 ip-src 127.0.0.1 ip-dst 128.0.0.1 + eth-src 11:11:11:11:11:11 eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions nvgre_encap / + queue index 0 / end + + testpmd> set nvgre-with-vlan ip-version ipv4 tni 4 ip-src 127.0.0.1 + ip-dst 128.0.0.1 vlan-tci 34 eth-src 11:11:11:11:11:11 + eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions nvgre_encap / + queue index 0 / end + +IPv6 NVGRE outer header:: + + testpmd> set nvgre ip-version ipv6 tni 4 ip-src ::1 ip-dst ::2222 + eth-src 11:11:11:11:11:11 eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions nvgre_encap / + queue index 0 / end + + testpmd> set nvgre-with-vlan ip-version ipv6 tni 4 ip-src ::1 ip-dst ::2222 + vlan-tci 34 eth-src 11:11:11:11:11:11 eth-dst 22:22:22:22:22:22 + testpmd> flow create 0 ingress pattern end actions nvgre_encap / + queue index 0 / end + +BPF Functions +-------------- + +The following sections show functions to load/unload eBPF based filters. + +bpf-load +~~~~~~~~ + +Load an eBPF program as a callback for partciular RX/TX queue:: + + testpmd> bpf-load rx|tx (portid) (queueid) (load-flags) (bpf-prog-filename) + +The available load-flags are: + +* ``J``: use JIT generated native code, otherwise BPF interpreter will be used. + +* ``M``: assume input parameter is a pointer to rte_mbuf, otherwise assume it is a pointer to first segment's data. + +* ``-``: none. + +.. note:: + + You'll need clang v3.7 or above to build bpf program you'd like to load + +For example: + +.. code-block:: console + + cd test/bpf + clang -O2 -target bpf -c t1.c + +Then to load (and JIT compile) t1.o at RX queue 0, port 1:: + +.. code-block:: console + + testpmd> bpf-load rx 1 0 J ./dpdk.org/test/bpf/t1.o + +To load (not JITed) t1.o at TX queue 0, port 0:: + +.. code-block:: console + + testpmd> bpf-load tx 0 0 - ./dpdk.org/test/bpf/t1.o + +bpf-unload +~~~~~~~~~~ + +Unload previously loaded eBPF program for partciular RX/TX queue:: + + testpmd> bpf-unload rx|tx (portid) (queueid) + +For example to unload BPF filter from TX queue 0, port 0: + +.. code-block:: console + + testpmd> bpf-load tx 0 0 - ./dpdk.org/test/bpf/t1.o diff --git a/doc/guides/tools/cryptoperf.rst b/doc/guides/tools/cryptoperf.rst index 3c0e7d94..c366af4e 100644 --- a/doc/guides/tools/cryptoperf.rst +++ b/doc/guides/tools/cryptoperf.rst @@ -181,7 +181,7 @@ The following are the appication command-line options: crypto_dpaa2_sec crypto_armv8 crypto_scheduler - crypto_mrvl + crypto_mvsam * ``--optype <name>`` diff --git a/doc/guides/tools/testbbdev.rst b/doc/guides/tools/testbbdev.rst index 5c7112de..234a64f5 100644 --- a/doc/guides/tools/testbbdev.rst +++ b/doc/guides/tools/testbbdev.rst @@ -21,7 +21,7 @@ The bbdevice drivers PMD which should be tested can be enabled by setting ``CONFIG_RTE_LIBRTE_PMD_<name>=y`` -Setting example for (*turbo_sw*) PMD +Setting example for (*baseband_turbo_sw*) PMD ``CONFIG_RTE_LIBRTE_PMD_BBDEV_TURBO_SW=y`` @@ -70,30 +70,6 @@ The following are the command-line options: ``-c TEST_CASE [TEST_CASE ...], --test_cases TEST_CASE [TEST_CASE ...]`` Defines test cases to run. If not specified all available tests are run. - The following tests can be run: - - * unittest - Small unit tests witch check basic functionality of bbdev library. - * latency - Test calculates three latency metrics: - - * offload_latency_tc - measures the cost of offloading enqueue and dequeue operations. - * offload_latency_empty_q_tc - measures the cost of offloading a dequeue operation from an empty queue. - checks how long last dequeueing if there is no operations to dequeue - * operation_latency_tc - measures the time difference from the first attempt to enqueue till the - first successful dequeue. - * validation - Test do enqueue on given vector and compare output after dequeueing. - * throughput - Test measures the achieved throughput on the available lcores. - Results are printed in million operations per second and million bits per second. - * interrupt - The same test as 'throughput' but uses interrupts instead of PMD to perform - the dequeue. - **Example usage:** ``./test-bbdev.py -c validation`` @@ -105,18 +81,18 @@ The following are the command-line options: ``-v TEST_VECTOR [TEST_VECTOR ...], --test_vector TEST_VECTOR [TEST_VECTOR ...]`` Specifies paths to the test vector files. If not specified path is set based on *$RTE_SDK* environment variable concatenated with - "*/app/test-bbdev/test_vectors/bbdev_vector_null.data*" and indicates default + "*/app/test-bbdev/test_vectors/bbdev_null.data*" and indicates default data file. **Example usage:** - ``./test-bbdev.py -v app/test-bbdev/test_vectors/bbdev_vector_td_test1.data`` - Fills vector based on bbdev_vector_td_test1.data file and runs all tests + ``./test-bbdev.py -v app/test-bbdev/test_vectors/turbo_dec_test1.data`` + Fills vector based on turbo_dec_test1.data file and runs all tests - ``./test-bbdev.py -v bbdev_vector_td_test1.data bbdev_vector_te_test2.data`` + ``./test-bbdev.py -v turbo_dec_test1.data turbo_enc_test2.data`` The bbdev test app is executed twice. First time vector is filled based on - *bbdev_vector_td_test1.data* file and second time based on - *bbdev_vector_te_test2.data* file. For both executions all tests are run. + *turbo_dec_test1.data* file and second time based on + *turb_enc_test2.data* file. For both executions all tests are run. ``-n NUM_OPS, --num_ops NUM_OPS`` Specifies number of operations to process on device. If not specified num_ops @@ -130,8 +106,54 @@ The following are the command-line options: Specifies operations enqueue/dequeue burst size. If not specified burst_size is set to 32. Maximum is 512. - -Parameter globbing +Test Cases +~~~~~~~~~~ + +There are 6 main test cases that can be executed using testbbdev tool: + +* Sanity checks [-c unittest] + - Performs sanity checks on BBDEV interface, validating basic functionality + +* Validation tests [-c validation] + - Performs full operation of enqueue and dequeue + - Compares the dequeued data buffer with a expected values in the test + vector (TV) being used + - Fails if any dequeued value does not match the data in the TV + +* Offload Cost measurement [-c offload] + - Measures the CPU cycles consumed from the receipt of a user enqueue + until it is put on the device queue + - The test measures 4 metrics + (a) *SW Enq Offload Cost*: Software only enqueue offload cost, the cycle + counts and time (us) from the point the enqueue API is called until + the point the operation is put on the accelerator queue. + (b) *Acc Enq Offload Cost*: The cycle count and time (us) from the + point the operation is put on the accelerator queue until the return + from enqueue. + (c) *SW Deq Offload Cost*: Software dequeue cost, the cycle counts and + time (us) consumed to dequeue one operation. + (d) *Empty Queue Enq Offload Cost*: The cycle count and time (us) + consumed to dequeue from an empty queue. + +* Latency measurement [-c latency] + - Measures the time consumed from the first enqueue until the first + appearance of a dequeued result + - This measurment represents the full latency of a bbdev operation + (encode or decode) to execute + +* Poll-mode Throughput measurement [-c throughput] + - Performs full operation of enqueue and dequeue + - Executes in poll mode + - Measures the achieved throughput on a subset or all available CPU cores + - Dequeued data is not validated against expected values stored in TV + - Results are printed in million operations per second and million bits + per second + +* Interrupt-mode Throughput [-c interrupt] + - Similar to Throughput test case, but using interrupts. No polling. + + +Parameter Globbing ~~~~~~~~~~~~~~~~~~ Thanks to the globbing functionality in python test-bbdev.py script allows to @@ -141,26 +163,63 @@ run tests with different set of vector files without giving all of them explicit .. code-block:: console - ./test-bbdev.py -v app/test-bbdev/test_vectors/bbdev_vector_*.data + ./test-bbdev.py -v app/test-bbdev/test_vectors/turbo_<enc/dec>_c<c>_k<k>_r<r>_e<e>_<extra-info>.data It runs all tests with following vectors: -- ``bbdev_vector_null.data`` +- ``bbdev_null.data`` + +- ``turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data`` + +- ``turbo_enc_c1_k40_r0_e1196_rm.data`` + +- ``turbo_enc_c2_k5952_r0_e17868_crc24b.data`` + +- ``turbo_dec_c1_k40_r0_e17280_sbd_negllr.data`` + +- ``turbo_dec_c1_k6144_r0_e34560_sbd_posllr.data`` + +- ``turbo_enc_c1_k40_r0_e272_rm.data`` + +- ``turbo_enc_c3_k4800_r2_e14412_crc24b.data`` + +- ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` + +- ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data`` + +- ``turbo_enc_c1_k6144_r0_e120_rm_rvidx.data`` + +- ``turbo_enc_c4_k4800_r2_e14412_crc24b.data`` + +- ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data`` + +- ``turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data`` + +- ``turbo_enc_c1_k6144_r0_e18444.data`` + +- ``turbo_dec_c1_k6144_r0_e34560_negllr.data`` + +- ``turbo_enc_c1_k40_r0_e1190_rm.data`` + +- ``turbo_enc_c1_k6144_r0_e18448_crc24a.data`` -- ``bbdev_vector_td_default.data`` +- ``turbo_dec_c1_k6144_r0_e34560_posllr.data`` -- ``bbdev_vector_te_default.data`` +- ``turbo_enc_c1_k40_r0_e1194_rm.data`` +- ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` .. code-block:: console - ./test-bbdev.py -v app/test-bbdev/test_vectors/bbdev_vector_t?_default.data + ./test-bbdev.py -v app/test-bbdev/turbo_*_default.data -It runs all tests with "default" vectors: +It runs all tests with "default" vectors. -- ``bbdev_vector_te_default.data`` +* ``turbo_dec_default.data`` is a soft link to + ``turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data`` -- ``bbdev_vector_td_default.data`` +* ``turbo_enc_default.data`` is a soft link to + ``turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data`` Running Tests @@ -174,9 +233,27 @@ x86_64-native-linuxapp-icc target: |-- app |-- test-bbdev |-- test_vectors - |-- bbdev_vector_null.data - |-- bbdev_vector_td_default.data - |-- bbdev_vector_te_default.data + |-- bbdev_null.data + |-- turbo_dec_c1_k6144_r0_e34560_sbd_negllr.data + |-- turbo_enc_c1_k40_r0_e1196_rm.data + |-- turbo_enc_c2_k5952_r0_e17868_crc24b.data + |-- turbo_dec_c1_k40_r0_e17280_sbd_negllr.data + |-- turbo_dec_c1_k6144_r0_e34560_sbd_posllr.data + |-- turbo_enc_c1_k40_r0_e272_rm.data + |-- turbo_enc_c3_k4800_r2_e14412_crc24b.data + |-- turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_high_snr.data + |-- turbo_dec_c2_k3136_r0_e4920_sbd_negllr_crc24b.data + |-- turbo_enc_c1_k6144_r0_e120_rm_rvidx.data + |-- turbo_enc_c4_k4800_r2_e14412_crc24b.data + |-- turbo_dec_c1_k6144_r0_e10376_crc24b_sbd_negllr_low_snr.data + |-- turbo_dec_c2_k3136_r0_e4920_sbd_negllr.data + |-- turbo_enc_c1_k6144_r0_e18444.data + |-- turbo_dec_c1_k6144_r0_e34560_negllr.data + |-- turbo_enc_c1_k40_r0_e1190_rm.data + |-- turbo_enc_c1_k6144_r0_e18448_crc24a.data + |-- turbo_dec_c1_k6144_r0_e34560_posllr.data + |-- turbo_enc_c1_k40_r0_e1194_rm.data + |-- turbo_enc_c1_k6144_r0_e32256_crc24b_rm.data |-- x86_64-native-linuxapp-icc |-- app @@ -188,43 +265,43 @@ All bbdev devices .. code-block:: console ./test-bbdev.py -p ../../x86_64-native-linuxapp-icc/app/testbbdev - -v ./test_vectors/bbdev_vector_td_default.data + -v turbo_dec_default.data It runs all available tests using the test vector filled based on -*bbdev_vector_td_default.data* file. +*turbo_dec_default.data* file. By default number of operations to process on device is set to 32, timeout is set to 300s and operations enqueue/dequeue burst size is set to 32. -Moreover a bbdev (*bbdev_null*) device will be created. +Moreover a bbdev (*baseband_null*) device will be created. -bbdev turbo_sw device -~~~~~~~~~~~~~~~~~~~~~ +baseband turbo_sw device +~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: console ./test-bbdev.py -p ../../x86_64-native-linuxapp-icc/app/testbbdev - -e="--vdev=turbo_sw" -t 120 -c validation - -v ./test_vectors/bbdev_vector_t?_default.data -n 64 -b 8 32 + -e="--vdev=baseband_turbo_sw" -t 120 -c validation + -v ./test_vectors/turbo_* -n 64 -b 8 32 It runs **validation** test for each vector file that matches the given pattern. Number of operations to process on device is set to 64 and operations timeout is set to 120s and enqueue/dequeue burst size is set to 8 and to 32. -Moreover a bbdev (*turbo_sw*) device will be created. +Moreover a bbdev (*baseband_turbo_sw*) device will be created. bbdev null device ~~~~~~~~~~~~~~~~~ -Executing bbdev null device with *bbdev_vector_null.data* helps in measuring the +Executing bbdev null device with *bbdev_null.data* helps in measuring the overhead introduced by the bbdev framework. .. code-block:: console - ./test-bbdev.py -e="--vdev=bbdev_null0" - -v ./test_vectors/bbdev_vector_null.data + ./test-bbdev.py -e="--vdev=baseband_null0" + -v ./test_vectors/bbdev_null.data **Note:** -bbdev_null device does not have to be defined explicitly as it is created by default. +baseband_null device does not have to be defined explicitly as it is created by default. diff --git a/doc/guides/tools/testeventdev.rst b/doc/guides/tools/testeventdev.rst index 77480ffe..46effd87 100644 --- a/doc/guides/tools/testeventdev.rst +++ b/doc/guides/tools/testeventdev.rst @@ -123,6 +123,36 @@ The following are the application command-line options: Use ethernet device as producer. +* ``--prod_type_timerdev`` + + Use event timer adapter as producer. + + * ``--prod_type_timerdev_burst`` + + Use burst mode event timer adapter as producer. + + * ``--timer_tick_nsec`` + + Used to dictate number of nano seconds between bucket traversal of the + event timer adapter. Refer `rte_event_timer_adapter_conf`. + + * ``--max_tmo_nsec`` + + Used to configure event timer adapter max arm timeout in nano seconds. + + * ``--expiry_nsec`` + + Dictate the number of nano seconds after which the event timer expires. + + * ``--nb_timers`` + + Number of event timers each producer core will generate. + + * ``--nb_timer_adptrs`` + + Number of event timer adapters to be used. Each adapter is used in + round robin manner by the producer cores. + Eventdev Tests -------------- @@ -347,6 +377,13 @@ Supported application command line options are following:: --fwd_latency --queue_priority --prod_type_ethdev + --prod_type_timerdev_burst + --prod_type_timerdev + --timer_tick_nsec + --max_tmo_nsec + --expiry_nsec + --nb_timers + --nb_timer_adptrs Example ^^^^^^^ @@ -365,6 +402,14 @@ Example command to run perf queue test with ethernet ports: sudo build/app/dpdk-test-eventdev --vdev=event_sw0 -- \ --test=perf_queue --plcores=2 --wlcore=3 --stlist=p --prod_type_ethdev +Example command to run perf queue test with event timer adapter: + +.. code-block:: console + + sudo build/app/dpdk-test-eventdev --vdev="event_octeontx" -- \ + --wlcores 4 --plcores 12 --test perf_queue --stlist=a \ + --prod_type_timerdev --fwd_latency + PERF_ATQ Test ~~~~~~~~~~~~~~~ @@ -431,6 +476,13 @@ Supported application command line options are following:: --worker_deq_depth --fwd_latency --prod_type_ethdev + --prod_type_timerdev_burst + --prod_type_timerdev + --timer_tick_nsec + --max_tmo_nsec + --expiry_nsec + --nb_timers + --nb_timer_adptrs Example ^^^^^^^ @@ -442,6 +494,14 @@ Example command to run perf ``all types queue`` test: sudo build/app/dpdk-test-eventdev --vdev=event_octeontx -- \ --test=perf_atq --plcores=2 --wlcore=3 --stlist=p --nb_pkts=0 +Example command to run perf ``all types queue`` test with event timer adapter: + +.. code-block:: console + + sudo build/app/dpdk-test-eventdev --vdev="event_octeontx" -- \ + --wlcores 4 --plcores 12 --test perf_atq --verbose 20 \ + --stlist=a --prod_type_timerdev --fwd_latency + PIPELINE_QUEUE Test ~~~~~~~~~~~~~~~~~~~ |
