aboutsummaryrefslogtreecommitdiffstats
path: root/src/plugins/dpdk/cryptodev
AgeCommit message (Collapse)AuthorFilesLines
2023-09-28dpdk-cryptodev: improve dequeue behavior, fix cache stats loggingPiotr Bronowski4-76/+88
This patch provides minor improvements to the logic governing dequeuing from the ring. Previously whenever a frame was dequeued we've been trying to dequeue from the ring another one till inflight == 0. Now threshold is set for 8 frames pending in the cache to be consumed by the vnet. This threshold has been chosen based on cache ring stats observation in the system under load. Some unnecessary logic for setting deq_tail has been removed. Also logging has been corrected, and cache ring logic simplied. Type: improvement Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I19f3daf5913006e9cb23e142a163f596e85f5bda
2023-09-07dpdk-cryptodev: fix cache ring stats cli commandPiotr Bronowski1-14/+12
The logic for calcuating processed elements in the cache ring was broken. In case tail and deq_tail equals and frame element pointed by the tile is not NULL it means there is exactly one processed element in the ring. Type: fix Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I69c978334fc952049393214ccc9cc5245351f7f7
2023-08-22dpdk-cryptodev: fix coverity issuesPiotr Bronowski2-5/+8
This patch addresses coverity issues CID 322716 and CID 322717. Type: fix Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I59d6f40c1af8e829d8cb3c042a52e144aeaf1e6b
2023-08-18dpdk-cryptodev: improve cryptodev cache ring implementationPiotr Bronowski4-239/+353
Sw ring is renamed to the cache ring. This name better reflects the puropse of this ring. We've introduced push/pop functions, as well as other utility functions which remove code repetition. Error handlig is improved: previously in case of an error all frame elements were marked as bad, now only these for which errors occured have the error status set. Unnecessary stats counters have been removed. Type: improvement Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I2fd42a529ac84ce5ad260611d6b35a861d441c79
2023-07-06dpdk-cryptodev: introduce sw_ring to the crypto op data pathPiotr Bronowski2-124/+235
This patch introduces sw_ring to the crypto op data path implementation, so that raw data path and crypto op data path use same mechanism of processing async frames. Crypto op ring has been removed from the implementation. Type: improvement Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: Id823f80a88cfa0ff40252616a36de8bb044c7f45
2023-06-29dpdk-cryptodev: sync mbuf length with corresponding vlib bufferPiotr Bronowski1-12/+21
When vlib buffer is processed on vnet side its length is corrected by cipher padding and icv_sz. These changes need to be reflected in the mbuf internals. Type: fix Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I0aa03f67f556dfc8f9a577ca1967210527221e02
2023-06-19dpdk-cryptodev: enq/deq scheme reworkPiotr Bronowski4-223/+210
This rework tries to address issues found on SPR QAT, for traffic reaching max possible throughoutput for single QAT PF packet drops were observed. Fix changes enq/deq scheme by utilizing software ring in enq call from VNET but enq and deq to QAT happens only in deq callback function what should enable better utlization of hardware resources. Type: improvement Signed-off-by: Dastin Wilski <dastin.wilski@gmail.com> Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I2e8c473d20a269fd5e93f0c8d1f8c8aa193712bd
2023-06-13dpdk-cryptodev: introduce sw_ringPiotr Bronowski2-0/+66
This patch introduces sw_ring. This ring is used in next set of patchas and plays role of a buffer for QAT, allowing collecting frame elements in case QAT queue is fully utilized, and assembling frame from QAT dequeued elements. Type: improvement Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Signed-off-by: Dastin Wilski <dastin.wilski@gmail.com> Change-Id: I20718e200986ab4dba5cbc31c05a904072a6981a
2023-06-01crypto: make crypto-dispatch node working in adaptive modeXiaoming Jiang1-2/+1
This patch can make crypto dispatch node adaptively switching between pooling and interrupt mode, and improve vpp overall performance. Type: improvement Signed-off-by: Xiaoming Jiang <jiangxiaoming@outlook.com> Change-Id: I845ed1d29ba9f3c507ea95a337f6dca7f8d6e24e
2023-04-25dpdk: code preparation for bumping to DPDK 22.11Xinyao Cai4-17/+106
This patch prepares code for bumping DPDK version to 22.11, but the DPDK version of this patch keeps at 22.07 for compatibility. the "no-dsa" parameter in DPDK configuration is removed, the "blacklist" parameter can be used to block the related DSA devices. Type: feature Signed-off-by: Xinyao Cai <xinyao.cai@intel.com> Change-Id: I08787c6584bba66383fc0a784963f33171196910
2023-03-29dpdk-cryptodev: fix name formatting of session poolsGabriel Oginski1-2/+2
Originally the name for each session pool is incorrectly prepared. It doesn't have right length. It is not null terminated. The fix corrects the name formatting for each session pool. Type: fix Signed-off-by: Gabriel Oginski <gabrielx.oginski@intel.com> Change-Id: I67da3d64702ccb27a5907825528f8c95d91040bb
2023-03-24dpdk-cryptodev: fix formatting name of poolsGabriel Oginski1-2/+2
Originally the name for each session pool can be incorrect prepared. The fix changes formatting for name for each session pool. Type: fix Signed-off-by: Gabriel Oginski <gabrielx.oginski@intel.com> Change-Id: I42e0752f9f46c5a42524ec7b863a7c9dd3c23110
2022-09-07dpdk-cryptodev: reduce request to enable asyncGabriel Oginski1-1/+0
Originally initialization cryptodev device(s) calls double request to enabled async mode and increased ref count twice for async mode. Due to this cannot be change any assigned async handlers to other async crypto engine. The fixes reduce double request to enable async mode in initialization cryptodev device(s) and VPP can be change assigned async handlers to other crypto engine after disabled all async feature, for example: ipsec, wireguard. Type: fix Signed-off-by: Gabriel Oginski <gabrielx.oginski@intel.com> Change-Id: If22e682c3c10de781d05c2e09b5420f75be151c3
2022-01-26dpdk: not having cryptodev resources should not produce warningsDamjan Marion1-4/+1
Type: fix Change-Id: Ifb2e4d93dcf8648b1bd66f4c0ee937295683bd87 Signed-off-by: Damjan Marion <damarion@cisco.com>
2022-01-24dpdk-cryptodev: add support chacha20-poly1305Gabriel Oginski4-6/+44
Originally cryptodev doesn't support chacha20-poly1305 with aad length 0. This patch add support in cryptodev for chacha20-poly1305 with aad length 0. This length is using in Wireguard. Type: improvement Signed-off-by: Gabriel Oginski <gabrielx.oginski@intel.com> Change-Id: I0608920bb557d7d071e7f9f37c80cf50bad81dcc
2022-01-12dpdk: bump to DPDK v21.11Damjan Marion3-7/+7
Type: feature This patch bumps dpdk version from 21.08 to 21.11 Change-Id: Id37fdba75f1ea4f4eac3c92226f3b1c539e1daca Signed-off-by: Dastin Wilski <dastin.wilski@gmail.com> Signed-off-by: Damjan Marion <damarion@cisco.com>
2022-01-12dpdk-cryptodev: run dequeue again in case of interrupt mode being usedDastin Wilski1-0/+6
Type: fix Change-Id: I7aa172e58c970c4971db6ef2ff5b199b7f3c0b99 Signed-off-by: Dastin Wilski <dastin.wilski@gmail.com>
2022-01-05crypto: encrypt/decrypt queues sw_schedulerJakub Wysocki2-20/+29
Type: improvement Previously multiple sw crypto scheduler queues per core design caused unaverage frame processing rate for each async op ID – the lower the op ID is the highly likely they are processed first. For example, when a RX core is feeding both encryption and decryption jobs of the same crypto algorithm to the queues at a high rate, in the mean time the crypto cores have no enough cycles to process all: the jobs in the decryption queue are less likely being processed, causing packet drop. To improve the situation this patch makes every core only owning a two queues, one for encrypt operations and one for decrypt. The queue is changed either after checking each core or after founding a frame to process. All crypto jobs with different algorithm are pushed to thoses queues and are treated evenly. In addition, the crypto async infra now uses unified dequeue handler, one per engine. Only the active engine will be registered its dequeue handler in crypto main. Signed-off-by: DariuszX Kazimierski <dariuszx.kazimierski@intel.com> Signed-off-by: PiotrX Kleski <piotrx.kleski@intel.com> Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Signed-off-by: Jakub Wysocki <jakubx.wysocki@intel.com> Change-Id: I517ee8e31633980de5e0dd4b05e1d5db5dea760e
2021-11-30dpdk: cryptodev AES-CTR+SHA1 support addedPiotrX Kleski1-1/+4
Type: improvement This patch adds AES-CTR-128/192/256 + SHA1 linked algo support to dpdk cryptodev. Signed-off-by: PiotrX Kleski <piotrx.kleski@intel.com> Change-Id: Idc162b29f4075ef8be9577abd3daf6de05f84faa
2021-11-30dpdk-cryptodev: scalable session countDastin Wilski2-96/+143
Originally cryptodev allocates mempools for seesion and session private data during its initialization. Moreover the size of these mempools are fixed resulting in limited session count (up to value specified in CRYPTODEV_NB_SESSION macro). This patch allows for session count to scale up by allocating new mempools as they are needed during session creation. Type: improvement Signed-off-by: Dastin Wilski <dastin.wilski@gmail.com> Change-Id: I6ae240b474d3089d3ff50ca5bc7ff48f149983db
2021-09-13dpdk: fix prefetch assert on ArmTianyu Li1-4/+4
CLIB_PREFETCH (cop[1], CLIB_CACHE_LINE_BYTES * 3, STORE); Note on 64 bytes cache line size arm machines, CLIB_CACHE_LINE_BYTES 128 CLIB_CACHE_PREFETCH_BYTES 6 above CLIB_PREFETCH () macro will be expand to ASSERT ((size) <= 4 * CLIB_CACHE_PREFETCH_BYTES); it will hit assert due to size (i.e. 3 * 128) > 4 * 64 Solution: Change to CLIB_PREFETCH (cop[1], sizeof(*cop[1]), STORE); Type: fix Signed-off-by: Tianyu Li <tianyu.li@arm.com> Reviewed-by: Lijian Zhang <lijian.zhang@arm.com> Change-Id: Id0981fd5bd2b25ff71db4197b25578d0b7a9803e
2021-07-15misc: replace CLIB_PREFETCH with clib_prefetch_{load,store}Damjan Marion2-9/+9
Type: refactor Change-Id: Id10cbf52e8f2dd809080a228d8fa282308be84ac Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-06-04dpdk: silence coverity warning on use of uninitialized valuepibr1-0/+1
Type: fix Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I76923ad5035498aae821db4fd42a127617476fbb
2021-05-27dpdk: bump to 21.05Fan Zhang1-0/+13
Type: feature This patch bumps DPDK version to 21.05 and updated VPP to accomodate the changes in DPDK latest version. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: If217441f70c9ab531196dca7ec7a486ec9931cff
2021-05-26dpdk: fix missing symbolFan Zhang1-2/+1
Type: fix This patch fixes the missing symbol of dpdk_plugin.so when creating symmetric key. The solution is to add dependency of libssl to dpdk cryptodev and disable cryptodev engine when libssl is not presented. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: I30aa6e3e3af1faefa82883bad613e1d82235a2ec
2021-05-07dpdk: fix cryptodev session handlerFan Zhang1-0/+4
Type: fix This patch fixes the possible segmentation fault in DPDK cryptodev when deleting session. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: Ie81de09f7250eb5c1ed6ee471363137d8372fe27
2021-04-23dpdk: support md5 async handler in dpdk cryptodevAlexander Chernavin1-0/+3
With this change, enable MD5 with AES-CBC support in dpdk cryptodev. Type: improvement Signed-off-by: Alexander Chernavin <achernavin@netgate.com> Change-Id: Ic587aaa1fa0dc102e36eb34f329ef21a16156f26
2021-04-09dpdk: selection of cryptodev engine data-pathFan Zhang5-2727/+2074
Type: improvement This patch combined cryptodev op and cryptodev raw API data paths into one and makes the engine run-timely select which data path is used: if all cryptodev devices support RTE_CRYPTODEV_FF_SYM_RAW_DP feature flag, the raw data path API is used, otherwise the traditional data path is used. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: Ibbd7c4405288bd9a48a34878954fd3040df7b4ad
2021-03-31dpdk: fix packet offset for GCM crypto opsRobert Shearman1-0/+1
The crypto op data offset passed into DPDK is relative to the mbuf buffer address plus the mbuf data offset, therefore the mbuf data offset needs to be set rather than left at whatever previous value it was at, which is likely to be incorrect and result in the wrong portion of the packet being encrypted/decrypted for GCM. The fe->crypto_start_offset field is relative to the start of the vlib buffer (as opposed to the current data pointer), so set the mbuf data_off field to VLIB_BUFFER_PRE_DATA_SIZE when performing a GCM crypto op enqueue to match the crypto_start_offset semantics. This then matches the behaviour in the non-GCM case. Type: fix Change-Id: I0ac2a44139387158765a3e04cfcaa5ee6f11d395 Signed-off-by: Robert Shearman <robertshearman@gmail.com>
2021-03-26vlib: introduce vlib_get_main_by_index(), vlib_get_n_threads()Damjan Marion1-1/+1
Type: improvement Change-Id: If3da7d4338470912f37ff1794620418d928fb77f Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-03-17dpdk: cryptodev session managementpibr1-108/+377
Ensure that dpdk_cryptodev engine async handlers are registered for the set of algorithms supported by all cryptodevices in the system. Type: improvement Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Change-Id: I8293d87c6f07e935a1a0637704f24918cd3e132a
2021-03-04dpdk: deprecate ipsec backendFan Zhang2-467/+578
Type: refactor DPDK crypto devices are now accessible via the async infra, so there is no need for the DPDK ipsec plugin. In addition this patch fixes the problem that cryptodev backend not working when master core and worker cores lies in different numa nodes. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: Ie8516bea706248c7bc25abac53a9c656bb8247d9
2021-02-19dpdk: fix cryptodev offset updateFan Zhang1-11/+8
Type: fix This patch fixes the missed crypto and integ offset update for every packet. Previously the offset is updated only when the key is changed. This is ok for encryption but not always true for decryption. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: Iccd0011f4ae488746ce487a14b94ddd24fb0c07c
2020-12-18dpdk: rebase cryptodev engine for dpdk 20.11Fan Zhang1-120/+126
Type: feature This patch rebase cryptodev engine for the new cryptodev raw APIs introduced in DPDK 20.11. Signed-off-by: Piotr Bronowski <PiotrX.Bronowski@intel.com> Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: I4da335379c5dfeb358017092086d305a01b652dc
2020-12-04crypto: fixed async frame enqueue race conditionPiotrX Kleski1-1/+0
Type: fix To avoid race condition happening in async crypto engines, async frame state and thread index set should happen before enqueue. In addition as the enqueue handler already returns the enqueue status, when an enqueue is failed, the async crypto engine shall not worry about setting the async frame state but let the submit_open_frame function to do just that. Signed-off-by: PiotrX Kleski <piotrx.kleski@intel.com> Reviewed-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: Ic1b0c94478b3cfd5fab98657218bbd70c46a220a
2020-10-21vlib: print logs to stderr if interactive or nosyslog setDamjan Marion2-8/+2
If VPP is started in interactive mode, instead of sending logs to syslog server we print them directly to stderr. Output is colorized, but that can be turned off with unix { nocolor } Type: improvement Change-Id: I9a0f0803e4cba2849a6efa0b6a86b9614ed33ced Signed-off-by: Damjan Marion <damarion@cisco.com>
2020-09-29dpdk: fix compileFan Zhang1-49/+13
Type: fix Since DPDK is now compiled by meson but some compiles in VPP is missing. This patch fixes that. - Fixes QAT PMD not compiled. QAT meson compile, even for sym crypto PMD, is happened in drive/compress/qat. Originally all PMDs in compressdev is disabled by default. This patch fixes that. - Fixes DPDK plugin version detection. DPDK meson build generates rte_build_config.h, which containing all version information in build-dpdk instead of rte_config.h in make. This patch uses the file to detect version data. - Removed SW crypto PMD auto-creation in cryptodev engine. In case the AESNI-MB PMD required shared library is missing. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: I7cd91abb5de303ff5e4c55cd05e011b57f883524
2020-09-09crypto: change cryptodev with new cryptodev APIFan Zhang1-0/+1536
Type: feature This patch updateds cryptodev engine uses new DPDK Cryptodev API planned to be upstreamed in DPDK 20.11. Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Signed-off-by: Piotr Bronowski <piotrX.bronowski@intel.com> Change-Id: I8dd1a8ac643f1e952deb787e466b76ea7aa5f420
2020-09-03crypto: SW scheduler async crypto enginePiotrX Kleski1-2/+4
Type: feature This patch adds new sw_scheduler async crypto engine. The engine transforms async frames info sync crypto ops and delegates them to active sync engines. With the patch it is possible to increase the single worker crypto throughput by offloading the crypto workload to multiple workers. By default all workers in the system will attend the crypto workload processing. However a worker's available cycles are limited. To avail more cycles to one worker to process other workload (e.g. the worker core that handles the RX/TX and IPSec stack processing), a useful cli command is added to remove itself (or add it back later) from the heavy crypto workload but only let other workers to process the crypto. The command is: - set sw_scheduler worker <idx> crypto <on|off> It also adds new interrupt mode to async crypto dispatch node. This mode signals the node when new frames are enqueued as opposed to polling mode that continuously calls dispatch node. New cli commands: - set crypto async dispatch [polling|interrupt] - show crypto async status (displays mode and nodes' states) Signed-off-by: PiotrX Kleski <piotrx.kleski@intel.com> Signed-off-by: DariuszX Kazimierski <dariuszx.kazimierski@intel.com> Reviewed-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: I332655f347bb9e3bc9c64166e86e393e911bdb39
2020-07-23dpdk: device_id sorted order for cryptodevVladimir Ratnikov1-0/+16
By default, VPP automatically assignes for each tunnel next available QAT device by order dev_id-que-pair. In most cases we have more than one device and it can greatly increase ipsec perfomance without any actions with configuration from user if we use all the devices first and first que-pairs Type: feature Signed-off-by: Vladimir Ratnikov <vratnikov@netgate.com> Change-Id: Iac9fe74768775459e22f69bb3706b542090a9375
2020-07-16crypto: fix coverity issue for cryptodevFan Zhang1-65/+103
- Fixes coverity issue #210160. - Fixes the possible issue in cryptodev when input node does not update mbuf, such as avf-input. - Fixes GCM ESN packet incorrect tag. - Code clean up to reduce binary size. Type: fix Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Signed-off-by: Dariusz Kazimierski <dariuszx.kazimierski@intel.com> Signed-off-by: Piotr Kleski <piotrx.kleski@intel.com> Change-Id: Ic05ae29855ac1f7a62e4af5831a4ed9faa8f561a
2020-06-15dpdk: cryptodev: fix non-null terminated stringsBenoît Ganne1-4/+4
Type: fix Change-Id: Ib6f423e24f1a8d8439cd7e8893e4605e10984d48 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2020-05-27dpdk: bump to DPDK v20.05Damjan Marion1-0/+7
Type: improvement Change-Id: I5f89fc3d994bd85d2c5138069ea2c58661814228 Signed-off-by: Damjan Marion <damarion@cisco.com>
2020-05-06crypto: fix coverity issuesFan Zhang1-5/+11
Fix coverity issues in crypto framework and cryptodev engine. Type: fix Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Change-Id: Ib261da0163c8182c803600db22c5a6dad5a19999
2020-04-30crypto: introduce async crypto infraFan Zhang2-0/+1402
Type: feature Signed-off-by: Damjan Marion <damarion@cisco.com> Signed-off-by: Filip Tehlar <ftehlar@cisco.com> Signed-off-by: Fan Zhang <roy.fan.zhang@intel.com> Signed-off-by: Piotr Bronowski <piotrx.bronowski@intel.com> Signed-off-by: Dariusz Kazimierski <dariuszx.kazimierski@intel.com> Signed-off-by: Piotr Kleski <piotrx.kleski@intel.com> Change-Id: I4c3fcccf55c36842b7b48aed260fef2802b5c54b