aboutsummaryrefslogtreecommitdiffstats
path: root/src/vlib/buffer_funcs.h
AgeCommit message (Collapse)AuthorFilesLines
2024-04-10vlib: fix vlib_buffer_free_inline() AVX-512 overflowBenoît Ganne1-1/+1
'queue' is declared on the stack and must be big enough to handle queue_size + max processed elements per loops. When AVX-512 was added, this additional max was not increased from 4 to 8 accordingly. Type: fix Fixes: 767a9ab4b1 Change-Id: I15f2f930fc316f92da2e4642f41852d856c06a48 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2023-12-21buffers: add vlib_buffer_unalloc_to_pool()Damjan Marion1-0/+17
Change-Id: Icd616989a525ef0be6c73a2ad91ae4871cf3921d Type: improvement Signed-off-by: Damjan Marion <damarion@cisco.com>
2023-10-17buffers: introduce vlib_buffer_template_tDamjan Marion1-17/+16
Type: improvement Change-Id: Ie86a5edf2ada21355543e9a0382052b16ff86927 Signed-off-by: Damjan Marion <damarion@cisco.com>
2023-03-10vlib: 512-bit SIMD version of vlib_buffer_freeLeyi Rong1-1/+69
Process 8 packets perf batch in vlib_buffer_free_inline() when CLIB_HAVE_VEC512 is enabled. Type: improvement Signed-off-by: Leyi Rong <leyi.rong@intel.com> Change-Id: I78b8a525bce25ee355c9bf0e0f651698a8c45bda
2022-06-29vlib: enqueue_to_next_with_aux implementationMohammed Hawari1-0/+11
Change-Id: I0e1bb39d765ec3efa7b28ca02fb7beeb23607e51 Type: improvement Signed-off-by: Mohammed Hawari <mohammed@hawari.fr>
2022-05-24vlib: implement aux data handoffMohammed Hawari1-3/+6
Type: improvement Change-Id: I20b41537a249a55f01004e45392b34adaa8fd792 Signed-off-by: Mohammed Hawari <mohammed@hawari.fr>
2021-11-08vppinfra: vectorized index to pointer functionDamjan Marion1-92/+29
Type: improvement Change-Id: I05e1a8fa31761b113355123429d72da18881d4b0 Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-08-27bufmon: add buffer monitoring pluginBenoît Ganne1-11/+8
This plugin allow to keep track of buffer usage in VPP graph nodes. The main use is to detect buffer leakages. Type: feature Change-Id: Iadcf4ab98207fab6e2fa375060879bc2a25b711e Signed-off-by: Benoît Ganne <bganne@cisco.com>
2021-08-20buffers: fix buffer linearizationBenoît Ganne1-95/+104
vlib_buffer_chain_linearize() truncates partial data in chained buffers in corner cases when current_data is negative. Strengthen test cases to reproduce the errors and fix it. Type: fix Change-Id: Ida621923711c5755508224bdc3842b31003c6c0b Signed-off-by: Benoît Ganne <bganne@cisco.com>
2021-05-14vlib: pass node runtime to vlib_buffer_enqueue_to_thread()Damjan Marion1-2/+3
Mechanical change for patch following this one... Type: improvement Change-Id: Iee12f3a8851f35569e6c039494a94fc36e83d20f Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-05-03vlib: multiarch vlib_frame_queue_dequeue()Damjan Marion1-0/+5
Fact that it needs to copy buffer indices justifies this move. Type: improvement Change-Id: I5eb815ccc4cca0ef70b092eb83a49b713efdcbeb Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-26vppinfra: clib_memcpy_u32() utilizing SIMD mask loads/storesDamjan Marion1-37/+1
Type: improvement Change-Id: I55c4688bd1feffae139ce12a66d15885373e5cd7 Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-21buffers: vlib_get_buffers() with 512-bit SIMDDamjan Marion1-5/+46
Type: improvement Change-Id: Id8ce3ffc1299a38171b82a7082454412c840a40c Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-21vlib: don't inline vlib_buffer_enqueue_to_*Damjan Marion1-0/+20
Function call in this case is not bad thing, as those functions are called once per frame. Type: improvement Change-Id: I7bd61cf746b905e2c8e3085ebb2ff001c0a52da8 Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-06buffers: performance improvementRadu Nicolau1-7/+21
Initialize the local variables as to prevent first fall through one by one processing; skip prefetching rather than doing one by one when count below 12. Type: improvement Signed-off-by: Radu Nicolau <radu.nicolau@intel.com> Change-Id: If02935da081e24f7cfe408811ccf6922cf982092
2020-08-31vppinfra: convert A_extend_to_B to B_from_A format of vector inlinesDamjan Marion1-6/+6
Make it shorter and same format when converting to biggor or smaller types. Type: refactor Change-Id: I443d67e18ae65d779b4d9a0dce5406f7d9f0e4ac Signed-off-by: Damjan Marion <damarion@cisco.com>
2020-05-21vlib: enforce buffer alloc return value checkDamjan Marion1-6/+6
Type: improvement Change-Id: I33cd702f9c4d995520509fdb16e3b8f8fd50e535 Signed-off-by: Damjan Marion <damarion@cisco.com>
2020-04-13buffers: configurable buffer fault injectorDave Barach1-0/+11
When configured at compile time via the cmake VPP_BUFFER_FAULT_INJECTOR option, the buffer allocator will appear to fail a certain fraction of the time. By default, the allocator succeeds 80% of the time. Detailed command line configuration options are available, but only when the image has been compiled with cmake option described above: vlib { buffer-alloc-success-rate [0.0 ... 1.0] buffer-alloc-success-seed <nnnn> } Modify vlib_buffer_pool_create(...) so 0 is always an invalid buffer index. Debug images: add checks for bad buffer index enqueues, and also verify that f->n_vectors doesn't accidentally map one or more instances of the frame poison pattern 0xfefefefe. Type: improvement Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Iab939858014463d1e664682805013d334d6fcbe5
2020-04-07l2: handle complete clone fail in l2_floodDave Barach1-2/+2
vlib_buffer_clone(...) may not manage to produce any buffer clones at all. vlib_buffer_clone_256 should not smash the original buffer reference count if no clones are produced. Type: fix Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I14d9d53637a220485c7a0036cfc75a4149b264ea
2020-03-20rdma: ring db only once per vector on txBenoît Ganne1-0/+18
Type: improvement Change-Id: I09b017d60bc38ff8a4a7a73aff42b27611bd8955 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2020-03-02rdma: improve rx loopDamjan Marion1-0/+18
Type: improvement Change-Id: If81847bc0c92d167ce03e1e94a2f8e18f8154af2 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-11-26misc: add gdb helpers for vlib buffersBenoît Ganne1-0/+7
Type: feature Change-Id: I7e619040857310fffd33c355f6c491a55f7a4a02 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-10-31vlib: only dump 1st buffer in chain by defaultBenoît Ganne1-1/+1
Several nodes include buffers in their traces, but only the 1st. When formatting the trace we must not try to iterate through all chained buffers. Default to display only the 1st buffer. Type: fix Change-Id: Ib3c668bbf4ab70ae68eba2ac402c7b7329825b70 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-10-28vlib: don't use vector for keeping buffer indices in the poolDamjan Marion1-22/+40
Type: refactor Change-Id: I72221b97d7e0bf5c93e20bbda4473ca67bfcdeb4 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-08-15vlib: copy trace_handle in vlib_buffer_copy/clone() functionsJohn Lo1-0/+2
Since vlib_buffer_copy() and vlib_buffer_clone() both preserve VLIB_BUFFER_IS_TRACED bit in flags field, it should also copy trace_handle which would add minimal overhead. Thus, callers of these functions do not have to call vlib_buffer_copy_trace_flags() to copy trace_handle. Type: refactor Signed-off-by: John Lo <loj@cisco.com> Change-Id: Iff6a3f81660dd62b36a2966033eb380305340310
2019-08-09vlib: fix vlib_buffer_copy to preserve buffer flags bitJohn Lo1-3/+8
Make vlib_buffer_copy() preserve buffer flags bit the same way as that of vlib_buffer_clone() so both are consistent. Type: fix Signed-off-by: John Lo <loj@cisco.com> Change-Id: I6c32aa1e88724b482ce2439d82019e690311b664
2019-06-20buffers: fix crashKlement Sekera1-3/+9
this change is being made to fix a crash when current_data < 0 in buffer linearization function Ticket: N/A Type: fix Fixes: f883f6a1132ad4bb7aa9d9a79d420274fbcf3b64 Change-Id: Ia4ede823f673780e0c30d075b091db42e183650d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-06-12buffers: fix vlib_buffer_free_no_nextDamjan Marion1-1/+1
Type: fix Fixes: 910d369 Change-Id: I0e8380cd2b0dc038a028d9cf2568741059de460f Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-04-01Enhance vlib_buffer_clone to change headroom space for packetJohn Lo1-7/+66
Add function vlib_buffer_clone_at_offset() so that the cloned packets' 1st buffer will have the original packet header copied at the specified offset (instead of current_data). This can be used to increase headroom in the cloned packet to allow large header encaps in case the stdandard 128B predata is not enough. The original vlib_buffer_clone() still works the same as before. Change-Id: I3b50c8ad7e3952568bc141710567c99a3dacddce Signed-off-by: John Lo <loj@cisco.com>
2019-03-20Re-enable aarch64 neon instruction in vlib_buffer_free_inlineLijian.Zhang1-3/+3
int vaddvq_u8 (uint8x16_t __a) is not appropriate to implement xxx_is_all_zero, as there may be overflow causing incorrect return value. Here's an example. u8x16 x = {0 <repeats 12 times>, 1, 255, 0, 0}; Change-Id: Ia6a10bdf8da360dec12db902d028751a1a77e9a4 Signed-off-by: Lijian Zhang <Lijian.Zhang@arm.com> Reviewed-by: Sirshak Das <Sirshak.Das@arm.com>
2019-03-13buffers: don't use clib_memcpy for copying buffer indicesDamjan Marion1-1/+37
Should be faster this way if n_indices is not constant value Change-Id: I6c34fd313daa2392199f3b9bd20d0cd6cf9ae21b Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-03-13buffers: fix ref_count mask in vlib_buffer_free_inlineDamjan Marion1-1/+1
Credits to: Lollita Liu <lollita.liu@ericsson.com> Change-Id: I40f7ad9cc5220db7ac4571cb14feaf22fc02aa2e Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-03-12ICMP46 error: Clone first buffer instead of "truncating" original bufferOle Troan1-0/+21
Previous code was walked buffer chain, effectively trying to "truncate" the chain, reset the length of first buffer and reused that as the ICMP error message. That could have issues in cases there were other users of the buffer chain. Update to clone the first buffer in chain, and use that for the ICMP error message instead. Change-Id: Ibc1a0bf2d854dae41874808c8297028ed93dd69d Signed-off-by: Ole Troan <ot@cisco.com>
2019-03-04buffers: support cases where numa node memory is not availableDamjan Marion1-2/+2
Change-Id: I550ef893e41d86310d4e5af16d6100b8e557b68a Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-02-25buffer chain linearizationKlement Sekera1-97/+120
Rewrite vlib_buffer_chain_linearize function so that it works as intended. Linearize buffer chains coming out of reassembly to work around some dpdk-tx issues. Note that this is not a complete workaround as a sufficiently large packet will still cause the resulting chain to be too long. Drop features from reassembly code which relies on knowing which and how many buffers were freed during linearization, buffer counts and tracing capabilities for these cases. Change-Id: Ic65de53ecb5c78cd96b178033f6a576ab4060ed1 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-02-14buffer: bug fix vlib_buffer_free_inlineLollita Liu1-6/+7
buffer_pool_index, in case of buffer_pool_index change, should free buffers in queue before setting buffer_pool_index again Change-Id: I846ead947a7b2d940c8fc747976239d608597391 Signed-off-by: Lollita Liu <lollita.liu@ericsson.com>
2019-02-09buffers: fix typoDamjan Marion1-4/+4
Change-Id: I4e836244409c98739a13092ee252542a2c5fe259 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-02-06buffers: make buffer data size configurable from startup configDamjan Marion1-9/+10
Example: buffers { default data-size 1536 } Change-Id: I5b4436850ca18025c9fdcfc7ed648c2c2732d660 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-31buffers: vallidate that buffer is allocated during buffer pool putDamjan Marion1-8/+5
Change-Id: I8044b34a37fe1994a8dfa1ca89929f3642c72e8d Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-31buffers: reinitialize metadata, add additional validationDamjan Marion1-13/+43
- DPDK overwrites metadata as part of rte_pktmbuf_init(...) so we need reinitialize it - additional checks added to ensure ref_count is never < 1 Change-Id: Ida336f81c4723e8f2e0ad4a70cb7b1ecfff978a0 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-30buffers: major cleanup and improvementsDamjan Marion1-187/+329
This patch introduces following changes: - deprecated free lists which are not used and not compatible with external buffer managers (i.e. DPDK) - introduces native support for per-numa buffer pools - significantly improves performance of buffer alloc and free Change-Id: I4a8e723ae47056717afd6cac0efe87cb731b5be7 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-29Fix error while compiling using 'vlib/vlib.h' dependencie.Dmitry Vakhrushev1-0/+2
This variable without any needed 'define' becoming unused. Change-Id: I661a75a78dba03abb861ed918ad1e634a8ecd0af Signed-off-by: Dmitry Vakhrushev <dmitry@netgate.com>
2019-01-24buffers: fix vector typesDamjan Marion1-1/+3
Change-Id: I9df96264e30806ac3daf7121f314f34f06232413 Signed-off-by: Damjan Marion <damjan.marion@gmail.com>
2019-01-23buffers: wrap vlib_buffer_t to union and expose vector typesDamjan Marion1-1/+12
Change-Id: I1c12e2941cae198ededbb65eb5be51a4eabe2c1b Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-20buffers: remove VLIB_BUFFER_DEFAULT_FREE_LIST macro and fl->n_data_bytesDamjan Marion1-33/+3
Change-Id: I0ba5175be077c40556f2a3ce629c5bbcd71e0a81 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-20buffers: remove vlib_buffer_delete_free_listDamjan Marion1-16/+0
Change-Id: I5fe01e918b534d1ac1d47f0d7610536f45c3158c Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-20buffers: don't init metadata, as it is already initializedDamjan Marion1-38/+1
Change-Id: Ia083050389853c25b069f0f8286d50d3f4aef527 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-20buffers: keep buffer_main in vlib_main_tDamjan Marion1-14/+22
Change-Id: I3bb1d9f83dd08f4b93acd4a281bfec0674e39c2e Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-19buffers: remove free-list information from buffer metadataDamjan Marion1-45/+6
Change-Id: I6048c6a51efa826ac333f7d15919cb87dd766d74 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-01-18buffers: remove freelist argument from vlib_buffer_add_dataDamjan Marion1-3/+2
Change-Id: Idb224b807373128b78dc46f177b2d2e0ed288349 Signed-off-by: Damjan Marion <damarion@cisco.com>