aboutsummaryrefslogtreecommitdiffstats
path: root/src/vppinfra/vector_avx2.h
AgeCommit message (Collapse)AuthorFilesLines
2021-05-05vppinfra: fix x86 packs / packus wrappersDamjan Marion1-10/+12
They both take signed value as input. Type: fix Change-Id: If3d8ec4e0b1c02d7d65262bdd9db49ff7fbfef39 Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-27vlib: improve enqueue_to_next buffer indices extractionDamjan Marion1-1/+20
Type: improvement Change-Id: Ib7b2fa7d821f6d2708f6dc378a0f36f68c843f57 Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-25vppinfra: AVX512 mask load/stores and compress storeDamjan Marion1-8/+0
Type: improvement Change-Id: Id6be598aade072653e408cca465e62931d060233 Signed-off-by: Damjan Marion <damarion@cisco.com>
2021-04-15vppinfra: correct intrinsic called by u16x16_from_u8x16Lijian.Zhang1-2/+2
u16x16_from_u8x16() and i16x16_from_i8x16() call intrisics _mm256_cvtepu8_epi64 and _mm256_cvtepi8_epi64. But they are not seems doing the right data conversion from the name of the wrappers. The correct intrinsics been called should be _mm256_cvtepu8_epi16 and _mm256_cvtepi8_epi16. Type: fix Change-Id: Id71de6ae1a266a370f11c33a46684202be766c43 Signed-off-by: Lijian Zhang <Lijian.Zhang@arm.com>
2020-08-31vppinfra: convert A_extend_to_B to B_from_A format of vector inlinesDamjan Marion1-2/+2
Make it shorter and same format when converting to biggor or smaller types. Type: refactor Change-Id: I443d67e18ae65d779b4d9a0dce5406f7d9f0e4ac Signed-off-by: Damjan Marion <damarion@cisco.com>
2020-07-15vppinfra: more vector inlinesDamjan Marion1-0/+25
Type: improvement Change-Id: Ie0de374b89ec3a17befecf3f08e94951597609ec Signed-off-by: Damjan Marion <damarion@cisco.com>
2020-03-16rdma: add Mellanox mlx5 Direct Verbs receive supportDamjan Marion1-0/+10
Type: feature Change-Id: I3f287ab536a482c366ad7df47e1c04e640992ebc Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-04-16vppinfra: more AVX2 and AVX512 inlinesDamjan Marion1-0/+26
Change-Id: I81bd967a580ae3b476dfd731e9933a9898568a91 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-04-08vppinfra: u32x8 transposeDamjan Marion1-0/+56
Change-Id: I7d39cb184f1f9ad24276183c29969327681a1f82 Signed-off-by: Damjan Marion <damarion@cisco.com>
2019-03-26ipsec: esp-encrypt reworkDamjan Marion1-0/+13
Change-Id: Ibe7f806b9d600994e83c9f1be526fdb0a1ef1833 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-11-20vppinfra: add 128 and 256 bit vector scatter/gather inlinesDamjan Marion1-0/+59
Change-Id: If6c65f16c6fba8beb90e189c1443c3d7d67ee02c Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-10-17bond: tx optimizationsDamjan Marion1-0/+12
Break up bond tx function into multiple small workloads: 1. parse the packet header and hash it based on the configured algorithm 2. optionally, trace the packet 3. convert the hash value from (1) to the slave port 4. update the buffers with the slave sw_if_index 5. Add the buffers to the queues 6. Create and send the frames old numbers ----------- Time 5.3, average vectors/node 223.74, last 128 main loops 40.00 per node 222.61 vector rates in 3.3627e6, out 6.6574e6, drop 3.3964e4, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call BondEthernet0-output active 68998 17662979 0 1.89e1 255.99 BondEthernet0-tx active 68998 17662979 0 2.60e1 255.99 TenGigabitEthernet3/0/1-output active 68998 8797416 0 1.03e1 127.50 TenGigabitEthernet3/0/1-tx active 68998 8797416 0 7.85e1 127.50 TenGigabitEthernet7/0/1-output active 68996 8865563 0 1.02e1 128.49 TenGigabitEthernet7/0/1-tx active 68996 8865563 0 7.65e1 128.49 new numbers ----------- BondEthernet0-output active 304064 77840384 0 2.29e1 256.00 BondEthernet0-tx active 304064 77840384 0 2.47e1 256.00 TenGigabitEthernet3/0/1-output active 304064 38765525 0 1.03e1 127.49 TenGigabitEthernet3/0/1-tx active 304064 38765525 0 7.66e1 127.49 TenGigabitEthernet7/0/1-output active 304064 39074859 0 1.01e1 128.51 Change-Id: I3ef9a52bfe235559dae09d055c03c5612c08a0f7 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-07-16vppinfra: AVX2 interleave functionsDamjan Marion1-3/+14
Change-Id: I8688f700fccd87484da3e202ca3a070cc14eb267 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-07-12Revert "vppinfra: AVX2 blend"Dave Barach1-6/+0
Causes clang validation failures. The patch did not actually pass validation; unfortunately it received a +1 from fd.io JJB - presumably due to a race condition This reverts commit 779c865cc6c7af5bb435d8b3465d80685370edb2. Change-Id: Ica3697f8f90e67d3eae4debc597f27d7d512004a Signed-off-by: Dave Barach <dbarach@cisco.com>
2018-07-12vppinfra: AVX2 blendDamjan Marion1-0/+6
Change-Id: Ie7a64318f10ebb535c98aff4e25cdfc48f60ff33 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-06-28ip: vectorized ip checksumDamjan Marion1-0/+28
Change-Id: Ida678e6f31daa8decb18189da712a350336326e2 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-06-27vppinfra: add vector horizontal add and byte swap (SSE4.2 & AVX2)Damjan Marion1-0/+16
Change-Id: I4e0fd487970796f0153a5b16333827d23b57deac Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-05-25Vectorized bihash_{48,40,24,16}_8 key compareDamjan Marion1-20/+25
bihash_48_8 case: Scalar code: 6 clocks SSE4.2 code: 3 clocks AVX2 code: 2.27 clocks AVX512 code: 1.5 clocks Change-Id: I40700175835a1e7321276e47eadbf9771d3c5a68 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-05-22vppinfra: add clib_count_equal_uXX and clib_memset_uXX functionsDamjan Marion1-2/+2
Change-Id: I56782652d8ef10304900cc293cfc0502689d800e Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-05-20vector functions cleanup and improvementsDamjan Marion1-4/+11
Remove functions which have native C equivalent (i.e. _is_equal can be replaced with ==, _add with +) Add SSE4.2, AVX-512 implementations of splat, load_unaligned, store_unaligned, is_all_zero, is_equal, is_all_equal Change-Id: Ie80b0e482e7a76248ad79399c2576468532354cd Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-05-18Add vlib_buffer_enqueue_to_next inline functionDamjan Marion1-0/+6
Change-Id: I1042c0fe179b57a00ce99c8d62cb1bdbe24d9184 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-05-17Add buffer pointer-to-index and index-to-pointer array functionsDamjan Marion1-0/+22
Change-Id: Ib3fcc3ceb7f315389bcdecbb7d9632540a5dd6ba Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-05-09dpdk: tx code reworkDamjan Marion1-0/+12
Change-Id: Ifea9c772e8784642433b92091f5769eb9ec06890 Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-04-25dpdk: complete rework of the dpdk-input nodeDamjan Marion1-0/+80
Change-Id: If174d189de40e6f9ffae99997bba93a2519d9fda Signed-off-by: Damjan Marion <damarion@cisco.com>
an class="p">(FIB_PATH_LIST_FLAG_NONE, paths); src->fes_entry_flags |= (fib_entry_get_flags(fib_entry_get_index(cover)) & FIB_ENTRY_FLAGS_RR_INHERITED); vec_free(paths); } /** * Source initialisation Function */ static void fib_entry_src_rr_init (fib_entry_src_t *src) { src->u.rr.fesr_cover = FIB_NODE_INDEX_INVALID; src->u.rr.fesr_sibling = FIB_NODE_INDEX_INVALID; } /* * use the path-list of the cover, unless it would form a loop. * that is unless the cover is via this entry. * If a loop were to form it would be a 1 level loop (i.e. X via X), * and there would be 2 locks on the path-list; one since its used * by the cover, and 1 from here. The first lock will go when the * cover is removed, the second, and last, when the covered walk * occurs during the cover's removal - this is not a place where * we can handle last lock gone. * In short, don't let the loop form. The usual rules of 'we must * let it form so we know when it breaks' don't apply here, since * the loop will break when the cover changes, and this function * will be called again when that happens. */ void fib_entry_src_rr_use_covers_pl (fib_entry_src_t *src, const fib_entry_t *fib_entry, const fib_entry_t *cover) { fib_node_index_t *entries = NULL; dpo_proto_t proto; fib_entry_src_t *s; proto = fib_proto_to_dpo(fib_entry->fe_prefix.fp_proto); vec_add1(entries, fib_entry_get_index(fib_entry)); if (fib_path_list_recursive_loop_detect(cover->fe_parent, &entries)) { src->fes_pl = fib_path_list_create_special(proto, FIB_PATH_LIST_FLAG_DROP, drop_dpo_get(proto)); } else { src->fes_pl = cover->fe_parent; vec_foreach (s,cover->fe_srcs) { if (s->fes_pl != cover->fe_parent) continue; src->fes_path_exts.fpel_exts = vec_dup (s->fes_path_exts.fpel_exts); break; } } vec_free(entries); } /* * Source activation. Called when the source is the new best source on the entry */ static int fib_entry_src_rr_activate (fib_entry_src_t *src, const fib_entry_t *fib_entry) { fib_entry_t *cover; /* * find the covering prefix. become a dependent thereof. * for IP there should always be a cover, though it may be the default route. * For MPLS there is never a cover. */ if (FIB_PROTOCOL_MPLS == fib_entry->fe_prefix.fp_proto) { src->fes_pl = fib_path_list_create_special(DPO_PROTO_MPLS, FIB_PATH_LIST_FLAG_DROP, NULL); fib_path_list_lock(src->fes_pl); return (!0); } src->u.rr.fesr_cover = fib_table_get_less_specific(fib_entry->fe_fib_index, &fib_entry->fe_prefix); ASSERT(FIB_NODE_INDEX_INVALID != src->u.rr.fesr_cover); cover = fib_entry_get(src->u.rr.fesr_cover); src->u.rr.fesr_sibling = fib_entry_cover_track(cover, fib_entry_get_index(fib_entry)); /* * if the cover is attached then install an attached-host path * (like an adj-fib). Otherwise inherit the forwarding from the cover */ if (FIB_ENTRY_FLAG_ATTACHED & fib_entry_get_flags_i(cover)) { fib_entry_src_rr_resolve_via_connected(src, fib_entry, cover); } else { fib_entry_src_rr_use_covers_pl(src, fib_entry, cover); } fib_path_list_lock(src->fes_pl); /* * return go for install */ return (!0); } /** * Source Deactivate. * Called when the source is no longer best source on the entry */ static void fib_entry_src_rr_deactivate (fib_entry_src_t *src, const fib_entry_t *fib_entry) { fib_entry_t *cover; /* * remove the dependency on the covering entry */ if (FIB_NODE_INDEX_INVALID != src->u.rr.fesr_cover) { fib_node_index_t *entries = NULL; cover = fib_entry_get(src->u.rr.fesr_cover); fib_entry_cover_untrack(cover, src->u.rr.fesr_sibling); src->u.rr.fesr_cover = FIB_NODE_INDEX_INVALID; if (FIB_NODE_INDEX_INVALID != cover->fe_parent) { fib_path_list_recursive_loop_detect(cover->fe_parent, &entries); vec_free(entries); } } fib_path_list_unlock(src->fes_pl); src->fes_pl = FIB_NODE_INDEX_INVALID; vec_free (src->fes_path_exts.fpel_exts); src->fes_entry_flags = FIB_ENTRY_FLAG_NONE; } fib_entry_src_cover_res_t fib_entry_src_rr_cover_change (fib_entry_src_t *src, const fib_entry_t *fib_entry) { fib_entry_src_cover_res_t res = { .install = !0, .bw_reason = FIB_NODE_BW_REASON_FLAG_NONE, }; if (FIB_NODE_INDEX_INVALID == src->u.rr.fesr_cover) { /* * the source may be added, but it is not active * if it is not tracking the cover. */ return (res); } /* * this function is called when this entry's cover has a more specific * entry inserted benaeth it. That does not necessarily mean that this * entry is covered by the new prefix. check that */ if (src->u.rr.fesr_cover != fib_table_get_less_specific(fib_entry->fe_fib_index, &fib_entry->fe_prefix)) { fib_entry_src_rr_deactivate(src, fib_entry); fib_entry_src_rr_activate(src, fib_entry); /* * dependent children need to re-resolve to the new forwarding info */ res.bw_reason = FIB_NODE_BW_REASON_FLAG_EVALUATE; } return (res); } /* * fib_entry_src_rr_cover_update * * This entry's cover has updated its forwarding info. This entry * will need to re-inheret. */ fib_entry_src_cover_res_t fib_entry_src_rr_cover_update (fib_entry_src_t *src, const fib_entry_t *fib_entry) { fib_entry_src_cover_res_t res = { .install = !0, .bw_reason = FIB_NODE_BW_REASON_FLAG_NONE, }; fib_node_index_t old_path_list; fib_entry_t *cover; if (FIB_NODE_INDEX_INVALID == src->u.rr.fesr_cover) { /* * the source may be added, but it is not active * if it is not tracking the cover. */ return (res); } cover = fib_entry_get(src->u.rr.fesr_cover); old_path_list = src->fes_pl; /* * if the ocver is attached then install an attached-host path * (like an adj-fib). Otherwise inherit the forwarding from the cover */ if (FIB_ENTRY_FLAG_ATTACHED & fib_entry_get_flags_i(cover)) { fib_entry_src_rr_resolve_via_connected(src, fib_entry, cover); } else { fib_entry_src_rr_use_covers_pl(src, fib_entry, cover); } fib_path_list_lock(src->fes_pl); fib_path_list_unlock(old_path_list); /* * dependent children need to re-resolve to the new forwarding info */ res.bw_reason = FIB_NODE_BW_REASON_FLAG_EVALUATE; return (res); } static u8* fib_entry_src_rr_format (fib_entry_src_t *src, u8* s) { return (format(s, " cover:%d", src->u.rr.fesr_cover)); } const static fib_entry_src_vft_t rr_src_vft = { .fesv_init = fib_entry_src_rr_init, .fesv_activate = fib_entry_src_rr_activate, .fesv_deactivate = fib_entry_src_rr_deactivate, .fesv_cover_change = fib_entry_src_rr_cover_change, .fesv_cover_update = fib_entry_src_rr_cover_update, .fesv_format = fib_entry_src_rr_format, }; void fib_entry_src_rr_register (void) { fib_entry_src_behaviour_register(FIB_SOURCE_BH_RR, &rr_src_vft); }