aboutsummaryrefslogtreecommitdiffstats
path: root/tests
AgeCommit message (Collapse)AuthorFilesLines
2023-09-04fix(hoststack): hardcode DUT1 for nginxVratko Polak2-2/+2
Previously, the Set up LDP or VCL Nginx on DUT node keyword was wrongly documenting its "dut" argument as type string, while using the value as a node dict instead. As most of the code already used direct dut1 variables, this patch eliminates the dut argument altogether and puts DUT1 into the keyword name. Change-Id: Ie29afc175b98c1b1cb554438da9178738662d108 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2023-07-14feat(crc): bump messages for cycle after 2306Vratko Polak6-12/+12
The supported VPP code is 380c62d7040bf285d449f1d05692141c40089470 which is the latest commit before stable/2306 diverged from master. + Separate messages into blocks with paths to .api sources. + Prepare memif for use_dma (currently always false). + Use srv6 test teardown action in srv6 device tests. - Do not use sr_policy_add_v2 from 37258 yet. - Leave a TODO to do it eventually, for now inputs are too different. - Do not use index based policer calls from 37873. - The policer_add_del is not deprecated yet and inputs are different. Change-Id: I350138f09542ec12ce4bec140203175940ccf9d4 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2023-06-21feat(infra): update 3n-alt cx6 iterative testsJuraj Linkeš2-0/+354
Add ipsec reassembly test cases to bring in line with 3n-icx. Add ipsec policy test cases with flow cache enabled for both inbound and outbound traffic. Remove GTPU hardware tests as the NIC doesn't support GTPU offload RX. Change-Id: I7b18914f983d5974b390f995ec638a64b9bbccbb Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2023-06-02job-spec: Replace TRex scale2m-tg tests by scale20k-tgTibor Frank2-16/+16
Change-Id: If7e00460fee4eee50479dc089c026670dcf3edf9 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2023-05-31"fix(test): Hoststack"Peter Mikus8-13/+13
Change-Id: I6ec27d787b59d4bc13b8c04f4217302cc3021581 Signed-off-by: pmikus <peter.mikus@protonmail.ch>
2023-05-30fix(test): Hoststackpmikus8-13/+13
Signed-off-by: pmikus <peter.mikus@protonmail.ch> Change-Id: I676696928fd1758c9dddf4f2580b9e924fe71b53
2023-05-29feat(tox): Cleanup docspmikus1-142/+0
Signed-off-by: pmikus <peter.mikus@protonmail.ch> Change-Id: Ia13e920ef11d364893a51957778e60084f4ac0f9
2023-05-29docs(var): Next releasepmikus1-0/+142
Signed-off-by: pmikus <peter.mikus@protonmail.ch> Change-Id: I5cd45dd59f13e1ca942f44341b5f37e257c2b51e
2023-05-24feat(reassembly): add few suites for ip reassemblyVratko Polak7-0/+1174
In tunneling scenarios, VPP needs to reassembly packet fragments before decapsulation. Conveniently, VPP also fragments the packets after encapsulation (at least for some encapsulation protocols) if they do not fit into MTU. So this change adds few suites, adapted from existing 3-node ones which additionally lower MTU on DUT1-DUT2 link. The intended packet size is 1518B. 64B packets will fit, and most of IMIX packets will also fit without fragmentation. VPP refuses to reassembly 9000B packets as they have too many fragments. + Add a keyword so suites need only one additional line of code. + Add new test tag REASSEMBLY for the added suites. + Tell autogen the one soak test case for reaseembly should be 1518B. - It is possible to increase fragment limit for 9000B, not done yet. - Ipsec policy suites are added but they should not be used. - VPP does fragmentation twice (before and after encapsulation). - VPP does not perform reassembly (only last fragment is decrypted). - TRex does not realize the packet from VPP is only a fragment. - The result is false pass with unfairly good performance. + Suites still included, to simplify verifying VPP fixes for the above. Change-Id: If33c60d767fea161d9e4ffabb8ded3d81c8f39ed Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2023-05-10feat(infra): Remove system.d dependencypmikus2-6/+5
Signed-off-by: pmikus <peter.mikus@protonmail.ch> Change-Id: Icb7b6124dcba7bb57c18ceb91120284f6fe02c2e
2023-04-20Add 100k tunnels ipsec policy mode with fastpath enabled test suiteYulong Pei1-0/+178
Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I3708253adf4c7421ff48eee6aefb735b39726359
2023-04-03Enable IPsec policy fast path inboundYulong Pei3-3/+6
This will greatly improve the performance of IPsec policy cases Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: Ib42b5650791be7fcc8649873eaf12616aeba5aec
2023-03-02fix(virtio): add vhost pluginVratko Polak5-10/+10
Previously, only vhost suites got the plugin. Fixes: 5e663cf4cfa7c3ce187dc4f223fa94dfb41f484b Change-Id: I020410581583c6226936aa9480913c751b5e8852 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2023-02-09Add 1M flows test suites of VPP IP4 and VPP IP6Yulong Pei4-0/+579
VPP routing with one million fib rules also is a key indicator for performance benchmark Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I19b52f6b96bdc5ed1e76305d258825fb17bd3af9
2023-02-02fix(perf/hoststack/quic): enable plugin_crypto_openssl.soDave Wallace4-9/+9
- convert hoststack tests to use app-api-socket instead of vpp-api-socket Change-Id: Ibcd87f3624e33168ed3afcb30267262efc927e7c Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-01-24Tune host stack performance testsFilip Tehlar2-16/+26
* remove evt_qs_memfd_seg and evt_qs_seg_size as they are obsolete * use app socket api instead of vpp's api * apply rx and tx descriptors constraints * increase event queue size for nginx tests Signed-off-by: Filip Tehlar <ftehlar@cisco.com> Change-Id: Ie7ac12b8f587e0b1962884127e8eb68a4e588aee Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-01-19Add 40k tunnels ipsec policy mode with fastpath enabled test suiteYulong Pei3-6/+182
In order to compare performance number with existed routing based ipip interface mode 40k tunnels ipsec test. Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I94816a8c18f08b539114d6ca9d4e1218a2b6ea5f
2022-12-14nginx: bump down to working 1_21_5 versionViliam Luc2-20/+20
Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I9ff7abda9961d92c98ab08c77e79406644c115fc
2022-12-06feat(model): Cleanup and introduce telemetrypmikus1-2/+2
- Due to divergence from original design path the RAW was never consumed. It adds too much code complexity and requires processing on both storage and compute. Removing entirely to make modeling efficient. - log (apparently SSH) section will never be consumed in the way it is coded in model. This section is also not part of model schema itself due to the point above. - Introducing telemetry section that is going to carry telemetry items required for CDash. Signed-off-by: pmikus <peter.mikus@protonmail.ch> Change-Id: I7e0256c6c9715de8ee559eed29dce96329aac97d
2022-11-25nginx: new stable 1.22.1 versionViliam Luc4-266/+20
- removed old 1_15_0 - removed old 1_14_2 + updated job_specs Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: Ie75cde28f485b809fe08f808f4fa7d31d52c3929
2022-11-14TAGS: minor update in TRex tests tagsViliam Luc12-25/+24
- fix ip6 scale tag - unified order of tags Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I173bd8369167f07a118f607a9c1957e354f772b2
2022-11-14CSIT-1879: Enable vxlan plugin in all vxlan test suitesSteven Luong37-40/+43
vxlan is going to a plugin. https://gerrit.fd.io/r/c/vpp/+/37511 Need to add vxlan_plugin.so to all vxlan test cases Signed-off-by: Steven Luong <sluong@cisco.com> Change-Id: Ie7f9c6c61eac4b85aeca769d467ddd0f7d5118f0 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-11-14CSIT-1880 - Enable vhost plugin in all vhost test suitesSteven Luong105-194/+204
vhost is going to a plugin. https://gerrit.fd.io/r/c/vpp/+/37488 Need to add vhost_plugin.so to all vhost test cases Signed-off-by: Steven Luong <sluong@cisco.com> Change-Id: I5c79795c1ba5ffcdc9d10697faffbc0b49b22abb
2022-10-03Add 10k tunnels IPSec policy mode test suitesYulong Pei4-0/+696
Also add 10k tunnels IPSec policy mode test with spd fast path enabled. spd fast path is to reduce policy adding time, refer to patchset [1] & [2] for details. [1] https://gerrit.fd.io/r/c/vpp/+/36095 [2] https://gerrit.fd.io/r/c/vpp/+/36097 Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: yulong pei <yulong.pei@intel.com> Change-Id: I211b38a60d273c46e68a5b5f712090037977c6e6
2022-09-30Add VPP wireguard async mode test suiteYulong Pei34-61/+1081
Add VPP wireguard async mode test suite to use QAT device for crypto. Also change keyword ipsechw to cryptohw in suite_setup.robot since currently crypto device is not only used by IPSec. Signed-off-by: Yulong Pei <yulong.pei@intel.com> Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Change-Id: Ibdadb3b09c04b7181415ffd4a248abddc6289075
2022-09-30Add wireguard multiple tunnels test suitesYulong Pei6-4/+849
Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I7abd546e67fdbe481b204bb6a1ec7e9c654dcdae
2022-09-28telemetry: linux telemetry with perf-statViliam Luc1-2/+2
Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I17ced17a309cc0ac21c5fc94e570c89a456339e2
2022-07-27fix: start testpmd and l3fwd in 3 cyclesViliam Luc4-12/+16
Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I935ee5cf5bd7e760ef29fdc338f981e4e7cc333f
2022-06-30job_specs: add wireguard perf tests to regressionYulong Pei1-1/+1
Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I9e499df973fd9990ab02fe7008d12e78b2ab089f
2022-06-28feat(telemetry): ReworkPeter Mikus81-147/+147
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0a84ce88b318c488ba2d519b20237c88b9f9f1e6
2022-06-17Add ipsec policy test suites with flow director or rss acceleratedYulong Pei3-0/+535
The test scenario is about two private networks which communication was protected by ipsec. The test suites are to show performance impovement about ipsec esp flow lookup that offloaded by NIC flow director and rss. Verified on 3n-clx and Intel E810 NIC environment, with 64B ipsec packet flow, performance improved ~31% with 1C2T, ~110% with 2C4T, ~250% with 4C8T. Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I30aec8c5115e5a6fbef88c11d1bef2624029d1b9
2022-06-15feat(tests): Remove PPS tests.Vratko Polak24-1803/+144
+ Delete nat44 pps suites. + Rename ip4base suites, testcases, suite tags (also tg) to -tput. + Also robot tags like UDP_PPS and TCP_PPS. - Keep traffic types and traffic profile names with pps. + They can be renamed in a different change if needed. + Update job specs. - Do not update methodology yet (avoid conflict with another Change). Ticket: CSIT-1829 Change-Id: Ic0dac441f6e8253edce88e057ab7fad1d3814c24 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-10fix(hoststack): distinguish two tcp tag typesVratko Polak4-8/+8
Previous change did not consider TCP_PPS and TCP_CPS robot tags are used by two different suite types (hoststack or ASTF). This fixes the unintended impact on hoststack. + Add HOSTSTACK tag to VSAP suites. - They could also get VSAP tag, but not needed for this Change. Fixes: 1daa6fdc0bae284dee1b61f34534e59b60b7526a Change-Id: Ic583b5ae336c9b74794706fefc232f221a243c87 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-10fix(wireguard): add missing test teardownVratko Polak1-0/+1
Fixes: 75ef277536c43096e7646430568d97d2cc489238 Change-Id: I94575bee0aa77907cc900b58b186c3612fb18d85 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-09Add Test Suite for VPP WireGuardCui,Cheng1-0/+168
Signed-off-by: Cui,Cheng <cheng.cui@intel.com> Change-Id: I71e257e5f16a887a40ff83d782a9ebdee9d05c34 Signed-off-by: Cui,Cheng <cheng.cui@intel.com>
2022-05-24fix(uti): Add TG type exportPeter Mikus14-28/+28
- This is actually bug not a feature. - AB to be added later - Tested on TREX and iPerf3 Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ib6f2d13e3b9401a9fb5759e42a8a310ee11b9d41
2022-05-18Add gtpu flow offload test suiteYulong Pei2-4/+150
The implementation of GTPU offload rx is to use ip4_gtpu flow MARK action of NIC and vpp flow REDIRECT_TO_NODE and BUFFER_ADVANCE fuction to direct received gtpu flow to gtpu4-flow-input graph node, skipped ethernet-input, ip4-input, ip4-lookup, ip4-local, ip4-udp-lookup normal graph node processing. Verified on 3n-clx and Intel E810 NIC environment, single core with 64B packet, performance improve ~33% that compare with pure software way. Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I2af4589448bdb1729e4ce206a8cf3a1239c61af8 Signed-off-by: Yulong Pei <yulong.pei@intel.com>
2022-05-03feat(astf): Support framesizes for ASTFVratko Polak34-356/+1125
- No support for IMIX. + Fix a bad bug in padding (most ASTF profiles had wrong frame sizes). + Fix a big typo in TCP PPS profiles (s->c was not data, just RST). + Control transaction size via ASTF_N_DATA_FRAMES env variable. - Default value 5 leads to transactions smaller than before. + It ensures transaction is one burst (per direction) even for jumbo. + Edit autogen to set supported frame sizes based on suite id. + Both TCP and UDP use the same values: + 64B for CPS (exact for UDP, nominal for TCP). + 100B, 1518B and 9000B for TPUT and PPS. - TCP TPUT achievable minimum is 70B. + Used 100B to leave room for possible IPv6 ASTF tests. + Separate function for code reused by vpp and trex tests. - I do not really like the new "copy and edit" approach added here. + But it is a quick edit, better autogen refactor is low priority. + Consider both established and transitory sessions as valid. - Mostly for compatibility with 2202 behavior and to avoid ramp-ups. - Assuming both session states have similar enough VPP CPU overhead. + Added a TODO to investigate and maybe reconsider later. + Update the state timeout value to 240s. + That is the default for TCP (for transitory state). - UDP could keep using 300s. + But I prefer UDP and TCP to behave as similarly as possible. + Use TRex tunables to get the exact frame size (for data packets). - It is not clear why the recipe for MSS has to be this complicated. + Move code away from profile init, as frame size is not known there. + Change internal profile API, so values related to MSS are passed. + Lower ramp-up rate for TCP TPUT tests. + Because without lower rate, jumbo fails on packet loss in ramp-up. + UDP TPUT ramp-up rate also lowered (just to keep suites more similar). + Distinguish one-direction and aggregated average frame size. + Update keyword documentation where the distiction matters. + One-direction is needed for turning bandwidth limit to TPS limit. + Aggregated is needed for correct NDRPDR bandwidth result value. - TCP TPUT will always be few percent below bidirectional maximum. + That is unavoidable, as one direction sends more control packets. + Add runtime consistency checks so future refactors are safer. + Fail if padding requested would be negative. + Fail if suite claims unexpected values for packets per transaction. + Edit the 4 types of ASTF profiles to keep them similar to each other. + Move UDP TPUT limit value from a field back to direct argument. + Stop pretending first UDP packet is not data. + Apply small improvements where convenient. + Replace "aggregate" with "aggregated" where possible. + To lower probability of any future typos in variable names. + Avoid calling Set Numeric Frame Sizes twice. + Code formatting, keyword documentation, code comments, ... + Add TODOs for less important code quality improvements. - Postpone updating of methodology pages to a subsequent change. Change-Id: I4b381e5210e69669f972326202fdcc5a2c9c923b Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-04-11feat(telemetry): T-Rex runtimePeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I2829b5931e5a63a33f524e1d14ffb20c27f4ce5a
2022-03-31fix(geneve): Restore session multiplicationVratko Polak5-10/+10
Previous edit was restoring the correct formatting for bold text in suite documentation by a mass replace of single asterisk to double asterisk. But in geneve suites, one usage of single asterisk was numeric multiplication, which was errorneously replaces with double asterisk (power operation). This restores the multiplication. Fixes: e82ebbd96e2ba73276e2f1b6d7f9c2d8a9442c3f Change-Id: I537ebe59dd8c04768a90a5b61a348c57e3dfd149 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-03-30feat(terraform): 1n_aws_c5nPeter Mikus13-26/+39
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I12d2d58de01693b18628cd73b5d8c36467506f32
2022-03-22fix(nat44det): reduce ramp-up rateVratko Polak6-18/+18
The previous values were too high for 9000b bandwidth. In order to avoid any packet loss, the new values are low enough even for x710 (10 Gbps). The ramp-up rate is also considerably lowered for lower scales, to keep the rate constant. Ramp-up duration slightly increased above the minimum needed to avoid time overheads, duration stretching and rounding errors. Ticket: CSIT-1811 Change-Id: Iefa8d4d9344e3876014dc5415a4acfedc5abf429 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-02-04Fix: Typo in Perf.NfvDensity.DcrMemif.ChainIpsecTibor Frank35-35/+35
Change-Id: I901ef0afce782bf69b7169e0dd08ce06b6f074b0 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2022-01-17fix(IPsec): fix policy testsJuraj Linkeš28-58/+58
Replace the hardcoded SPD inbound/outbound ranges with values derived from test inputs. Add the necessary routes now that the tunnel endpoints are not in the same subnet. Also add ip neighbor entry on DUT2 for the same reason. Also replace ipsec sa dump with show ipsec all in teardown of tests where both SAs and SPDs are configured to improve troubleshooting. Change-Id: I7d89a99fcf457a701c87bf6ac07364b62802677d Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2021-12-22add 2-node ipsec hwasync casesxinfeng zhao11-0/+1837
Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Change-Id: I1590d438216e4c886aae03f83364837153623dbc
2021-12-15UTI: Export resultsVratko Polak22-9/+83
+ Model version 1.0.0. - Only some result types are exported. + MRR, NDRPDR and SOAK. - Other result types to be added later. + In contrast, all test types are detected. + Convert custom classes to JSON-serializable equivalents. + Sort dict keys before converting to JSON. + Override the order for some known keys. + Export sets as sorted arrays. + Convert to info content from serialized raw content. + Also export outputs for suite setups and teardowns. + Info files for setup/teardown exist only temporarily. + The data is merged into suite.info.json file. + This simplifies presentation of total suite duration. + Define model via JSON schema: - Just test case, suite setup/teardown/suite to be added later. - Just info, raw to be added later. + Proper descriptions. + Json is generated from yaml. + This is a convenience for maintainers. + The officially used schema is the .json one. + TODOs written into a separate .txt file. + Validate exported instance against the schema. + Include format checking. + Update CSIT requirements for validation dependencies. + This needs python-dateutil==2.8.2, only a patch bump. + Compute bandwidth also for soak tests. + This unifies with NDRPDR to simplify schema definition. - PAL may need an update for parsing soak test message. + Include SSH log items, raw output only. + Generate all outputs in a single filesystem tree. + Move raw outputs into test_output_raw.tar.xz. + Rename existing tar with suites to generated_robot_files.tar.xz. Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-12-15docs: small update structure fixViliam Luc9-22/+19
Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: Ib0a853f7bd8cdbb0dee747e4f601bc4e54d006e9
2021-12-13doc: fixing documentation structureViliam Luc678-13631/+17013
When building documentation using sphinx we see ~1200 similar warnings [0] [0] - https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-verify-tox-master-ubuntu2004-x86_64/3289/doc_verify.log.gz These warning are harmless and can be fixed later Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I1ac1099d38935971d47491dde905715345d3935c
2021-11-29Remove executable flag from suite filesVratko Polak4-0/+0
If ./{filename} does not execute, {filename} should not be flagged as executable. No .robot file is executable. Change-Id: I7add9d223ad0e587bc041a4cca6bac222a89e078 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-11-24NAT44det: Decrease ramp-up rate at big scalesVratko Polak2-4/+4
Some testbeds occasionally miss one session with the previous value. + Increase ramp-up duration proportionally. Change-Id: I5300127d7e8fbc670c37ae392ab514b077b3c9b1 Signed-off-by: Vratko Polak <vrpolak@cisco.com>