aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot
AgeCommit message (Collapse)AuthorFilesLines
2022-10-03Add 10k tunnels IPSec policy mode test suitesYulong Pei1-0/+14
Also add 10k tunnels IPSec policy mode test with spd fast path enabled. spd fast path is to reduce policy adding time, refer to patchset [1] & [2] for details. [1] https://gerrit.fd.io/r/c/vpp/+/36095 [2] https://gerrit.fd.io/r/c/vpp/+/36097 Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: yulong pei <yulong.pei@intel.com> Change-Id: I211b38a60d273c46e68a5b5f712090037977c6e6 (cherry picked from commit d65da6a72887238eae5dd3b7eb4e53055ef516c3)
2022-10-03feat(soak): add warmup and statsVratko Polak1-0/+24
Specifically on 2n-icx, soak tests are suffering from the first search trial (at half of max rate) reporting unsent packets, thus polluting the results, see CSIT-1867. This adds a plain trial (no stats, result ignored), fixing the issue. To keep the structure similar to NDRPDR tests, stats trial is done after the search, at the discovered critical rate. Ticket: CSIT-1867 Change-Id: I12ff6f58c4eb174594e667bfa86ba1585164b8ef Signed-off-by: Vratko Polak <vrpolak@cisco.com> (cherry picked from commit 141f1d7e939ebe7ca73d4d27c554cdfd8fc3f86d)
2022-09-30Add VPP wireguard async mode test suiteYulong Pei2-1/+10
Add VPP wireguard async mode test suite to use QAT device for crypto. Also change keyword ipsechw to cryptohw in suite_setup.robot since currently crypto device is not only used by IPSec. Signed-off-by: Yulong Pei <yulong.pei@intel.com> Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Change-Id: Ibdadb3b09c04b7181415ffd4a248abddc6289075 (cherry picked from commit 7df86999bfaee9e5a0fa0c8ee6f5bc897343de9a)
2022-08-16fix: add missing closing ENDViliam Luc2-2/+2
Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I5352181e9cc427378ca919bf7459f0b7d1a62bd3
2022-08-16fix(robot): replace Run Keyword UnlessVratko Polak9-39/+39
Use Run Keyword If, with the condition negated. This silences the deprecation warning seen since Robot version bump. Change-Id: I9a88ff551994df3e129d71327f08207133a958a6 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-08-12fix(hoststack): add a missing endVratko Polak1-0/+1
Change-Id: I07de784de4bfd1e1fa4230567331e55a72d1a7de Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-07-27fix: start testpmd and l3fwd in 3 cyclesViliam Luc1-93/+0
Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: I935ee5cf5bd7e760ef29fdc338f981e4e7cc333f
2022-06-28feat(telemetry): ReworkPeter Mikus1-4/+4
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0a84ce88b318c488ba2d519b20237c88b9f9f1e6
2022-06-20fix(uti): export correct units in cps ndrpdrVratko Polak1-1/+1
Compute Bandwidth keyword returns value in Gbps, as that is convenient for test messages. Export Search Bound keyword expects value in bps, as that is convenient for JSON export. While the conversion was present in Display Single PPS Bound keyword, it was missing from Display Single CPS Bound keyword. Now added. Change-Id: I5bb5934beeecfa2ee1b77dd1b1bfc788d034c845 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-20style(robot): capitalize two keywords properlyVratko Polak1-6/+6
This is just to make the subsequent fix more readable. Change-Id: I7cebcaed3f658cd026f05713ea2f94dce60786c0 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-17fix: testpmd and l3fwd check stateViliam Luc1-1/+9
+ print testpmd and l3fwd pid after start Testpmd and l3fwd utility can be started but link might not be ready yet. This fix start the utility on all DUTs and do the check later. Signed-off-by: Viliam Luc <vluc@cisco.com> Change-Id: If476e22f206d9a6a0dd399879a88eafedca92bb6
2022-06-14fix(vlan): do not apply strip offloadVratko Polak2-23/+1
Some tests (bonding) are failing as VPP refuses startup.conf. That is because the config parameter support has been removed: https://gerrit.fd.io/r/c/vpp/+/34822 That is VPP commit 8c8531c15c9195c2c82695eff6e70887de67fd88. According to commit message, it was just a workaround for Cisco ENIC cards, which are no longer tested in CSIT. + Delete 1 robot and 1 python keyword as they become unused. Change-Id: If53e324a27cfbd4659e49ab8bcd03253642dde96 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-10fix(hoststack): distinguish two tcp tag typesVratko Polak1-4/+8
Previous change did not consider TCP_PPS and TCP_CPS robot tags are used by two different suite types (hoststack or ASTF). This fixes the unintended impact on hoststack. + Add HOSTSTACK tag to VSAP suites. - They could also get VSAP tag, but not needed for this Change. Fixes: 1daa6fdc0bae284dee1b61f34534e59b60b7526a Change-Id: Ic583b5ae336c9b74794706fefc232f221a243c87 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-06-09Add Test Suite for VPP WireGuardCui,Cheng1-0/+58
Signed-off-by: Cui,Cheng <cheng.cui@intel.com> Change-Id: I71e257e5f16a887a40ff83d782a9ebdee9d05c34 Signed-off-by: Cui,Cheng <cheng.cui@intel.com>
2022-06-01fix(core): TG exportPeter Mikus1-0/+1
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I80f9f32f10d46eccf1157949a184ec4467d071cd
2022-05-26fix(core): TG exportPeter Mikus1-1/+1
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I60313cd62a878bb713832f14121a938a81c954ed
2022-05-24fix(uti): Add TG type exportPeter Mikus2-17/+33
- This is actually bug not a feature. - AB to be added later - Tested on TREX and iPerf3 Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ib6f2d13e3b9401a9fb5759e42a8a310ee11b9d41
2022-05-18Add gtpu flow offload test suiteYulong Pei1-2/+15
The implementation of GTPU offload rx is to use ip4_gtpu flow MARK action of NIC and vpp flow REDIRECT_TO_NODE and BUFFER_ADVANCE fuction to direct received gtpu flow to gtpu4-flow-input graph node, skipped ethernet-input, ip4-input, ip4-lookup, ip4-local, ip4-udp-lookup normal graph node processing. Verified on 3n-clx and Intel E810 NIC environment, single core with 64B packet, performance improve ~33% that compare with pure software way. Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Signed-off-by: Yulong Pei <yulong.pei@intel.com> Change-Id: I2af4589448bdb1729e4ce206a8cf3a1239c61af8 Signed-off-by: Yulong Pei <yulong.pei@intel.com>
2022-05-03feat(astf): Support framesizes for ASTFVratko Polak3-75/+188
- No support for IMIX. + Fix a bad bug in padding (most ASTF profiles had wrong frame sizes). + Fix a big typo in TCP PPS profiles (s->c was not data, just RST). + Control transaction size via ASTF_N_DATA_FRAMES env variable. - Default value 5 leads to transactions smaller than before. + It ensures transaction is one burst (per direction) even for jumbo. + Edit autogen to set supported frame sizes based on suite id. + Both TCP and UDP use the same values: + 64B for CPS (exact for UDP, nominal for TCP). + 100B, 1518B and 9000B for TPUT and PPS. - TCP TPUT achievable minimum is 70B. + Used 100B to leave room for possible IPv6 ASTF tests. + Separate function for code reused by vpp and trex tests. - I do not really like the new "copy and edit" approach added here. + But it is a quick edit, better autogen refactor is low priority. + Consider both established and transitory sessions as valid. - Mostly for compatibility with 2202 behavior and to avoid ramp-ups. - Assuming both session states have similar enough VPP CPU overhead. + Added a TODO to investigate and maybe reconsider later. + Update the state timeout value to 240s. + That is the default for TCP (for transitory state). - UDP could keep using 300s. + But I prefer UDP and TCP to behave as similarly as possible. + Use TRex tunables to get the exact frame size (for data packets). - It is not clear why the recipe for MSS has to be this complicated. + Move code away from profile init, as frame size is not known there. + Change internal profile API, so values related to MSS are passed. + Lower ramp-up rate for TCP TPUT tests. + Because without lower rate, jumbo fails on packet loss in ramp-up. + UDP TPUT ramp-up rate also lowered (just to keep suites more similar). + Distinguish one-direction and aggregated average frame size. + Update keyword documentation where the distiction matters. + One-direction is needed for turning bandwidth limit to TPS limit. + Aggregated is needed for correct NDRPDR bandwidth result value. - TCP TPUT will always be few percent below bidirectional maximum. + That is unavoidable, as one direction sends more control packets. + Add runtime consistency checks so future refactors are safer. + Fail if padding requested would be negative. + Fail if suite claims unexpected values for packets per transaction. + Edit the 4 types of ASTF profiles to keep them similar to each other. + Move UDP TPUT limit value from a field back to direct argument. + Stop pretending first UDP packet is not data. + Apply small improvements where convenient. + Replace "aggregate" with "aggregated" where possible. + To lower probability of any future typos in variable names. + Avoid calling Set Numeric Frame Sizes twice. + Code formatting, keyword documentation, code comments, ... + Add TODOs for less important code quality improvements. - Postpone updating of methodology pages to a subsequent change. Change-Id: I4b381e5210e69669f972326202fdcc5a2c9c923b Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-04-11feat(telemetry): T-Rex runtimePeter Mikus1-1/+36
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I2829b5931e5a63a33f524e1d14ffb20c27f4ce5a
2022-04-07Revert "feat(core): Add dpdk log level"Peter Mikus1-2/+2
This reverts commit 22bfd741614145d86751df2d61ff77af9db63627. Reason for revert: OMFG this patch is causing 90% perf degradation on AWS Change-Id: Id081f61ae849678b6b877fbf03309c4b1b0c6df6 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2022-04-05VppConfigGenerator: disable vpp cli pagerDave Wallace1-1/+2
- Robot framework logs contain 'show pci' output from VPP that is truncated due to the VPP cli pager. Add UNIX cli-no-pager configuration to the default startup config to fix this issue. Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I80f2cd045063245098f22eb638a9796c5031fecf
2022-03-21feat(core): Add dpdk log levelPeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I8af9bae81ec83a3195ae83febd978882ebc2474d
2022-03-09fix(lisp): repair a typoVratko Polak1-2/+2
+ Unify the order of interfaces, along 0-->1 path. Ticket: CSIT-1806 Fixes: 14a71b74b414df7616ccb5ada3d50ecb90d96bae Change-Id: Ia20d05b4e16daf60260fcbaea806406fa654a94e Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-02-16chore(api): use the non-deprecated nat init callVratko Polak1-2/+3
Change-Id: Ic1949c46aa339e66a47a0ebbc428499c5cf9305e Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-02-08fix(iperf3): Compare string literals using !=Vratko Polak1-3/+3
The previous code leads to warnings from Robot. Change-Id: I323f2abb277ae26d43ee5540a3b21e01a8dd9ef1 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-01-17fix(IPsec): fix policy testsJuraj Linkeš1-2/+12
Replace the hardcoded SPD inbound/outbound ranges with values derived from test inputs. Add the necessary routes now that the tunnel endpoints are not in the same subnet. Also add ip neighbor entry on DUT2 for the same reason. Also replace ipsec sa dump with show ipsec all in teardown of tests where both SAs and SPDs are configured to improve troubleshooting. Change-Id: I7d89a99fcf457a701c87bf6ac07364b62802677d Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2021-12-15UTI: Export resultsVratko Polak7-9/+72
+ Model version 1.0.0. - Only some result types are exported. + MRR, NDRPDR and SOAK. - Other result types to be added later. + In contrast, all test types are detected. + Convert custom classes to JSON-serializable equivalents. + Sort dict keys before converting to JSON. + Override the order for some known keys. + Export sets as sorted arrays. + Convert to info content from serialized raw content. + Also export outputs for suite setups and teardowns. + Info files for setup/teardown exist only temporarily. + The data is merged into suite.info.json file. + This simplifies presentation of total suite duration. + Define model via JSON schema: - Just test case, suite setup/teardown/suite to be added later. - Just info, raw to be added later. + Proper descriptions. + Json is generated from yaml. + This is a convenience for maintainers. + The officially used schema is the .json one. + TODOs written into a separate .txt file. + Validate exported instance against the schema. + Include format checking. + Update CSIT requirements for validation dependencies. + This needs python-dateutil==2.8.2, only a patch bump. + Compute bandwidth also for soak tests. + This unifies with NDRPDR to simplify schema definition. - PAL may need an update for parsing soak test message. + Include SSH log items, raw output only. + Generate all outputs in a single filesystem tree. + Move raw outputs into test_output_raw.tar.xz. + Rename existing tar with suites to generated_robot_files.tar.xz. Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-12-01feat(Performance): Add 2M/1G hugepagespmikus3-6/+8
+ Add ability to switch between hugepages. Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I84d8eae28ed414a32e5ba82e6c9ed10d7f0ef9cb
2021-11-08vpp_device: bind to vfio-pci before running testsJuraj Linkeš1-1/+2
In rare cases, binding the whole /dev/vfio folder will result in unusable VFs: notice dpdk EAL: Cannot open /dev/vfio/151: Device or resource busy [0], section 4.3.1. provides some clues as to what's going on and how to avoid the failure. Mounting /dev/vfio reset the file descriptors of all devices under /dev/vfio. Vfio-pci creates a device when an interface is bound to it. The rare failure then occurs when /dev/vfio is mounted while a process is using the file descriptors result in that process using invalid file descriptors (or file descriptors belonging to a different VF). Fix the issue by binding i40e and ice VFs to vfio-pci before containers are created and make sure that the VFs are not unbound later in testing. Only bind DUT VFs since the TG uses the kernel driver. [0]: https://connect.redhat.com/sites/default/files/2021-03/Cloud Native Network Function Requirements.pdf Ticket: CSIT-1794 Change-Id: I83db91b29d16669fb034b141ad247f6f796fdf64 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2021-10-25IPsec: add nth SPD entry outbound flow cache TCsJuraj Linkeš1-0/+7
Add 1, 10, 100 and 1000 SPD entries using IPv4 outbound flow cache optimization. Change-Id: I7abb65a82454c17ef754cb11386186610f0c27e8 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2021-10-08Add flow test suitesxinfeng zhao2-0/+56
The comms DDP package needs to be installed to support all flow protocols Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Change-Id: I6ab1bd8beb9edb8c9889a0ed9a999080ca3cef3d
2021-10-06docstring: Docstring warnings fixed.Viliam Luc1-1/+1
Change-Id: I8679c819b7c6ecd9e680bdc6edeafd2124def0b5 Signed-off-by: Viliam Luc <vluc@cisco.com>
2021-10-04FIX: Set correct teardown rate in NDRPDRVratko Polak1-1/+1
It should be in transactions per second, so target rate (min rate). Change-Id: I56a12d65466320fe5e4b2ee7d3b9946c9b3c414f Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-09-29MLRsearch: Increase timeout to 20 minutesVratko Polak1-2/+2
There are sporadic failures due to timeout. An earlier edit changed how the timeout is applied (now it is for the whole search, previously it was for final phase only) so failures are not unexpected. Increasing timeout value from 12 minutes to 20 minutes (both for NDRPDR and for NDR part of RECONF) as the frequency shows it will not prolong runs much. The timeout is increased both in Robot call site and in Python keyword default argument. Change-Id: I3f07125ba9eeb90ce5819b1450044441b8b3977a Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-09-17Performance keywords: Use matching runtime rateVratko Polak1-0/+1
Previously, runtime rate was not modified, so max rate was always used, even in ndrpdr with two stats trials. Now, each call to "Send traffic at specified rate" keyword sets the given rate as runtime rate. Change-Id: I3eb41eeadd6b64d23056c67c23a25f9565bb0424 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-09-10back-to-back tests: add TG testsViliam Luc1-0/+32
Change-Id: I9d028294deb1e31b6d185deb1c7523e0226a0ada Signed-off-by: Viliam Luc <vluc@cisco.com>
2021-08-24Telemetry: vpp devicepmikus2-30/+7
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I80efb278cf824dd8b1d38bb3ff3df43366e7295a
2021-08-24Fix DPDK nb_cores allocationpmikus2-38/+19
+ Migrate the call to main python util Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I8d00165071d4764121740c1c77819b8ffc65b3ba
2021-08-20Scale tests: Use exec commands and VAT executorVratko Polak1-1/+1
This is the same appoach as in ipsec tests. This is the fastest way to fix scale tests. - Some argument combinations are not supported, e.g. multipath=False. + Make multipath=True default. - Devicetest using local=True only works with PAPI. + But that was true already in rls2106, so no need to fix here. + Add two more formats to NetworkIncrement. + Check address is aligned, unless call side disables that. + Only one keyword for two devicetests disables the check. + Papi history condition to print first and last. + Minor docstring edits. Change-Id: I32d24e2f821b412b5841a50d2f87e1c86345b1aa Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-07-28IPsec: add nth matching SPD entry outbound TCsJuraj Linkeš1-5/+5
Add testcases with plain ipv4 forwarding with 1, 10, 100 and 1000 SPD entries on outbound traffic in both directions both directions. Only match the last SPD entry and process others before the matching entry. Add testcases only without flow cache optimization. Refactor the Python functions that add SPD entries: - Unify the args in functions that add one and multiple entries. - For multiple entries, add the ability to pass an object that will handle how values in each iteration (i.e. for each entry) are modified. Change-Id: I061922eec6acc75a4e115202c07e72d89bf1f4d3 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2021-07-20FIX: Mellanoxpmikus1-5/+6
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I6aba0123c1f79e5c2acbac07503f74fc0050f472
2021-06-17Core: Rework CPU allocationpmikus7-187/+47
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I6826add7b3032041632c3952c45a3c64409400b0
2021-06-16ip6lisp: Do not fail on string overhead valueVratko Polak1-1/+2
+ Set overhead in those suites to numeric values. + Change the library to tolerate string representations anyway. Change-Id: Ic6215840f7797801c994a38db5637999eb85a034 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-06-11FIX: GSO testspmikus1-3/+4
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I00372b5528752186a814e83af092e1b0123e4683
2021-06-10FIX: Pylint reducepmikus17-17/+17
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I909942dbb920df7f0fe15c0c92cda92c3cd8d8ad
2021-06-10Add gtpu sw performance test casesxinfeng zhao1-0/+62
Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Reviewed-by: Yulong Pei <yulong.pei@intel.com> Change-Id: Ibe4420529192559a3cdbb84e0a67859750a9e0d5
2021-06-10Dpdk in VM: Increase num_mbufsVratko Polak2-7/+9
Since switching to rxq ratio 1:1, the previous value was not enough for 4c vhost tests. + Set the 1:1 rxq ratio also for DPDK (outside vhost) tests. - Still not sure why those tests were not failing. + Set the 1:1 rxq ratio also for VPP container tests. Ticket: CSIT-1783 Change-Id: Id9d136a638eb212a8105a6288e7847f3c35fefb0 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-06-04Performance: Add AF_XDP testspmikus3-6/+67
- enabling for fortville, columbiaville - enabling experimental for mlx Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I1b7ceb54769f4a0089ac7309350499e60c5cca0a
2021-05-28Framework: Telemetry retakepmikus2-115/+62
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I2f019a083916aec9f7816266f6ad5b92dcc31fa0