aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot/shared
AgeCommit message (Collapse)AuthorFilesLines
2021-12-15UTI: Export resultsVratko Polak5-2/+28
+ Model version 1.0.0. - Only some result types are exported. + MRR, NDRPDR and SOAK. - Other result types to be added later. + In contrast, all test types are detected. + Convert custom classes to JSON-serializable equivalents. + Sort dict keys before converting to JSON. + Override the order for some known keys. + Export sets as sorted arrays. + Convert to info content from serialized raw content. + Also export outputs for suite setups and teardowns. + Info files for setup/teardown exist only temporarily. + The data is merged into suite.info.json file. + This simplifies presentation of total suite duration. + Define model via JSON schema: - Just test case, suite setup/teardown/suite to be added later. - Just info, raw to be added later. + Proper descriptions. + Json is generated from yaml. + This is a convenience for maintainers. + The officially used schema is the .json one. + TODOs written into a separate .txt file. + Validate exported instance against the schema. + Include format checking. + Update CSIT requirements for validation dependencies. + This needs python-dateutil==2.8.2, only a patch bump. + Compute bandwidth also for soak tests. + This unifies with NDRPDR to simplify schema definition. - PAL may need an update for parsing soak test message. + Include SSH log items, raw output only. + Generate all outputs in a single filesystem tree. + Move raw outputs into test_output_raw.tar.xz. + Rename existing tar with suites to generated_robot_files.tar.xz. Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-12-01feat(Performance): Add 2M/1G hugepagespmikus3-6/+8
+ Add ability to switch between hugepages. Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I84d8eae28ed414a32e5ba82e6c9ed10d7f0ef9cb
2021-11-08vpp_device: bind to vfio-pci before running testsJuraj Linkeš1-1/+2
In rare cases, binding the whole /dev/vfio folder will result in unusable VFs: notice dpdk EAL: Cannot open /dev/vfio/151: Device or resource busy [0], section 4.3.1. provides some clues as to what's going on and how to avoid the failure. Mounting /dev/vfio reset the file descriptors of all devices under /dev/vfio. Vfio-pci creates a device when an interface is bound to it. The rare failure then occurs when /dev/vfio is mounted while a process is using the file descriptors result in that process using invalid file descriptors (or file descriptors belonging to a different VF). Fix the issue by binding i40e and ice VFs to vfio-pci before containers are created and make sure that the VFs are not unbound later in testing. Only bind DUT VFs since the TG uses the kernel driver. [0]: https://connect.redhat.com/sites/default/files/2021-03/Cloud Native Network Function Requirements.pdf Ticket: CSIT-1794 Change-Id: I83db91b29d16669fb034b141ad247f6f796fdf64 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2021-10-08Add flow test suitesxinfeng zhao2-0/+56
The comms DDP package needs to be installed to support all flow protocols Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Change-Id: I6ab1bd8beb9edb8c9889a0ed9a999080ca3cef3d
2021-09-10back-to-back tests: add TG testsViliam Luc1-0/+32
Change-Id: I9d028294deb1e31b6d185deb1c7523e0226a0ada Signed-off-by: Viliam Luc <vluc@cisco.com>
2021-08-24Telemetry: vpp devicepmikus2-30/+7
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I80efb278cf824dd8b1d38bb3ff3df43366e7295a
2021-08-24Fix DPDK nb_cores allocationpmikus1-1/+1
+ Migrate the call to main python util Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I8d00165071d4764121740c1c77819b8ffc65b3ba
2021-07-20FIX: Mellanoxpmikus1-5/+6
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I6aba0123c1f79e5c2acbac07503f74fc0050f472
2021-06-17Core: Rework CPU allocationpmikus3-138/+24
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I6826add7b3032041632c3952c45a3c64409400b0
2021-06-10FIX: Pylint reducepmikus3-3/+3
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I909942dbb920df7f0fe15c0c92cda92c3cd8d8ad
2021-06-10Dpdk in VM: Increase num_mbufsVratko Polak1-3/+4
Since switching to rxq ratio 1:1, the previous value was not enough for 4c vhost tests. + Set the 1:1 rxq ratio also for DPDK (outside vhost) tests. - Still not sure why those tests were not failing. + Set the 1:1 rxq ratio also for VPP container tests. Ticket: CSIT-1783 Change-Id: Id9d136a638eb212a8105a6288e7847f3c35fefb0 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-06-04Performance: Add AF_XDP testspmikus3-6/+67
- enabling for fortville, columbiaville - enabling experimental for mlx Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I1b7ceb54769f4a0089ac7309350499e60c5cca0a
2021-05-26Add dpdk max-simd-bitwidth configuration.xinfeng zhao1-0/+1
Signed-off-by: xinfeng zhao <xinfengx.zhao@intel.com> Reviewed-by: Yulong Pei <yulong.pei@intel.com> Change-Id: Ie7b9912eacdb60689012936580d3d774f7e470ae
2021-05-26rxq_ratio default value changed to 1viliam.luc1-2/+2
Change-Id: I0c973ceffc13f4f3c5f959d56cb0890b87091a82 Signed-off-by: Viliam Luc <viliam.luc@pantheon.tech>
2021-05-20perf: add TCP Nginx+LDPRELOAD suitesxizhanx3-2/+59
1. Suite steup add download nginx 2. Add nginx-1.14.2/1.15.0 ldp test suite 3. Add NginxUtils,NginxConfigGenerator method 4. Taskset the PID of nginx to the unused cores in VPP and these cores are under NIC's NUMA ID 5. cleanup add Kill Processes - nohup Signed-off-by: xizhanx <xix.zhang@intel.com> Change-Id: Idbf0e4ec3bf63e88281a8e3e34f52e00a6801c85 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-31Device: Add stats test with AVFVratko Polak1-0/+28
+ Extract the stats keyword into a library. Change-Id: Icfaaa32c2a97d479ba0339acf34b865006949a89 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-02-19Add test suites for crypto sw scheduler engineYulong Pei1-0/+23
This patch is to add test suites for vpp plugin crypto_sw_scheduler, IPsec sync mode is to do crypto and packet forward work in same worker cores, crypto_sw_scheduler can schedule crypto work to other async crypto cores to improve whole crypto processing capability. This test suites configure fixed 1 rx queues per port, then measure IPsec performance with 1, 2, 3 crypto cores. This patchset include 1, 2, 4, 8 ipsec tunnels test cases. +Vratko help to change to count total physical cores instead of previous only count crypto cores in test cases. Change-Id: I0e67182e3d13273890a23703d838101900e25126 Signed-off-by: Yulong Pei <yulong.pei@intel.com> Signed-off-by: Vratko Polak <vrpolak@cisco.com> Signed-off-by: pmikus <pmikus@cisco.com>
2021-02-19Framework: GSO testpmikus1-0/+69
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I874568aebbe50c38e634438eb1201ca737bd18ad
2021-01-04API: deprecated COP APIsTomas Alexy1-1/+1
Signed-off-by: Tomas Alexy <tomas.alexy@pantheon.tech> Change-Id: Ifb97088a26f3c2ab2ecee28ed43e6e957a844c64
2021-01-04GSO: TAP/VHOST use casepmikus4-27/+142
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ibafaaca4438587284d1e255c764f0701002941e8
2020-12-18API: deprecated bond APIsJan Gelety1-2/+2
Jira: CSIT-1597 Change-Id: I77b746584851331d46eca1caafd4468d18f1e78d Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-12-16API: deprecated IPSEC APIsJan Gelety1-0/+9
Jira: CSIT-1597 + add ipsec_sa tear down action Change-Id: I4d1e6f26c14e61e8ddc6f29fbf5528a034c84eb0 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-12-03vpp-device: GENEVE tunnel test, l3 modeJan Gelety2-87/+140
Jira: CSIT-1769 Change-Id: I98510cd8e627d7347f77d0032b7bac28f2e36c61 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-12-08Revert "vpp-device: GENEVE tunnel test, l3 mode"Peter Mikus2-140/+87
This reverts commit a9f54ca5080aeef17686f300a6807bf9b46b7c90. Reason for revert: DO NOT MERGE BROKEN TESTS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! https://logs.fd.io/production/vex-yul-rot-jenkins-1/csit-vpp-device-master-ubuntu1804-1n-skx/9918/archives/log.html.gz Change-Id: I2ce2970a43e5dd7487eeb54d1ccbb149e06cf8fa Signed-off-by: pmikus <pmikus@cisco.com>
2020-12-03vpp-device: GENEVE tunnel test, l3 modeJan Gelety2-87/+140
Jira: CSIT-1769 Change-Id: I0ab906966f760b809b8431530c010a8f0835d62c Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-12-02API: deprecated NAT APIsJan Gelety1-4/+3
Jira: CSIT-1597 Change-Id: Ib48d20ba087bc9828970c3acc081ac6e4447c20c Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-11-25Add rxq_thr optionpmikus1-1/+2
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I46750f8571d86dc13df95252e8894c4b988efd5c
2020-10-29Support existing test types with ASTFVratko Polak2-6/+35
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-10-29tests: Add GSO enabled testsMartin Balaz2-5/+6
Change-Id: I409b060f64ae7c6787448ae519fd76d8384e9ffb Signed-off-by: Martin Balaz <martin.balaz@pantheon.tech>
2020-10-19Tests: Remove GBPpmikus1-1/+0
- Specs - Tests - Library - TODO: PAL Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I8890940d56c7a9924d4a5f9365b6bb881610db14
2020-09-30Framework: main heap size value multiplicationJan Gelety1-1/+1
- default value for main heap size set to 2G - allow allocation of n x 2G in tests that need it Change-Id: I02dafe6bc61649d2823486a3e983f66f5a213a88 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-24test: nat44det - add session number checkJan Gelety1-1/+1
- some tests need to reduce rate for ramp-up phase - some tests need to extend trail duration in ramp-up phase - removed 2n1l-10ge2p1x710-ethip4udp-nat44det-h1-p63-s63 suite as nat out ports are randomly selected from available port range so T-Rex stateless is not able to provide required out2in traffic Change-Id: I1145496610d202f81d911e68aa819844d7600918 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-23FIX: Race conditionpmikus1-0/+2
avf tests works, but vfio-pci tests are following, the interfaces must be in down state (ideally unbind, as vpp cannot pick them). Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I77af85ec4239059a5455ef68683ca129548bd7bd
2020-09-22Framework: Heapsize configurationpmikus1-3/+4
- https://gerrit.fd.io/r/c/vpp/+/28829 - https://gerrit.fd.io/r/c/vpp/+/28896 - ip6 to be added later Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I7b03b8ace5c5c7582d4a2f66bfd788a3ec80a916
2020-09-07VPP-DEV API COV: Add NAT44-ED testsJan Gelety1-0/+53
Jira: CSIT-1755 Change-Id: I34baa22a49f44da3fa80d91fa2f4132c982fe610 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-03Framework: Code aligmentspmikus2-5/+17
+ Unifying code structures - To easily plug another DUT + New PCI PassThrough templates + Improved perf stat on cores allocated in test. Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I325f17b977314f93cb91818feddfddf3e607eb8a
2020-08-08Fix: NAT44 deterministic moved to separate det44 pluginJan Gelety1-4/+5
- align CSIT code with VPP code changes for NAT44 deterministic (DET44) feature - align test names according to snat44ed tests - remove obsolete 3-node nat tests - remove 2n1l-10ge2p1x710-ethip4udp-snat44det-h1048576-p63-s66060288 tests (not enough memory for such high number of sessions) Change-Id: I9a22b99b4cfa56d18e9c7ef9c58296e202567d42 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-07Perf: NAT44 endpoint-dependent mode - udp, part IJan Gelety2-36/+76
- continuation of https://gerrit.fd.io/r/c/csit/+/26898 as there was reached limit of changes (1000) Jira: CSIT-1711 - udp synthetic profiles w/o data packets - udp cps perf tests, phase I (no special "search cps" KW) Part I means that we are using MRR tests to collect traffic data until there is ready new CPS test type with corresponding algorithm. Change-Id: I0d30feb9ecf1d0bff937152656f8eb422f831378 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-17Framework: Profiles loadpmikus2-1/+64
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Iab84aff31a23bb9d8e1165f5314004803fd8a501
2020-07-23T-Rex: Add advanced stateful modeJan Gelety1-1/+1
- provide base routines to run T-Rex in advanced stateful mode Change-Id: Ib0dc5f2919c370753335f6446860683dc4b12d93 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-31Framework: Perf stat capturingPeter Mikus1-2/+3
Change-Id: I3bbe1fe0073ddeead5219993675f24955e8c3dfd Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-07-16Remove remains of WRK testsTibor Frank2-49/+0
Change-Id: If61783fb717757c6189f06924412bd079e15a08f Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-06-30make graph node variant can be configured from gerrit triggerYulong Pei1-0/+1
for example, below will trigger VPP with default code path test in CSIT, "csit-2n-clx-perftest mrrAND64bANDnic_intel-xxv710ANDeth-l2bdbasemaclrnNOTdrv_avf" and below will trigger VPP with avx512 code path test in CSIT, "csit-2n-clx-perftest icl mrrAND64bANDnic_intel-xxv710ANDeth-l2bdbasemaclrnNOTdrv_avf" Change-Id: Id3abf932203ce00f34f8643c5bc4d5fc4d3e6efb Signed-off-by: Yulong Pei <yulong.pei@intel.com>
2020-05-07perf: refactor 'setup suite topology interfaces'Dave Wallace1-58/+33
- and 'setup suite topology interfaces no tg' to use a common keyword to create suite variables using the required topology information. Change-Id: I46894948bc86eb7ce72d036e5b84f09c5c1385db Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-05-04FIX: IPsecvagrant1-4/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Iacd7aa49d9832233033fe18fc228ccd45c12b4db
2020-05-04VPP-DEV API Coverages: SRv6Jan Gelety1-0/+61
Jira: CSIT-1698 Change-Id: I6d9154284990df8877850e4014716510016e485b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-29FIX: dot1qip4vxlan-l2bdPeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ia4415090263357d16cfd5f5bc93931ded5688557
2020-04-27Performance: MLX best practicePeter Mikus1-2/+11
- Part II Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I869a3d7e051dcbae3a230d4646219e70e5de2e87
2020-04-27FIX: NF_density IPv4 profilePeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I471c0cb98c24416d893248b55b376b03a4b8830a
2020-04-23FIX: Mellanox jumbo framesPeter Mikus1-0/+20
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I84b3c07e22a313e96ac59fc7818960c502507651