aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot
AgeCommit message (Collapse)AuthorFilesLines
2020-01-29Ndrpdr: Remove unused latency measurementsVratko Polak1-23/+14
+ Improve keyword Documentation. + Reorder measurements, higher loads first. Change-Id: I1aeab1c8ada268422e9f421e075aff2ac9eec23f Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-29perf: QUIC transport hoststack test suiteDave Wallace1-3/+20
Change-Id: I73f4be7ea315c7a5dcce46e1bd3034bcb0a97ee2 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-28perf: hoststack iperf3 test tuningDave Wallace4-28/+76
Change-Id: I53425f57fe9ecef9cff2c94642cc7cb24537a961 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-28Check interval validity in reconf testsVratko Polak1-0/+2
Without this, if the search part returns min_rate, then reconf part reports few seconds with a small loss count. But min_rate means there is a bug or misconfiguration, so the tests should fail, instead of returning misleading results. Change-Id: I566608b40ae11d6aaa3fe022a32da97901e1999c Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-28Perform some cleanup discovered by GPL workVratko Polak1-12/+0
+ Remove one __init__.robot file with no function. + Remove executable flags from traffic profiles. + Improve indentation in some traffic profiles. Change-Id: I099e63069003fba788976a9470a909ccff51a88c Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-27Increase min_rate for NDRPDR testsVratko Polak1-2/+4
Latency measurements break for loads less than 9 kpps per direction. Setting min_rate to 90 kpps prevents 10% latency trial from breaking. 90 kpps should be enough for any type of test. When some test finds a lower value, this patch will make the fail with a message saying the loss rate at minimal load is too high (instead of a cryptic error from 10% measurement without this patch). Change-Id: I97b8428aa5044da23343b86447857b3a101f6baa Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-27Restore PDR stat run for NDRPDRVratko Polak1-2/+6
Before [0], the NDRPDR test ended with two measurements that perform runtime stats collection, each 2 seconds long. [0] then decreased that duration to one second, and removed the PDR measurement. This change re-adds the PDR measurement, while keeping the duration at 1 second. [0] https://gerrit.fd.io/r/c/csit/+/24427 Change-Id: I76828f11863a9e278cdd0cf60f2792d5b614ce43 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-23FIX: Various issuesPeter Mikus2-80/+4
- TrafficProfile - IPsec jumbo - TDLK leftover - Trex scapy server Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0601ea75ed6e7318c46124fb02e9c61066089791
2020-01-21Ndrpdr: Add latency trials for smaller loadsVratko Polak1-11/+78
Change-Id: I46bd4b58bdd02a3deed38c158214e1e9977dde05 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-17perf: add TCP Iperf3+LDPRELOAD test suiteDave Wallace2-33/+144
Change-Id: Icff49fb31cce342a2a4ae799e844ec91f9e5e366 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-16FIX: L2XC with vhost and VLANJan Gelety1-0/+3
Change-Id: Ia87abadba431099eeacd8b50ce5d116597a1f393 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-10Trending: new daily setJan Gelety1-8/+19
New daily sets are prepared based on information in https://gerrit.fd.io/r/c/csit/+/24073/1/docs/job_specs/perf_tests_job_specs.md and previous test set definitions in docs/job_specs/test_select_list_[2n|3n]_[clx|skx|hsw|tsh|dnv].md files. - mrr-daily-2n-clx: 510 TCs (incl. nfv_density), expected exec. time 8:50h - mrr-daily-2n-skx: 525 TCs (incl. nfv_density), expected exec. time 7:55h - mrr-daily-3n-skx: 393 TCs (incl. nfv_density), expected exec. time 11:00h - mrr-daily-3n-hsw: 177 TCs (incl. nfv_density), expected exec. time 7:10h - mrr-daily-3n-tsh: 204 TCs, expected exec. time 21:00h - mrr-daily-2n-dnv: 84 TCs, expected exec. time 2:25h - mrr-daily-3n-dnv: 144 TCs, expected exec. time 6:35h + add some missing test suites + add trex-sl-2n-ethip4udp-1000u15p.py T-Rex traffic profile + correction of TS and TC names and tags in directory tests/vpp/perf/nfv_density/chain_ipsec Change-Id: Icfc86e9af97ed8dd8ccd2a34355c99aad69a28c0 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-10Autogen: Generate also NIC drivers.Vratko Polak1-11/+19
+ Disallowed -avf- (or -rdma-) as "template" suites. + GBP suite switched to DPDK driver in repo. + Each NIC has its own list of supported drivers, in Constants. + Updated tag expressions for daily jobs: + Feature, ipsec, memif, scale, srv6, tunnels, vhost and vts are tested only with vfio-pci. + Other (base, dot1q, dot1ad) tested with all drivers. + Setup actions currently depend on driver, generated. - The performance_rdma action is trivial for now. - Several tests fail, to be fixed later, e.g. by performance_rdma. + Reconf tests are also supported. + Added DRV_VFIO_PCI tags missing, mainly in density tests. - Vhost suites (density, reconf) are failing, but suites look good. - TCP suites do not support NIC drivers yet. - DPDK obviously not supported. + Use Python 3 in regenerate scripts. + Fix typos binded => bound. + File open modes set either u"rt" or u"wt" everywhere. + Remove a trailing space in an environment variable name. Change-Id: I290470675dc5c9e88b2eaa5ab6285ecd9ed7827a Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-15Hoststack perf infrastructure refactoringDave Wallace6-17/+496
- DUT only topology (hoststack test apps are co-located with vpp) - Make vpp app specific keywords generic where applicable - Add IP4 Prefix to topology file - Support running wrk in linux namespace - Refactor namespace cleanup - Remove redundant namespace creation code - Refactor test/keyword dirs: tcp -> hoststack - Add hoststack utility keywords - Refactor wrk suite setup/teardown - Update tests with recent perf infra changes Change-Id: Ia1cf07978d579393eef94923819a87c8c1f36f34 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-09FIX: Container testPeter Mikus1-1/+0
+ We should check if container is UP + We should adjust privileges. Change-Id: I3f604b7b9740943b4fd3b0574c18d2729e88b816 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-01-09DMM: RemoveTibor Frank1-65/+0
Change-Id: Ibbfbed79e473c804390802ae1ecd737b50c06aa3 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-01-07Vhost: Add GSO optionPeter Mikus2-6/+17
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Iba9af71c293645b480203af72fcf940cbe9ccb3a
2019-12-20VTS: Unify the testsPeter Mikus1-72/+0
- Converting to 2n as they were always 2n (with l2xc on 3rd node) - Removing KW and converting to layered approach Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ie349c50f72eb362815e7c5ede076d421ab386e76
2019-12-20Add VPP NSIM Plugin KeywordsDave Wallace1-0/+62
Change-Id: I1e080a50a68ded8eebc5e6093c55d06281726684 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-12-18python3: fix RF 'only continuation marker' warningDave Wallace1-5/+5
Change-Id: I250ffd196045bc623d5d38f30f16a08bcaea137c Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-12-17Telemetry: Add more operational dataPeter Mikus2-108/+33
+ Add both NDR and PDR telemetry capture + Speedup sockets + Adjust privileges Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ia6fd5d405e6fb410651d8b705c921653753aea10
2019-12-13FIX: eth interfaces need numa_node information in topology fileJan Gelety1-12/+4
Change-Id: I9a3c6947da8ab5033e29b39833c97bcdafdc3ca9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-16Remove supervisord dependency from containersPeter Mikus1-2/+1
+ Startup config stream-line Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ia12e48202a5fc88461d3d24ac9b9599e7175b1d8
2019-12-13Python3: refactor ':FOR' statementsDave Wallace1-1/+2
Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I76835e3d3acf6955e328f30427f9dd0098947e41
2019-12-11Introduce VPP-IPsec container tests.Ludovit Mikula5-42/+184
Change-Id: Ie64d662e81879bd52785e0188450d998bf056bda Signed-off-by: Ludovit Mikula <ludovit.mikula@pantheon.tech>
2019-12-03Remove: VIRLPeter Mikus1-28/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I2b1672412e19f15cb46246fffa931351cd3d518b
2019-11-28FIX: aclJan Gelety1-3/+3
Change-Id: Ieb1569a1dce9fd7c00041ba00f72c67142a581ed Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-28Python3: resources and librariesJan Gelety36-1856/+1314
Change-Id: I1392c06b1d64f62b141d24c0d42a8e36913b15e2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-21NSH_SFC: RemoveTibor Frank1-81/+0
Change-Id: Ib7f9dff7bede4f8ec3148f234109132be920de82 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2019-11-19HONEYCOMB: RemovePeter Mikus29-6855/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I60d80564050cb786c7b1e0a08f31f8292f82197e
2019-11-13LIGATO: RemovePeter Mikus3-93/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0c9b8b07ed3a113e24a38faaddd5ed06fae67768
2019-11-13Add vpp loadbalancer l3dsr/nat4 mode test suiteshaiyanx1.zhang1-13/+109
Change-Id: I03603108fe7128fb4bde4abddb6a0959dd5b5c73 Signed-off-by: haiyanx1.zhang <haiyanx1.zhang@intel.com>
2019-11-11FIX: do not use ping in vpp-device testsJan Gelety3-90/+85
Change-Id: Ic11f863e7ffad70d45e6e8dc39c70efc62cecc96 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-10-30Framework: Add show log as tear down actionPeter Mikus1-0/+1
+ We can see what is happening when debug level is increased + Big caveat of VPP is that it does not allow to configure default global level. Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I5ea1a618161cdb3170cab022d7451e52271f83c0
2019-10-25FIX: Limitation for NICsPeter Mikus1-2/+4
Change-Id: I8e0a14f59e20d61096e069bb80a0e2c00977ac5c Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-10-25Introduce pre-initialize driver layerPeter Mikus4-264/+188
+ This will help unify driver base differences between TCs. + Decrease amount of logic needed for suite generator. Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I1e84ba361dc1e829f0612c58a61096e2633ce0c5
2019-10-19Tests: Introduce driver layerPeter Mikus1-10/+32
+ Small bugfixes in various suites Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I051af0a6da7340fe3f617400370c3846ca5241ab
2019-10-16VPPD: GBP test casePeter Mikus1-3/+4
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I020cdb6ced70a9c22773dfbe1662aecd2b744d2d
2019-10-09Add vpp loadbalancer maglev mode test suitehaiyanX1.zhang2-0/+65
Change-Id: I61555ba566efef0a2151db9a30bf7f5d9ccad1df Signed-off-by: haiyanx1.zhang <haiyanx1.zhang@intel.com>
2019-09-26CSIT-1471: Policer - VAT API to PAPIpremkumarv4-52/+80
Change-Id: I6c7eff8d9cbb196ccb684a45579ac59942351cfe Signed-off-by: premkumarv <preym17@gmail.com>
2019-09-24Add support for HDRhistogramVratko Polak1-1/+1
+ Enable hdrh in trex server. + Append hdrh coded output after min/avg/max/. + Read (not show nor decode) hdrh value in PAL. + Also, remove old ndrpdrdisc code. Change-Id: I99d99f10386a621772b5419ca1f36080fa15aca7 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-24Reconf tests: Fix async measurementsVratko Polak1-3/+8
TRex does not zero the server counters. It copies the values to use as reference, and subtracts them when asked for results. But the reference is stored in the client (not the server). And CSIT uses different scripts to start and stop async traffic, which means different clients. This patch introduces a workaround. Async start will return xstats objects to use as reference, and async stop will use the objects to compute the correct results. The xstats objects are stored in TrafficGenerator instance. Sync measurement does not export the counters, to shorten logs. Other improvements: + Make stop_traffic_on_tg return measurement results directly. + Rename --async to --async_start as "async" is reserved in Python 3.7 + Minor pylint, docstring and typo fixes. Change-Id: I5fc56a0763afb7d62cfa7c0651f96b6867de3e15 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-19LISP: 3-node (func) to 2-node (dev) converstionNidhy4-1/+300
Change-Id: I0fbe4d9ef1b30a1ac1382ea43b5b52aeb8a7d39f Signed-off-by: Nidhy <nidhyanandhan.a@hcl.com>
2019-09-12Support unidirection in all searchesVratko Polak1-61/+84
+ Rename bool unidirection to int traffic_directions. + Rename "untagged" to "initial" for bandwidth calculation. + Fix latency measurement for unidirectional traffic. + Remove duplicate colon in soak test message. + Edit PAL to accept both forms. + Fix minor documentation issues. Change-Id: I6c76f2dc090ae493f2fbd7e9ccd45229d2306dea Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-10Refactor getting telemetryPeter Mikus6-81/+115
+ Ability to get stats from CNF via SocketPAPI - Remove obsolete functions Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I4d1b32a7279244592be96644e4f8a530c4f29a15
2019-09-06Read environment variables in Constants.pyVratko Polak5-63/+12
Instead of using EnsureGlobalVariable, which is clunky to use from Python. As a consequence, all caps variables from Constants.py are used directly and tests/__init__.robot and robot_enhancements.robot are deleted. + Rename the CRC global kill switch based on ci-man review. Change-Id: I10723792475bc83352bf8c3b7f7946ecf885a194 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-05FIX: Use vhost macs when testpmd_mac forwarding used in VMJan Gelety2-8/+58
Change-Id: I291c3dc852e7c1bde6fb6102a9168cfec9992682 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-08-27Delete unused SR keywordsVratko Polak1-58/+0
Trim both Python (L1) and Robot (L2) keywords, as no test performs SR related deletions. Change-Id: Ic76d870c3f58e927ea222b350e149d0770985072 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-08-23Introduce reconfig suites, for dot1q+ip4+vxlanVratko Polak4-60/+223
Ticket: CSIT-1551 + Config copied from other suites. + Layer keywords prepared for additional chains. + Renamed "create" argument to "vlan_per_chain". + TrafficGenerator result extraction improvements. + Also unified TG type and subtype checking. + Throughput estimated as PDR for zero PLR. + Actual Qemu start is skipped. + Parse xstats instead of stats in trex stop scipt. - Because stats are unconditionally cleared on each script connect. + Remember start values to get better ReceiveMeasurementResult at stop. + Expose loss count and time as appropriate test message. + Autogen support. + Multiple chain amounts and core densities. - Only 1 additional chain. + Proper TAGs. Documented. + Clarified min framesize tags are for TG-DUT. + Added a TODO concerning overhead values. + Added 118B tag. + Robot performance keywords improvements. + Added missing return value documentation. + Frame size as required test variable, not an explicit argument. + MLRsearch keyword to store result (not into test message). - TODO: De-duplicate: main keyword to measure, minor to set test message. - But implementation uses the fact default arguments are different. + Keywords for starting and stopping background traffic. - No checks whether start-stop pair is complete. + Multiple minor indentation and logging improvements. - Missing / TODOs: - Support also vlan_per_chain=True? - Support dot1q without vxlan? - Support also deletion? - How to package config steps so -reconf can be generated from -ndrpdr? Change-Id: If0270697f9ffa837c34ef802917220a53f156703 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-08-23Improve PLRsearch yet againVratko Polak1-1/+1
Logic improvements for rls1908 and post-pylint fixes: + Reduce search time to 30 minutes. + Use average instead alternating loads. + Rework log/exp avg/stdev from two estimates. + Introduce and use pessimistic variance of dual trackers. + Introduce safe_exp to use when None does not skip code. + Use dot relative imports (instead of disabling pylint). + Complete docstrings for simpler functions. + Append docstrings to named tuples. + Somewhat unify docstrings related to the same arguments. + Slightly improve intentation. + State named tuples as rtype where used. + Add returns and rtype to __repr__ where missing. + Return what docstring say (or update them). + Explicit copying for vector and matrix values. Change-Id: I884c68b4839c5df5e8bef95e463666599603a0ff Signed-off-by: Vratko Polak <vrpolak@cisco.com>