aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot/shared
AgeCommit message (Collapse)AuthorFilesLines
2020-10-26Support existing test types with ASTFVratko Polak2-6/+35
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com> Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-23FIX: Race conditionpmikus1-0/+2
avf tests works, but vfio-pci tests are following, the interfaces must be in down state (ideally unbind, as vpp cannot pick them). Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I77af85ec4239059a5455ef68683ca129548bd7bd
2020-09-07VPP-DEV API COV: Add NAT44-ED testsJan Gelety1-0/+53
Jira: CSIT-1755 Change-Id: I34baa22a49f44da3fa80d91fa2f4132c982fe610 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-03Framework: Code aligmentspmikus2-5/+17
+ Unifying code structures - To easily plug another DUT + New PCI PassThrough templates + Improved perf stat on cores allocated in test. Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I325f17b977314f93cb91818feddfddf3e607eb8a
2020-08-08Fix: NAT44 deterministic moved to separate det44 pluginJan Gelety1-4/+5
- align CSIT code with VPP code changes for NAT44 deterministic (DET44) feature - align test names according to snat44ed tests - remove obsolete 3-node nat tests - remove 2n1l-10ge2p1x710-ethip4udp-snat44det-h1048576-p63-s66060288 tests (not enough memory for such high number of sessions) Change-Id: I9a22b99b4cfa56d18e9c7ef9c58296e202567d42 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-07Perf: NAT44 endpoint-dependent mode - udp, part IJan Gelety2-36/+76
- continuation of https://gerrit.fd.io/r/c/csit/+/26898 as there was reached limit of changes (1000) Jira: CSIT-1711 - udp synthetic profiles w/o data packets - udp cps perf tests, phase I (no special "search cps" KW) Part I means that we are using MRR tests to collect traffic data until there is ready new CPS test type with corresponding algorithm. Change-Id: I0d30feb9ecf1d0bff937152656f8eb422f831378 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-17Framework: Profiles loadpmikus2-1/+64
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Iab84aff31a23bb9d8e1165f5314004803fd8a501
2020-07-23T-Rex: Add advanced stateful modeJan Gelety1-1/+1
- provide base routines to run T-Rex in advanced stateful mode Change-Id: Ib0dc5f2919c370753335f6446860683dc4b12d93 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-31Framework: Perf stat capturingPeter Mikus1-2/+3
Change-Id: I3bbe1fe0073ddeead5219993675f24955e8c3dfd Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-07-16Remove remains of WRK testsTibor Frank2-49/+0
Change-Id: If61783fb717757c6189f06924412bd079e15a08f Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-06-30make graph node variant can be configured from gerrit triggerYulong Pei1-0/+1
for example, below will trigger VPP with default code path test in CSIT, "csit-2n-clx-perftest mrrAND64bANDnic_intel-xxv710ANDeth-l2bdbasemaclrnNOTdrv_avf" and below will trigger VPP with avx512 code path test in CSIT, "csit-2n-clx-perftest icl mrrAND64bANDnic_intel-xxv710ANDeth-l2bdbasemaclrnNOTdrv_avf" Change-Id: Id3abf932203ce00f34f8643c5bc4d5fc4d3e6efb Signed-off-by: Yulong Pei <yulong.pei@intel.com>
2020-05-07perf: refactor 'setup suite topology interfaces'Dave Wallace1-58/+33
- and 'setup suite topology interfaces no tg' to use a common keyword to create suite variables using the required topology information. Change-Id: I46894948bc86eb7ce72d036e5b84f09c5c1385db Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-05-04FIX: IPsecvagrant1-4/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Iacd7aa49d9832233033fe18fc228ccd45c12b4db
2020-05-04VPP-DEV API Coverages: SRv6Jan Gelety1-0/+61
Jira: CSIT-1698 Change-Id: I6d9154284990df8877850e4014716510016e485b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-29FIX: dot1qip4vxlan-l2bdPeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ia4415090263357d16cfd5f5bc93931ded5688557
2020-04-27Performance: MLX best practicePeter Mikus1-2/+11
- Part II Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I869a3d7e051dcbae3a230d4646219e70e5de2e87
2020-04-27FIX: NF_density IPv4 profilePeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I471c0cb98c24416d893248b55b376b03a4b8830a
2020-04-23FIX: Mellanox jumbo framesPeter Mikus1-0/+20
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I84b3c07e22a313e96ac59fc7818960c502507651
2020-04-23Performance: DPDK refactorPeter Mikus2-3/+3
+ Rework BASH scripts (more code to python). + Move BASH into libraries. + Allows RDMA usage. + Fix 9000B tests. + Rename confusing l2fwd -> testpmd. + Fix suite setup. + Fix PCI whitelist to not accidentally pickup wrong interface. + Fix deprecated DPDK cli arguments. - MLX5 jumbo are disabled on NIC (i will increase separately). https://jenkins.fd.io/job/csit-dpdk-perf-verify-master-2n-clx/6/console (l3fwd still broken) - MLX5 IMIX seems to be some TRex issue with IMIX for mlx5 (i will handle separately) Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I31d1b67305fa247cb5e1f57e739d3ef30dc1a04b
2020-04-22FIX: NF_density MAC addressesPeter Mikus2-12/+12
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0e773060c344fda7b06937a81a53863a96a4373a
2020-04-20FIX: NF_density testsPeter Mikus1-5/+7
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic305d135b14697332436d6afd76173eff0c3874b
2020-04-17FIX: Proper NUMA computePeter Mikus1-1/+1
+ FIX for Vratko's FIX Change-Id: Ia6ff137120d84bdd2b9e02e3e935c62d1c732b7b Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-04-17Fix numa detectionVratko Polak1-1/+1
On 3n-hsw, xl-710 is numa 1. The old code returned 0, causing performance regression. Fixes: 14a71b74b414df7616ccb5ada3d50ecb90d96bae Change-Id: I81d3f711b6a37438f6411e29baafae9dec1c00a4 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-04-17VPP-DEV API Coverages: IPSEC interfaceJan Gelety1-3/+58
+ some pylint fixies Change-Id: I650ce16282ae953a1a5ee96e810702c01f71efd6 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-08FIX: Add show log output when test failed.Peter Mikus1-1/+3
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I6bdce8156ba214be210c195a9f3effd718301a86
2020-04-06Improve pf layerPeter Mikus8-955/+513
+ Merge single/double link + Introduce _pf{n}[0] variables so we can access physical function same way as virtual function + Cleanup code by moving complex logic to python + Prepare code for multiple vf functions Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic2e74a38bfa146441357de8f0916aeb638941c49
2020-03-18FIX: qsz typoPeter Mikus1-2/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I56e87868de0bb45e863f1d806e2efc3f5ec5df96
2020-03-10Make RXQs/TXQs configurablePeter Mikus1-9/+13
Change-Id: Ib30bc4697fcba93a6723ee492a59a0523425f623 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-02-25FIX: check if t-rex is running at test setup of all perf testsJan Gelety1-1/+8
Change-Id: I9af632035a1415666b2470c62a41d1b6acbf33c8 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-02-25FIX: avoid t-rex suite tear down action to fail successfull testsJan Gelety1-1/+1
- try to kill t-rex only if it is running - do not fail the whole test suite when t-rex test suite tear down action has failed Change-Id: I3748a08a3da4dd3617cb54e8fdd344d49566fc7d Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-28perf: hoststack iperf3 test tuningDave Wallace3-3/+16
Change-Id: I53425f57fe9ecef9cff2c94642cc7cb24537a961 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-23FIX: Various issuesPeter Mikus1-0/+4
- TrafficProfile - IPsec jumbo - TDLK leftover - Trex scapy server Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0601ea75ed6e7318c46124fb02e9c61066089791
2020-01-17perf: add TCP Iperf3+LDPRELOAD test suiteDave Wallace1-1/+1
Change-Id: Icff49fb31cce342a2a4ae799e844ec91f9e5e366 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-10Autogen: Generate also NIC drivers.Vratko Polak1-11/+19
+ Disallowed -avf- (or -rdma-) as "template" suites. + GBP suite switched to DPDK driver in repo. + Each NIC has its own list of supported drivers, in Constants. + Updated tag expressions for daily jobs: + Feature, ipsec, memif, scale, srv6, tunnels, vhost and vts are tested only with vfio-pci. + Other (base, dot1q, dot1ad) tested with all drivers. + Setup actions currently depend on driver, generated. - The performance_rdma action is trivial for now. - Several tests fail, to be fixed later, e.g. by performance_rdma. + Reconf tests are also supported. + Added DRV_VFIO_PCI tags missing, mainly in density tests. - Vhost suites (density, reconf) are failing, but suites look good. - TCP suites do not support NIC drivers yet. - DPDK obviously not supported. + Use Python 3 in regenerate scripts. + Fix typos binded => bound. + File open modes set either u"rt" or u"wt" everywhere. + Remove a trailing space in an environment variable name. Change-Id: I290470675dc5c9e88b2eaa5ab6285ecd9ed7827a Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-15Hoststack perf infrastructure refactoringDave Wallace3-11/+112
- DUT only topology (hoststack test apps are co-located with vpp) - Make vpp app specific keywords generic where applicable - Add IP4 Prefix to topology file - Support running wrk in linux namespace - Refactor namespace cleanup - Remove redundant namespace creation code - Refactor test/keyword dirs: tcp -> hoststack - Add hoststack utility keywords - Refactor wrk suite setup/teardown - Update tests with recent perf infra changes Change-Id: Ia1cf07978d579393eef94923819a87c8c1f36f34 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-09FIX: Container testPeter Mikus1-1/+0
+ We should check if container is UP + We should adjust privileges. Change-Id: I3f604b7b9740943b4fd3b0574c18d2729e88b816 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-01-07Vhost: Add GSO optionPeter Mikus2-6/+17
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Iba9af71c293645b480203af72fcf940cbe9ccb3a
2019-12-17Telemetry: Add more operational dataPeter Mikus1-0/+2
+ Add both NDR and PDR telemetry capture + Speedup sockets + Adjust privileges Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ia6fd5d405e6fb410651d8b705c921653753aea10
2019-12-13FIX: eth interfaces need numa_node information in topology fileJan Gelety1-12/+4
Change-Id: I9a3c6947da8ab5033e29b39833c97bcdafdc3ca9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-16Remove supervisord dependency from containersPeter Mikus1-2/+1
+ Startup config stream-line Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ia12e48202a5fc88461d3d24ac9b9599e7175b1d8
2019-12-11Introduce VPP-IPsec container tests.Ludovit Mikula4-42/+168
Change-Id: Ie64d662e81879bd52785e0188450d998bf056bda Signed-off-by: Ludovit Mikula <ludovit.mikula@pantheon.tech>
2019-12-03Remove: VIRLPeter Mikus1-28/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I2b1672412e19f15cb46246fffa931351cd3d518b
2019-11-28Python3: resources and librariesJan Gelety11-991/+442
Change-Id: I1392c06b1d64f62b141d24c0d42a8e36913b15e2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-13LIGATO: RemovePeter Mikus3-93/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0c9b8b07ed3a113e24a38faaddd5ed06fae67768
2019-11-11FIX: do not use ping in vpp-device testsJan Gelety1-22/+31
Change-Id: Ic11f863e7ffad70d45e6e8dc39c70efc62cecc96 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-10-30Framework: Add show log as tear down actionPeter Mikus1-0/+1
+ We can see what is happening when debug level is increased + Big caveat of VPP is that it does not allow to configure default global level. Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I5ea1a618161cdb3170cab022d7451e52271f83c0
2019-10-25Introduce pre-initialize driver layerPeter Mikus3-239/+188
+ This will help unify driver base differences between TCs. + Decrease amount of logic needed for suite generator. Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I1e84ba361dc1e829f0612c58a61096e2633ce0c5
2019-10-19Tests: Introduce driver layerPeter Mikus1-10/+32
+ Small bugfixes in various suites Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I051af0a6da7340fe3f617400370c3846ca5241ab
2019-10-16VPPD: GBP test casePeter Mikus1-3/+4
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I020cdb6ced70a9c22773dfbe1662aecd2b744d2d
2019-10-09Add vpp loadbalancer maglev mode test suitehaiyanX1.zhang1-0/+1
Change-Id: I61555ba566efef0a2151db9a30bf7f5d9ccad1df Signed-off-by: haiyanx1.zhang <haiyanx1.zhang@intel.com>