aboutsummaryrefslogtreecommitdiffstats
path: root/tests
AgeCommit message (Collapse)AuthorFilesLines
2020-10-29Support existing test types with ASTFVratko Polak116-1166/+3012
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-09-12Test: Add ramp-up phase to nat44ed-udir testsJan Gelety5-10/+39
Change-Id: I3aa50ec1ef9b0445014daa31e767323060f4a03f Signed-off-by: Jan Gelety <jgelety@cisco.com> (cherry picked from commit d68be735d882bafcb672ebb27a66efbcabbeb02d) Change-Id: Iad67c8445e18b22eccbea25d75b91827b398775f Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-12Framework: Add possibility to send ramp-up trafficJan Gelety6-61/+43
Change-Id: Ie24184ca4ac2d6c7abc32f0f103e10bc402ad93b Signed-off-by: Jan Gelety <jgelety@cisco.com> (cherry picked from commit 61044d391d6e8d6b47d0d4f156071bd61cd278df) Change-Id: I50e2a674784688e6eeea566fc2bc4d45a8ecfb8b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-30test: compute max_translations_per_thread based on worker threads numberJan Gelety15-15/+90
Change-Id: I1c638aef886bf37a9feb4a29e4949c7c8f19b717 Signed-off-by: Jan Gelety <jgelety@cisco.com> (cherry picked from commit d99951620507d354c4803eb1ee26609d992b70b3) Change-Id: Iaf5b4dffe603b0cf5cf0430fc6ca20acb7a01fda Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-11test: nat44det - add session number checkJan Gelety7-165/+81
- some tests need to reduce rate for ramp-up phase - some tests need to extend trail duration in ramp-up phase - removed 2n1l-10ge2p1x710-ethip4udp-nat44det-h1-p63-s63 suite as nat out ports are randomly selected from available port range so T-Rex stateless is not able to provide required out2in traffic Change-Id: I1145496610d202f81d911e68aa819844d7600918 Signed-off-by: Jan Gelety <jgelety@cisco.com> (cherry picked from commit 3b408b7ea702dd3817442186035121fe862cbf7f) Change-Id: I53da8c086373d06e0842e5563964d9287c0fa403 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-23Fix: use nat44 not snat44Jan Gelety21-229/+229
Change-Id: If141566f76d5b2f309953233e693c34fef8ee12f Signed-off-by: Jan Gelety <jgelety@cisco.com> (cherry picked from commit a0362eb00573cda326b7067f020359d5913e76fd) Change-Id: I6fe01e18ca126800ffba344800b622562894ee05 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-21Tests: nat44ed-unipmikus5-0/+825
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Iee488e2244a4e253471310bc7fb9640c69c6b0cb (cherry picked from commit d3f265ab0e44baa9fe223643240253f9d54c1b4c)
2020-09-11Add ipsec async mode performance test casesYulong Pei20-0/+3018
In VPP 20.05, vpp added async crypto engine that support to use QAT hardware to do encryption and decryption, vnet/ipsec enabled async mode to use async crypto engine. Current async crypto engine also use dpdk_cryptodev as async handlers, in the future it may add other native QAT driver as async handlers. Note that async crypto engine is to support vnet/ipsec, it is different with current existing dpdk backend which itself has ESP implementation in plugins/dpdk/ipsec. Change-Id: I4e6eaa7ca1eddb8b1c45212de0684fb26907119b Signed-off-by: Yulong Pei <yulong.pei@intel.com> (cherry picked from commit d12f510caf3bb83695488684eb07de79b3e753b9)
2020-09-07VPP-DEV API COV: Add NAT44-DET testsMartin Balaz2-0/+222
Jira: CSIT-1756 Signed-off-by: Martin Balaz <martin.balaz@pantheon.tech> Change-Id: Ia6fdc6df91f36ffcabd428326195bb9d8f967e52
2020-09-07VPP-DEV API COV: Add NAT44-ED testsJan Gelety2-0/+234
Jira: CSIT-1755 Change-Id: I34baa22a49f44da3fa80d91fa2f4132c982fe610 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-03Framework: Bump DPDK 20.08pmikus2-5/+5
+ DPDK 20.08 + Migrate make -> meson + Fix all trending issues Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I31dcb22627c0f8d17ec63c5b138a2da958b006f4
2020-08-09job_specs: use test tag ethip4udp-snat44det-h1-p1-s1 not ethip4udp-ip4base-nat44Jan Gelety1-140/+0
Change-Id: Idb7b415995f0a3474e379b0c023f64a96e886a55 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-08Fix: NAT44 deterministic moved to separate det44 pluginJan Gelety23-2532/+105
- align CSIT code with VPP code changes for NAT44 deterministic (DET44) feature - align test names according to snat44ed tests - remove obsolete 3-node nat tests - remove 2n1l-10ge2p1x710-ethip4udp-snat44det-h1048576-p63-s66060288 tests (not enough memory for such high number of sessions) Change-Id: I9a22b99b4cfa56d18e9c7ef9c58296e202567d42 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-20Framework: use 'stl' in trex stateless profile namesJan Gelety522-522/+522
Change-Id: I74641cc89d2f25d50b67d51bf2567082b420aabb Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-19Perf: NAT44 endpoint-dependent mode - tcp, part IJan Gelety10-0/+1645
Jira: CSIT-1736 - tcp synthetic profiles w/o data packets - tcp cps perf tests, phase I (no special "search cps" KW) Change-Id: I52be34b0fdd51d7a33c8c5de9b46d7064c48f7fa Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-07Perf: NAT44 endpoint-dependent mode - udp, part IJan Gelety21-286/+1838
- continuation of https://gerrit.fd.io/r/c/csit/+/26898 as there was reached limit of changes (1000) Jira: CSIT-1711 - udp synthetic profiles w/o data packets - udp cps perf tests, phase I (no special "search cps" KW) Part I means that we are using MRR tests to collect traffic data until there is ready new CPS test type with corresponding algorithm. Change-Id: I0d30feb9ecf1d0bff937152656f8eb422f831378 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-23T-Rex: Add advanced stateful modeJan Gelety53-53/+53
- provide base routines to run T-Rex in advanced stateful mode Change-Id: Ib0dc5f2919c370753335f6446860683dc4b12d93 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-31Framework: Perf stat capturingPeter Mikus2-2/+2
Change-Id: I3bbe1fe0073ddeead5219993675f24955e8c3dfd Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-07-30Introduce per DUT configurable statspmikus3-9/+45
- test/suite/global - binary logic is not working Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ia3d81cbf2c5f04d1093a0a408c84a9ffc6f3eef0
2020-07-16Add ADL plugin to cop testsVratko Polak8-8/+8
The following VPP change renames and moves COP functionality into a brand new plugin: https://gerrit.fd.io/r/c/vpp/+/27916 In a subsequent change we should rename tests and start using the new API messages. This change is just a minimal edit to avoid test failures. Change-Id: I1c4b81060dd533ef447f3cc44443b71c5e524d59 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-06-10NAT44 EI testsMaros Mullner16-0/+2504
Signed-off-by: Maros Mullner <maros.mullner@pantheon.tech> Change-Id: Ib5f58f60a1409ed139e2846793bf52fdc02a6571
2020-06-09Remove leading tc[nn] from test namesJuraj Linkeš560-5657/+5657
The test names are unique without it and the information doesn't add anything extra. Change-Id: Idc7d6d1d21c8c05691e1757227a0a3787406d370 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2020-05-25FIX: vts perf testsJan Gelety3-12/+12
Change-Id: Ie144c22575a7976da0c77e787e450355b73b0006 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-05-15Performance: Tests with virtio driver in VMPeter Mikus69-67/+834
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I20e01dfe83a961dc8202d33783a678d38e71cff2
2020-05-07perf: refactor 'setup suite topology interfaces'Dave Wallace8-9/+8
- and 'setup suite topology interfaces no tg' to use a common keyword to create suite variables using the required topology information. Change-Id: I46894948bc86eb7ce72d036e5b84f09c5c1385db Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-05-04perf: remove hoststack wrk cps/rps test suites.Dave Wallace2-196/+0
- The VSAP project will be adding hoststack connections-per-second and requests-per-second tests of the hoststack using the Apache 'ab' test tools. - WRK infra in /resources/libraries will be cleaned up separately as a part of the hoststack + LDP + nginx test suite. Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: Ic1d2f4299e8b6ae6be84283f22f6e28dd05bd80f
2020-05-04VPP-DEV API Coverages: SRv6Jan Gelety12-15/+694
Jira: CSIT-1698 Change-Id: I6d9154284990df8877850e4014716510016e485b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-28CSIT-1597 API cleanup: lispJan Gelety9-9/+13
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/24663 - update vpp stable to version 20.05-rc0~637 - remove unused L1 and L2 lisp KWs Change-Id: I2672b6a375ad70c82f331dcc991c145e868108b9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-23FIX: Mellanox jumbo framesPeter Mikus4-0/+4
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I84b3c07e22a313e96ac59fc7818960c502507651
2020-04-23Performance: DPDK refactorPeter Mikus12-520/+41
+ Rework BASH scripts (more code to python). + Move BASH into libraries. + Allows RDMA usage. + Fix 9000B tests. + Rename confusing l2fwd -> testpmd. + Fix suite setup. + Fix PCI whitelist to not accidentally pickup wrong interface. + Fix deprecated DPDK cli arguments. - MLX5 jumbo are disabled on NIC (i will increase separately). https://jenkins.fd.io/job/csit-dpdk-perf-verify-master-2n-clx/6/console (l3fwd still broken) - MLX5 IMIX seems to be some TRex issue with IMIX for mlx5 (i will handle separately) Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I31d1b67305fa247cb5e1f57e739d3ef30dc1a04b
2020-04-20FIX: VTS testsPeter Mikus3-6/+6
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ib815175360ca565ce147f802e5dea0908b7507ee
2020-04-17VPP-DEV API Coverages: IPSEC interfaceJan Gelety37-106/+307
+ some pylint fixies Change-Id: I650ce16282ae953a1a5ee96e810702c01f71efd6 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-06Improve pf layerPeter Mikus545-995/+2087
+ Merge single/double link + Introduce _pf{n}[0] variables so we can access physical function same way as virtual function + Cleanup code by moving complex logic to python + Prepare code for multiple vf functions Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic2e74a38bfa146441357de8f0916aeb638941c49
2020-03-19perf: Fix broken hoststack testsDave Wallace2-2/+2
- Rename NSIM attribute names as changed in b9f4ba11 Change-Id: I6bc232c9954cfd9004d1d0cf22446957e78a641a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-03-12rls2001 perf: fix hoststack test packet sizesDave Wallace8-24/+25
- TCP packet size is 1460B not 9000B - QUIC packet size is 1280B not 9000B Change-Id: I6604a74fa533db4ac782782c85ea54038688627a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-03-10Make RXQs/TXQs configurablePeter Mikus551-60/+1705
Change-Id: Ib30bc4697fcba93a6723ee492a59a0523425f623 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-03-09Fix on 2n suiteVratko Polak1-1/+1
Wrong tag said it is 3. Change-Id: I3ea6ffbec3e38a11b721318266bcb2ea451d2d5a Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-03-03perf: Clean up Hoststack testsDave Wallace10-46/+347
- Update test names with clients/streams - Convert test results to JSON output * iperf3 results include bits_per_second * vpp_echo results include both client and server output which includes time in seconds and rx_data/tx_data in bytes which can be used to calculate the average bits per second. Tx and Rx data will always be the same: BPS = (client tx_data * 8) / ((client time + server time) / 2) - Fix WRK test results data formatting errors Change-Id: Ie2aeb665e3cc0739b16f97ba2628eebe6e041d22 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-02-25FIX: check if t-rex is running at test setup of all perf testsJan Gelety498-703/+703
Change-Id: I9af632035a1415666b2470c62a41d1b6acbf33c8 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-02-10FIX: Detection if l2fwd/l3fwd is up/downJan Gelety3-12/+8
Change-Id: Ide5de222e8314a0ea0be59f9a478f8d59147f722 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-22FIX: policerJan Gelety3-6/+15
Change-Id: I23910b74c8720245b43067ac8c68880291b4b1c2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-02-07FIX: Detection if testpmd/l3fwd is upPeter Mikus3-102/+122
Change-Id: Ibd2e038332fe2bdf0e5bd69bf7376a2a7357e901 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-02-04Ipsec: Use new plugin name in reconf testsVratko Polak40-40/+40
This edit is not suitable for rls2001. Change-Id: I18ea22346d5996e78034f35d74a87c125010a146 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-02-04Add more reconf tests, for IPsecVratko Polak41-1/+6101
- Not adding nf_density tests. - Not adding hardware ipsec tests. - Not adding -policy- tests. - Using old crypto_ia32_plugin.so plugin name. + Suitable for cherry-picking to rls2001. Change-Id: Ibf44d6d91e2afa2320637ecd9eb69d5d5dc364aa Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-31T-Rex: CPU pinningPeter Mikus3-10/+9
+ Detect NUMA + Pin based on numa location Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ife350f8c70e5437ac7c1413c7753f2a2f62777d9
2020-01-30perf: Add hoststack NSIM+LDPRELOAD+IPERF3 test suiteDave Wallace1-0/+66
Change-Id: Ia7a876b1aa240676e1f2d23618c1d4e09ead14f0 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-29perf: QUIC transport hoststack test suiteDave Wallace1-0/+74
Change-Id: I73f4be7ea315c7a5dcce46e1bd3034bcb0a97ee2 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-29IPSEC: Change plugin namingPeter Mikus104-104/+104
https://gerrit.fd.io/r/c/vpp/+/24574 Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic2b7c925ceba1e16de87c64003ecceeea69c681c
2020-01-28Ipsec: Unify first line of Local Template docVratko Polak62-94/+90
Seeing differences when diff-ing between suites is distracting. + Bump copyright year, even for files with no change. Change-Id: Iaca79647821dd8233bdbe6b0ac8b14fdb04060a8 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-28perf: hoststack iperf3 test tuningDave Wallace1-18/+11
Change-Id: I53425f57fe9ecef9cff2c94642cc7cb24537a961 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>