aboutsummaryrefslogtreecommitdiffstats
path: root/tests/vpp/perf
AgeCommit message (Collapse)AuthorFilesLines
2021-01-04API: deprecated COP APIsTomas Alexy4-84/+82
Signed-off-by: Tomas Alexy <tomas.alexy@pantheon.tech> Change-Id: Ifb97088a26f3c2ab2ecee28ed43e6e957a844c64
2021-01-04GSO: TAP/VHOST use casepmikus5-0/+484
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ibafaaca4438587284d1e255c764f0701002941e8
2020-12-17PAPI: Cache connected client instancesVratko Polak3-3/+5
Disconnect+connect cycle is expensive and slow. This change tracks connected client instances so later "connect" to the same target uses it. Explicit disconnects are allowed (and executed before VPP ends), but once again disconnected instances are cached and reused, as creating a new instance is more expensive than just connect. + Add missing checks on interfaces being up to appropriate keyword. + Use appropriate keywords. + Add a comment explaining why a simpler keyword is not appropriate. + Improve VPP checks in containers. + Fix the vppctl check to actually work. + Add PAPI check to ensure VPP is really ready. + Delay/reorder checks to make them faster with multiple containers. + Leave some TODOs to improve various lifecycles later. + As we do not stop VPP in test/suite teardown: + One final disconnect is needed, added to __init__.robot teardowns. - Import of the final disconnect keyword is ugly, but it works. - We could use a hashable class for distinguishing node+socket pairs. - Are we connecting to VPP inside VMs? Change-Id: I49cd726740c3e8cae1591c7c84b85a447241228f Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-12-16API: deprecated IPSEC APIsJan Gelety124-124/+124
Jira: CSIT-1597 + add ipsec_sa tear down action Change-Id: I4d1e6f26c14e61e8ddc6f29fbf5528a034c84eb0 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-11-26Re-enable 9000B tests for ipsecVratko Polak160-0/+1920
Ticket: CSIT-1771 VPP-1207 VPP-1675 Change-Id: I8ba2d62054361e72b833943327434ea071e7e568 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-10-29Support existing test types with ASTFVratko Polak114-1162/+2952
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-10-23CSIT-1597 NAT44 API: dynamic configJan Gelety15-77/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/29463 - update vpp stable to version 21.01-rc0~283-g5f4f2081c Change-Id: I079c10e4537448c3b078f22c3fe4ed266a5e2e2c Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-10-19Tests: Remove GBPpmikus1-147/+0
- Specs - Tests - Library - TODO: PAL Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I8890940d56c7a9924d4a5f9365b6bb881610db14
2020-10-15FIX: vpp not started with configuration for nat44ed with 16M sessionsJan Gelety1-1/+1
- after merge of commit vppinfra: use heap to store bihash data | https://gerrit.fd.io/r/c/vpp/+/29099 it is necessary to allocate 14GB of memory for main heap in case of nat44ed test with 16.5M sessions Change-Id: I65dccdd4f50382b1b40cdffd2d9031ffb7d3bfa1 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-12Test: Add ramp-up phase to nat44ed-udir testsJan Gelety5-10/+35
Change-Id: I3aa50ec1ef9b0445014daa31e767323060f4a03f Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-30Framework: Add possibility to send ramp-up trafficJan Gelety6-61/+43
Change-Id: Ie24184ca4ac2d6c7abc32f0f103e10bc402ad93b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-30test: compute max_translations_per_thread based on worker threads numberJan Gelety15-15/+90
Change-Id: I1c638aef886bf37a9feb4a29e4949c7c8f19b717 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-30Framework: main heap size value multiplicationJan Gelety8-0/+16
- default value for main heap size set to 2G - allow allocation of n x 2G in tests that need it Change-Id: I02dafe6bc61649d2823486a3e983f66f5a213a88 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-24test: nat44det - add session number checkJan Gelety7-165/+81
- some tests need to reduce rate for ramp-up phase - some tests need to extend trail duration in ramp-up phase - removed 2n1l-10ge2p1x710-ethip4udp-nat44det-h1-p63-s63 suite as nat out ports are randomly selected from available port range so T-Rex stateless is not able to provide required out2in traffic Change-Id: I1145496610d202f81d911e68aa819844d7600918 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-23Fix: use nat44 not snat44Jan Gelety17-221/+221
Change-Id: If141566f76d5b2f309953233e693c34fef8ee12f Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-22test: lisp moved to pluginJan Gelety4-4/+4
- lisp is moving to plugin: https://gerrit.fd.io/r/c/vpp/+/28972 Jira: CSIT-1597 Change-Id: I362f1931b5fc441da9c8a6dda933a66bfed8c4b8 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-21Tests: nat44ed-unipmikus5-0/+825
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Iee488e2244a4e253471310bc7fb9640c69c6b0cb
2020-09-11Add ipsec async mode performance test casesYulong Pei20-0/+3018
In VPP 20.05, vpp added async crypto engine that support to use QAT hardware to do encryption and decryption, vnet/ipsec enabled async mode to use async crypto engine. Current async crypto engine also use dpdk_cryptodev as async handlers, in the future it may add other native QAT driver as async handlers. Note that async crypto engine is to support vnet/ipsec, it is different with current existing dpdk backend which itself has ESP implementation in plugins/dpdk/ipsec. Change-Id: I4e6eaa7ca1eddb8b1c45212de0684fb26907119b Signed-off-by: Yulong Pei <yulong.pei@intel.com>
2020-09-03Framework: Bump DPDK 20.08pmikus1-1/+1
+ DPDK 20.08 + Migrate make -> meson + Fix all trending issues Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I31dcb22627c0f8d17ec63c5b138a2da958b006f4
2020-08-09job_specs: use test tag ethip4udp-snat44det-h1-p1-s1 not ethip4udp-ip4base-nat44Jan Gelety1-140/+0
Change-Id: Idb7b415995f0a3474e379b0c023f64a96e886a55 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-08Fix: NAT44 deterministic moved to separate det44 pluginJan Gelety23-2532/+105
- align CSIT code with VPP code changes for NAT44 deterministic (DET44) feature - align test names according to snat44ed tests - remove obsolete 3-node nat tests - remove 2n1l-10ge2p1x710-ethip4udp-snat44det-h1048576-p63-s66060288 tests (not enough memory for such high number of sessions) Change-Id: I9a22b99b4cfa56d18e9c7ef9c58296e202567d42 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-20Framework: use 'stl' in trex stateless profile namesJan Gelety518-518/+518
Change-Id: I74641cc89d2f25d50b67d51bf2567082b420aabb Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-19Perf: NAT44 endpoint-dependent mode - tcp, part IJan Gelety10-0/+1645
Jira: CSIT-1736 - tcp synthetic profiles w/o data packets - tcp cps perf tests, phase I (no special "search cps" KW) Change-Id: I52be34b0fdd51d7a33c8c5de9b46d7064c48f7fa Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-07Perf: NAT44 endpoint-dependent mode - udp, part IJan Gelety20-285/+1836
- continuation of https://gerrit.fd.io/r/c/csit/+/26898 as there was reached limit of changes (1000) Jira: CSIT-1711 - udp synthetic profiles w/o data packets - udp cps perf tests, phase I (no special "search cps" KW) Part I means that we are using MRR tests to collect traffic data until there is ready new CPS test type with corresponding algorithm. Change-Id: I0d30feb9ecf1d0bff937152656f8eb422f831378 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-23T-Rex: Add advanced stateful modeJan Gelety53-53/+53
- provide base routines to run T-Rex in advanced stateful mode Change-Id: Ib0dc5f2919c370753335f6446860683dc4b12d93 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-31Framework: Perf stat capturingPeter Mikus1-1/+1
Change-Id: I3bbe1fe0073ddeead5219993675f24955e8c3dfd Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-07-30Introduce per DUT configurable statspmikus1-3/+17
- test/suite/global - binary logic is not working Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ia3d81cbf2c5f04d1093a0a408c84a9ffc6f3eef0
2020-07-16Add ADL plugin to cop testsVratko Polak4-4/+4
The following VPP change renames and moves COP functionality into a brand new plugin: https://gerrit.fd.io/r/c/vpp/+/27916 In a subsequent change we should rename tests and start using the new API messages. This change is just a minimal edit to avoid test failures. Change-Id: I1c4b81060dd533ef447f3cc44443b71c5e524d59 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-06-10NAT44 EI testsMaros Mullner16-0/+2504
Signed-off-by: Maros Mullner <maros.mullner@pantheon.tech> Change-Id: Ib5f58f60a1409ed139e2846793bf52fdc02a6571
2020-06-09Remove leading tc[nn] from test namesJuraj Linkeš511-5564/+5564
The test names are unique without it and the information doesn't add anything extra. Change-Id: Idc7d6d1d21c8c05691e1757227a0a3787406d370 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2020-05-25FIX: vts perf testsJan Gelety3-12/+12
Change-Id: Ie144c22575a7976da0c77e787e450355b73b0006 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-05-15Performance: Tests with virtio driver in VMPeter Mikus65-60/+827
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I20e01dfe83a961dc8202d33783a678d38e71cff2
2020-05-07perf: refactor 'setup suite topology interfaces'Dave Wallace8-9/+8
- and 'setup suite topology interfaces no tg' to use a common keyword to create suite variables using the required topology information. Change-Id: I46894948bc86eb7ce72d036e5b84f09c5c1385db Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-05-04perf: remove hoststack wrk cps/rps test suites.Dave Wallace2-196/+0
- The VSAP project will be adding hoststack connections-per-second and requests-per-second tests of the hoststack using the Apache 'ab' test tools. - WRK infra in /resources/libraries will be cleaned up separately as a part of the hoststack + LDP + nginx test suite. Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: Ic1d2f4299e8b6ae6be84283f22f6e28dd05bd80f
2020-05-04VPP-DEV API Coverages: SRv6Jan Gelety6-15/+13
Jira: CSIT-1698 Change-Id: I6d9154284990df8877850e4014716510016e485b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-28CSIT-1597 API cleanup: lispJan Gelety4-4/+8
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/24663 - update vpp stable to version 20.05-rc0~637 - remove unused L1 and L2 lisp KWs Change-Id: I2672b6a375ad70c82f331dcc991c145e868108b9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-20FIX: VTS testsPeter Mikus3-6/+6
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ib815175360ca565ce147f802e5dea0908b7507ee
2020-04-17VPP-DEV API Coverages: IPSEC interfaceJan Gelety35-106/+105
+ some pylint fixies Change-Id: I650ce16282ae953a1a5ee96e810702c01f71efd6 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-06Improve pf layerPeter Mikus508-871/+1881
+ Merge single/double link + Introduce _pf{n}[0] variables so we can access physical function same way as virtual function + Cleanup code by moving complex logic to python + Prepare code for multiple vf functions Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic2e74a38bfa146441357de8f0916aeb638941c49
2020-03-19perf: Fix broken hoststack testsDave Wallace2-2/+2
- Rename NSIM attribute names as changed in b9f4ba11 Change-Id: I6bc232c9954cfd9004d1d0cf22446957e78a641a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-03-12rls2001 perf: fix hoststack test packet sizesDave Wallace8-24/+25
- TCP packet size is 1460B not 9000B - QUIC packet size is 1280B not 9000B Change-Id: I6604a74fa533db4ac782782c85ea54038688627a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-03-10Make RXQs/TXQs configurablePeter Mikus508-11/+1535
Change-Id: Ib30bc4697fcba93a6723ee492a59a0523425f623 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-03-09Fix on 2n suiteVratko Polak1-1/+1
Wrong tag said it is 3. Change-Id: I3ea6ffbec3e38a11b721318266bcb2ea451d2d5a Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-03-03perf: Clean up Hoststack testsDave Wallace10-46/+347
- Update test names with clients/streams - Convert test results to JSON output * iperf3 results include bits_per_second * vpp_echo results include both client and server output which includes time in seconds and rx_data/tx_data in bytes which can be used to calculate the average bits per second. Tx and Rx data will always be the same: BPS = (client tx_data * 8) / ((client time + server time) / 2) - Fix WRK test results data formatting errors Change-Id: Ie2aeb665e3cc0739b16f97ba2628eebe6e041d22 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-02-25FIX: check if t-rex is running at test setup of all perf testsJan Gelety498-703/+703
Change-Id: I9af632035a1415666b2470c62a41d1b6acbf33c8 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-22FIX: policerJan Gelety2-4/+10
Change-Id: I23910b74c8720245b43067ac8c68880291b4b1c2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-02-04Ipsec: Use new plugin name in reconf testsVratko Polak40-40/+40
This edit is not suitable for rls2001. Change-Id: I18ea22346d5996e78034f35d74a87c125010a146 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-02-04Add more reconf tests, for IPsecVratko Polak41-1/+6101
- Not adding nf_density tests. - Not adding hardware ipsec tests. - Not adding -policy- tests. - Using old crypto_ia32_plugin.so plugin name. + Suitable for cherry-picking to rls2001. Change-Id: Ibf44d6d91e2afa2320637ecd9eb69d5d5dc364aa Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-31T-Rex: CPU pinningPeter Mikus1-2/+2
+ Detect NUMA + Pin based on numa location Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ife350f8c70e5437ac7c1413c7753f2a2f62777d9
2020-01-30perf: Add hoststack NSIM+LDPRELOAD+IPERF3 test suiteDave Wallace1-0/+66
Change-Id: Ia7a876b1aa240676e1f2d23618c1d4e09ead14f0 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>