aboutsummaryrefslogtreecommitdiffstats
path: root/tests/vpp/device
AgeCommit message (Collapse)AuthorFilesLines
2020-10-29Support existing test types with ASTFVratko Polak2-8/+2
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-09-23Fix: use nat44 not snat44Jan Gelety4-8/+8
Change-Id: If141566f76d5b2f309953233e693c34fef8ee12f Signed-off-by: Jan Gelety <jgelety@cisco.com> (cherry picked from commit a0362eb00573cda326b7067f020359d5913e76fd) Change-Id: I6fe01e18ca126800ffba344800b622562894ee05 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-07VPP-DEV API COV: Add NAT44-DET testsMartin Balaz2-0/+222
Jira: CSIT-1756 Signed-off-by: Martin Balaz <martin.balaz@pantheon.tech> Change-Id: Ia6fdc6df91f36ffcabd428326195bb9d8f967e52
2020-09-07VPP-DEV API COV: Add NAT44-ED testsJan Gelety2-0/+234
Jira: CSIT-1755 Change-Id: I34baa22a49f44da3fa80d91fa2f4132c982fe610 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-07-31Framework: Perf stat capturingPeter Mikus1-1/+1
Change-Id: I3bbe1fe0073ddeead5219993675f24955e8c3dfd Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-07-30Introduce per DUT configurable statspmikus1-2/+16
- test/suite/global - binary logic is not working Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ia3d81cbf2c5f04d1093a0a408c84a9ffc6f3eef0
2020-07-16Add ADL plugin to cop testsVratko Polak4-4/+4
The following VPP change renames and moves COP functionality into a brand new plugin: https://gerrit.fd.io/r/c/vpp/+/27916 In a subsequent change we should rename tests and start using the new API messages. This change is just a minimal edit to avoid test failures. Change-Id: I1c4b81060dd533ef447f3cc44443b71c5e524d59 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-06-09Remove leading tc[nn] from test namesJuraj Linkeš45-45/+45
The test names are unique without it and the information doesn't add anything extra. Change-Id: Idc7d6d1d21c8c05691e1757227a0a3787406d370 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2020-05-15Performance: Tests with virtio driver in VMPeter Mikus4-7/+7
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I20e01dfe83a961dc8202d33783a678d38e71cff2
2020-05-04VPP-DEV API Coverages: SRv6Jan Gelety6-0/+681
Jira: CSIT-1698 Change-Id: I6d9154284990df8877850e4014716510016e485b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-28CSIT-1597 API cleanup: lispJan Gelety5-5/+5
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/24663 - update vpp stable to version 20.05-rc0~637 - remove unused L1 and L2 lisp KWs Change-Id: I2672b6a375ad70c82f331dcc991c145e868108b9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-17VPP-DEV API Coverages: IPSEC interfaceJan Gelety2-0/+202
+ some pylint fixies Change-Id: I650ce16282ae953a1a5ee96e810702c01f71efd6 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-06Improve pf layerPeter Mikus37-124/+206
+ Merge single/double link + Introduce _pf{n}[0] variables so we can access physical function same way as virtual function + Cleanup code by moving complex logic to python + Prepare code for multiple vf functions Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic2e74a38bfa146441357de8f0916aeb638941c49
2020-03-10Make RXQs/TXQs configurablePeter Mikus37-37/+148
Change-Id: Ib30bc4697fcba93a6723ee492a59a0523425f623 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-01-22FIX: policerJan Gelety1-2/+5
Change-Id: I23910b74c8720245b43067ac8c68880291b4b1c2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-31T-Rex: CPU pinningPeter Mikus1-2/+1
+ Detect NUMA + Pin based on numa location Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ife350f8c70e5437ac7c1413c7753f2a2f62777d9
2020-01-29IPSEC: Change plugin namingPeter Mikus4-4/+4
https://gerrit.fd.io/r/c/vpp/+/24574 Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic2b7c925ceba1e16de87c64003ecceeea69c681c
2020-01-10Support suite tags in autogenVratko Polak37-1/+38
+ Include a script to add suite tags to many suites at once. + Add suite tags also to device tests (not covered by autogen). Change-Id: I514ee6178e22999b43460028fe2696738b012f04 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-07Update of VPP_STABLE_VER filesJan Gelety1-1/+1
- use new vpp ref build - ubuntu 18.04: 20.01-rc0~983-g78565f38e - use new vpp ref build - centos7: 20.01-rc0~983_g78565f3~b8651 + remove EXPECTED_FAILING tag from tc01-64B-ethip4-l2patch-dev test Change-Id: Iab47a66003926024f87e028b1b1d9136b8fb4ec4 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-02Revert "L2Patch: Remove EXPECTED_FAILING"Peter Mikus1-1/+1
This reverts commit 09c5a6b8e1c6efed8826ef34aa64809226e80edb. Reason for revert: CSIT has not yet the latest VPP version Change-Id: Ibaa2c00c639bacef1561898daf9485c3a68efec4 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-01-02L2Patch: Remove EXPECTED_FAILINGPeter Mikus1-1/+1
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ie950370c47403872597f8857edd651df2552ccb2
2019-12-01CSIT-VPP-DEV: move tap tests back to critical poolJan Gelety3-3/+1
Change-Id: Ic99e828588d561a07c169502dff8ca19ac98400f Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-10VPPD: Add L2patchPeter Mikus1-0/+75
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Iebdc03e56cf9ecf0322ff5c2c8698e9e3daf4407
2019-12-10CSIT-1597 API cleanup: classifyJan Gelety3-1/+3
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/21551 - move vpp stable build to 20.01-rc0~821 - tap tests tagged 'EXPECTED_FAILING' until necessary vpp api change https://gerrit.fd.io/r/c/vpp/+/21706 and csit code is adapted Change-Id: If4b9b7b5bc207612d08aa8aa4469a5361e058fc3 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-28Python3: resources and librariesJan Gelety37-334/+336
Change-Id: I1392c06b1d64f62b141d24c0d42a8e36913b15e2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-11FIX: do not use ping in vpp-device testsJan Gelety19-149/+131
Change-Id: Ic11f863e7ffad70d45e6e8dc39c70efc62cecc96 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-10-25Introduce pre-initialize driver layerPeter Mikus36-144/+212
+ This will help unify driver base differences between TCs. + Decrease amount of logic needed for suite generator. Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I1e84ba361dc1e829f0612c58a61096e2633ce0c5
2019-10-19Tests: Introduce driver layerPeter Mikus36-56/+106
+ Small bugfixes in various suites Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I051af0a6da7340fe3f617400370c3846ca5241ab
2019-10-16VPPD: GBP test casePeter Mikus1-0/+88
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I020cdb6ced70a9c22773dfbe1662aecd2b744d2d
2019-09-26CSIT-1471: Policer - VAT API to PAPIpremkumarv4-9/+33
Change-Id: I6c7eff8d9cbb196ccb684a45579ac59942351cfe Signed-off-by: premkumarv <preym17@gmail.com>
2019-09-19LISP: 3-node (func) to 2-node (dev) converstionNidhy5-0/+417
Change-Id: I0fbe4d9ef1b30a1ac1382ea43b5b52aeb8a7d39f Signed-off-by: Nidhy <nidhyanandhan.a@hcl.com>
2019-08-20Align NF_denisty testsPeter Mikus1-1/+2
+ Apply layer approach for interfaces Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I6c1898b9833f9f42b8912f1b521e30093360d7ea
2019-08-14FIX: Remove eth_avf and portX_vifY interfaces from topo in TC/TS tear downJan Gelety1-0/+1
Change-Id: I90901c4ab14dfa76dc3e1f786a4b986479e0ba47 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-08-09Clean: L2 bridge domainPeter Mikus1-1/+2
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I3721900b91a9c8e9db8766bdcb9771b6bcf95bca
2019-08-06VPPD: Dot1QPeter Mikus1-0/+74
Change-Id: I0d3c925ea4a6896a0df98db6ddaf4238e6291bf1 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-25VPPD: VXLANPeter Mikus2-0/+147
Change-Id: Iac518341e6a0fff4a31ad3692ccc5697690db8be Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-24NF_density dot1qip4udpvxlanPeter Mikus1-1/+3
+ NF_density compatible + Baseline VM tests compatible + Scalability NxM - 3n variant is missing in VXLAN and DOT1Q due to requests to have it asymmetrical - TODO Change-Id: Ia817d7be00622b594b451bc07756240d1a379e04 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-19FIX: naming convention fixes for TOXPeter Mikus4-4/+4
Change-Id: I0644fbd0e136dee3a91fd9406155f971b3c726eb Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-19VAT API to PAPI CONVERSIONpremkumarv1-2/+2
Change-Id: I9c87cdfb99780f398da5cd90351b303caa89bda9 Signed-off-by: premkumarv <preym17@gmail.com>
2019-07-19FIX: COP ipv6 missing FIBPeter Mikus2-0/+4
- From unknown reason VAT was more tolerant Change-Id: Ia740843c67604a18de40d14cc448138b9f079b41 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-17Refactor VPP Device VM vhost testsjuraj.linkes9-451/+338
* replace the current VM image with kernel img * rework keyword usage to make it consistent with performance tests * remove resources/libraries/robot/shared/qemu.robot as it's not used anywhere anymore Change-Id: Ia5bc19e9e6ed9af031e4d9b5c0c89431fb49fd33 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-07-12VPPD: Cop, iACL, PolicerPeter Mikus8-0/+608
- Alignment with PERF Change-Id: Id7b2cb91c88e44cf9265c3376330f6aa579abf7a Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-12VPPD: Merge IPv4/IPv6Peter Mikus8-436/+24
- Preparation for coverage.sh Change-Id: Id299df9cd57ee69116686c8bc85eaba1cc353122 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-11FIX: Various naming convention typosPeter Mikus21-21/+21
Change-Id: Ib55d9f3993cb63d3130a6bd338d612bde0235d5d Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-11Aligning FUNC/PERF keywords - Data drivenPeter Mikus24-541/+943
Change-Id: I233c8908c724556db09c60de6efa9237068fb4c3 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-09CSIT-1539 Aligning FUNC/PERF keywords - Phase IIIPeter Mikus14-603/+404
Change-Id: Ibd3e99390bf004173d3d126946b95ffe9ba8f196 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-05Remove pkt_trace as global variablePeter Mikus1-2/+0
- Use explicit parameters Change-Id: I21cbc8cf13ec5e455b20c685c4427a53a655f71f Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-03FIX: use len parameter name in prefix structure in ip_route_add_delJan Gelety2-2/+4
+ new vpp stabel build with fixed ipv6 address incrementation Change-Id: Id3f4104a48e45b6fd70ccbd47ad9e3c6925220fd Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-06-17Add packet trace to teardown of all vpp-device func testsJan Gelety4-4/+4
Change-Id: I26ad1f9a481d3d7ecdb0ad4e85d521756302e09b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-06-25Align suite/test teardown/setupPeter Mikus19-286/+271
+ Phase II - aligning FUNC/PERF setup. + 2n/3n compatibility of suite setup. + Same structure of testcase for DEV/PERF. + DEVICE tests are now running VPP startup config in same way as PERF. + DEVICE is using COREKEEPER! - DEVICE tests are running topology compute twice for backward compatibility. TODO to remove. - Still phase III needed to align TAGS, LIBRARIES and remove OLD setup phase. Change-Id: Ib6e1b0ebf4abebf3cbe73b4fda5cac953c2a7270 Signed-off-by: Peter Mikus <pmikus@cisco.com>