aboutsummaryrefslogtreecommitdiffstats
path: root/resources/api
AgeCommit message (Collapse)AuthorFilesLines
2020-10-29Support existing test types with ASTFVratko Polak1-2/+2
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-09-11Add ipsec async mode performance test casesYulong Pei1-0/+2
In VPP 20.05, vpp added async crypto engine that support to use QAT hardware to do encryption and decryption, vnet/ipsec enabled async mode to use async crypto engine. Current async crypto engine also use dpdk_cryptodev as async handlers, in the future it may add other native QAT driver as async handlers. Note that async crypto engine is to support vnet/ipsec, it is different with current existing dpdk backend which itself has ESP implementation in plugins/dpdk/ipsec. Change-Id: I4e6eaa7ca1eddb8b1c45212de0684fb26907119b Signed-off-by: Yulong Pei <yulong.pei@intel.com> (cherry picked from commit d12f510caf3bb83695488684eb07de79b3e753b9)
2020-09-07VPP-DEV API COV: Add NAT44-ED testsJan Gelety1-20/+20
Jira: CSIT-1755 Change-Id: I34baa22a49f44da3fa80d91fa2f4132c982fe610 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-09job_specs: use test tag ethip4udp-snat44det-h1-p1-s1 not ethip4udp-ip4base-nat44Jan Gelety1-1/+1
Change-Id: Idb7b415995f0a3474e379b0c023f64a96e886a55 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-08Fix: NAT44 deterministic moved to separate det44 pluginJan Gelety1-2/+11
- align CSIT code with VPP code changes for NAT44 deterministic (DET44) feature - align test names according to snat44ed tests - remove obsolete 3-node nat tests - remove 2n1l-10ge2p1x710-ethip4udp-snat44det-h1048576-p63-s66060288 tests (not enough memory for such high number of sessions) Change-Id: I9a22b99b4cfa56d18e9c7ef9c58296e202567d42 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-08-07Perf: NAT44 endpoint-dependent mode - udp, part IJan Gelety1-0/+16
- continuation of https://gerrit.fd.io/r/c/csit/+/26898 as there was reached limit of changes (1000) Jira: CSIT-1711 - udp synthetic profiles w/o data packets - udp cps perf tests, phase I (no special "search cps" KW) Part I means that we are using MRR tests to collect traffic data until there is ready new CPS test type with corresponding algorithm. Change-Id: I0d30feb9ecf1d0bff937152656f8eb422f831378 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-05-11CSIT-1597 API cleanup: ipsecJan Gelety1-240/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/26276 - update vpp stable to version 20.05-rc0~727 Change-Id: I39a0b5e60fac6a74aff2426f6a448c0e117ab647 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-29API: Add collection for 26276/7Jan Gelety1-0/+238
Change-Id: Ib65e3268e04fd4a7908e3e7c3c4217807f47a38b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-28CSIT-1597 API cleanup: lispJan Gelety1-258/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/24663 - update vpp stable to version 20.05-rc0~637 - remove unused L1 and L2 lisp KWs Change-Id: I2672b6a375ad70c82f331dcc991c145e868108b9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-28FIX: crc for create_vhost_user_ifJan Gelety1-1/+1
Change-Id: Ia43679c440da62ba44ad74d4c1eb9003b4ba0a63 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-28API: Add collection for 24663/6Jan Gelety1-0/+247
Change-Id: I009df88e0a315603b5f9d72eb1d1a729d1a8a4cd Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-04-02CSIT-1597 API cleanup: virtioJan Gelety1-247/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/26086 - update vpp stable to version 20.05-rc0~630 Change-Id: I3e3f00ba81e62f07c5d287300dfaff6cee18a1b4 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-30API: Add collection for 26086/8Jan Gelety1-0/+245
Change-Id: I41867bdb395a72733c1f616b981e952087b017ef Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-29Update of VPP_STABLE_VER filesJan Gelety1-249/+2
- use new vpp ref build - ubuntu 18.04: 20.05-rc0~532-g4fde4ae03 - use new vpp ref build - centos7: 20.05-rc0~532_g4fde4ae~b9253 Change-Id: If6a5478ca9b89050beae8f2dc4ce4e91b328aa21 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-29API: Add collection for 26256/7Jan Gelety1-4/+4
Change-Id: Ib5d8f9f0c8e8920a2d19e89d4d5518f4882ff32d Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-26API: Add collection for 25923/9Jan Gelety1-0/+247
Change-Id: I48da04747ffcbdb0b4452d11ee52204e966eb78b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-27CSIT-1597 API cleanup: aclJan Gelety1-249/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/26184 - update vpp stable to version 20.05-rc0~424 Change-Id: Ie621a29db25755151bfee76a045a423fd1267ada Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-25API: Add collection for 25787/15Jan Gelety1-0/+247
Change-Id: Id33dbc63832848f951fc6478472e9d6dd614bff9 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-24CSIT-1597 API cleanup: srv6Jan Gelety1-249/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/25647 - update vpp stable to version 20.05-rc0~398 Change-Id: I5ebc84a702264b0b37e3886b48ec0713d174f880 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-23API: Add collection for 25647/3Jan Gelety1-0/+247
Change-Id: I01398df1d8d0274ff7aafc4cef0f431f19bef099 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-17CSIT-1597 API cleanup: vpeJan Gelety1-249/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/25567 - update vpp stable to version 20.05-rc0~352 Change-Id: I205d7d3de48d480e070d3e2d3e3ab83480b69c77 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-11API: Add collection for 25567/1Jan Gelety1-0/+247
Change-Id: I16ca645cf55771a986696e94a9543d4e8aa1c7be Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-16CSIT-1597 API cleanup: vxlanJan Gelety1-249/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/25569 - update vpp stable to version 20.05-rc0~345 - some pylint fixies in Policer.py Change-Id: Iba2c877d0e74526c69826c5ec4afd64ed2055d31 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-16API: Add collection for 25569/4Jan Gelety1-0/+247
Change-Id: If5edd443d281616c1c7a7225b2a606f3a3255bd0 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-10CSIT-1597 API cleanup: policerJan Gelety1-249/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/25566 - update vpp stable to version 20.05-rc0~327 Change-Id: I6af2e4e21c9190e87a63c5a9c8529bbc7f8adf1a Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-10API: Add collection for 25566/5Jan Gelety1-0/+247
Change-Id: Id5bbd395304c391493dc46a6c82fe395f14340ae Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-10CSIT-1597 API cleanup: ipsecJan Gelety1-249/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/25529 - update vpp stable to version 20.05-rc0~312-g287d5e109 Change-Id: I6c7b3520f4bb306c3b0b59247b4ba2d5f170686c Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-10API: Add collection for 25529/2Jan Gelety1-0/+248
Change-Id: Iac159cb60d4afbc23fc7edffefd82fc3e2a64940 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-09CSIT-1597 API cleanup: l2Jan Gelety1-248/+4
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/22970 - update vpp stable to version 20.05-rc0~307 Change-Id: I3b4c69ee60b0add12edeb8d5e6ac6006b519593b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-03-09API: Add collection for 22970/10Jan Gelety1-0/+244
Change-Id: I8b635b14950ca148969831614fc7f32d52ef79d1 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-02-26CRC comments: SW crypto is enoughVratko Polak1-3/+3
Change-Id: I4ec4a56d54850a7654bdc1a6330bd473bd997608 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-02-10FIX: set host physical interface mac address to rdma interfaceJan Gelety1-0/+2
Reason is that the traffic generator is initialized with MAC addresses of physical interfaces in the suite setup while rdma interfaces are created later. Change-Id: I2cf182cde20f14778624740a1f0467096c336306 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-31Update CRC commentsVratko Polak1-15/+18
+ Link to current stable build. + Remove policer comment (as it is already a devicetest). + Add nsim comment. + Include perftest tag pattern to run a drv-avf test. + Switch from 2n-skx to 2n-clx to test also rdma-core driver. + Add tag for vfio-pci driver for the rest of tag expressions. Change-Id: Id2f2ee0525e27437166431045a174b3097dff52b Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-30perf: Add hoststack NSIM+LDPRELOAD+IPERF3 test suiteDave Wallace1-0/+4
Change-Id: Ia7a876b1aa240676e1f2d23618c1d4e09ead14f0 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-01-14Update comments in crc file.Vratko Polak1-5/+4
+ L2patch is already covered. + RDMA is not device testable yet. Change-Id: I1dcccfcb340a8fe61c20d3516340e26db293cfaa Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-13CSIT-1597 API cleanup: rdmaJan Gelety1-241/+4
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/24225 - update vpp stable to version 20.01-rc0~1011 Change-Id: Ibbfdbe867cf89715619efc5c5507859983025ece Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-01-08API: Add collection for 24225/3Jan Gelety1-8/+8
Change-Id: I5c4593d05dbcb4d918611dc7520432a446dbf664 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-12API: Add collection for 22970/5Jan Gelety1-0/+237
Change-Id: If865526d833a267eba85e6ae4107e4d0da0dfd0b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-12CSIT-1597, CSIT-1647 API cleanup: gbpJan Gelety1-241/+4
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/23835 - update vpp stable to version 20.01-rc0~843 Change-Id: Idb1299cbbba578c09542c23858fb1fc55d090f26 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-12API: Add collection for 23835/1Jan Gelety1-12/+12
- update vpp stable to version 20.01-rc0~841 Change-Id: I05fc3878a606852c8348b61b40d8000411709d33 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-01API: Add collection for 23830/1Jan Gelety1-6/+6
- update vpp stable to version 20.01-rc0~839 Change-Id: Ic79bccf7cce384f5058218398bbef16cf1c9d40c Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-01API: Add collection for 23829/1Jan Gelety1-0/+237
Change-Id: I4b2a7bda48fba3050a35966f7a86c8dc7ba62d36 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-01CSIT-1597 API cleanup: vhostJan Gelety1-241/+2
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/22277 - update vpp stable to version 20.01-rc0~831 Change-Id: Icb3801d6a7bc738cf48bc5ecbb0767f115dde964 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-01API: Add collection for 22277/5Jan Gelety1-0/+239
Change-Id: Icd82e53ba5836302b036dbe41070d11b2ea4d401 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-11CSIT-1597 API cleanup: tapJan Gelety1-241/+4
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/21706 - cover API cahnges in VPP: https://gerrit.fd.io/r/c/vpp/+/23887 - update vpp stable to version 20.01-rc0~828 Change-Id: I70d88741a02729421d4b31010910138838811518 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-30API: Add collection for 23887/2Jan Gelety1-6/+6
Change-Id: I3c2124aebbab8cac12a7fa661e75b68b70286240 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-10API: Add collection for 21706/17Jan Gelety1-0/+237
Change-Id: Ia4b278a48130426423f33a7cec1e586e8e552a00 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-12-10CSIT-1597 API cleanup: classifyJan Gelety1-241/+4
- cover API changes in VPP: https://gerrit.fd.io/r/c/vpp/+/21551 - move vpp stable build to 20.01-rc0~821 - tap tests tagged 'EXPECTED_FAILING' until necessary vpp api change https://gerrit.fd.io/r/c/vpp/+/21706 and csit code is adapted Change-Id: If4b9b7b5bc207612d08aa8aa4469a5361e058fc3 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-29API: Add collection for 21551/27Jan Gelety1-2/+2
Change-Id: Id4b97a28013741d08d2293714e7d1a9034b0ceed Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-11-14CRC: Update comments after lb-nat4 testsVratko Polak1-9/+5
Change-Id: I5530421ab8b4dd6404d9a7d1939a890b408c0df9 Signed-off-by: Vratko Polak <vrpolak@cisco.com>