aboutsummaryrefslogtreecommitdiffstats
path: root/docs/tag_documentation.rst
AgeCommit message (Collapse)AuthorFilesLines
2020-10-29Support existing test types with ASTFVratko Polak1-27/+74
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-05-15Performance: Tests with virtio driver in VMPeter Mikus1-4/+10
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I20e01dfe83a961dc8202d33783a678d38e71cff2
2020-03-10Make RXQs/TXQs configurablePeter Mikus1-0/+10
Change-Id: Ib30bc4697fcba93a6723ee492a59a0523425f623 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2020-01-15Remove everything related to TLDKVratko Polak1-4/+0
- Leftovers from kubernetes found, but not removed here. Change-Id: If8cb9269d0f3e69f642d7fe02c59122e17925a4d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-01-09DMM: RemoveTibor Frank1-4/+0
Change-Id: Ibbfbed79e473c804390802ae1ecd737b50c06aa3 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2019-11-13LIGATO: RemovePeter Mikus1-14/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I0c9b8b07ed3a113e24a38faaddd5ed06fae67768
2019-11-13Add vpp loadbalancer l3dsr/nat4 mode test suiteshaiyanx1.zhang1-2/+10
Change-Id: I03603108fe7128fb4bde4abddb6a0959dd5b5c73 Signed-off-by: haiyanx1.zhang <haiyanx1.zhang@intel.com>
2019-10-25Introduce pre-initialize driver layerPeter Mikus1-0/+10
+ This will help unify driver base differences between TCs. + Decrease amount of logic needed for suite generator. Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I1e84ba361dc1e829f0612c58a61096e2633ce0c5
2019-10-09Add vpp loadbalancer maglev mode test suitehaiyanX1.zhang1-0/+4
Change-Id: I61555ba566efef0a2151db9a30bf7f5d9ccad1df Signed-off-by: haiyanx1.zhang <haiyanx1.zhang@intel.com>
2019-09-21Fixing typo tag_documentation for LISPNidhy1-1/+1
Change-Id: Idd715cb0d3e6b60c660b95ac618e8e114419491b Signed-off-by: Nidhy <nidhyanandhan.a@hcl.com>
2019-09-19LISP: 3-node (func) to 2-node (dev) converstionNidhy1-0/+20
Change-Id: I0fbe4d9ef1b30a1ac1382ea43b5b52aeb8a7d39f Signed-off-by: Nidhy <nidhyanandhan.a@hcl.com>
2019-08-23Introduce reconfig suites, for dot1q+ip4+vxlanVratko Polak1-14/+32
Ticket: CSIT-1551 + Config copied from other suites. + Layer keywords prepared for additional chains. + Renamed "create" argument to "vlan_per_chain". + TrafficGenerator result extraction improvements. + Also unified TG type and subtype checking. + Throughput estimated as PDR for zero PLR. + Actual Qemu start is skipped. + Parse xstats instead of stats in trex stop scipt. - Because stats are unconditionally cleared on each script connect. + Remember start values to get better ReceiveMeasurementResult at stop. + Expose loss count and time as appropriate test message. + Autogen support. + Multiple chain amounts and core densities. - Only 1 additional chain. + Proper TAGs. Documented. + Clarified min framesize tags are for TG-DUT. + Added a TODO concerning overhead values. + Added 118B tag. + Robot performance keywords improvements. + Added missing return value documentation. + Frame size as required test variable, not an explicit argument. + MLRsearch keyword to store result (not into test message). - TODO: De-duplicate: main keyword to measure, minor to set test message. - But implementation uses the fact default arguments are different. + Keywords for starting and stopping background traffic. - No checks whether start-stop pair is complete. + Multiple minor indentation and logging improvements. - Missing / TODOs: - Support also vlan_per_chain=True? - Support dot1q without vxlan? - Support also deletion? - How to package config steps so -reconf can be generated from -ndrpdr? Change-Id: If0270697f9ffa837c34ef802917220a53f156703 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-06-19Add scale ipsec tunnel testsPeter Mikus1-2/+2
- 5K - 10K Change-Id: Iaa2c77d07c3686e09fe327c663d7d7fc42024034 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-05-23CSIT-1506: Rework IPsec base testcasesPeter Mikus1-4/+24
- Add: aes-128-gcm aes-128-gcm aes-256-gcm aes-256-gcm aes-128-cbc hmac-sha-256 aes-256-cbc hmac-sha-512 - Remove: hmac-sha1 - Scale will follow in next patch Change-Id: I789f71cf66cf61b8dbb3c6dbe9b6fdc79866ac33 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-26NF density tests with dtc=0.5 and dtcr=2Peter Mikus1-0/+10
Change-Id: Icff556142280ad0b6261e0a2bfb71672ee6b3807 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-08Upgrade autogen to NICs and search typesVratko Polak1-0/+28
+ All perf suites updated, as autogen change is backward incompatible. + Only x710 -ndrpdr suites remain in git repository. + Removed retry feature of run_tests. + Renamed topology_type and traffic_type to traffic_profile. + Renamed framesize to frame_size; nic_model to nic_name. + Reordered Variables table to start with nic_name and overhead. + Fixed wrong overhead value in some cbc-sha1 suites. + Fixed some suite tags. ++ Introduced tags to distinguish suites, such as IPSECINT. + Bound crypto hardware to NIC name. + Implemented NIC limit hiding. + Also search and teardown related arguments are now hidden. + Main measurement keywords updated to new arg handling. + Max rate related keywords moved to performance_limits.robot + Teardowns unified. + Generated tests are archived. ++ Generated directory is .gitignore-d. + Regenerator raises an exception on seeing non-compatible suite. ++ Relatively helpful message should be seen in exception. + Suite and template Documentation has generated parts. +- With short NIC names only. + Autogen checker also upgraded. + Tag expressions replaced with long files to ensure analogous tests. Change-Id: I60e9a999187e7da1f60d0eb4fb02afa14682aa46 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-04-05Fixes for Taishan testbedjuraj.linkes1-4/+0
* rename NICs in Taishan testbed to Intel-X520-DA2 * fix testbed NIC selection * remove unnecessary 82599es tests * blacklist unsupported tests on Taishan testbed * lowercase previously created bash local variables Change-Id: I3b7f59e2edc00929fab2f5a0fce483f236f83075 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-03-01Add ARM Taishan testbed33 perftestStanislav Chlebec1-0/+4
Change-Id: Iaa0e7fbca7de26c7dce8b389cc3494f628a2ecd0 Signed-off-by: Stanislav Chlebec <stanislav.chlebec@pantheon.tech> Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-01-29Add 2n1l-10ge2p1x710-eth-l2xcbase-eth-2vhostvr1024-1vm-soakTibor Frank1-0/+4
Change-Id: I6a045225b0ff3cc4cd9880ee05861fd7bf36c61a Signed-off-by: Tibor Frank <tifrank@cisco.com>
2019-01-23Updated RF tag documentation file with svc density tags, other minor edits.Maciek Konstantynowicz1-47/+60
Change-Id: Iece74d5e7e380db344005a6035252bbe18925646 Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
2019-01-21CSIT-1390 Implement multichain configuration for l2bd with vhost/l3fwd.Peter Mikus1-0/+21
Change-Id: I36ab0bc7c9eada4c3f7bb5cb8f53de7d267ff910 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2018-11-28Add trial duration info to tag documentationVratko Polak1-1/+3
Change-Id: Iad77972ab6a3f4eb4c2e3c09aebfcce0fc5f999b Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2018-11-19Report: Edit the chapters' titlesTibor Frank1-2/+2
Change-Id: I23cedb3ad0f188e9c7570f549e397f4c42404fab Signed-off-by: Tibor Frank <tifrank@cisco.com>
2018-10-10Add VXLAN scale perf testsJan Gelety1-0/+48
Jira: CSIT-1273 Change-Id: Ic2a41661c158384a5aaa7d4e73f30ffca13ddf82 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2018-09-20CSIT-1205 Create AVF driver test - Add Tests L2BD L2XCPeter Mikus1-0/+8
Change-Id: Ia74a87346fe2c0a170b881b510f1b0558a538f3c Signed-off-by: Peter Mikus <pmikus@cisco.com>
2018-08-24VPP link bonding perf tests with two links between DUTsJan Gelety1-0/+8
Jira: CSIT-1247 Change-Id: Ia6a3f7682eb9a35040682913c0e33a1d61d1e6fa Signed-off-by: Jan Gelety <jgelety@cisco.com>
2018-08-08CSIT-1231: Cleanup CSIT RF Tags DescriptionsTibor Frank1-27/+49
Change-Id: Ib57014bf2461518eadcd156b16f8ee26eeb86785 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2018-07-09CSIT-1142 Change thread perf test TAGsPeter Mikus1-18/+23
Due to automatization of SMT detection this change is suppose to remove static thread/core tags in favor of dynamic one. Leaving the static tags for number of physical cores to be able to select TCs. Change-Id: I7f99f605821f363e45c333f46d1dea786693521b Signed-off-by: Peter Mikus <pmikus@cisco.com>
2018-06-15CSIT-1085 eth-l2xcbase-eth-1memif-1dcrPeter Mikus1-0/+6
Test case definitions Change-Id: I03736ef003d3839859dc450745323362b21aa75e Signed-off-by: Peter Mikus <pmikus@cisco.com>
2018-06-01VPP link bonding - lacp modeJan Gelety1-0/+8
CSIT-1072 CSIT-1083 CSIT-1084 Change-Id: Icaf0e09af633d301306a84251f47822e95b411a2 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2018-05-31DPDK link bonding: xor modeJan Gelety1-0/+16
CSIT-1073 CSIT-1074 CSIT-1077 Change-Id: I248e079021c30988210fd792b32d5afdca71aadb Signed-off-by: Jan Gelety <jgelety@cisco.com>
2018-05-04CSIT-992: Add two suites using optimized searchVratko Polak1-0/+6
+ Add 10ge2p1x520-ethip4-ip4base-ndrpdr.robot + Add 10ge2p1x520-eth-l2bdbasemaclrn-eth-2vhostvr1024-1vm-ndrpdr.robot + Make sure both IMIX and 9000B test cases are there. + Document NDRPDR test tag. + Execute some of NDRPDR tests daily. Change-Id: I66427d49cb6b8304fde4e8d91606de33b624186a Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2018-04-23CSIT-1009: SRv6 proxy testsJan Gelety1-0/+20
- SRv6 with endpoint to SR-unaware Service Function via static proxy - SRv6 with endpoint to SR-unaware Service Function via dynamic proxy - SRv6 with endpoint to SR-unaware Service Function via masquerading proxy Change-Id: I6a6062cb41d810bf9e27dacfd866181bfde4c693 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2018-02-23Raw results chk keywords and initial TCPeter Mikus1-0/+10
- Implementation of Display raw results - Implementation of Traffic should pass with maximum rate - Code optimizations - L2BD, L2XC, IP4, IP6 TC Change-Id: I0b50e995a853ed3c8f88323ccc650e47b8ed2496 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2018-01-11CSIT Report updatePeter Mikus1-0/+4
Change-Id: I42efd42ec577723ef2b17a1753ae744986bb66ff Signed-off-by: Peter Mikus <pmikus@cisco.com>
2018-01-10CSIT-866: wrk onboarding in CSITTibor Frank1-0/+16
- CSIT-867: Low Level Description - CSIT-868: wrk traffic profile - parsing - CSIT-869: wrk implementation into CSIT Change-Id: I65e1037f5ae05b3a5b2020e4a6c54462766ae1b4 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2017-11-13Introduce HC_NSH tagMarek Gradzki1-0/+4
NSH is optional HC component so possibility to exclude it (e.g. for local testing) might be usefull. Change-Id: Ib2496009f8839364e50d255fbe231f3addc45bc4 Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
2017-10-20Rename all container based test following name conventionsPeter Mikus1-1/+34
Change-Id: I84f6da2bede47e7ae35e6b4911d425d2402dbd1b Signed-off-by: Peter Mikus <pmikus@cisco.com>
2017-08-28FIX: SNAT -> NAT renaming of APIPeter Mikus1-2/+6
- Based on https://gerrit.fd.io/r/#/c/8146/ Change-Id: I774483c7e2b419a9b9b6be78812ecf53a5c91eab Signed-off-by: Peter Mikus <pmikus@cisco.com>
2017-08-02CSIT-744 Update report content for proper parsingPeter Mikus1-1/+1
Follow the good practices for formatting. Update the content of release report RST files to use the proper directives like :cite:, :footnote:, :abbr:, :captions: etc.With these markings we will be able to format output in proper way and style it in various sphinx output builders. Change-Id: Ibbb538d43c3bd364a6acdcc990097a477f49de00 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2017-08-01CSIT-699: Add l2bd+dot1q perf testsJan Gelety1-6/+9
Change-Id: I3d8cbb235918331b3c60147f7870a4aa90036748 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2017-07-18CSIT-703: Add ACL 2t2c perf test casesJan Gelety1-0/+47
Change-Id: I51f6015681d1fc62da56cf1cca110f7a4c82ba10 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2017-07-17FIX: remove previous QEMU build when needed to change qszJan Gelety1-0/+12
Change-Id: I949f8800b45354c205c5a5d36ab2abeeb93409dc Signed-off-by: Jan Gelety <jgelety@cisco.com>
2017-07-04CSIT-702 Update static content after optimizationsPeter Mikus1-18/+5
Update the mechanics of script for generating report to reflect the latest changes. Update some static content. Change-Id: If60f7b74245138cc3044a49c46ff06d5850732de Signed-off-by: Peter Mikus <pmikus@cisco.com>
2017-07-04CSIT-651 Add keywords and template for memifPeter Mikus1-0/+3
Add keywords and template for memif Change-Id: I8bd5cfe345260750a74930c8ef55690be5f2dd5e Signed-off-by: Peter Mikus <pmikus@cisco.com>
2017-06-07 TLDK udwfwd test case:Qun Wan1-0/+4
- ip fragement for ipv4 - ip reassemble for ipv4 - ip checksum for ipv4 - ip checksum for ipv6 - ip fragment for ipv6 added the genpcap.py script to generate the pcap files and corresponding test scripts Change-Id: I01329dbb0ecd3c3a5dd7a63ef1dea01b2e717da5 Signed-off-by: qun wan <qun.wan@intel.com>
2017-05-17Func test reorg - decrease exec time per vpp patchJan Gelety1-0/+5
- remove tests covered by make test - remove some negative tests Change-Id: I35d641562ee85888f7d1dd60dd892689b1e9c52d Signed-off-by: Jan Gelety <jgelety@cisco.com> Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
2017-05-17CSIT-562 HC Test: Persistence suite rework, part1selias1-0/+17
- rename keywords that check operational data, to distinguish from keywords that check configuration data - modify existing persistence cases so that they don't affect each other - add feature-specific persistence tests for: Interface Management, L2 bridge - move Interface Management variables to separate file so they can be reused - add new tag HC_PERSIST to execute persistence tests separately - update existing Honeycomb tags to comply with CSIT conventions More feature-specific tests to be added in part2. Change-Id: I8657da4bb9e9a3d5261c04f5f1999b2572f605ef Signed-off-by: selias <samelias@cisco.com>
2017-05-11CSIT-574: IPSec Scale File Splitsjan.hradil1-0/+4
split file into two separate files based on number of tunnels values 1 or 1000 corrected name errors - scale -> base Change-Id: I7114c777f3cc3d7f4951a1f59e54f3a7bbe404ec Signed-off-by: jan.hradil <jan.hradil@pantheon.tech>
2017-05-10CSIT-573: Split SNAT scale tests to multiple filesjan.hradil1-0/+20
split file with many test cases into mutliple files based on users and ports added SRC_USER tags removed empty lines at EOF Change-Id: I19c79cdf8d7de4147c98bc061559af1fde10ab8c Signed-off-by: jan.hradil <jan.hradil@pantheon.tech>