aboutsummaryrefslogtreecommitdiffstats
path: root/docs
diff options
context:
space:
mode:
authorVratko Polak <vrpolak@cisco.com>2020-10-28 12:50:10 +0100
committerVratko Polak <vrpolak@cisco.com>2020-10-29 20:26:41 +0000
commit3ccdfe158ce748d1892b66d755643fa88fe5920e (patch)
tree691c1181442e82d958eddc14355a31e4ae023c2b /docs
parente3bcfbf0390898a81b4fdcbc105938102cfec6a2 (diff)
Support existing test types with ASTF
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Diffstat (limited to 'docs')
-rw-r--r--docs/tag_documentation.rst101
1 files changed, 74 insertions, 27 deletions
diff --git a/docs/tag_documentation.rst b/docs/tag_documentation.rst
index 36550d4f5c..ee709d0f44 100644
--- a/docs/tag_documentation.rst
+++ b/docs/tag_documentation.rst
@@ -189,6 +189,36 @@ Scaling Tags
Traffic stream with 100 000 unique flows (100 IPs/users x 1000 UDP ports) in
one direction.
+.. topic:: HOSTS_1024
+
+ Stateless or stateful traffic stream with 1024 client source IP4 addresses,
+ usually with 63 flow differing in source port number. Could be UDP or TCP.
+ If NAT is used, the clients are inside. Outside IP range can differ.
+
+.. topic:: HOSTS_4096
+
+ Stateless or stateful traffic stream with 4096 client source IP4 addresses,
+ usually with 63 flow differing in source port number. Could be UDP or TCP.
+ If NAT is used, the clients are inside. Outside IP range can differ.
+
+.. topic:: HOSTS_16384
+
+ Stateless or stateful traffic stream with 16384 client source IP4 addresses,
+ usually with 63 flow differing in source port number. Could be UDP or TCP.
+ If NAT is used, the clients are inside. Outside IP range can differ.
+
+.. topic:: HOSTS_65536
+
+ Stateless or stateful traffic stream with 65536 client source IP4 addresses,
+ usually with 63 flow differing in source port number. Could be UDP or TCP.
+ If NAT is used, the clients are inside. Outside IP range can differ.
+
+.. topic:: HOSTS_262144
+
+ Stateless or stateful traffic stream with 262144 client source IP4 addresses
+ usually with 63 flow differing in source port number. Could be UDP or TCP.
+ If NAT is used, the clients are inside. Outside IP range can differ.
+
Test Category Tags
------------------
@@ -278,11 +308,12 @@ Test Type Tags
.. topic:: BASE
Baseline test cases, no encapsulation, no feature(s) configured in tests.
+ No scaling whatsoever, beyond minimum needed for RSS.
.. topic:: IP4BASE
IPv4 baseline test cases, no encapsulation, no feature(s) configured in
- tests.
+ tests. Minimal number of routes. Other quantities may be scaled.
.. topic:: IP6BASE
@@ -306,7 +337,8 @@ Test Type Tags
.. topic:: SCALE
- Scale test cases.
+ Scale test cases. Other tags specify which quantities are scaled.
+ Also applies if scaling is set on TG only (e.g. DUT works as IP4BASE).
.. topic:: ENCAP
@@ -316,21 +348,56 @@ Test Type Tags
At least one feature is configured in test cases. Use also feature tag(s).
+.. topic:: UDP
+
+ Tests which use any kind of UDP traffic (STL or ASTF profile).
+
.. topic:: TCP
- Tests which use TCP.
+ Tests which use any kind of TCP traffic (STL or ASTF profile).
+
+..
+ TODO: Should we define tags STL and ASTF?
+
+.. topic:: UDP_UDIR
+
+ Tests which use unidirectional UDP traffic (STL profile only).
+
+.. topic:: UDP_BIDIR
+
+ Tests which use bidirectional UDP traffic (STL profile only).
+
+.. topic:: UDP_CPS
+
+ Tests which measure connections per second on minimal UDP pseudoconnections.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
.. topic:: TCP_CPS
- Performance tests which measure connections per second using http requests.
+ Tests which measure connections per second on empty TCP connections.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
+
+.. topic:: UDP_PPS
-.. topic:: TCP_RPS
+ Tests which measure packets per second on lightweight UDP transactions.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
- Performance tests which measure requests per second using http requests.
+.. topic:: TCP_PPS
+
+ Tests which measure packets per second on lightweight TCP transactions.
+ This implies ASTF traffic profile is used.
+ This tag selects specific output processing in PAL.
.. topic:: HTTP
- Tests which use HTTP.
+ Tests which use traffic formed of valid HTTP requests (and responses).
+
+..
+ TODO: Add HTTP tag to the current hoststack tests.
+ TODO: Document other tags already used by hoststack tests.
.. topic:: NF_DENSITY
@@ -867,23 +934,3 @@ Multi-Threading Tags
8 worker threads each pinned to a separate logical core within 4 dedicated
physical cores. 4 receive queues per interface. Main thread pinned to core
1.
-
-Honeycomb Tags
---------------
-
-.. topic:: HC_FUNC
-
- Honeycomb functional test cases.
-
-.. topic:: HC_NSH
-
- Honeycomb NSH test cases.
-
-.. topic:: HC_PERSIST
-
- Honeycomb persistence test cases.
-
-.. topic:: HC_REST_ONLY
-
- (Exclusion tag) Honeycomb test cases that cannot be run in Netconf mode
- using ODL client for Restfconf -> Netconf translation.