Support existing test types with ASTF

+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
 + Update existing cps traffic profiles.
 + Add missing traffic profiles.
 + UDP:
  + Single burst of 32 packets was confirmed as safe enough for TRex.
  + Maybe 64 could work, but not enough testing for that.
  + Multiple bursts have lead to reduced TRex performance,
    as overlaping bursts (from different client instances)
    tend to fill up the buffers.
 + TCP:
  + Data size set to 11111 bytes, completely arbitrarily.
  + Results look reasonable, so I have kept that.
  - MSS not set at all
 - No tested support for frame size other than 64B.
  - Frame size does not even factor into TCP profiles.
 + So other frame sizes are skipped in autogen.
+ Update tags in related suites.
 - HOSTS_{n} and SRC_USER_{n} should be unified.
 - Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
 + Resetter is called (if set) before each measurement.
 + If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
 + Transaction corresponds to profile.
 + TRex multiplier argument sets target rate in transactions per second.
 + The familiar STL traffic:
  + Bidirectional is considered to be 2 packets per transaction.
  + Unidirectional is considered to be 1 packet per transaction.
 + The newer ASTF traffic:
  + 4 subtypes, each has different number of packets per transaction.
 + For max rate computation:
  + Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
 + Use the server side value.
  - Client side value is higher by an overhead.
  - TRex is not sending traffic during that time.
 + Remove delays from traffic profiles.
  - Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
 + Only for ASTF profiles, each ASTF profile has limited scale.
 + Scale defined in suite variables.
 + For TRex to send all transactions provided duration value is ignored.
  + The appropriate value is computed in TrafficGenerator.
  + An ad-hoc time constant is added to match the TRex client side time overhead.
  + The profile driver receives the computed duration.
  + Measurement for PLRsearch add a sleep if the computed duration is smaller.
 + Alternative argument for search algos if scale is limited.
  + Both need higher timeout to accomodate big scales.
  + MLRsearch can afford fewer phases.
 + Added a parameter to optionally shorten the duration.
  + Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
 - Polling could eliminate the time overhead value.
 + But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
 + The stop has several steps so that:
  + The traffic is really stopped entirely.
  + Late packets do not count (maybe as errors).
  + Stats are preserved to read for results (and cleared afterwards).
 + Several quantities added to ReceiveRateMeasurement:
  + Original target duration is preserved (algos need that).
  + Input estimate (tps) for early search iterations.
  + Output estimate (maybe pps) for MRR output.
  + Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
 - TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
 + Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
 + Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
 + Rename argument names to min_rate and max_rate.
 + Use relative receive rate in initial phase.
+ PLRsearch improvements:
 + Careful computation when output (pps) does not match input (tps).
 + Use geometric distribution (instead of Poisson).
  + Helps agains math errors.
  + This should improve estimate stability.
  - But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
 + send_traffic_on_tg now calls the full set_rate_provider_defaults.
 + _send_traffic_on_tg_internal for the logic without provider defaults.
  + As the internal function is re-used by measure() without affecting defaults.
 + Move _parse_traffic_results just before get_measurement_result.
  + As the latter uses fields set bu the former, it is now easier to read.
 + Multiple sources for approximate duration.
  + Tried from more precise to more available.
 + Includes logic for _pps tests (added in later change).
 + Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
 + performance_actions.robot:
  + Hosts Additional Statistics Action For * keywords.
 + performance_display.robot:
  + Hosts keyword for displaying and verifying results.
  + Change test message to use the correct unit (pps or cps).
 + performance_limits.robot renamed to performance_vars.robot
  + Added many keywords, mostly for accessing test variables.
   + Moved variables for Policer into a new keyword there.
  + Some keywords need sophisticated logic.
  - Other are basically Get Variable Value.
   + But in future more logic can be added, without editing callers.
  + Documentation for the new keywords acts as a documentation for test variables.
 + performance_utils.robot has the rest.
  + Eliminated arguments if the value is in test variable.
  + Small improvements to documentation.
  - Still not enough cleanup with respect to arguments and test variables.
 + Keywords are sorted alphabetically now in each one.
+ Suites:
 + Unified variables table:
  + No colons in comments.
  + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
 + Add -cps to existing cps suite names.
  + Remove "trial data overwrite".
  + Compute max rate as in STL suites.
 + Each NAT suite has ip4base suite to compare results to.
  - Those act as indirect TRex calibration.
  - VPP does not lose packets in those.
 + Latency in ASTF suites is disabled hard.
  - As we do not support latency in ASTF profiles yet.
 + Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
 + Prefer to use argument names.
 + In Python, also the last argument is followed by comma.
 + It makes renaming and reordering easier.
 + Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.

Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

1 files changed, 35 insertions, 61 deletions

diff --git a/GPL/tools/trex/trex_stl_profile.py b/GPL/tools/trex/trex_stl_profile.py
index ac09b8dda1..ea001b58b6 100644
--- a/GPL/tools/trex/trex_stl_profile.py
+++ b/GPL/tools/trex/trex_stl_profile.py
@@ -60,8 +60,17 @@ def fmt_latency(lat_min, lat_avg, lat_max, hdrh):
 
 
 def simple_burst(
-        profile_file, duration, framesize, rate, warmup_time, port_0, port_1,
-        latency, async_start=False, traffic_directions=2, force=False):
+        profile_file,
+        duration,
+        framesize,
+        rate,
+        port_0,
+        port_1,
+        latency,
+        async_start=False,
+        traffic_directions=2,
+        force=False,
+    ):
     """Send traffic and measure packet loss and latency.
 
     Procedure:
@@ -83,7 +92,6 @@ def simple_burst(
     :param framesize: Frame size.
     :param duration: Duration of traffic run in seconds (-1=infinite).
     :param rate: Traffic rate [percentage, pps, bps].
-    :param warmup_time: Traffic warm-up time in seconds, 0 = disable.
     :param port_0: Port 0 on the traffic generator.
     :param port_1: Port 1 on the traffic generator.
     :param latency: With latency stats.
@@ -94,7 +102,6 @@ def simple_burst(
     :type framesize: int or str
     :type duration: float
     :type rate: str
-    :type warmup_time: float
     :type port_0: int
     :type port_1: int
     :type latency: bool
@@ -105,8 +112,7 @@ def simple_burst(
     client = None
     total_rcvd = 0
     total_sent = 0
-    approximated_duration = 0
-    approximated_rate = 0
+    approximated_duration = 0.0
     lost_a = 0
     lost_b = 0
     lat_a = u"-1/-1/-1/"
@@ -162,40 +168,6 @@ def simple_burst(
         ports = [port_0]
         if traffic_directions > 1:
             ports.append(port_1)
-        # Warm-up phase:
-        if warmup_time > 0:
-            # Clear the stats before injecting:
-            client.clear_stats()
-
-            # Choose rate and start traffic:
-            client.start(
-                ports=ports, mult=rate, duration=warmup_time, force=force,
-                core_mask=STLClient.CORE_MASK_PIN
-            )
-
-            # Block until done:
-            time_start = time.monotonic()
-            client.wait_on_traffic(ports=ports, timeout=warmup_time+30)
-            time_stop = time.monotonic()
-            approximated_duration = time_stop - time_start
-
-            if client.get_warnings():
-                for warning in client.get_warnings():
-                    print(warning)
-
-            # Read the stats after the test:
-            stats = client.get_stats()
-
-            print(u"##### Warmup statistics #####")
-            print(json.dumps(stats, indent=4, separators=(u",", u": ")))
-
-            lost_a = stats[port_0][u"opackets"] - stats[port_1][u"ipackets"]
-            if traffic_directions > 1:
-                lost_b = stats[port_1][u"opackets"] - stats[port_0][u"ipackets"]
-
-            print(f"\npackets lost from {port_0} --> {port_1}: {lost_a} pkts")
-            if traffic_directions > 1:
-                print(f"packets lost from {port_1} --> {port_0}: {lost_b} pkts")
 
         # Clear the stats before injecting:
         client.clear_stats()
@@ -204,8 +176,11 @@ def simple_burst(
 
         # Choose rate and start traffic:
         client.start(
-            ports=ports, mult=rate, duration=duration, force=force,
-            core_mask=STLClient.CORE_MASK_PIN
+            ports=ports,
+            mult=rate,
+            duration=duration,
+            force=force,
+            core_mask=STLClient.CORE_MASK_PIN,
         )
 
         if async_start:
@@ -254,10 +229,6 @@ def simple_burst(
             else:
                 total_sent = stats[port_0][u"opackets"]
                 total_rcvd = stats[port_1][u"ipackets"]
-            try:
-                approximated_rate = total_sent / approximated_duration
-            except ZeroDivisionError:
-                pass
 
             print(f"\npackets lost from {port_0} --> {port_1}: {lost_a} pkts")
             if traffic_directions > 1:
@@ -275,13 +246,14 @@ def simple_burst(
             if client:
                 client.disconnect()
             print(
-                f"rate={rate!r}, total_received={total_rcvd}, "
-                f"total_sent={total_sent}, frame_loss={lost_a + lost_b}, "
-                f"target_duration={duration!r}, "
-                f"approximated_duration={approximated_duration!r}, "
-                f"approximated_rate={approximated_rate}, "
-                f"latency_stream_0(usec)={lat_a}, "
-                f"latency_stream_1(usec)={lat_b}, "
+                f"rate={rate!r}; "
+                f"total_received={total_rcvd}; "
+                f"total_sent={total_sent}; "
+                f"frame_loss={lost_a + lost_b}; "
+                f"target_duration={duration!r}; "
+                f"approximated_duration={approximated_duration!r}; "
+                f"latency_stream_0(usec)={lat_a}; "
+                f"latency_stream_1(usec)={lat_b}; "
             )
 
 
@@ -309,10 +281,6 @@ def main():
         help=u"Traffic rate with included units (pps)."
     )
     parser.add_argument(
-        u"-w", u"--warmup_time", type=float, default=5.0,
-        help=u"Traffic warm-up time in seconds, 0 = disable."
-    )
-    parser.add_argument(
         u"--port_0", required=True, type=int,
         help=u"Port 0 on the traffic generator."
     )
@@ -345,10 +313,16 @@ def main():
         framesize = args.frame_size
 
     simple_burst(
-        profile_file=args.profile, duration=args.duration, framesize=framesize,
-        rate=args.rate, warmup_time=args.warmup_time, port_0=args.port_0,
-        port_1=args.port_1, latency=args.latency, async_start=args.async_start,
-        traffic_directions=args.traffic_directions, force=args.force
+        profile_file=args.profile,
+        duration=args.duration,
+        framesize=framesize,
+        rate=args.rate,
+        port_0=args.port_0,
+        port_1=args.port_1,
+        latency=args.latency,
+        async_start=args.async_start,
+        traffic_directions=args.traffic_directions,
+        force=args.force,
     )