Support existing test types with ASTF

+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
 + Update existing cps traffic profiles.
 + Add missing traffic profiles.
 + UDP:
  + Single burst of 32 packets was confirmed as safe enough for TRex.
  + Maybe 64 could work, but not enough testing for that.
  + Multiple bursts have lead to reduced TRex performance,
    as overlaping bursts (from different client instances)
    tend to fill up the buffers.
 + TCP:
  + Data size set to 11111 bytes, completely arbitrarily.
  + Results look reasonable, so I have kept that.
  - MSS not set at all
 - No tested support for frame size other than 64B.
  - Frame size does not even factor into TCP profiles.
 + So other frame sizes are skipped in autogen.
+ Update tags in related suites.
 - HOSTS_{n} and SRC_USER_{n} should be unified.
 - Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
 + Resetter is called (if set) before each measurement.
 + If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
 + Transaction corresponds to profile.
 + TRex multiplier argument sets target rate in transactions per second.
 + The familiar STL traffic:
  + Bidirectional is considered to be 2 packets per transaction.
  + Unidirectional is considered to be 1 packet per transaction.
 + The newer ASTF traffic:
  + 4 subtypes, each has different number of packets per transaction.
 + For max rate computation:
  + Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
 + Use the server side value.
  - Client side value is higher by an overhead.
  - TRex is not sending traffic during that time.
 + Remove delays from traffic profiles.
  - Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
 + Only for ASTF profiles, each ASTF profile has limited scale.
 + Scale defined in suite variables.
 + For TRex to send all transactions provided duration value is ignored.
  + The appropriate value is computed in TrafficGenerator.
  + An ad-hoc time constant is added to match the TRex client side time overhead.
  + The profile driver receives the computed duration.
  + Measurement for PLRsearch add a sleep if the computed duration is smaller.
 + Alternative argument for search algos if scale is limited.
  + Both need higher timeout to accomodate big scales.
  + MLRsearch can afford fewer phases.
 + Added a parameter to optionally shorten the duration.
  + Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
 - Polling could eliminate the time overhead value.
 + But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
 + The stop has several steps so that:
  + The traffic is really stopped entirely.
  + Late packets do not count (maybe as errors).
  + Stats are preserved to read for results (and cleared afterwards).
 + Several quantities added to ReceiveRateMeasurement:
  + Original target duration is preserved (algos need that).
  + Input estimate (tps) for early search iterations.
  + Output estimate (maybe pps) for MRR output.
  + Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
 - TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
 + Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
 + Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
 + Rename argument names to min_rate and max_rate.
 + Use relative receive rate in initial phase.
+ PLRsearch improvements:
 + Careful computation when output (pps) does not match input (tps).
 + Use geometric distribution (instead of Poisson).
  + Helps agains math errors.
  + This should improve estimate stability.
  - But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
 + send_traffic_on_tg now calls the full set_rate_provider_defaults.
 + _send_traffic_on_tg_internal for the logic without provider defaults.
  + As the internal function is re-used by measure() without affecting defaults.
 + Move _parse_traffic_results just before get_measurement_result.
  + As the latter uses fields set bu the former, it is now easier to read.
 + Multiple sources for approximate duration.
  + Tried from more precise to more available.
 + Includes logic for _pps tests (added in later change).
 + Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
 + performance_actions.robot:
  + Hosts Additional Statistics Action For * keywords.
 + performance_display.robot:
  + Hosts keyword for displaying and verifying results.
  + Change test message to use the correct unit (pps or cps).
 + performance_limits.robot renamed to performance_vars.robot
  + Added many keywords, mostly for accessing test variables.
   + Moved variables for Policer into a new keyword there.
  + Some keywords need sophisticated logic.
  - Other are basically Get Variable Value.
   + But in future more logic can be added, without editing callers.
  + Documentation for the new keywords acts as a documentation for test variables.
 + performance_utils.robot has the rest.
  + Eliminated arguments if the value is in test variable.
  + Small improvements to documentation.
  - Still not enough cleanup with respect to arguments and test variables.
 + Keywords are sorted alphabetically now in each one.
+ Suites:
 + Unified variables table:
  + No colons in comments.
  + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
 + Add -cps to existing cps suite names.
  + Remove "trial data overwrite".
  + Compute max rate as in STL suites.
 + Each NAT suite has ip4base suite to compare results to.
  - Those act as indirect TRex calibration.
  - VPP does not lose packets in those.
 + Latency in ASTF suites is disabled hard.
  - As we do not support latency in ASTF profiles yet.
 + Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
 + Prefer to use argument names.
 + In Python, also the last argument is followed by comma.
 + It makes renaming and reordering easier.
 + Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.

Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

4 files changed, 89 insertions, 32 deletions

diff --git a/resources/libraries/python/MLRsearch/AbstractMeasurer.py b/resources/libraries/python/MLRsearch/AbstractMeasurer.py
index 622b8fdba6..82116f2e43 100644
--- a/resources/libraries/python/MLRsearch/AbstractMeasurer.py
+++ b/resources/libraries/python/MLRsearch/AbstractMeasurer.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2019 Cisco and/or its affiliates.
+# Copyright (c) 2020 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -24,7 +24,7 @@ class AbstractMeasurer(metaclass=ABCMeta):
         """Perform trial measurement and return the result.
 
         :param duration: Trial duration [s].
-        :param transmit_rate: Target transmit rate [pps].
+        :param transmit_rate: Target transmit rate [tps].
         :type duration: float
         :type transmit_rate: float
         :returns: Structure containing the result of the measurement.
diff --git a/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py b/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py
index f4f2d3f096..f2bf04e1b1 100644
--- a/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py
+++ b/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2019 Cisco and/or its affiliates.
+# Copyright (c) 2020 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -30,16 +30,19 @@ class AbstractSearchAlgorithm(metaclass=ABCMeta):
 
     @abstractmethod
     def narrow_down_ndr_and_pdr(
-            self, fail_rate, line_rate, packet_loss_ratio):
+            self, min_rate, max_rate, packet_loss_ratio):
         """Perform measurements to narrow down intervals, return them.
 
         This will be renamed when custom loss ratio lists are supported.
 
-        :param fail_rate: Minimal target transmit rate [pps].
-        :param line_rate: Maximal target transmit rate [pps].
+        :param min_rate: Minimal target transmit rate [tps].
+            Usually, tests are set to fail if search reaches this or below.
+        :param max_rate: Maximal target transmit rate [tps].
+            Usually computed from line rate and various other limits,
+            to prevent failures or duration stretching in Traffic Generator.
         :param packet_loss_ratio: Fraction of packets lost, for PDR [1].
-        :type fail_rate: float
-        :type line_rate: float
+        :type min_rate: float
+        :type max_rate: float
         :type packet_loss_ratio: float
         :returns: Structure containing narrowed down intervals
             and their measurements.
diff --git a/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py b/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py
index 2db65ef755..87dc784cbc 100644
--- a/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py
+++ b/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py
@@ -157,7 +157,7 @@ class MultipleLossRatioSearch(AbstractSearchAlgorithm):
         :returns: The relative width of double logarithmic size.
         :rtype: float
         """
-        return 1.999 * relative_width - relative_width * relative_width
+        return 1.99999 * relative_width - relative_width * relative_width
         # The number should be 2.0, but we want to avoid rounding errors,
         # and ensure half of double is not larger than the original value.
 
@@ -255,52 +255,50 @@ class MultipleLossRatioSearch(AbstractSearchAlgorithm):
             1.0 - MultipleLossRatioSearch.half_relative_width(relative_width)
         )
 
-    def narrow_down_ndr_and_pdr(
-            self, minimum_transmit_rate, maximum_transmit_rate,
-            packet_loss_ratio):
+    def narrow_down_ndr_and_pdr(self, min_rate, max_rate, packet_loss_ratio):
         """Perform initial phase, create state object, proceed with next phases.
 
-        :param minimum_transmit_rate: Minimal target transmit rate [pps].
-        :param maximum_transmit_rate: Maximal target transmit rate [pps].
+        :param min_rate: Minimal target transmit rate [tps].
+        :param max_rate: Maximal target transmit rate [tps].
         :param packet_loss_ratio: Fraction of packets lost, for PDR [1].
-        :type minimum_transmit_rate: float
-        :type maximum_transmit_rate: float
+        :type min_rate: float
+        :type max_rate: float
         :type packet_loss_ratio: float
         :returns: Structure containing narrowed down intervals
             and their measurements.
         :rtype: NdrPdrResult.NdrPdrResult
         :raises RuntimeError: If total duration is larger than timeout.
         """
-        minimum_transmit_rate = float(minimum_transmit_rate)
-        maximum_transmit_rate = float(maximum_transmit_rate)
+        minimum_transmit_rate = float(min_rate)
+        maximum_transmit_rate = float(max_rate)
         packet_loss_ratio = float(packet_loss_ratio)
-        line_measurement = self.measurer.measure(
+        max_measurement = self.measurer.measure(
             self.initial_trial_duration, maximum_transmit_rate)
         initial_width_goal = self.final_relative_width
         for _ in range(self.number_of_intermediate_phases):
             initial_width_goal = self.double_relative_width(initial_width_goal)
         max_lo = maximum_transmit_rate * (1.0 - initial_width_goal)
-        mrr = max(
-            minimum_transmit_rate, min(max_lo, line_measurement.receive_rate)
-        )
+        mrr = max(minimum_transmit_rate, min(
+            max_lo, max_measurement.relative_receive_rate
+        ))
         mrr_measurement = self.measurer.measure(
             self.initial_trial_duration, mrr
         )
         # Attempt to get narrower width.
         if mrr_measurement.loss_fraction > 0.0:
             max2_lo = mrr * (1.0 - initial_width_goal)
-            mrr2 = min(max2_lo, mrr_measurement.receive_rate)
+            mrr2 = min(max2_lo, mrr_measurement.relative_receive_rate)
         else:
             mrr2 = mrr / (1.0 - initial_width_goal)
         if minimum_transmit_rate < mrr2 < maximum_transmit_rate:
-            line_measurement = mrr_measurement
+            max_measurement = mrr_measurement
             mrr_measurement = self.measurer.measure(
                 self.initial_trial_duration, mrr2)
             if mrr2 > mrr:
-                line_measurement, mrr_measurement = \
-                    (mrr_measurement, line_measurement)
+                max_measurement, mrr_measurement = \
+                    (mrr_measurement, max_measurement)
         starting_interval = ReceiveRateInterval(
-            mrr_measurement, line_measurement)
+            mrr_measurement, max_measurement)
         starting_result = NdrPdrResult(starting_interval, starting_interval)
         state = self.ProgressState(
             starting_result, self.number_of_intermediate_phases,
diff --git a/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py b/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py
index 31a6f8202e..c732e66026 100644
--- a/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py
+++ b/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2019 Cisco and/or its affiliates.
+# Copyright (c) 2020 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -17,18 +17,39 @@
 class ReceiveRateMeasurement:
     """Structure defining the result of single Rr measurement."""
 
-    def __init__(self, duration, target_tr, transmit_count, loss_count):
+    def __init__(
+            self, duration, target_tr, transmit_count, loss_count,
+            approximated_duration=0.0, partial_transmit_count=0):
         """Constructor, normalize primary and compute secondary quantities.
 
+        If approximated_duration is nonzero, it is stored.
+        If approximated_duration is zero, duration value is stored.
+        Either way, additional secondary quantities are computed
+        from the store value.
+
+        If there is zero transmit_count, fractions are set to zero.
+
+        In some cases, traffic generator does not attempt all the needed
+        transactions. In that case, nonzero partial_transmit_count
+        holds (an estimate of) count of the actually attempted transactions.
+        This is used to populate some secondary quantities.
+
+        TODO: Use None instead of zero?
+
         :param duration: Measurement duration [s].
         :param target_tr: Target transmit rate [pps].
             If bidirectional traffic is measured, this is bidirectional rate.
         :param transmit_count: Number of packets transmitted [1].
         :param loss_count: Number of packets transmitted but not received [1].
+        :param approximated_duration: Estimate of the actual time of the trial.
+        :param partial_transmit_count: Estimate count of actually attempted
+            transactions.
         :type duration: float
         :type target_tr: float
         :type transmit_count: int
         :type loss_count: int
+        :type approximated_duration: float
+        :type partial_transmit_count: int
         """
         self.duration = float(duration)
         self.target_tr = float(target_tr)
@@ -38,8 +59,41 @@ class ReceiveRateMeasurement:
         self.transmit_rate = transmit_count / self.duration
         self.loss_rate = loss_count / self.duration
         self.receive_rate = self.receive_count / self.duration
-        self.loss_fraction = float(self.loss_count) / self.transmit_count
-        # TODO: Do we want to store also the real time (duration + overhead)?
+        self.loss_fraction = (
+            float(self.loss_count) / self.transmit_count
+            if self.transmit_count > 0 else 1.0
+        )
+        self.receive_fraction = (
+            float(self.receive_count) / self.transmit_count
+            if self.transmit_count > 0 else 0.0
+        )
+        self.approximated_duration = (
+            float(approximated_duration) if approximated_duration
+            else self.duration
+        )
+        self.approximated_receive_rate = (
+            self.receive_count / self.approximated_duration
+            if self.approximated_duration > 0.0 else 0.0
+        )
+        # If the traffic generator is unreliable and sends less packets,
+        # the absolute receive rate might be too low for next target.
+        self.partial_transmit_count = (
+            int(partial_transmit_count) if partial_transmit_count
+            else self.transmit_count
+        )
+        self.partial_receive_fraction = (
+            float(self.receive_count) / self.partial_transmit_count
+            if self.partial_transmit_count > 0 else 0.0
+        )
+        self.partial_receive_rate = (
+            self.target_tr * self.partial_receive_fraction
+        )
+        # We use relative packet ratios in order to support cases
+        # where target_tr is in transactions per second,
+        # but there are multiple packets per transaction.
+        self.relative_receive_rate = (
+            self.target_tr * self.receive_count / self.transmit_count
+        )
 
     def __str__(self):
         """Return string reporting input and loss fraction."""
@@ -51,4 +105,6 @@ class ReceiveRateMeasurement:
         return f"ReceiveRateMeasurement(duration={self.duration!r}," \
             f"target_tr={self.target_tr!r}," \
             f"transmit_count={self.transmit_count!r}," \
-            f"loss_count={self.loss_count!r})"
+            f"loss_count={self.loss_count!r}," \
+            f"approximated_duration={self.approximated_duration!r}," \
+            f"partial_transmit_count={self.partial_transmit_count!r})"