From 1daa6fdc0bae284dee1b61f34534e59b60b7526a Mon Sep 17 00:00:00 2001
From: Vratko Polak <vrpolak@cisco.com>
Date: Mon, 25 Apr 2022 10:22:05 +0200
Subject: feat(astf): Support framesizes for ASTF

- No support for IMIX.
+ Fix a bad bug in padding (most ASTF profiles had wrong frame sizes).
+ Fix a big typo in TCP PPS profiles (s->c was not data, just RST).
+ Control transaction size via ASTF_N_DATA_FRAMES env variable.
 - Default value 5 leads to transactions smaller than before.
 + It ensures transaction is one burst (per direction) even for jumbo.
+ Edit autogen to set supported frame sizes based on suite id.
 + Both TCP and UDP use the same values:
  + 64B for CPS (exact for UDP, nominal for TCP).
  + 100B, 1518B and 9000B for TPUT and PPS.
   - TCP TPUT achievable minimum is 70B.
   + Used 100B to leave room for possible IPv6 ASTF tests.
 + Separate function for code reused by vpp and trex tests.
 - I do not really like the new "copy and edit" approach added here.
  + But it is a quick edit, better autogen refactor is low priority.
+ Consider both established and transitory sessions as valid.
 - Mostly for compatibility with 2202 behavior and to avoid ramp-ups.
 - Assuming both session states have similar enough VPP CPU overhead.
 + Added a TODO to investigate and maybe reconsider later.
+ Update the state timeout value to 240s.
 + That is the default for TCP (for transitory state).
 - UDP could keep using 300s.
  + But I prefer UDP and TCP to behave as similarly as possible.
+ Use TRex tunables to get the exact frame size (for data packets).
 - It is not clear why the recipe for MSS has to be this complicated.
 + Move code away from profile init, as frame size is not known there.
 + Change internal profile API, so values related to MSS are passed.
+ Lower ramp-up rate for TCP TPUT tests.
 + Because without lower rate, jumbo fails on packet loss in ramp-up.
 + UDP TPUT ramp-up rate also lowered (just to keep suites more similar).
+ Distinguish one-direction and aggregated average frame size.
 + Update keyword documentation where the distiction matters.
 + One-direction is needed for turning bandwidth limit to TPS limit.
 + Aggregated is needed for correct NDRPDR bandwidth result value.
 - TCP TPUT will always be few percent below bidirectional maximum.
  + That is unavoidable, as one direction sends more control packets.
+ Add runtime consistency checks so future refactors are safer.
 + Fail if padding requested would be negative.
 + Fail if suite claims unexpected values for packets per transaction.
+ Edit the 4 types of ASTF profiles to keep them similar to each other.
 + Move UDP TPUT limit value from a field back to direct argument.
 + Stop pretending first UDP packet is not data.
+ Apply small improvements where convenient.
 + Replace "aggregate" with "aggregated" where possible.
  + To lower probability of any future typos in variable names.
 + Avoid calling Set Numeric Frame Sizes twice.
 + Code formatting, keyword documentation, code comments, ...
 + Add TODOs for less important code quality improvements.
- Postpone updating of methodology pages to a subsequent change.

Change-Id: I4b381e5210e69669f972326202fdcc5a2c9c923b
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
---
 resources/libraries/python/Constants.py            |   8 +
 resources/libraries/python/NATUtil.py              |  28 +--
 resources/libraries/python/TrafficGenerator.py     |  19 +-
 resources/libraries/python/autogen/Regenerator.py  |  64 ++++--
 .../robot/performance/performance_display.robot    |  22 +-
 .../robot/performance/performance_utils.robot      |   6 +-
 .../robot/performance/performance_vars.robot       | 235 +++++++++++++++------
 7 files changed, 266 insertions(+), 116 deletions(-)

(limited to 'resources/libraries')

diff --git a/resources/libraries/python/Constants.py b/resources/libraries/python/Constants.py
index 6bcf5413d9..bfbbfd7471 100644
--- a/resources/libraries/python/Constants.py
+++ b/resources/libraries/python/Constants.py
@@ -269,6 +269,14 @@ class Constants:
         u"PERF_TRIAL_ASTF_DELAY", 0.112
     )
 
+    # Number of data frames in TPUT transaction, used both by TCP and UDP.
+    # The value should be 33 to keep historic continuity for UDP TPUT tests,
+    # but we are limited by TRex window of 48 KiB, so for 9000B tests
+    # it means we can send only 5 full data frames in a burst.
+    # https://github.com/cisco-system-traffic-generator/
+    # trex-core/blob/v2.88/src/44bsd/tcp_var.h#L896-L903
+    ASTF_N_DATA_FRAMES = get_int_from_env(u"ASTF_N_DATA_FRAMES", 5)
+
     # Extended debug (incl. vpp packet trace, linux perf stat, ...).
     # Full list is available as suite variable (__init__.robot) or is
     # override by test.
diff --git a/resources/libraries/python/NATUtil.py b/resources/libraries/python/NATUtil.py
index ceed560a04..841bd2e683 100644
--- a/resources/libraries/python/NATUtil.py
+++ b/resources/libraries/python/NATUtil.py
@@ -278,7 +278,7 @@ class NATUtil:
 
     @staticmethod
     def get_nat44_sessions_number(node, proto):
-        """Get number of established NAT44 sessions from NAT44 mapping data.
+        """Get number of expected NAT44 sessions from NAT44 mapping data.
 
         This keyword uses output from a CLI command,
         so it can start failing when VPP changes the output format.
@@ -287,17 +287,21 @@ class NATUtil:
         The current implementation supports both 2202 and post-2202 format.
         (The Gerrit number changing the output format is 34877.)
 
-        For TCP proto, the post-2202 format includes "timed out"
-        established sessions into its count of total sessions.
+        For TCP proto, the expected state after rampup is
+        some number of sessions in transitory state (VPP has seen the FINs),
+        and some number of sessions in established state (meaning
+        some FINs were lost in the last trial).
+        While the two states may need slightly different number of cycles
+        to process next packet, the current implementation considers
+        both of them the "fast path", so they are both counted as expected.
+
         As the tests should fail if a session is timed-out,
-        the logic substracts timed out sessions for the resturned value.
+        the logic substracts timed out sessions for the returned value
+        (only available for post-2202 format).
 
-        The 2202 output reports most of TCP sessions as in "transitory" state,
-        as opposed to "established", but the previous CSIT logic tolerated that.
-        Ideally, whis keyword would add establised and transitory sessions
-        (but without CLOSED and WAIT_CLOSED sessions) and return that.
-        The current implementation simply returns "total tcp sessions" value,
-        to preserve the previous CSIT behavior for 2202 output.
+        TODO: Investigate if it is worth to insert additional rampup trials
+        in TPUT tests to ensure all sessions are transitory before next
+        measurement.
 
         :param node: DUT node.
         :param proto: Required protocol - TCP/UDP/ICMP.
@@ -328,9 +332,7 @@ class NATUtil:
                     found = True
                 continue
             # Proto is found, find the line we are interested in.
-            if proto_l == u"tcp" and u"established" not in line:
-                continue
-            if u"total" not in line and u"established" not in line:
+            if u"total" not in line:
                 raise RuntimeError(f"show nat summary: no {proto} total.")
             # We have the line with relevant numbers.
             total_part, timed_out_part = line.split(u"(", 1)
diff --git a/resources/libraries/python/TrafficGenerator.py b/resources/libraries/python/TrafficGenerator.py
index b2748f74ba..03e3890959 100644
--- a/resources/libraries/python/TrafficGenerator.py
+++ b/resources/libraries/python/TrafficGenerator.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2021 Cisco and/or its affiliates.
+# Copyright (c) 2022 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -643,6 +643,9 @@ class TrafficGenerator(AbstractMeasurer):
         )
         command_line.add_with_value(u"duration", f"{computed_duration!r}")
         command_line.add_with_value(u"frame_size", self.frame_size)
+        command_line.add_with_value(
+            u"n_data_frames", Constants.ASTF_N_DATA_FRAMES
+        )
         command_line.add_with_value(u"multiplier", multiplier)
         command_line.add_with_value(u"port_0", p_0)
         command_line.add_with_value(u"port_1", p_1)
@@ -811,7 +814,7 @@ class TrafficGenerator(AbstractMeasurer):
             use_latency=False,
             ramp_up_rate=None,
             ramp_up_duration=None,
-            state_timeout=300.0,
+            state_timeout=240.0,
             ramp_up_only=False,
         ):
         """Send traffic from all configured interfaces on TG.
@@ -1215,7 +1218,7 @@ class TrafficGenerator(AbstractMeasurer):
         The target_tr field of ReceiveRateMeasurement is in
         transactions per second. Transmit count and loss count units
         depend on the transaction type. Usually they are in transactions
-        per second, or aggregate packets per second.
+        per second, or aggregated packets per second.
 
         TODO: Fail on running or already reported measurement.
 
@@ -1382,7 +1385,7 @@ class TrafficGenerator(AbstractMeasurer):
             use_latency=False,
             ramp_up_rate=None,
             ramp_up_duration=None,
-            state_timeout=300.0,
+            state_timeout=240.0,
         ):
         """Store values accessed by measure().
 
@@ -1451,7 +1454,7 @@ class OptimizedSearch:
     """Class to be imported as Robot Library, containing search keywords.
 
     Aside of setting up measurer and forwarding arguments,
-    the main business is to translate min/max rate from unidir to aggregate.
+    the main business is to translate min/max rate from unidir to aggregated.
     """
 
     @staticmethod
@@ -1475,7 +1478,7 @@ class OptimizedSearch:
             use_latency=False,
             ramp_up_rate=None,
             ramp_up_duration=None,
-            state_timeout=300.0,
+            state_timeout=240.0,
             expansion_coefficient=4.0,
     ):
         """Setup initialized TG, perform optimized search, return intervals.
@@ -1616,7 +1619,7 @@ class OptimizedSearch:
             use_latency=False,
             ramp_up_rate=None,
             ramp_up_duration=None,
-            state_timeout=300.0,
+            state_timeout=240.0,
     ):
         """Setup initialized TG, perform soak search, return avg and stdev.
 
@@ -1674,7 +1677,7 @@ class OptimizedSearch:
         :type ramp_up_rate: float
         :type ramp_up_duration: float
         :type state_timeout: float
-        :returns: Average and stdev of estimated aggregate rate giving PLR.
+        :returns: Average and stdev of estimated aggregated rate giving PLR.
         :rtype: 2-tuple of float
         """
         tg_instance = BuiltIn().get_library_instance(
diff --git a/resources/libraries/python/autogen/Regenerator.py b/resources/libraries/python/autogen/Regenerator.py
index 3011b06897..4474996ef1 100644
--- a/resources/libraries/python/autogen/Regenerator.py
+++ b/resources/libraries/python/autogen/Regenerator.py
@@ -17,6 +17,7 @@ TODO: How can we check each suite id is unique,
       when currently the suite generation is run on each directory separately?
 """
 
+import copy
 import sys
 
 from glob import glob
@@ -116,6 +117,39 @@ def check_suite_tag(suite_tag, prolog):
         raise ValueError(f"Suite tag found {found} times for {suite_tag}")
 
 
+def filter_and_edit_kwargs_for_astf(suite_id, kwargs):
+    """Return possibly edited kwargs, or None if to be skipped.
+
+    This is a code block used in few places.
+    Kwargs is (a copy of) one item from tc_kwargs_list.
+    Currently, the editable field is frame_size,
+    to be increased to for tests with data (not just CPS).
+
+    :param suite_id: Suite ID.
+    :param kwargs: Key-value pairs used to construct one testcase.
+    :type suite_id: str
+    :type tc_kwargs_list: dict
+    :returns: Edited kwargs.
+    :rtype Optional[dict]
+    """
+    if u"-cps-" in suite_id:
+        # Contrary to UDP, there is no place to affect frame size
+        # in TCP CPS tests. Actual frames are close to min size.
+        # UDP uses the min value too, for fairer comparison to TCP.
+        if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
+            return None
+    elif (u"-pps-" in suite_id or u"-tput-" in suite_id):
+        if u"imix" in str(kwargs[u"frame_size"]).lower():
+            # ASTF does not support IMIX (yet).
+            return None
+        if kwargs[u"frame_size"] in MIN_FRAME_SIZE_VALUES:
+            # Minimal (TRex) TCP data frame is 80B for IPv4.
+            # In future, we may want to have also IPv6 TCP.
+            # UDP uses the same value, for fairer comparison to TCP.
+            kwargs[u"frame_size"] = 100
+    return kwargs
+
+
 def add_default_testcases(testcase, iface, suite_id, file_out, tc_kwargs_list):
     """Add default testcases to file.
 
@@ -130,7 +164,9 @@ def add_default_testcases(testcase, iface, suite_id, file_out, tc_kwargs_list):
     :type file_out: file
     :type tc_kwargs_list: dict
     """
-    for kwargs in tc_kwargs_list:
+    for kwas in tc_kwargs_list:
+        # We may edit framesize for ASTF, the copy should be local.
+        kwargs = copy.deepcopy(kwas)
         # TODO: Is there a better way to disable some combinations?
         emit = True
         if kwargs[u"frame_size"] == 9000:
@@ -156,14 +192,8 @@ def add_default_testcases(testcase, iface, suite_id, file_out, tc_kwargs_list):
                 emit = False
             if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
                 emit = False
-        if (
-                u"-cps-" in suite_id
-                or u"-pps-" in suite_id
-                or u"-tput-" in suite_id
-        ):
-            if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
-                emit = False
-        if emit:
+        kwargs = filter_and_edit_kwargs_for_astf(suite_id, kwargs)
+        if emit and kwargs is not None:
             file_out.write(testcase.generate(**kwargs))
 
 
@@ -207,17 +237,11 @@ def add_trex_testcases(testcase, suite_id, file_out, tc_kwargs_list):
     :type file_out: file
     :type tc_kwargs_list: dict
     """
-    for kwargs in tc_kwargs_list:
-        # TODO: Is there a better way to disable some combinations?
-        emit = True
-        if (
-                u"-cps-" in suite_id
-                or u"-pps-" in suite_id
-                or u"-tput-" in suite_id
-        ):
-            if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
-                emit = False
-        if emit:
+    for kwas in tc_kwargs_list:
+        # We may edit framesize for ASTF, the copy should be local.
+        kwargs = copy.deepcopy(kwas)
+        kwargs = filter_and_edit_kwargs_for_astf(suite_id, kwargs)
+        if kwargs is not None:
             file_out.write(testcase.generate(**kwargs))
 
 
diff --git a/resources/libraries/robot/performance/performance_display.robot b/resources/libraries/robot/performance/performance_display.robot
index db2b522091..a6df6f7b3a 100644
--- a/resources/libraries/robot/performance/performance_display.robot
+++ b/resources/libraries/robot/performance/performance_display.robot
@@ -1,4 +1,4 @@
-# Copyright (c) 2021 Cisco and/or its affiliates.
+# Copyright (c) 2022 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -46,19 +46,19 @@
 | | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other}
 | | Fail | ${message}
 
-| Compute bandwidth
+| Compute Bandwidth
 | | [Documentation]
 | | ... | Compute (bidir) bandwidth from given (unidir) transaction rate.
 | | ...
-| | ... | This keyword reads "ppta" and "avg_frame_size" set elsewhere.
-| | ... | The implementation should work for both pps and cps rates.
+| | ... | This keyword reads \${ppta} and \${avg_aggregated_frame_size} set
+| | ... | elsewhere. The implementation should work for both pps and cps rates.
 | | ... |
 | | ... | *Arguments:*
 | | ... | - tps - Transaction rate (unidirectional) [tps]. Type: float
 | | ...
 | | ... | *Returns:*
 | | ... | - Computed bandwidth in Gbps.
-| | ... | - Computed aggregate packet rate in pps.
+| | ... | - Computed aggregated packet rate in pps.
 | |
 | | ... | *Example:*
 | |
@@ -68,7 +68,7 @@
 | |
 | | ${ppta} = | Get Packets Per Transaction Aggregated
 | | ${pps} = | Evaluate | ${tps} * ${ppta}
-| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9
+| | ${bandwidth} = | Evaluate | ${pps} * (${avg_aggregated_frame_size}+20)*8/1e9
 | | Return From Keyword | ${bandwidth} | ${pps}
 
 | Display Reconfig Test Message
@@ -96,7 +96,7 @@
 | Display result of NDRPDR search
 | | [Documentation]
 | | ... | Display result of NDR+PDR search, both quantities, both bounds,
-| | ... | aggregate in units given by trasaction type, e.g. by default
+| | ... | aggregated, in units given by trasaction type, e.g. by default
 | | ... | in packet per seconds and Gbps total bandwidth
 | | ... | (for initial packet size).
 | | ... |
@@ -115,7 +115,7 @@
 | | ... | - transaction_type - String identifier to determine how to count
 | | ... | transactions. Default is "packet".
 | | ... | *Arguments:*
-| | ... | - result - Measured result data. Aggregate rate, tps or pps.
+| | ... | - result - Measured result data. Aggregated rate, tps or pps.
 | | ... | Type: NdrPdrResult
 | |
 | | ... | *Example:*
@@ -175,7 +175,7 @@
 | | ... | it is in transactions per second. Bidirectional traffic
 | | ... | transaction is understood as having 2 packets, for this purpose.
 | | ... |
-| | ... | Pps values are aggregate in packet per seconds,
+| | ... | Pps values are aggregated, in packet per seconds
 | | ... | and Gbps total bandwidth (for initial packet size).
 | | ... |
 | | ... | Througput is calculated as:
@@ -231,8 +231,8 @@
 
 | Display single pps bound
 | | [Documentation]
-| | ... | Display one pps bound of NDR+PDR search,
-| | ... | aggregate in packet per seconds and Gbps total bandwidth
+| | ... | Display one pps bound of NDR+PDR search, aggregated,
+| | ... | in packet per seconds and Gbps total bandwidth
 | | ... | (for initial packet size).
 | | ... |
 | | ... | The bound to display is given as target transfer rate, it is assumed
diff --git a/resources/libraries/robot/performance/performance_utils.robot b/resources/libraries/robot/performance/performance_utils.robot
index 8350db0205..c90fc7434d 100644
--- a/resources/libraries/robot/performance/performance_utils.robot
+++ b/resources/libraries/robot/performance/performance_utils.robot
@@ -46,7 +46,7 @@
 | | ... | Display results as formatted test message.
 | | ... | Fail if computed lower bound is 110% of the minimal rate or less.
 | | ... | Input rates are unidirectional, in transaction per second.
-| | ... | Reported result may contain aggregate pps rates, depending on test.
+| | ... | Reported result may contain aggregated pps rates, depending on test.
 | | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
 | |
 | | ... | *Test (or broader scope) variables read:*
@@ -105,7 +105,7 @@
 | | ... | Display findings as a formatted test message.
 | | ... | Fail if a resulting lower bound has too high loss ratio.
 | | ... | Input rates are unidirectional, in transaction per second.
-| | ... | Reported result may contain aggregate pps rates, depending on test.
+| | ... | Reported result may contain aggregated pps rates, depending on test.
 | | ... | Additional latency measurements are performed for smaller loads,
 | | ... | even if latency stream is disabled in search. Their results
 | | ... | are also displayed.
@@ -468,7 +468,7 @@
 | | ... | *Arguments:*
 | | ... | - trial_duration - Duration of single trial [s].
 | | ... | Type: float
-| | ... | - rate - Target aggregate transmit rate [bps] / Bits per second
+| | ... | - rate - Target aggregated transmit rate [bps] / Bits per second.
 | | ... | Type: float
 | | ... | - frame_size - L2 Frame Size [B].
 | | ... | Type: integer or string
diff --git a/resources/libraries/robot/performance/performance_vars.robot b/resources/libraries/robot/performance/performance_vars.robot
index 4f2cc50fa3..f75d585e4b 100644
--- a/resources/libraries/robot/performance/performance_vars.robot
+++ b/resources/libraries/robot/performance/performance_vars.robot
@@ -1,4 +1,4 @@
-# Copyright (c) 2021 Cisco and/or its affiliates.
+# Copyright (c) 2022 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -155,11 +155,11 @@
 | Get Packets Per Transaction Aggregated
 | | [Documentation]
 | | ... | Return value of \${packets_per_transaction_aggregated};
-| | ... | if not defined, assume traffic is symmetric and compute
-| | ... | from unidirectional values.
+| | ... | if not defined, assume traffic is symmetric (or unidirectional)
+| | ... | and compute from unidirectional values.
 | |
-| | ... | The return value is used when reporting PPS values from TPS found
-| | ... | by some search (e.g. NDRPDR).
+| | ... | The return value is used when reporting PPS (and bandwidth) values
+| | ... | from TPS found by some search (e.g. NDRPDR).
 | | ... | Return type: integer.
 | |
 | | ... | *Example:*
@@ -169,7 +169,6 @@
 | | ${ppta} = | Get Variable Value | \${packets_per_transaction_aggregated}
 | | ... | ${0}
 | | Return From Keyword If | "${ppta}" != "0" | ${ppta}
-| | # TODO: Insert TCP computation from packet size here.
 | | ${pptad} = | Get Packets Per Transaction And Direction
 | | ${traffic_directions} = | Get Traffic Directions
 | | # We do not support ASTF profiles with multiple transactions, yet.
@@ -181,8 +180,9 @@
 | | ... | Return value of \${packets_per_transaction_and_direction},
 | | ... | or ${1} if not defined.
 | |
-| | ... | The return value is used when computing max rate (TPS),
-| | ... | so for asymmetric transaction use the more numerous direction.
+| | ... | The return value is used when computing max rate (TPS)
+| | ... | from packet level (pps or bps) limits.
+| | ... | For asymmetric transactions, use the more numerous direction.
 | | ... | Return type: integer.
 | |
 | | ... | *Example:*
@@ -392,19 +392,14 @@
 
 | Set Jumbo
 | | [Documentation]
-| | ... | For jumbo frames detection, the maximal packet size is relevant,
-| | ... | encapsulation overhead (if any) has effect.
-| |
-| | ... | This keyword computes jumbo boolean (some suites need that for
-| | ... | configuration decisions).
+| | ... | Call Set Numeric Frame Sizes and set jumbo based on max framesize.
 | | ... | To streamline suite autogeneration, both input and output values
 | | ... | are communicated as test (or broader scope) variables,
 | | ... | instead of explicit arguments and return values.
 | |
 | | ... | *Test (or broader scope) variables read:*
 | | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
+| | ... | - frame_size - Framesize. Type: integer or string
 | |
 | | ... | *Test variables set:*
 | | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
@@ -414,32 +409,22 @@
 | |
 | | ... | \| Set Jumbo \|
 | |
-| | # Already called by Set Max Rate And Jumbo, but some suites (e.g. device)
-| | # are calling this directly.
+| | # Some suites (e.g. device) are not calling Set Max Rate And Jumbo.
 | | Set Numeric Frame Sizes
-| | ${jumbo} = | Set Variable If | ${max_frame_size} < 1522
-| | ... | ${False} | ${True}
+| | ${jumbo} = | Evaluate | ${max_frame_size} >= 1522
 | | Set Test Variable | \${jumbo}
 
 | Set Max Rate And Jumbo
 | | [Documentation]
-| | ... | Input framesize can be either integer in case of a single packet
-| | ... | in stream, or IMIX string defining mix of packets.
-| | ... | For jumbo frames detection, the maximal packet size is relevant.
-| | ... | For maximal transmit rate, the average packet size is relevant.
-| | ... | In both cases, encapsulation overhead (if any) has effect.
-| | ... | The maximal rate is computed from NIC name.
-| | ... | The implementation works by mapping from exact
-| | ... | whitelisted NIC names.
-| | ... | The mapping is hardcoded in nic_limits.yaml
-| | ... | TODO: Make the mapping from NIC names case insensistive.
-| |
 | | ... | This keyword computes maximal unidirectional transmit rate
 | | ... | and jumbo boolean (some suites need that for configuration decisions).
 | | ... | To streamline suite autogeneration, both input and output values
 | | ... | are communicated as test (or broader scope) variables,
 | | ... | instead of explicit arguments and return values.
 | |
+| | ... | For correctly applying bandwidth limit, average frame size is used,
+| | ... | see Set Numeric Frame Sizes keyword documentation for details.
+| |
 | | ... | If this keyword detects the test is interested in (unidirectional)
 | | ... | transactons per second maximal rate (tps), that is returned (not pps).
 | |
@@ -448,8 +433,13 @@
 | | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
 | | ... | string
-| | ... | - packets_per_transaction_and_direction - Pps-tps conversion.
-| | ... | Optional, default 1.
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| | ... | - TEST_TAGS - Robot tags of this test. Type: list of string
 | |
 | | ... | *Test variables set:*
 | | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
@@ -457,8 +447,11 @@
 | | ... | Type: float
 | | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
 | | ... | enabled. Type: boolean
-| | ... | avg_frame_size - Average frame size including overhead. Type: float
-| | ... | max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - avg_directional_frame_size - Average frame size including overhead
+| | ... | for the more loaded direction. Type: float
+| | ... | - avg_aggregated_frame_size - Average frame size including overhead
+| | ... | across both traffic directions. Type: float
 | |
 | | ... | *Example:*
 | |
@@ -470,68 +463,188 @@
 | | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name}
 | | ${bps_limit} = | Get From Dictionary
 | | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name}
-| | Set Numeric Frame Sizes
-| | # We need to add 20B (Ethernet preamble and inter-frame gap)
-| | # to avg_frame_size
-| | ${rate} = | Evaluate | ${bps_limit} / ((${avg_frame_size} + 20.0) * 8)
+| | # Set Jumbo also calls Set Numeric Frame Sizes.
+| | Set Jumbo
+| | # We need to add 20B (Ethernet preamble and inter-frame gap).
+| | ${adfs} = | Get Variable Value | \${avg_directional_frame_size}
+| | ${rate} = | Evaluate | ${bps_limit} / ((${adfs} + 20.0) * 8)
 | | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit}
 | | ... | ${pps_limit} | ${rate}
 | | ${pptad} = | Get Packets Per Transaction And Direction
 | | ${max_rate} = | Evaluate | ${max_rate} / ${pptad}
 | | Set Test Variable | \${max_rate}
-| | Set Jumbo
 
 | Set Numeric Frame Sizes
 | | [Documentation]
 | | ... | Framesize can be either integer in case of a single packet
 | | ... | in stream, or set of packets in case of IMIX type or simmilar.
 | | ... | For jumbo decisions, we need a numeric size of the biggest packet.
-| | ... | For max rate decisions, we need a numeric average packet size.
-| | ... | This keyword computes both and sets them as test variables.
+| | ... | For bandwidth limit decisions, we need a numeric average packet size
+| | ... | in the more bit intensive direction if traffic is non-symmetric.
+| | ... | Computation of max_rate assumes it is also the more pps direction
+| | ... | (so it can have smaller average frame size than the aggregated one).
+| | ... | Average (across both directions) frame size is also used
+| | ... | for displaying the bidirectional bandwidth forwarded.
+| | ... | This keyword computes all three values (accounting for overheads)
+| | ... | and sets them as test variables.
 | |
 | | ... | Each suite sets a value named \${overhead},
 | | ... | which describes by how many bytes the frames on DUT-DUT link
 | | ... | are larger (due to encapsulation) than those
-| | ... | on the primary TG-DUT link. But for some suites that value
+| | ... | on the primary TG-DUT link. For some suites that value
 | | ... | can be negaive (if TG-DUT is encapsulated more heavily).
 | | ... | For calculations in this keyword, we need largest sizes
 | | ... | across links, so zero is used if \${overhead} is negative.
 | |
+| | ... | The other overhead is from TCP control packets (only IPv4 supported).
+| | ... | TCP_CPS tests have SYN frames of length 78B and other frames 70B.
+| | ... | The more loaded is client-to-server direction with 1 SYN and 3 other,
+| | ... | across both directions it is 2 SYN and 5 other.
+| | ... | TCP_PPS and TCP_TPUT tests have one other control packet less
+| | ... | (in the less loaded direction), but they do contain data frames.
+| |
 | | ... | *Test variables read:*
 | | ... | - frame_size - Framesize. Type: integer or string
 | | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| | ... | - TEST_TAGS - Robot tags of this test. Type: list of string
 | |
 | | ... | *Test variables set*
-| | ... | avg_frame_size - Average frame size including overhead. Type: float
-| | ... | max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - avg_directional_frame_size - Average frame size including overhead
+| | ... | for the more loaded direction. Type: float
+| | ... | - avg_aggregated_frame_size - Average frame size including overhead
+| | ... | across both traffic directions. Type: float
 | |
 | | ... | *Example:*
 | |
 | | ... | \| Set Numeric Frame Sizes \|
 | |
+| | ${bare_max_frame_size} = | Run Keyword If
+| | ... | '${frame_size}' == 'IMIX_v4_1' | Set Variable | ${1518.0}
+| | ... | ELSE | Convert To Number | ${frame_size}
+| | ${bafs} = | Run Keyword If
+| | ... | '${frame_size}' == 'IMIX_v4_1' | Set Variable | ${353.8333333333333}
+| | ... | ELSE | Convert To Number | ${frame_size}
+| | # Long boolean formula in 2 lines.
+| | ${is_tcp_pps} = | Evaluate | 'TCP_PPS' in ${TEST_TAGS}
+| | ${is_tcp_tput} = | Evaluate | ${is_tcp_pps} or 'TCP_TPUT' in ${TEST_TAGS}
+| | ${avg_dir_frame_size} | ${avg_agg_frame_size} = | Run Keyword If
+| | ... | 'TCP_CPS' in ${TEST_TAGS} | Apply Tcp Cps Proto Overhead | ${bafs}
+| | ... | ELSE IF | ${is_tcp_tput} | Apply Tcp Tput Proto Overhead | ${bafs}
+| | ... | ELSE | Set Variable | ${bafs} | ${bafs}
 | | ${max_overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
-| | ${bare_avg_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
-| | ... | Set Variable | ${353.83333}
-| | ... | ELSE
-| | ... | Convert To Number | ${frame_size}
-| | # Do not use $max_overhead (without braces), that does not tolerate string.
-| | ${avg_frame_size} = | Evaluate | ${bare_avg_frame_size} + ${max_overhead}
-| | Set Test Variable | \${avg_frame_size}
-| | ${bare_max_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
-| | ... | Set Variable | ${1518}
-| | ... | ELSE
-| | ... | Convert To Number | ${frame_size}
-| | ${max_frame_size} = | Evaluate | $bare_max_frame_size + $max_overhead
-| | Set Test Variable | ${max_frame_size}
+| | ${mfs} = | Evaluate | ${bare_max_frame_size} + ${max_overhead}
+| | ${adfs} = | Evaluate | ${avg_dir_frame_size} + ${max_overhead}
+| | ${aafs} = | Evaluate | ${avg_agg_frame_size} + ${max_overhead}
+| | Set Test Variable | \${max_frame_size} | ${mfs}
+| | Set Test Variable | \${avg_directional_frame_size} | ${adfs}
+| | Set Test Variable | \${avg_aggregated_frame_size} | ${aafs}
+
+| Apply Tcp Cps Proto Overhead
+| | [Documentation]
+| | ... | Recompute average frame size for TCP CPS test cases.
+| |
+| | ... | This is contitionally called from Set Numeric Frame Sizes.
+| | ... | In Robot Framework it is more convenient to wrap such a block
+| | ... | as a standalone keyword to Run Keyword If.
+| |
+| | ... | *Test variables read:*
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Usually set globally. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| |
+| | ... | *Arguments:*
+| | ... | - bare_avg_frame_size - Average numeric framesize without overheads.
+| |
+| | ... | *Returns:*
+| | ... | - avg_dir_frame_size - Average framesize for more loaded direction.
+| | ... | - avg_agg_frame_size - Average framesize across both directions.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${adfs} \| \${aafs} = \| Apply Tcp Cps Proto Overhead \| \${bafs}
+| |
+| | [Arguments] | ${bare_avg_frame_size}
+| |
+| | # Increase max_frame_size for TCP tests if used for more than just jumbo.
+| | Run Keyword If | ${bare_avg_frame_size} != 64
+| | ... | Fail | TCP_CPS tests are only supported for (nominal) 64B frames.
+| | # TODO: Unify with packets_per_transaction_* variables when adding PCAP.
+| | ${pptad} = | Get Packets Per Transaction And Direction
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${avg_dir_frame_size} = | Evaluate | (78.0 * 1 + 70.0 * 3) / (1 + 3)
+| | Run Keyword If | '${pptad}' != '4'
+| | ... | Fail | TCP CPS with pptad '${pptad}' != '4'.
+| | ${avg_agg_frame_size} = | Evaluate | (78.0 * 2 + 70.0 * 5) / (2 + 5)
+| | Run Keyword If | '${ppta}' != '7'
+| | ... | Fail | TCP CPS with ppta '${ppta}' != '7'.
+| | Return From Keyword | ${avg_dir_frame_size} | ${avg_agg_frame_size}
+
+| Apply Tcp Tput Proto Overhead
+| | [Documentation]
+| | ... | Recompute average frame size for TCP TPUT (or PPS) test cases.
+| |
+| | ... | This is contitionally called from Set Numeric Frame Sizes.
+| | ... | In Robot Framework it is more convenient to wrap such a block
+| | ... | as a standalone keyword to Run Keyword If.
+| |
+| | ... | *Test variables read:*
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Usually set globally. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| |
+| | ... | *Arguments:*
+| | ... | - bare_framesize - Average numeric framesize without overheads.
+| |
+| | ... | *Returns:*
+| | ... | - avg_dir_frame_size - Average framesize for more loaded direction.
+| | ... | - avg_agg_frame_size - Average framesize across both directions.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${adfs} \| \${aafs} = \| Apply Tcp Cps Proto Overhead \| \${bafs}
+| |
+| | [Arguments] | ${bare_framesize}
+| |
+| | # TODO: Unify with packets_per_transaction_* variables when adding PCAP.
+| | ${pptad} = | Get Packets Per Transaction And Direction
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | # Long float formula in 4 lines.
+| | ${numerator} = | Evaluate | ${bare_framesize} * ${ASTF_N_DATA_FRAMES}
+| | ${numerator} = | Evaluate | 78.0 * 1 + 70.0 * 3 + ${numerator}
+| | ${denominator} = | Evaluate | 1 + 3 + ${ASTF_N_DATA_FRAMES}
+| | ${avg_dir_frame_size} = | Evaluate | ${numerator} / ${denominator}
+| | Run Keyword If | '${pptad}' != '${denominator}'
+| | ... | Fail | TCP TPUT with pptad '${pptad}' != '${denominator}'.
+| | # Long float formula in 4 lines.
+| | ${numerator} = | Evaluate | ${bare_framesize} * 2 * ${ASTF_N_DATA_FRAMES}
+| | ${numerator} = | Evaluate | 78.0 * 2 + 70.0 * 4 + ${numerator}
+| | ${denominator} = | Evaluate | 2 + 4 + 2 * ${ASTF_N_DATA_FRAMES}
+| | ${avg_agg_frame_size} = | Evaluate | ${numerator} / ${denominator}
+| | Run Keyword If | '${ppta}' != '${denominator}'
+| | ... | Fail | TCP TPUT with ppta '${ppta}' != '${denominator}'.
+| | Return From Keyword | ${avg_dir_frame_size} | ${avg_agg_frame_size}
 
 | Set Rates For Policer
 | | [Documentation]
 | | ... | Policer tests need these values,
-| | ... | currently computed from \${avg_frame_size}.
+| | ... | currently computed from \${avg_directional_frame_size}.
 | | ... | TODO: Verify the units match and computation is correct.
 | |
 | | ... | *Test (or broader scope) variables read:*
-| | ... | - avg_frame_size - Average L2 Frame Size [B]. Type: float
+| | ... | - avg_directional_frame_size - Average L2 Frame Size [B]. Type: float
 | | ... | Set by Set Max Rate And Jumbo keyword.
 | |
 | | ... | *Test variables set:*
@@ -542,5 +655,5 @@
 | |
 | | ... | \| Set Rates For Policer \|
 | |
-| | Set Test Variable | \${eb} | ${avg_frame_size}
-| | Set Test Variable | \${cb} | ${avg_frame_size}
+| | Set Test Variable | \${eb} | ${avg_directional_frame_size}
+| | Set Test Variable | \${cb} | ${avg_directional_frame_size}
-- 
cgit 1.2.3-korg