aboutsummaryrefslogtreecommitdiffstats
path: root/resources
diff options
context:
space:
mode:
authorVratko Polak <vrpolak@cisco.com>2022-04-25 10:22:05 +0200
committerVratko Polak <vrpolak@cisco.com>2022-05-03 15:23:47 +0000
commit1daa6fdc0bae284dee1b61f34534e59b60b7526a (patch)
tree6314e011a8f029c4332586d5fd05b51d1a69a89c /resources
parentcd417be7f836eb9346fad4f87bd4f75dc1d9a429 (diff)
feat(astf): Support framesizes for ASTF
- No support for IMIX. + Fix a bad bug in padding (most ASTF profiles had wrong frame sizes). + Fix a big typo in TCP PPS profiles (s->c was not data, just RST). + Control transaction size via ASTF_N_DATA_FRAMES env variable. - Default value 5 leads to transactions smaller than before. + It ensures transaction is one burst (per direction) even for jumbo. + Edit autogen to set supported frame sizes based on suite id. + Both TCP and UDP use the same values: + 64B for CPS (exact for UDP, nominal for TCP). + 100B, 1518B and 9000B for TPUT and PPS. - TCP TPUT achievable minimum is 70B. + Used 100B to leave room for possible IPv6 ASTF tests. + Separate function for code reused by vpp and trex tests. - I do not really like the new "copy and edit" approach added here. + But it is a quick edit, better autogen refactor is low priority. + Consider both established and transitory sessions as valid. - Mostly for compatibility with 2202 behavior and to avoid ramp-ups. - Assuming both session states have similar enough VPP CPU overhead. + Added a TODO to investigate and maybe reconsider later. + Update the state timeout value to 240s. + That is the default for TCP (for transitory state). - UDP could keep using 300s. + But I prefer UDP and TCP to behave as similarly as possible. + Use TRex tunables to get the exact frame size (for data packets). - It is not clear why the recipe for MSS has to be this complicated. + Move code away from profile init, as frame size is not known there. + Change internal profile API, so values related to MSS are passed. + Lower ramp-up rate for TCP TPUT tests. + Because without lower rate, jumbo fails on packet loss in ramp-up. + UDP TPUT ramp-up rate also lowered (just to keep suites more similar). + Distinguish one-direction and aggregated average frame size. + Update keyword documentation where the distiction matters. + One-direction is needed for turning bandwidth limit to TPS limit. + Aggregated is needed for correct NDRPDR bandwidth result value. - TCP TPUT will always be few percent below bidirectional maximum. + That is unavoidable, as one direction sends more control packets. + Add runtime consistency checks so future refactors are safer. + Fail if padding requested would be negative. + Fail if suite claims unexpected values for packets per transaction. + Edit the 4 types of ASTF profiles to keep them similar to each other. + Move UDP TPUT limit value from a field back to direct argument. + Stop pretending first UDP packet is not data. + Apply small improvements where convenient. + Replace "aggregate" with "aggregated" where possible. + To lower probability of any future typos in variable names. + Avoid calling Set Numeric Frame Sizes twice. + Code formatting, keyword documentation, code comments, ... + Add TODOs for less important code quality improvements. - Postpone updating of methodology pages to a subsequent change. Change-Id: I4b381e5210e69669f972326202fdcc5a2c9c923b Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Diffstat (limited to 'resources')
-rw-r--r--resources/libraries/python/Constants.py8
-rw-r--r--resources/libraries/python/NATUtil.py28
-rw-r--r--resources/libraries/python/TrafficGenerator.py19
-rw-r--r--resources/libraries/python/autogen/Regenerator.py64
-rw-r--r--resources/libraries/robot/performance/performance_display.robot22
-rw-r--r--resources/libraries/robot/performance/performance_utils.robot6
-rw-r--r--resources/libraries/robot/performance/performance_vars.robot235
7 files changed, 266 insertions, 116 deletions
diff --git a/resources/libraries/python/Constants.py b/resources/libraries/python/Constants.py
index 6bcf5413d9..bfbbfd7471 100644
--- a/resources/libraries/python/Constants.py
+++ b/resources/libraries/python/Constants.py
@@ -269,6 +269,14 @@ class Constants:
u"PERF_TRIAL_ASTF_DELAY", 0.112
)
+ # Number of data frames in TPUT transaction, used both by TCP and UDP.
+ # The value should be 33 to keep historic continuity for UDP TPUT tests,
+ # but we are limited by TRex window of 48 KiB, so for 9000B tests
+ # it means we can send only 5 full data frames in a burst.
+ # https://github.com/cisco-system-traffic-generator/
+ # trex-core/blob/v2.88/src/44bsd/tcp_var.h#L896-L903
+ ASTF_N_DATA_FRAMES = get_int_from_env(u"ASTF_N_DATA_FRAMES", 5)
+
# Extended debug (incl. vpp packet trace, linux perf stat, ...).
# Full list is available as suite variable (__init__.robot) or is
# override by test.
diff --git a/resources/libraries/python/NATUtil.py b/resources/libraries/python/NATUtil.py
index ceed560a04..841bd2e683 100644
--- a/resources/libraries/python/NATUtil.py
+++ b/resources/libraries/python/NATUtil.py
@@ -278,7 +278,7 @@ class NATUtil:
@staticmethod
def get_nat44_sessions_number(node, proto):
- """Get number of established NAT44 sessions from NAT44 mapping data.
+ """Get number of expected NAT44 sessions from NAT44 mapping data.
This keyword uses output from a CLI command,
so it can start failing when VPP changes the output format.
@@ -287,17 +287,21 @@ class NATUtil:
The current implementation supports both 2202 and post-2202 format.
(The Gerrit number changing the output format is 34877.)
- For TCP proto, the post-2202 format includes "timed out"
- established sessions into its count of total sessions.
+ For TCP proto, the expected state after rampup is
+ some number of sessions in transitory state (VPP has seen the FINs),
+ and some number of sessions in established state (meaning
+ some FINs were lost in the last trial).
+ While the two states may need slightly different number of cycles
+ to process next packet, the current implementation considers
+ both of them the "fast path", so they are both counted as expected.
+
As the tests should fail if a session is timed-out,
- the logic substracts timed out sessions for the resturned value.
+ the logic substracts timed out sessions for the returned value
+ (only available for post-2202 format).
- The 2202 output reports most of TCP sessions as in "transitory" state,
- as opposed to "established", but the previous CSIT logic tolerated that.
- Ideally, whis keyword would add establised and transitory sessions
- (but without CLOSED and WAIT_CLOSED sessions) and return that.
- The current implementation simply returns "total tcp sessions" value,
- to preserve the previous CSIT behavior for 2202 output.
+ TODO: Investigate if it is worth to insert additional rampup trials
+ in TPUT tests to ensure all sessions are transitory before next
+ measurement.
:param node: DUT node.
:param proto: Required protocol - TCP/UDP/ICMP.
@@ -328,9 +332,7 @@ class NATUtil:
found = True
continue
# Proto is found, find the line we are interested in.
- if proto_l == u"tcp" and u"established" not in line:
- continue
- if u"total" not in line and u"established" not in line:
+ if u"total" not in line:
raise RuntimeError(f"show nat summary: no {proto} total.")
# We have the line with relevant numbers.
total_part, timed_out_part = line.split(u"(", 1)
diff --git a/resources/libraries/python/TrafficGenerator.py b/resources/libraries/python/TrafficGenerator.py
index b2748f74ba..03e3890959 100644
--- a/resources/libraries/python/TrafficGenerator.py
+++ b/resources/libraries/python/TrafficGenerator.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2021 Cisco and/or its affiliates.
+# Copyright (c) 2022 Cisco and/or its affiliates.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at:
@@ -643,6 +643,9 @@ class TrafficGenerator(AbstractMeasurer):
)
command_line.add_with_value(u"duration", f"{computed_duration!r}")
command_line.add_with_value(u"frame_size", self.frame_size)
+ command_line.add_with_value(
+ u"n_data_frames", Constants.ASTF_N_DATA_FRAMES
+ )
command_line.add_with_value(u"multiplier", multiplier)
command_line.add_with_value(u"port_0", p_0)
command_line.add_with_value(u"port_1", p_1)
@@ -811,7 +814,7 @@ class TrafficGenerator(AbstractMeasurer):
use_latency=False,
ramp_up_rate=None,
ramp_up_duration=None,
- state_timeout=300.0,
+ state_timeout=240.0,
ramp_up_only=False,
):
"""Send traffic from all configured interfaces on TG.
@@ -1215,7 +1218,7 @@ class TrafficGenerator(AbstractMeasurer):
The target_tr field of ReceiveRateMeasurement is in
transactions per second. Transmit count and loss count units
depend on the transaction type. Usually they are in transactions
- per second, or aggregate packets per second.
+ per second, or aggregated packets per second.
TODO: Fail on running or already reported measurement.
@@ -1382,7 +1385,7 @@ class TrafficGenerator(AbstractMeasurer):
use_latency=False,
ramp_up_rate=None,
ramp_up_duration=None,
- state_timeout=300.0,
+ state_timeout=240.0,
):
"""Store values accessed by measure().
@@ -1451,7 +1454,7 @@ class OptimizedSearch:
"""Class to be imported as Robot Library, containing search keywords.
Aside of setting up measurer and forwarding arguments,
- the main business is to translate min/max rate from unidir to aggregate.
+ the main business is to translate min/max rate from unidir to aggregated.
"""
@staticmethod
@@ -1475,7 +1478,7 @@ class OptimizedSearch:
use_latency=False,
ramp_up_rate=None,
ramp_up_duration=None,
- state_timeout=300.0,
+ state_timeout=240.0,
expansion_coefficient=4.0,
):
"""Setup initialized TG, perform optimized search, return intervals.
@@ -1616,7 +1619,7 @@ class OptimizedSearch:
use_latency=False,
ramp_up_rate=None,
ramp_up_duration=None,
- state_timeout=300.0,
+ state_timeout=240.0,
):
"""Setup initialized TG, perform soak search, return avg and stdev.
@@ -1674,7 +1677,7 @@ class OptimizedSearch:
:type ramp_up_rate: float
:type ramp_up_duration: float
:type state_timeout: float
- :returns: Average and stdev of estimated aggregate rate giving PLR.
+ :returns: Average and stdev of estimated aggregated rate giving PLR.
:rtype: 2-tuple of float
"""
tg_instance = BuiltIn().get_library_instance(
diff --git a/resources/libraries/python/autogen/Regenerator.py b/resources/libraries/python/autogen/Regenerator.py
index 3011b06897..4474996ef1 100644
--- a/resources/libraries/python/autogen/Regenerator.py
+++ b/resources/libraries/python/autogen/Regenerator.py
@@ -17,6 +17,7 @@ TODO: How can we check each suite id is unique,
when currently the suite generation is run on each directory separately?
"""
+import copy
import sys
from glob import glob
@@ -116,6 +117,39 @@ def check_suite_tag(suite_tag, prolog):
raise ValueError(f"Suite tag found {found} times for {suite_tag}")
+def filter_and_edit_kwargs_for_astf(suite_id, kwargs):
+ """Return possibly edited kwargs, or None if to be skipped.
+
+ This is a code block used in few places.
+ Kwargs is (a copy of) one item from tc_kwargs_list.
+ Currently, the editable field is frame_size,
+ to be increased to for tests with data (not just CPS).
+
+ :param suite_id: Suite ID.
+ :param kwargs: Key-value pairs used to construct one testcase.
+ :type suite_id: str
+ :type tc_kwargs_list: dict
+ :returns: Edited kwargs.
+ :rtype Optional[dict]
+ """
+ if u"-cps-" in suite_id:
+ # Contrary to UDP, there is no place to affect frame size
+ # in TCP CPS tests. Actual frames are close to min size.
+ # UDP uses the min value too, for fairer comparison to TCP.
+ if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
+ return None
+ elif (u"-pps-" in suite_id or u"-tput-" in suite_id):
+ if u"imix" in str(kwargs[u"frame_size"]).lower():
+ # ASTF does not support IMIX (yet).
+ return None
+ if kwargs[u"frame_size"] in MIN_FRAME_SIZE_VALUES:
+ # Minimal (TRex) TCP data frame is 80B for IPv4.
+ # In future, we may want to have also IPv6 TCP.
+ # UDP uses the same value, for fairer comparison to TCP.
+ kwargs[u"frame_size"] = 100
+ return kwargs
+
+
def add_default_testcases(testcase, iface, suite_id, file_out, tc_kwargs_list):
"""Add default testcases to file.
@@ -130,7 +164,9 @@ def add_default_testcases(testcase, iface, suite_id, file_out, tc_kwargs_list):
:type file_out: file
:type tc_kwargs_list: dict
"""
- for kwargs in tc_kwargs_list:
+ for kwas in tc_kwargs_list:
+ # We may edit framesize for ASTF, the copy should be local.
+ kwargs = copy.deepcopy(kwas)
# TODO: Is there a better way to disable some combinations?
emit = True
if kwargs[u"frame_size"] == 9000:
@@ -156,14 +192,8 @@ def add_default_testcases(testcase, iface, suite_id, file_out, tc_kwargs_list):
emit = False
if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
emit = False
- if (
- u"-cps-" in suite_id
- or u"-pps-" in suite_id
- or u"-tput-" in suite_id
- ):
- if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
- emit = False
- if emit:
+ kwargs = filter_and_edit_kwargs_for_astf(suite_id, kwargs)
+ if emit and kwargs is not None:
file_out.write(testcase.generate(**kwargs))
@@ -207,17 +237,11 @@ def add_trex_testcases(testcase, suite_id, file_out, tc_kwargs_list):
:type file_out: file
:type tc_kwargs_list: dict
"""
- for kwargs in tc_kwargs_list:
- # TODO: Is there a better way to disable some combinations?
- emit = True
- if (
- u"-cps-" in suite_id
- or u"-pps-" in suite_id
- or u"-tput-" in suite_id
- ):
- if kwargs[u"frame_size"] not in MIN_FRAME_SIZE_VALUES:
- emit = False
- if emit:
+ for kwas in tc_kwargs_list:
+ # We may edit framesize for ASTF, the copy should be local.
+ kwargs = copy.deepcopy(kwas)
+ kwargs = filter_and_edit_kwargs_for_astf(suite_id, kwargs)
+ if kwargs is not None:
file_out.write(testcase.generate(**kwargs))
diff --git a/resources/libraries/robot/performance/performance_display.robot b/resources/libraries/robot/performance/performance_display.robot
index db2b522091..a6df6f7b3a 100644
--- a/resources/libraries/robot/performance/performance_display.robot
+++ b/resources/libraries/robot/performance/performance_display.robot
@@ -1,4 +1,4 @@
-# Copyright (c) 2021 Cisco and/or its affiliates.
+# Copyright (c) 2022 Cisco and/or its affiliates.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at:
@@ -46,19 +46,19 @@
| | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other}
| | Fail | ${message}
-| Compute bandwidth
+| Compute Bandwidth
| | [Documentation]
| | ... | Compute (bidir) bandwidth from given (unidir) transaction rate.
| | ...
-| | ... | This keyword reads "ppta" and "avg_frame_size" set elsewhere.
-| | ... | The implementation should work for both pps and cps rates.
+| | ... | This keyword reads \${ppta} and \${avg_aggregated_frame_size} set
+| | ... | elsewhere. The implementation should work for both pps and cps rates.
| | ... |
| | ... | *Arguments:*
| | ... | - tps - Transaction rate (unidirectional) [tps]. Type: float
| | ...
| | ... | *Returns:*
| | ... | - Computed bandwidth in Gbps.
-| | ... | - Computed aggregate packet rate in pps.
+| | ... | - Computed aggregated packet rate in pps.
| |
| | ... | *Example:*
| |
@@ -68,7 +68,7 @@
| |
| | ${ppta} = | Get Packets Per Transaction Aggregated
| | ${pps} = | Evaluate | ${tps} * ${ppta}
-| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9
+| | ${bandwidth} = | Evaluate | ${pps} * (${avg_aggregated_frame_size}+20)*8/1e9
| | Return From Keyword | ${bandwidth} | ${pps}
| Display Reconfig Test Message
@@ -96,7 +96,7 @@
| Display result of NDRPDR search
| | [Documentation]
| | ... | Display result of NDR+PDR search, both quantities, both bounds,
-| | ... | aggregate in units given by trasaction type, e.g. by default
+| | ... | aggregated, in units given by trasaction type, e.g. by default
| | ... | in packet per seconds and Gbps total bandwidth
| | ... | (for initial packet size).
| | ... |
@@ -115,7 +115,7 @@
| | ... | - transaction_type - String identifier to determine how to count
| | ... | transactions. Default is "packet".
| | ... | *Arguments:*
-| | ... | - result - Measured result data. Aggregate rate, tps or pps.
+| | ... | - result - Measured result data. Aggregated rate, tps or pps.
| | ... | Type: NdrPdrResult
| |
| | ... | *Example:*
@@ -175,7 +175,7 @@
| | ... | it is in transactions per second. Bidirectional traffic
| | ... | transaction is understood as having 2 packets, for this purpose.
| | ... |
-| | ... | Pps values are aggregate in packet per seconds,
+| | ... | Pps values are aggregated, in packet per seconds
| | ... | and Gbps total bandwidth (for initial packet size).
| | ... |
| | ... | Througput is calculated as:
@@ -231,8 +231,8 @@
| Display single pps bound
| | [Documentation]
-| | ... | Display one pps bound of NDR+PDR search,
-| | ... | aggregate in packet per seconds and Gbps total bandwidth
+| | ... | Display one pps bound of NDR+PDR search, aggregated,
+| | ... | in packet per seconds and Gbps total bandwidth
| | ... | (for initial packet size).
| | ... |
| | ... | The bound to display is given as target transfer rate, it is assumed
diff --git a/resources/libraries/robot/performance/performance_utils.robot b/resources/libraries/robot/performance/performance_utils.robot
index 8350db0205..c90fc7434d 100644
--- a/resources/libraries/robot/performance/performance_utils.robot
+++ b/resources/libraries/robot/performance/performance_utils.robot
@@ -46,7 +46,7 @@
| | ... | Display results as formatted test message.
| | ... | Fail if computed lower bound is 110% of the minimal rate or less.
| | ... | Input rates are unidirectional, in transaction per second.
-| | ... | Reported result may contain aggregate pps rates, depending on test.
+| | ... | Reported result may contain aggregated pps rates, depending on test.
| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
| |
| | ... | *Test (or broader scope) variables read:*
@@ -105,7 +105,7 @@
| | ... | Display findings as a formatted test message.
| | ... | Fail if a resulting lower bound has too high loss ratio.
| | ... | Input rates are unidirectional, in transaction per second.
-| | ... | Reported result may contain aggregate pps rates, depending on test.
+| | ... | Reported result may contain aggregated pps rates, depending on test.
| | ... | Additional latency measurements are performed for smaller loads,
| | ... | even if latency stream is disabled in search. Their results
| | ... | are also displayed.
@@ -468,7 +468,7 @@
| | ... | *Arguments:*
| | ... | - trial_duration - Duration of single trial [s].
| | ... | Type: float
-| | ... | - rate - Target aggregate transmit rate [bps] / Bits per second
+| | ... | - rate - Target aggregated transmit rate [bps] / Bits per second.
| | ... | Type: float
| | ... | - frame_size - L2 Frame Size [B].
| | ... | Type: integer or string
diff --git a/resources/libraries/robot/performance/performance_vars.robot b/resources/libraries/robot/performance/performance_vars.robot
index 4f2cc50fa3..f75d585e4b 100644
--- a/resources/libraries/robot/performance/performance_vars.robot
+++ b/resources/libraries/robot/performance/performance_vars.robot
@@ -1,4 +1,4 @@
-# Copyright (c) 2021 Cisco and/or its affiliates.
+# Copyright (c) 2022 Cisco and/or its affiliates.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at:
@@ -155,11 +155,11 @@
| Get Packets Per Transaction Aggregated
| | [Documentation]
| | ... | Return value of \${packets_per_transaction_aggregated};
-| | ... | if not defined, assume traffic is symmetric and compute
-| | ... | from unidirectional values.
+| | ... | if not defined, assume traffic is symmetric (or unidirectional)
+| | ... | and compute from unidirectional values.
| |
-| | ... | The return value is used when reporting PPS values from TPS found
-| | ... | by some search (e.g. NDRPDR).
+| | ... | The return value is used when reporting PPS (and bandwidth) values
+| | ... | from TPS found by some search (e.g. NDRPDR).
| | ... | Return type: integer.
| |
| | ... | *Example:*
@@ -169,7 +169,6 @@
| | ${ppta} = | Get Variable Value | \${packets_per_transaction_aggregated}
| | ... | ${0}
| | Return From Keyword If | "${ppta}" != "0" | ${ppta}
-| | # TODO: Insert TCP computation from packet size here.
| | ${pptad} = | Get Packets Per Transaction And Direction
| | ${traffic_directions} = | Get Traffic Directions
| | # We do not support ASTF profiles with multiple transactions, yet.
@@ -181,8 +180,9 @@
| | ... | Return value of \${packets_per_transaction_and_direction},
| | ... | or ${1} if not defined.
| |
-| | ... | The return value is used when computing max rate (TPS),
-| | ... | so for asymmetric transaction use the more numerous direction.
+| | ... | The return value is used when computing max rate (TPS)
+| | ... | from packet level (pps or bps) limits.
+| | ... | For asymmetric transactions, use the more numerous direction.
| | ... | Return type: integer.
| |
| | ... | *Example:*
@@ -392,19 +392,14 @@
| Set Jumbo
| | [Documentation]
-| | ... | For jumbo frames detection, the maximal packet size is relevant,
-| | ... | encapsulation overhead (if any) has effect.
-| |
-| | ... | This keyword computes jumbo boolean (some suites need that for
-| | ... | configuration decisions).
+| | ... | Call Set Numeric Frame Sizes and set jumbo based on max framesize.
| | ... | To streamline suite autogeneration, both input and output values
| | ... | are communicated as test (or broader scope) variables,
| | ... | instead of explicit arguments and return values.
| |
| | ... | *Test (or broader scope) variables read:*
| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
+| | ... | - frame_size - Framesize. Type: integer or string
| |
| | ... | *Test variables set:*
| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
@@ -414,32 +409,22 @@
| |
| | ... | \| Set Jumbo \|
| |
-| | # Already called by Set Max Rate And Jumbo, but some suites (e.g. device)
-| | # are calling this directly.
+| | # Some suites (e.g. device) are not calling Set Max Rate And Jumbo.
| | Set Numeric Frame Sizes
-| | ${jumbo} = | Set Variable If | ${max_frame_size} < 1522
-| | ... | ${False} | ${True}
+| | ${jumbo} = | Evaluate | ${max_frame_size} >= 1522
| | Set Test Variable | \${jumbo}
| Set Max Rate And Jumbo
| | [Documentation]
-| | ... | Input framesize can be either integer in case of a single packet
-| | ... | in stream, or IMIX string defining mix of packets.
-| | ... | For jumbo frames detection, the maximal packet size is relevant.
-| | ... | For maximal transmit rate, the average packet size is relevant.
-| | ... | In both cases, encapsulation overhead (if any) has effect.
-| | ... | The maximal rate is computed from NIC name.
-| | ... | The implementation works by mapping from exact
-| | ... | whitelisted NIC names.
-| | ... | The mapping is hardcoded in nic_limits.yaml
-| | ... | TODO: Make the mapping from NIC names case insensistive.
-| |
| | ... | This keyword computes maximal unidirectional transmit rate
| | ... | and jumbo boolean (some suites need that for configuration decisions).
| | ... | To streamline suite autogeneration, both input and output values
| | ... | are communicated as test (or broader scope) variables,
| | ... | instead of explicit arguments and return values.
| |
+| | ... | For correctly applying bandwidth limit, average frame size is used,
+| | ... | see Set Numeric Frame Sizes keyword documentation for details.
+| |
| | ... | If this keyword detects the test is interested in (unidirectional)
| | ... | transactons per second maximal rate (tps), that is returned (not pps).
| |
@@ -448,8 +433,13 @@
| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
| | ... | string
-| | ... | - packets_per_transaction_and_direction - Pps-tps conversion.
-| | ... | Optional, default 1.
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| | ... | - TEST_TAGS - Robot tags of this test. Type: list of string
| |
| | ... | *Test variables set:*
| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
@@ -457,8 +447,11 @@
| | ... | Type: float
| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
| | ... | enabled. Type: boolean
-| | ... | avg_frame_size - Average frame size including overhead. Type: float
-| | ... | max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - avg_directional_frame_size - Average frame size including overhead
+| | ... | for the more loaded direction. Type: float
+| | ... | - avg_aggregated_frame_size - Average frame size including overhead
+| | ... | across both traffic directions. Type: float
| |
| | ... | *Example:*
| |
@@ -470,68 +463,188 @@
| | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name}
| | ${bps_limit} = | Get From Dictionary
| | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name}
-| | Set Numeric Frame Sizes
-| | # We need to add 20B (Ethernet preamble and inter-frame gap)
-| | # to avg_frame_size
-| | ${rate} = | Evaluate | ${bps_limit} / ((${avg_frame_size} + 20.0) * 8)
+| | # Set Jumbo also calls Set Numeric Frame Sizes.
+| | Set Jumbo
+| | # We need to add 20B (Ethernet preamble and inter-frame gap).
+| | ${adfs} = | Get Variable Value | \${avg_directional_frame_size}
+| | ${rate} = | Evaluate | ${bps_limit} / ((${adfs} + 20.0) * 8)
| | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit}
| | ... | ${pps_limit} | ${rate}
| | ${pptad} = | Get Packets Per Transaction And Direction
| | ${max_rate} = | Evaluate | ${max_rate} / ${pptad}
| | Set Test Variable | \${max_rate}
-| | Set Jumbo
| Set Numeric Frame Sizes
| | [Documentation]
| | ... | Framesize can be either integer in case of a single packet
| | ... | in stream, or set of packets in case of IMIX type or simmilar.
| | ... | For jumbo decisions, we need a numeric size of the biggest packet.
-| | ... | For max rate decisions, we need a numeric average packet size.
-| | ... | This keyword computes both and sets them as test variables.
+| | ... | For bandwidth limit decisions, we need a numeric average packet size
+| | ... | in the more bit intensive direction if traffic is non-symmetric.
+| | ... | Computation of max_rate assumes it is also the more pps direction
+| | ... | (so it can have smaller average frame size than the aggregated one).
+| | ... | Average (across both directions) frame size is also used
+| | ... | for displaying the bidirectional bandwidth forwarded.
+| | ... | This keyword computes all three values (accounting for overheads)
+| | ... | and sets them as test variables.
| |
| | ... | Each suite sets a value named \${overhead},
| | ... | which describes by how many bytes the frames on DUT-DUT link
| | ... | are larger (due to encapsulation) than those
-| | ... | on the primary TG-DUT link. But for some suites that value
+| | ... | on the primary TG-DUT link. For some suites that value
| | ... | can be negaive (if TG-DUT is encapsulated more heavily).
| | ... | For calculations in this keyword, we need largest sizes
| | ... | across links, so zero is used if \${overhead} is negative.
| |
+| | ... | The other overhead is from TCP control packets (only IPv4 supported).
+| | ... | TCP_CPS tests have SYN frames of length 78B and other frames 70B.
+| | ... | The more loaded is client-to-server direction with 1 SYN and 3 other,
+| | ... | across both directions it is 2 SYN and 5 other.
+| | ... | TCP_PPS and TCP_TPUT tests have one other control packet less
+| | ... | (in the less loaded direction), but they do contain data frames.
+| |
| | ... | *Test variables read:*
| | ... | - frame_size - Framesize. Type: integer or string
| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| | ... | - TEST_TAGS - Robot tags of this test. Type: list of string
| |
| | ... | *Test variables set*
-| | ... | avg_frame_size - Average frame size including overhead. Type: float
-| | ... | max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - max_frame_size - Maximal frame size including overhead. Type: float
+| | ... | - avg_directional_frame_size - Average frame size including overhead
+| | ... | for the more loaded direction. Type: float
+| | ... | - avg_aggregated_frame_size - Average frame size including overhead
+| | ... | across both traffic directions. Type: float
| |
| | ... | *Example:*
| |
| | ... | \| Set Numeric Frame Sizes \|
| |
+| | ${bare_max_frame_size} = | Run Keyword If
+| | ... | '${frame_size}' == 'IMIX_v4_1' | Set Variable | ${1518.0}
+| | ... | ELSE | Convert To Number | ${frame_size}
+| | ${bafs} = | Run Keyword If
+| | ... | '${frame_size}' == 'IMIX_v4_1' | Set Variable | ${353.8333333333333}
+| | ... | ELSE | Convert To Number | ${frame_size}
+| | # Long boolean formula in 2 lines.
+| | ${is_tcp_pps} = | Evaluate | 'TCP_PPS' in ${TEST_TAGS}
+| | ${is_tcp_tput} = | Evaluate | ${is_tcp_pps} or 'TCP_TPUT' in ${TEST_TAGS}
+| | ${avg_dir_frame_size} | ${avg_agg_frame_size} = | Run Keyword If
+| | ... | 'TCP_CPS' in ${TEST_TAGS} | Apply Tcp Cps Proto Overhead | ${bafs}
+| | ... | ELSE IF | ${is_tcp_tput} | Apply Tcp Tput Proto Overhead | ${bafs}
+| | ... | ELSE | Set Variable | ${bafs} | ${bafs}
| | ${max_overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
-| | ${bare_avg_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
-| | ... | Set Variable | ${353.83333}
-| | ... | ELSE
-| | ... | Convert To Number | ${frame_size}
-| | # Do not use $max_overhead (without braces), that does not tolerate string.
-| | ${avg_frame_size} = | Evaluate | ${bare_avg_frame_size} + ${max_overhead}
-| | Set Test Variable | \${avg_frame_size}
-| | ${bare_max_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
-| | ... | Set Variable | ${1518}
-| | ... | ELSE
-| | ... | Convert To Number | ${frame_size}
-| | ${max_frame_size} = | Evaluate | $bare_max_frame_size + $max_overhead
-| | Set Test Variable | ${max_frame_size}
+| | ${mfs} = | Evaluate | ${bare_max_frame_size} + ${max_overhead}
+| | ${adfs} = | Evaluate | ${avg_dir_frame_size} + ${max_overhead}
+| | ${aafs} = | Evaluate | ${avg_agg_frame_size} + ${max_overhead}
+| | Set Test Variable | \${max_frame_size} | ${mfs}
+| | Set Test Variable | \${avg_directional_frame_size} | ${adfs}
+| | Set Test Variable | \${avg_aggregated_frame_size} | ${aafs}
+
+| Apply Tcp Cps Proto Overhead
+| | [Documentation]
+| | ... | Recompute average frame size for TCP CPS test cases.
+| |
+| | ... | This is contitionally called from Set Numeric Frame Sizes.
+| | ... | In Robot Framework it is more convenient to wrap such a block
+| | ... | as a standalone keyword to Run Keyword If.
+| |
+| | ... | *Test variables read:*
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Usually set globally. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| |
+| | ... | *Arguments:*
+| | ... | - bare_avg_frame_size - Average numeric framesize without overheads.
+| |
+| | ... | *Returns:*
+| | ... | - avg_dir_frame_size - Average framesize for more loaded direction.
+| | ... | - avg_agg_frame_size - Average framesize across both directions.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${adfs} \| \${aafs} = \| Apply Tcp Cps Proto Overhead \| \${bafs}
+| |
+| | [Arguments] | ${bare_avg_frame_size}
+| |
+| | # Increase max_frame_size for TCP tests if used for more than just jumbo.
+| | Run Keyword If | ${bare_avg_frame_size} != 64
+| | ... | Fail | TCP_CPS tests are only supported for (nominal) 64B frames.
+| | # TODO: Unify with packets_per_transaction_* variables when adding PCAP.
+| | ${pptad} = | Get Packets Per Transaction And Direction
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${avg_dir_frame_size} = | Evaluate | (78.0 * 1 + 70.0 * 3) / (1 + 3)
+| | Run Keyword If | '${pptad}' != '4'
+| | ... | Fail | TCP CPS with pptad '${pptad}' != '4'.
+| | ${avg_agg_frame_size} = | Evaluate | (78.0 * 2 + 70.0 * 5) / (2 + 5)
+| | Run Keyword If | '${ppta}' != '7'
+| | ... | Fail | TCP CPS with ppta '${ppta}' != '7'.
+| | Return From Keyword | ${avg_dir_frame_size} | ${avg_agg_frame_size}
+
+| Apply Tcp Tput Proto Overhead
+| | [Documentation]
+| | ... | Recompute average frame size for TCP TPUT (or PPS) test cases.
+| |
+| | ... | This is contitionally called from Set Numeric Frame Sizes.
+| | ... | In Robot Framework it is more convenient to wrap such a block
+| | ... | as a standalone keyword to Run Keyword If.
+| |
+| | ... | *Test variables read:*
+| | ... | - ASTF_N_DATA_FRAMES - Number of data frames per transaction
+| | ... | and direction. Usually set globally. Type: integer
+| | ... | - packets_per_transaction_and_direction - May be unset.
+| | ... | See Get Packets Per Transaction And Direction keyword. Type: integer
+| | ... | - packets_per_transaction_aggregated - May be unset.
+| | ... | See Get Packets Per Transaction Aggregated keyword. Type: integer
+| |
+| | ... | *Arguments:*
+| | ... | - bare_framesize - Average numeric framesize without overheads.
+| |
+| | ... | *Returns:*
+| | ... | - avg_dir_frame_size - Average framesize for more loaded direction.
+| | ... | - avg_agg_frame_size - Average framesize across both directions.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${adfs} \| \${aafs} = \| Apply Tcp Cps Proto Overhead \| \${bafs}
+| |
+| | [Arguments] | ${bare_framesize}
+| |
+| | # TODO: Unify with packets_per_transaction_* variables when adding PCAP.
+| | ${pptad} = | Get Packets Per Transaction And Direction
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | # Long float formula in 4 lines.
+| | ${numerator} = | Evaluate | ${bare_framesize} * ${ASTF_N_DATA_FRAMES}
+| | ${numerator} = | Evaluate | 78.0 * 1 + 70.0 * 3 + ${numerator}
+| | ${denominator} = | Evaluate | 1 + 3 + ${ASTF_N_DATA_FRAMES}
+| | ${avg_dir_frame_size} = | Evaluate | ${numerator} / ${denominator}
+| | Run Keyword If | '${pptad}' != '${denominator}'
+| | ... | Fail | TCP TPUT with pptad '${pptad}' != '${denominator}'.
+| | # Long float formula in 4 lines.
+| | ${numerator} = | Evaluate | ${bare_framesize} * 2 * ${ASTF_N_DATA_FRAMES}
+| | ${numerator} = | Evaluate | 78.0 * 2 + 70.0 * 4 + ${numerator}
+| | ${denominator} = | Evaluate | 2 + 4 + 2 * ${ASTF_N_DATA_FRAMES}
+| | ${avg_agg_frame_size} = | Evaluate | ${numerator} / ${denominator}
+| | Run Keyword If | '${ppta}' != '${denominator}'
+| | ... | Fail | TCP TPUT with ppta '${ppta}' != '${denominator}'.
+| | Return From Keyword | ${avg_dir_frame_size} | ${avg_agg_frame_size}
| Set Rates For Policer
| | [Documentation]
| | ... | Policer tests need these values,
-| | ... | currently computed from \${avg_frame_size}.
+| | ... | currently computed from \${avg_directional_frame_size}.
| | ... | TODO: Verify the units match and computation is correct.
| |
| | ... | *Test (or broader scope) variables read:*
-| | ... | - avg_frame_size - Average L2 Frame Size [B]. Type: float
+| | ... | - avg_directional_frame_size - Average L2 Frame Size [B]. Type: float
| | ... | Set by Set Max Rate And Jumbo keyword.
| |
| | ... | *Test variables set:*
@@ -542,5 +655,5 @@
| |
| | ... | \| Set Rates For Policer \|
| |
-| | Set Test Variable | \${eb} | ${avg_frame_size}
-| | Set Test Variable | \${cb} | ${avg_frame_size}
+| | Set Test Variable | \${eb} | ${avg_directional_frame_size}
+| | Set Test Variable | \${cb} | ${avg_directional_frame_size}