Support existing test types with ASTF

+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
 + Update existing cps traffic profiles.
 + Add missing traffic profiles.
 + UDP:
  + Single burst of 32 packets was confirmed as safe enough for TRex.
  + Maybe 64 could work, but not enough testing for that.
  + Multiple bursts have lead to reduced TRex performance,
    as overlaping bursts (from different client instances)
    tend to fill up the buffers.
 + TCP:
  + Data size set to 11111 bytes, completely arbitrarily.
  + Results look reasonable, so I have kept that.
  - MSS not set at all
 - No tested support for frame size other than 64B.
  - Frame size does not even factor into TCP profiles.
 + So other frame sizes are skipped in autogen.
+ Update tags in related suites.
 - HOSTS_{n} and SRC_USER_{n} should be unified.
 - Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
 + Resetter is called (if set) before each measurement.
 + If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
 + Transaction corresponds to profile.
 + TRex multiplier argument sets target rate in transactions per second.
 + The familiar STL traffic:
  + Bidirectional is considered to be 2 packets per transaction.
  + Unidirectional is considered to be 1 packet per transaction.
 + The newer ASTF traffic:
  + 4 subtypes, each has different number of packets per transaction.
 + For max rate computation:
  + Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
 + Use the server side value.
  - Client side value is higher by an overhead.
  - TRex is not sending traffic during that time.
 + Remove delays from traffic profiles.
  - Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
 + Only for ASTF profiles, each ASTF profile has limited scale.
 + Scale defined in suite variables.
 + For TRex to send all transactions provided duration value is ignored.
  + The appropriate value is computed in TrafficGenerator.
  + An ad-hoc time constant is added to match the TRex client side time overhead.
  + The profile driver receives the computed duration.
  + Measurement for PLRsearch add a sleep if the computed duration is smaller.
 + Alternative argument for search algos if scale is limited.
  + Both need higher timeout to accomodate big scales.
  + MLRsearch can afford fewer phases.
 + Added a parameter to optionally shorten the duration.
  + Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
 - Polling could eliminate the time overhead value.
 + But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
 + The stop has several steps so that:
  + The traffic is really stopped entirely.
  + Late packets do not count (maybe as errors).
  + Stats are preserved to read for results (and cleared afterwards).
 + Several quantities added to ReceiveRateMeasurement:
  + Original target duration is preserved (algos need that).
  + Input estimate (tps) for early search iterations.
  + Output estimate (maybe pps) for MRR output.
  + Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
 - TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
 + Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
 + Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
 + Rename argument names to min_rate and max_rate.
 + Use relative receive rate in initial phase.
+ PLRsearch improvements:
 + Careful computation when output (pps) does not match input (tps).
 + Use geometric distribution (instead of Poisson).
  + Helps agains math errors.
  + This should improve estimate stability.
  - But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
 + send_traffic_on_tg now calls the full set_rate_provider_defaults.
 + _send_traffic_on_tg_internal for the logic without provider defaults.
  + As the internal function is re-used by measure() without affecting defaults.
 + Move _parse_traffic_results just before get_measurement_result.
  + As the latter uses fields set bu the former, it is now easier to read.
 + Multiple sources for approximate duration.
  + Tried from more precise to more available.
 + Includes logic for _pps tests (added in later change).
 + Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
 + performance_actions.robot:
  + Hosts Additional Statistics Action For * keywords.
 + performance_display.robot:
  + Hosts keyword for displaying and verifying results.
  + Change test message to use the correct unit (pps or cps).
 + performance_limits.robot renamed to performance_vars.robot
  + Added many keywords, mostly for accessing test variables.
   + Moved variables for Policer into a new keyword there.
  + Some keywords need sophisticated logic.
  - Other are basically Get Variable Value.
   + But in future more logic can be added, without editing callers.
  + Documentation for the new keywords acts as a documentation for test variables.
 + performance_utils.robot has the rest.
  + Eliminated arguments if the value is in test variable.
  + Small improvements to documentation.
  - Still not enough cleanup with respect to arguments and test variables.
 + Keywords are sorted alphabetically now in each one.
+ Suites:
 + Unified variables table:
  + No colons in comments.
  + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
 + Add -cps to existing cps suite names.
  + Remove "trial data overwrite".
  + Compute max rate as in STL suites.
 + Each NAT suite has ip4base suite to compare results to.
  - Those act as indirect TRex calibration.
  - VPP does not lose packets in those.
 + Latency in ASTF suites is disabled hard.
  - As we do not support latency in ASTF profiles yet.
 + Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
 + Prefer to use argument names.
 + In Python, also the last argument is followed by comma.
 + It makes renaming and reordering easier.
 + Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.

Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

5 files changed, 1222 insertions, 705 deletions

diff --git a/resources/libraries/robot/performance/performance_actions.robot b/resources/libraries/robot/performance/performance_actions.robot
new file mode 100644
index 0000000000..2cf954a5b3
--- /dev/null
+++ b/resources/libraries/robot/performance/performance_actions.robot
@@ -0,0 +1,92 @@
+# Copyright (c) 2020 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Variables | resources/libraries/python/Constants.py
+| Resource | resources/libraries/robot/performance/performance_utils.robot
+|
+| Documentation
+| ... | Performance suite keywords - Actions related to performance tests.
+
+*** Keywords ***
+| Additional Statistics Action For bash-perf-stat
+| | [Documentation]
+| | ... | Additional Statistics Action for bash command "perf stat".
+| |
+| | Run Keyword If | ${extended_debug}==${True}
+| | ... | Perf Stat On All DUTs | ${nodes} | cpu_list=${cpu_alloc_str}
+
+| Additional Statistics Action For clear-show-runtime-with-traffic
+| | [Documentation]
+| | ... | Additional Statistics Action for clear and show runtime counters with
+| | ... | running traffic.
+| |
+| | ... | See documentation of the called keyword for required test variables.
+| |
+| | Clear and show runtime counters with running traffic
+
+| Additional Statistics Action For noop
+| | [Documentation]
+| | ... | Additional Statistics Action for no operation.
+| |
+| | No operation
+
+| Additional Statistics Action For vpp-clear-runtime
+| | [Documentation]
+| | ... | Additional Statistics Action for clear VPP runtime.
+| |
+| | VPP Clear Runtime On All DUTs | ${nodes}
+
+| Additional Statistics Action For vpp-clear-stats
+| | [Documentation]
+| | ... | Additional Statistics Action for clear VPP statistics.
+| |
+| | Clear Statistics On All DUTs | ${nodes}
+
+| Additional Statistics Action For vpp-enable-elog
+| | [Documentation]
+| | ... | Additional Statistics Action for enable VPP elog trace.
+| |
+| | VPP Enable Elog Traces On All DUTs | ${nodes}
+
+| Additional Statistics Action For vpp-enable-packettrace
+| | [Documentation]
+| | ... | Additional Statistics Action for enable VPP packet trace.
+| |
+| | Run Keyword If | ${extended_debug}==${True}
+| | ... | VPP Enable Traces On All DUTs | ${nodes} | fail_on_error=${False}
+
+| Additional Statistics Action For vpp-show-elog
+| | [Documentation]
+| | ... | Additional Statistics Action for show VPP elog trace.
+| |
+| | Show Event Logger On All DUTs | ${nodes}
+
+| Additional Statistics Action For vpp-show-packettrace
+| | [Documentation]
+| | ... | Additional Statistics Action for show VPP packet trace.
+| |
+| | Run Keyword If | ${extended_debug}==${True}
+| | ... | Show Packet Trace On All Duts | ${nodes} | maximum=${100}
+
+| Additional Statistics Action For vpp-show-runtime
+| | [Documentation]
+| | ... | Additional Statistics Action for show VPP runtime.
+| |
+| | VPP Show Runtime On All DUTs | ${nodes}
+
+| Additional Statistics Action For vpp-show-stats
+| | [Documentation]
+| | ... | Additional Statistics Action for show VPP statistics.
+| |
+| | Show Statistics On All DUTs | ${nodes}
diff --git a/resources/libraries/robot/performance/performance_display.robot b/resources/libraries/robot/performance/performance_display.robot
new file mode 100644
index 0000000000..e8dfdbbfbd
--- /dev/null
+++ b/resources/libraries/robot/performance/performance_display.robot
@@ -0,0 +1,240 @@
+# Copyright (c) 2020 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Documentation
+| ... | Performance suite keywords - Displaying results as test messages.
+| ... | This includes checks to fail test.
+
+*** Keywords ***
+| Check NDRPDR interval validity
+| | [Documentation]
+| | ... | Extract loss ratio of lower bound of the interval.
+| | ... | Fail if it does not reach the allowed value.
+| |
+| | ... | *Arguments:*
+| | ... | - interval - Measured interval. Type: ReceiveRateInterval
+| | ... | - packet_loss_ratio - Accepted loss (0.0 for NDR). Type: float
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Check NDRPDR interval validity \| \${result.pdr_interval} \
+| | ... | \| \${0.005} \|
+| |
+| | [Arguments] | ${interval} | ${packet_loss_ratio}=${0.0}
+| |
+| | ${lower_bound} = | Set Variable | ${interval.measured_low}
+| | ${lower_bound_lf} = | Set Variable | ${lower_bound.loss_fraction}
+| | Return From Keyword If | ${lower_bound_lf} <= ${packet_loss_ratio}
+| | Set Test Variable | \${rate_for_teardown} | ${lower_bound_lf}
+| | ${message}= | Catenate | SEPARATOR=${SPACE}
+| | ... | Minimal rate loss fraction ${lower_bound_lf}
+| | ... | does not reach target ${packet_loss_ratio}.
+| | ${message_zero} = | Set Variable | Zero packets forwarded!
+| | ${message_other} = | Set Variable | ${lower_bound.loss_count} packets lost.
+| | ${message} = | Set Variable If | ${lower_bound_lf} >= 1.0
+| | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other}
+| | Fail | ${message}
+
+| Display Reconfig Test Message
+| | [Documentation]
+| | ... | Display the number of packets lost (bidirectionally)
+| | ... | due to reconfiguration under traffic.
+| |
+| | ... | *Arguments:*
+| | ... | - result - Result of bidirectional measurtement.
+| | ... | Type: ReceiveRateMeasurement
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Display Reconfig Test Message \| \${result} \|
+| |
+| | [Arguments] | ${result}
+| |
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${packet_rate} = | Evaluate | ${result.target_tr} * ${ppta}
+| | ${packet_loss} = | Set Variable | ${result.loss_count}
+| | ${time_loss} = | Evaluate | ${packet_loss} / ${packet_rate}
+| | Set Test Message | Packets lost due to reconfig: ${packet_loss}
+| | Set Test Message | ${\n}Implied time lost: ${time_loss} | append=yes
+
+| Display result of NDRPDR search
+| | [Documentation]
+| | ... | Display result of NDR+PDR search, both quantities, both bounds,
+| | ... | aggregate in units given by trasaction type, e.g. by default
+| | ... | in packet per seconds and Gbps total bandwidth
+| | ... | (for initial packet size).
+| | ... |
+| | ... | The bound to display is encoded as target rate, it is assumed
+| | ... | it is in transactions per second. Bidirectional traffic
+| | ... | transaction is understood as having 2 packets, for this purpose.
+| | ... |
+| | ... | Througput is calculated as:
+| | ... | Sum of measured rate over streams
+| | ... | Bandwidth is calculated as:
+| | ... | (Throughput * (L2 Frame Size + IPG) * 8)
+| | ... | If the results contain latency data, display them for lower bounds.
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - frame_size_num - L2 Frame Size [B]. Type: integer or float
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
+| | ... | *Arguments:*
+| | ... | - result - Measured result data. Aggregate rate, tps or pps.
+| | ... | Type: NdrPdrResult
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Display result of NDRPDR search \| \${result} \|
+| |
+| | [Arguments] | ${result}
+| |
+| | Display single bound | NDR_LOWER
+| | ... | ${result.ndr_interval.measured_low.target_tr}
+| | ... | ${result.ndr_interval.measured_low.latency}
+| | Display single bound | NDR_UPPER
+| | ... | ${result.ndr_interval.measured_high.target_tr}
+| | Display single bound | PDR_LOWER
+| | ... | ${result.pdr_interval.measured_low.target_tr}
+| | ... | ${result.pdr_interval.measured_low.latency}
+| | Display single bound | PDR_UPPER
+| | ... | ${result.pdr_interval.measured_high.target_tr}
+
+| Display result of soak search
+| | [Documentation]
+| | ... | Display result of soak search, avg+-stdev, as upper/lower bounds.
+| | ... | See Display single bound for units used.
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
+| | ... | string
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
+| | ... | *Arguments:*
+| | ... | - avg - Estimated average critical load [pps]. Type: float
+| | ... | - stdev - Standard deviation of critical load [pps]. Type: float
+| |
+| | ... | *Returns:*
+| | ... | - Lower and upper bound of critical load [pps]. Type: 2-tuple of float
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Display result of soak search \| \${100000} \| \${100} \|
+| |
+| | [Arguments] | ${avg} | ${stdev}
+| |
+| | ${avg} = | Convert To Number | ${avg}
+| | ${stdev} = | Convert To Number | ${stdev}
+| | ${lower} = | Evaluate | ${avg} - ${stdev}
+| | ${upper} = | Evaluate | ${avg} + ${stdev}
+| | Display single bound | PLRsearch lower bound | ${lower}
+| | Display single bound | PLRsearch upper bound | ${upper}
+| | Return From Keyword | ${lower} | ${upper}
+
+| Display single bound
+| | [Documentation]
+| | ... | Compute and display one bound of NDR+PDR (or soak) search result.
+| | ... | If the latency string is present, it is displayed as well.
+| | ... |
+| | ... | The bound to display is given as target transfer rate, it is assumed
+| | ... | it is in transactions per second. Bidirectional traffic
+| | ... | transaction is understood as having 2 packets, for this purpose.
+| | ... |
+| | ... | Pps values are aggregate in packet per seconds,
+| | ... | and Gbps total bandwidth (for initial packet size).
+| | ... |
+| | ... | Througput is calculated as:
+| | ... | Sum of measured rate over streams
+| | ... | Bandwidth is calculated as:
+| | ... | (Throughput * (L2 Frame Size + IPG) * 8)
+| | ... | If the results contain latency data, display them for lower bounds.
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
+| | ... | *Arguments:*
+| | ... | - text - Flavor text describing which bound is this. Type: string
+| | ... | - tps - Transaction rate [tps]. Type: float
+| | ... | - latency - Latency data to display if non-empty. Type: string
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Display single bound \| NDR lower bound \| \${12345.67} \
+| | ... | \| latency=\${EMPTY} \|
+| |
+| | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY}
+| |
+| | ${transaction_type} = | Get Transaction Type
+| | Run Keyword And Return If | """_cps""" in """${transaction_type}"""
+| | ... | Display single cps bound | ${text} | ${tps} | ${latency}
+| | Display single pps bound | ${text} | ${tps} | ${latency}
+
+| Display single cps bound
+| | [Documentation]
+| | ... | Display one bound of NDR+PDR search for CPS tests.
+| | ... | The bounds are expressed as transactions per second.
+| | ... | If the latency string is present, it is displayed as well.
+| |
+| | ... | *Arguments:*
+| | ... | - text - Flavor text describing which bound is this. Type: string
+| | ... | - tps - Transaction rate [tps]. Type: float
+| | ... | - latency - Latency data to display if non-empty. Type: string
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Display single cps bound \| NDR lower bound \| \${12345.67} \
+| | ... | \| latency=\${EMPTY} \|
+| |
+| | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY}
+| |
+| | Set Test Message | ${\n}${text}: ${tps} CPS | append=yes
+| | Return From Keyword If | not """${latency}"""
+| | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency}
+| | ... | append=yes
+
+| Display single pps bound
+| | [Documentation]
+| | ... | Display one pps bound of NDR+PDR search,
+| | ... | aggregate in packet per seconds and Gbps total bandwidth
+| | ... | (for initial packet size).
+| | ... |
+| | ... | The bound to display is given as target transfer rate, it is assumed
+| | ... | it is in transactions per second. Bidirectional traffic
+| | ... | transaction is understood as having 2 packets, for this purpose.
+| | ... |
+| | ... | Througput is calculated as:
+| | ... | Sum of measured rates over streams
+| | ... | Bandwidth is calculated as:
+| | ... | (Throughput * (L2 Frame Size + IPG) * 8)
+| | ... | If the latency string is present, it is displayed as well.
+| |
+| | ... | *Arguments:*
+| | ... | - text - Flavor text describing which bound is this. Type: string
+| | ... | - tps - Transaction rate [tps]. Type: float
+| | ... | - latency - Latency data to display if non-empty. Type: string
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Display single pps bound \| NDR lower bound \| \${12345.67} \
+| | ... | \| latency=\${EMPTY} \|
+| |
+| | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY}
+| |
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${pps} = | Evaluate | ${tps} * ${ppta}
+| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9
+| | Set Test Message | ${\n}${text}: ${pps} pps, | append=yes
+| | Set Test Message | ${bandwidth} Gbps (initial) | append=yes
+| | Return From Keyword If | not """${latency}"""
+| | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency}
+| | ... | append=yes
diff --git a/resources/libraries/robot/performance/performance_limits.robot b/resources/libraries/robot/performance/performance_limits.robot
deleted file mode 100644
index 332ed9368b..0000000000
--- a/resources/libraries/robot/performance/performance_limits.robot
+++ /dev/null
@@ -1,153 +0,0 @@
-# Copyright (c) 2020 Cisco and/or its affiliates.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at:
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-*** Settings ***
-| Documentation | Performance suite keywords - Handling NIC and TG limits.
-| Library | Collections
-| Variables | ${CURDIR}/../../python/Constants.py
-
-*** Keywords ***
-| Get Average Frame Size
-| | [Documentation]
-| | ... | Framesize can be either integer in case of a single packet
-| | ... | in stream, or set of packets in case of IMIX type or simmilar.
-| |
-| | ... | *Arguments:*
-| | ... | - frame_size - Framesize. Type: integer or string
-| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
-| |
-| | ... | *Returns:*
-| | ... | Average frame size including overhead. Type: float
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Get Average Frame Size \| IMIX_v4_1 \|
-| |
-| | [Arguments] | ${frame_size} | ${overhead}=${0}
-| |
-| | ${frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
-| | ... | Set Variable | ${353.83333}
-| | ... | ELSE
-| | ... | Convert To Number | ${frame_size}
-| | ${avg_frame_size} = | Evaluate | ${frame_size} + ${overhead}
-| | Return From Keyword | ${avg_frame_size}
-
-| Get Maximum Frame Size
-| | [Documentation]
-| | ... | Framesize can be either integer in case of a single packet
-| | ... | in stream, or set of packets in case of IMIX type or simmilar.
-| |
-| | ... | *Arguments:*
-| | ... | - frame_size - Framesize. Type: integer or string
-| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
-| |
-| | ... | *Returns:*
-| | ... | Maximum frame size including overhead. Type: float
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Get Maximum Frame Size \| IMIX_v4_1 \|
-| |
-| | [Arguments] | ${frame_size} | ${overhead}=${0}
-| |
-| | ${frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
-| | ... | Set Variable | ${1518}
-| | ... | ELSE
-| | ... | Convert To Number | ${frame_size}
-| | ${max_frame_size} = | Evaluate | ${frame_size} + ${overhead}
-| | Return From Keyword | ${max_frame_size}
-
-| Set Max Rate And Jumbo
-| | [Documentation]
-| | ... | Input framesize can be either integer in case of a single packet
-| | ... | in stream, or IMIX string defining mix of packets.
-| | ... | For jumbo frames detection, the maximal packet size is relevant.
-| | ... | For maximal transmit rate, the average packet size is relevant.
-| | ... | In both cases, encapsulation overhead (if any) has effect.
-| | ... | The maximal rate is computed from NIC name.
-| | ... | The implementation works by mapping from exact
-| | ... | whitelisted NIC names.
-| | ... | The mapping is hardcoded in nic_limits.yaml
-| | ... | TODO: Make the mapping from NIC names case insensistive.
-| |
-| | ... | TODO: Make pps limit also definable per NIC.
-| |
-| | ... | This keyword computes maximal unidirectional transmit rate
-| | ... | and jumbo boolean (some suites need that for configuration decisions).
-| | ... | To streamline suite autogeneration, both input and output values
-| | ... | are communicated as test (or broader scope) variables,
-| | ... | instead of explicit arguments and return values.
-| |
-| | ... | *Test (or broader scope) variables read:*
-| | ... | - nic_name - Name of bottleneck NIC. Type: string
-| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
-| |
-| | ... | *Test variables set:*
-| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
-| | ... | Type: float
-| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
-| | ... | enabled. Type: boolean
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Set Test Variable \| \${frame_size} \| IMIX_v4_1 \|
-| | ... | \| Set Max Rate And Jumbo \|
-| |
-| | # Negative overhead is possible, if DUT-DUT traffic is less encapsulated
-| | # than TG-DUT traffic.
-| | # TODO: Re-check overhead values in suites with both traffics encapsulated.
-| | # TODO: Improve layered setup to detect encap/decap and update overhead.
-| | ${overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
-| | ${pps_limit} = | Get From Dictionary
-| | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name}
-| | ${bps_limit} = | Get From Dictionary
-| | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name}
-| | # swo := size_with_overhead
-| | ${avg_swo} = | Get Average Frame Size | ${frame_size} | ${overhead}
-| | ${rate} = | Evaluate | ${bps_limit} / ((${avg_swo} + 20.0) * 8)
-| | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit}
-| | ... | ${pps_limit} | ${rate}
-| | Set Test Variable | \${max_rate}
-| | Set Jumbo
-
-| Set Jumbo
-| | [Documentation]
-| | ... | For jumbo frames detection, the maximal packet size is relevant,
-| | ... | encapsulation overhead (if any) has effect.
-| |
-| | ... | This keyword computes jumbo boolean (some suites need that for
-| | ... | configuration decisions).
-| | ... | To streamline suite autogeneration, both input and output values
-| | ... | are communicated as test (or broader scope) variables,
-| | ... | instead of explicit arguments and return values.
-| |
-| | ... | *Test (or broader scope) variables read:*
-| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
-| |
-| | ... | *Test variables set:*
-| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
-| | ... | enabled. Type: boolean
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Set Jumnbo \|
-| |
-| | ${overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
-| | ${max_swo} = | Get Maximum Frame Size | ${frame_size} | ${overhead}
-| | ${jumbo} = | Set Variable If | ${max_swo} < 1522
-| | ... | ${False} | ${True}
-| | Set Test Variable | \${jumbo}
diff --git a/resources/libraries/robot/performance/performance_utils.robot b/resources/libraries/robot/performance/performance_utils.robot
index 69402dcd41..5cf6dba84c 100644
--- a/resources/libraries/robot/performance/performance_utils.robot
+++ b/resources/libraries/robot/performance/performance_utils.robot
@@ -22,20 +22,120 @@
 | Library | resources.libraries.python.TrafficGenerator.TGDropRateSearchImpl
 | Library | resources.libraries.python.Trace
 | Variables | resources/libraries/python/Constants.py
+| Resource | resources/libraries/robot/performance/performance_actions.robot
+| Resource | resources/libraries/robot/performance/performance_display.robot
+| Resource | resources/libraries/robot/performance/performance_vars.robot
 |
 | Documentation
 | ... | Performance suite keywords - utilities to find and verify NDR and PDR.
+| ... | See performance_vars.robot for values accessed via there.
 
 *** Variables ***
-| ${trial_duration}= | ${PERF_TRIAL_DURATION}
-| ${trial_multiplicity}= | ${PERF_TRIAL_MULTIPLICITY}
-| ${extended_debug}= | ${EXTENDED_DEBUG}
 | # Variable holding multiplicator of main heap size. By default it is set to 1
 | # that means the main heap size will be set to 2G. Some tests may require more
 | # memory for IP FIB (e.g. nat44det tests with 4M or 16M sessions).
 | ${heap_size_mult}= | ${1}
 
 *** Keywords ***
+| Clear and show runtime counters with running traffic
+| | [Documentation]
+| | ... | Start traffic at specified rate then clear runtime counters on all
+| | ... | DUTs. Wait for specified amount of time and capture runtime counters
+| | ... | on all DUTs. Finally stop traffic.
+| |
+| | ... | TODO: Support resetter if this is not the first trial-ish action?
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Clear and show runtime counters with running traffic \|
+| |
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${runtime_duration} = | Get Runtime Duration
+| | ${runtime_rate} = | Get Runtime Rate
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | ${use_latency} = | Get Use Latency
+| | # Duration of -1 means we will stop traffic manually.
+| | Send traffic on tg
+| | ... | duration=${-1}
+| | ... | rate=${runtime_rate}
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | async_call=${True}
+| | ... | ppta=${ppta}
+| | ... | use_latency=${use_latency}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_type=${transaction_type}
+| | ... | duration_limit=${0.0}
+| | FOR | ${action} | IN | @{pre_run_stats}
+| | | Run Keyword | Additional Statistics Action For ${action}
+| | END
+| | Sleep | ${runtime_duration}
+| | FOR | ${action} | IN | @{post_run_stats}
+| | | Run Keyword | Additional Statistics Action For ${action}
+| | END
+| | Stop traffic on tg
+
+| Find critical load using PLRsearch
+| | [Documentation]
+| | ... | Find boundaries for troughput (of hardcoded target loss ratio)
+| | ... | using PLRsearch algorithm.
+| | ... | Display results as formatted test message.
+| | ... | Fail if computed lower bound is 110% of the minimal rate or less.
+| | ... | Input rates are understood as uni-directional,
+| | ... | reported result contains aggregate rates.
+| | ... | Currently, the min_rate value is hardcoded to match test teardowns.
+| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - traffic_profile - Name of module defining traffc for measurements.
+| | ... | Type: string
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
+| | ... | string
+| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
+| | ... | Type: float
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Find critical load using PLR search \|
+| |
+| | # Get values via performance_vars.
+| | ${max_rate} = | Get Max Rate
+| | ${min_rate} = | Get Min Rate
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${resetter} = | Get Resetter
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | ${use_latency} = | Get Use Latency
+| | ${average} | ${stdev} = | Perform soak search
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | minimum_transmit_rate=${min_rate}
+| | ... | maximum_transmit_rate=${max_rate}
+| | ... | plr_target=${1e-7}
+| | ... | tdpt=${0.1}
+| | ... | initial_count=${50}
+| | ... | ppta=${ppta}
+| | ... | resetter=${resetter}
+| | ... | timeout=${1800.0}
+| | ... | trace_enabled=${False}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_type=${transaction_type}
+| | ... | use_latency=${use_latency}
+| | ${lower} | ${upper} = | Display result of soak search
+| | ... | ${average} | ${stdev}
+| | Set Test Variable | \${rate for teardown} | ${lower}
+| | Should Not Be True | 1.1 * ${min_rate} > ${lower}
+| | ... | Lower bound ${lower} too small for unidirectional minimum ${min_rate}.
+
 | Find NDR and PDR intervals using optimized search
 | | [Documentation]
 | | ... | Find boundaries for RFC2544 compatible NDR and PDR values
@@ -45,386 +145,243 @@
 | | ... | Input rates are understood as uni-directional,
 | | ... | reported result contains aggregate rates.
 | | ... | Additional latency measurements are performed for smaller loads,
-| | ... | their results are also displayed.
+| | ... | even if latency stream is disabled in search. Their results
+| | ... | are also displayed.
 | | ... | Finally, two measurements for runtime stats are done (not displayed).
 | | ... | Currently, the min_rate value is hardcoded to 90kpps,
 | | ... | allowing measurement at 10% of the discovered rate
 | | ... | without breaking latency streams.
+| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
 | |
 | | ... | *Test (or broader scope) variables read:*
 | | ... | - traffic_profile - Name of module defining traffc for measurements.
 | | ... | Type: string
 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
 | | ... | string
-| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
+| | ... | - max_rate - Calculated maximal transmit rate [tps].
 | | ... | Type: float
-| |
-| | ... | *Arguments:*
-| | ... | - packet_loss_ratio - Accepted loss during search. Type: float
-| | ... | - final_relative_width - Maximal width multiple of upper. Type: float
-| | ... | - final_trial_duration - Duration of final trials [s]. Type: float
-| | ... | - initial_trial_duration - Duration of initial trials [s]. Type: float
-| | ... | - intermediate phases - Number of intermediate phases [1].
-| | ... | Type: integer
-| | ... | - timeout - Fail if search duration is longer [s]. Type: float
-| | ... | - doublings - How many doublings to do when expanding [1].
-| | ... | Type: integer
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - latency_duration - Duration for latency-specific trials. Type: float
-| | ... | - latency - False to disable latency measurement; default value: True.
-| | ... | Type: boolean
+| | ... | - resetter - Callable to reset DUT state before each trial.
+| | ... | - transaction_scale - Number of ASTF transaction (zero if unlimited).
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
+| | ... | - disable_latency - If true, skip anything related to latency.
+| | ... | Useful if transaction_scale is high and TPS is low. Default: false.
 | |
 | | ... | *Example:*
 | |
-| | ... | \| Find NDR and PDR intervals using optimized search \| \${0.005} \
-| | ... | \| \${0.005} \| \${30.0} \| \${1.0} \| \${2} \| \${600.0} \| \${2} \
-| | ... | \| \${2} \| ${5.0} \|
-| |
-| | [Arguments] | ${packet_loss_ratio}=${0.005}
-| | ... | ${final_relative_width}=${0.005} | ${final_trial_duration}=${30.0}
-| | ... | ${initial_trial_duration}=${1.0}
-| | ... | ${number_of_intermediate_phases}=${2} | ${timeout}=${720.0}
-| | ... | ${doublings}=${2} | ${traffic_directions}=${2}
-| | ... | ${latency_duration}=${PERF_TRIAL_LATENCY_DURATION}
-| | ... | ${latency}=${True}
-| |
-| | # Latency measurements will need more than 9000 pps.
-| | ${result} = | Perform optimized ndrpdr search | ${frame_size}
-| | ... | ${traffic_profile} | ${9001} | ${max_rate}
-| | ... | ${packet_loss_ratio} | ${final_relative_width}
-| | ... | ${final_trial_duration} | ${initial_trial_duration}
-| | ... | ${number_of_intermediate_phases} | timeout=${timeout}
-| | ... | doublings=${doublings} | traffic_directions=${traffic_directions}
-| | ... | latency=${latency}
+| | ... | \| Find NDR and PDR intervals using optimized search \|
+| |
+| | # Get values via performance_vars.
+| | ${disable_latency} = | Get Disable Latency
+| | ${max_rate} = | Get Max Rate
+| | ${min_rate} = | Get Min Rate
+| | # \${packet_loss_ratio} is used twice so it is worth a variable.
+| | ${packet_loss_ratio} = | Get Packet Loss Ratio
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${resetter} = | Get Resetter
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | ${use_latency} = | Get Use Latency
+| | ${result} = | Perform optimized ndrpdr search
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | minimum_transmit_rate=${min_rate}
+| | ... | maximum_transmit_rate=${max_rate}
+| | ... | packet_loss_ratio=${packet_loss_ratio}
+| | ... | final_relative_width=${0.005}
+| | ... | final_trial_duration=${30.0}
+| | ... | initial_trial_duration=${1.0}
+| | ... | number_of_intermediate_phases=${2}
+| | ... | timeout=${720.0}
+| | ... | doublings=${2}
+| | ... | ppta=${ppta}
+| | ... | resetter=${resetter}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_type=${transaction_type}
+| | ... | use_latency=${use_latency}
 | | Display result of NDRPDR search | ${result}
 | | Check NDRPDR interval validity | ${result.pdr_interval}
 | | ... | ${packet_loss_ratio}
 | | Check NDRPDR interval validity | ${result.ndr_interval}
-| | ${pdr_sum}= | Set Variable | ${result.pdr_interval.measured_low.target_tr}
-| | ${pdr_per_stream}= | Evaluate | ${pdr_sum} / float(${traffic_directions})
-| | ${ndr_sum}= | Set Variable | ${result.ndr_interval.measured_low.target_tr}
-| | ${ndr_per_stream}= | Evaluate | ${ndr_sum} / float(${traffic_directions})
-| | ${rate}= | Evaluate | 0.9 * ${pdr_per_stream}
-| | Run Keyword If | ${latency}
-| | ... | Measure and show latency at specified rate | Latency at 90% PDR:
-| | ... | ${latency_duration} | ${rate} | ${framesize}
-| | ... | ${traffic_profile} | ${traffic_directions}
-| | ${rate}= | Evaluate | 0.5 * ${pdr_per_stream}
-| | Run Keyword If | ${latency}
-| | ... | Measure and show latency at specified rate | Latency at 50% PDR:
-| | ... | ${latency_duration} | ${rate} | ${framesize}
-| | ... | ${traffic_profile} | ${traffic_directions}
-| | ${rate}= | Evaluate | 0.1 * ${pdr_per_stream}
-| | Run Keyword If | ${latency}
-| | ... | Measure and show latency at specified rate | Latency at 10% PDR:
-| | ... | ${latency_duration} | ${rate} | ${framesize}
-| | ... | ${traffic_profile} | ${traffic_directions}
-| | Run Keyword If | ${latency}
-| | ... | Measure and show latency at specified rate | Latency at 0% PDR:
-| | ... | ${latency_duration} | ${0} | ${framesize}
-| | ... | ${traffic_profile} | ${traffic_directions}
-| | # Finally, trials with runtime and other stats.
+| | ${pdr} = | Set Variable | ${result.pdr_interval.measured_low.target_tr}
+| | ${ndr} = | Set Variable | ${result.ndr_interval.measured_low.target_tr}
 | | # We expect NDR and PDR to have different-looking stats.
 | | Send traffic at specified rate
-| | ... | ${1.0} | ${pdr_per_stream} | ${framesize} | ${traffic_profile}
-| | ... | traffic_directions=${traffic_directions}
-| | Send traffic at specified rate
-| | ... | ${1.0} | ${ndr_per_stream} | ${framesize} | ${traffic_profile}
-| | ... | traffic_directions=${traffic_directions}
+| | ... | rate=${pdr}
+| | ... | trial_duration=${1.0}
+| | ... | trial_multiplicity=${1}
+| | ... | use_latency=${use_latency}
+| | ... | duration_limit=${1.0}
+| | Run Keyword If | ${ndr} != ${pdr}
+| | ... | Send traffic at specified rate
+| | ... | rate=${ndr}
+| | ... | trial_duration=${1.0}
+| | ... | trial_multiplicity=${1}
+| | ... | use_latency=${use_latency}
+| | ... | duration_limit=${1.0}
+| | Return From Keyword If | ${disable_latency}
+| | ${rate} = | Evaluate | 0.9 * ${pdr}
+| | Measure and show latency at specified rate | Latency at 90% PDR: | ${rate}
+| | ${rate} = | Evaluate | 0.5 * ${pdr}
+| | Measure and show latency at specified rate | Latency at 50% PDR: | ${rate}
+| | ${rate} = | Evaluate | 0.1 * ${pdr}
+| | Measure and show latency at specified rate | Latency at 10% PDR: | ${rate}
+| | Measure and show latency at specified rate | Latency at 0% PDR: | ${0.0}
 
 | Find Throughput Using MLRsearch
 | | [Documentation]
-| | ... | Find and return lower bound PDR (zero PLR by default)
+| | ... | Find and return lower bound NDR (zero PLR)
 | | ... | aggregate throughput using MLRsearch algorithm.
 | | ... | Input rates are understood as uni-directional.
 | | ... | Currently, the min_rate value is hardcoded to match test teardowns.
+| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
 | |
 | | ... | *Test (or broader scope) variables read:*
 | | ... | - traffic_profile - Name of module defining traffc for measurements.
 | | ... | Type: string
 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
 | | ... | string
-| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
+| | ... | - max_rate - Calculated maximal transmit rate [tps].
 | | ... | Type: float
-| |
-| | ... | *Arguments:*
-| | ... | - packet_loss_ratio - Accepted loss during search. Type: float
-| | ... | - final_relative_width - Maximal width multiple of upper. Type: float
-| | ... | - final_trial_duration - Duration of final trials [s]. Type: float
-| | ... | - initial_trial_duration - Duration of initial trials [s]. Type: float
-| | ... | - intermediate phases - Number of intermediate phases [1].
-| | ... | Type: integer
-| | ... | - timeout - Fail if search duration is longer [s]. Type: float
-| | ... | - doublings - How many doublings to do when expanding [1].
-| | ... | Type: integer
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - latency - True to enable latency measurement; default value: False.
-| | ... | Type: boolean
+| | ... | - resetter - Callable to reset DUT state before each trial.
+| | ... | - transaction_scale - Number of ASTF transaction (zero if unlimited).
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
 | |
 | | ... | *Returns:*
 | | ... | - Lower bound for bi-directional throughput at given PLR. Type: float
 | |
 | | ... | *Example:*
 | |
-| | ... | \| \${throughpt}= \| Find Throughput Using MLRsearch \| \${0} \
-| | ... | \| \${0.001} \| \${10.0}\| \${1.0} \| \${1} \| \${720.0} \| \${2} \
-| | ... | \| \${2} \|
-| |
-| | [Arguments] | ${packet_loss_ratio}=${0.0}
-| | ... | ${final_relative_width}=${0.001} | ${final_trial_duration}=${10.0}
-| | ... | ${initial_trial_duration}=${1.0}
-| | ... | ${number_of_intermediate_phases}=${1} | ${timeout}=${720.0}
-| | ... | ${doublings}=${2} | ${traffic_directions}=${2} | ${latency}=${False}
-| |
-| | ${result} = | Perform optimized ndrpdr search | ${frame_size}
-| | ... | ${traffic_profile} | ${10000} | ${max_rate}
-| | ... | ${packet_loss_ratio} | ${final_relative_width}
-| | ... | ${final_trial_duration} | ${initial_trial_duration}
-| | ... | ${number_of_intermediate_phases} | timeout=${timeout}
-| | ... | doublings=${doublings} | traffic_directions=${traffic_directions}
-| | ... | latency=${latency}
+| | ... | \| \${throughpt}= \| Find Throughput Using MLRsearch \|
+| |
+| | ${max_rate} = | Get Max Rate
+| | ${min_rate} = | Get Min Rate
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${resetter} = | Get Resetter
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | ${use_latency} = | Get Use Latency
+| | ${result} = | Perform optimized ndrpdr search
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | minimum_transmit_rate=${min_rate}
+| | ... | maximum_transmit_rate=${max_rate}
+| | ... | packet_loss_ratio=${0.0}
+| | ... | final_relative_width=${0.001}
+| | ... | final_trial_duration=${10.0}
+| | ... | initial_trial_duration=${1.0}
+| | ... | number_of_intermediate_phases=${1}
+| | ... | timeout=${720}
+| | ... | doublings=${2}
+| | ... | ppta=${ppta}
+| | ... | resetter=${resetter}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_type=${transaction_type}
+| | ... | use_latency=${use_latency}
 | | Check NDRPDR interval validity | ${result.pdr_interval}
-| | ... | ${packet_loss_ratio}
+| | ... | ${0.0}
 | | Return From Keyword | ${result.pdr_interval.measured_low.target_tr}
 
-| Find critical load using PLRsearch
-| | [Documentation]
-| | ... | Find boundaries for troughput (of given target loss ratio)
-| | ... | using PLRsearch algorithm.
-| | ... | Display results as formatted test message.
-| | ... | Fail if computed lower bound 110% of the minimal rate or less.
-| | ... | Input rates are understood as uni-directional,
-| | ... | reported result contains aggregate rates.
-| | ... | Currently, the min_rate value is hardcoded to match test teardowns.
-| |
-| | ... | *Test (or broader scope) variables read:*
-| | ... | - traffic_profile - Name of module defining traffc for measurements.
-| | ... | Type: string
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
-| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
-| | ... | Type: float
-| |
-| | ... | *Arguments:*
-| | ... | - packet_loss_ratio - Accepted loss during search. Type: float
-| | ... | - timeout - Stop when search duration is longer [s]. Type: float
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - latency - True to enable latency measurement; default value: False.
-| | ... | Type: boolean
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Find critical load using PLR search \| \${1e-7} \| \${120} \
-| | ... | \| \${2} \|
-| |
-| | [Arguments] | ${packet_loss_ratio}=${1e-7} | ${timeout}=${1800.0}
-| | ... | ${traffic_directions}=${2} | ${latency}=${False}
-| |
-| | ${min_rate} = | Set Variable | ${10000}
-| | ${average} | ${stdev} = | Perform soak search | ${frame_size}
-| | ... | ${traffic_profile} | ${min_rate} | ${max_rate}
-| | ... | ${packet_loss_ratio} | timeout=${timeout}
-| | ... | traffic_directions=${traffic_directions} | latency=${latency}
-| | ${lower} | ${upper} = | Display result of soak search
-| | ... | ${average} | ${stdev}
-| | Should Not Be True | 1.1 * ${traffic_directions} * ${min_rate} > ${lower}
-| | ... | Lower bound ${lower} too small for unidirectional minimum ${min_rate}.
-
-| Display single bound
-| | [Documentation]
-| | ... | Display one bound of NDR+PDR search,
-| | ... | in packet per seconds (total and per stream)
-| | ... | and Gbps total bandwidth (for initial packet size).
-| | ... | Througput is calculated as:
-| | ... | Sum of measured rates over streams
-| | ... | Bandwidth is calculated as:
-| | ... | (Throughput * (L2 Frame Size + IPG) * 8)
-| | ... | The given result should contain latency data as well.
-| |
-| | ... | *Arguments:*
-| | ... | - text - Flavor text describing which bound is this. Type: string
-| | ... | - rate_total - Total (not per stream) measured Tr [pps]. Type: float
-| | ... | - frame_size - L2 Frame Size [B]. Type: integer
-| | ... | - latency - Latency data to display if non-empty. Type: string
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Display single bound \| NDR lower bound \| \${12345.67} \
-| | ... | \| \${64} \| latency=\${EMPTY} \|
-| |
-| | [Arguments] | ${text} | ${rate_total} | ${frame_size} | ${latency}=${EMPTY}
-| |
-| | ${bandwidth_total} = | Evaluate | ${rate_total} * (${frame_size}+20)*8 / 1e9
-| | Set Test Message | ${\n}${text}: ${rate_total} pps, | append=yes
-| | Set Test Message | ${bandwidth_total} Gbps (initial) | append=yes
-| | Return From Keyword If | not """${latency}"""
-| | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency}
-| | ... | append=yes
-
-| Display Reconfig Test Message
-| | [Documentation]
-| | ... | Display the number of packets lost (bidirectionally)
-| | ... | due to reconfiguration under traffic.
-| |
-| | ... | *Arguments:*
-| | ... | - result - Result of bidirectional measurtement.
-| | ... | Type: ReceiveRateMeasurement
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Display Reconfig Test Message \| \${result} \|
-| |
-| | [Arguments] | ${result}
-| |
-| | Set Test Message | Packets lost due to reconfig: ${result.loss_count}
-| | ${time_lost} = | Evaluate | ${result.loss_count} / ${result.target_tr}
-| | Set Test Message | ${\n}Implied time lost: ${time_lost} | append=yes
-
-| Display result of NDRPDR search
-| | [Documentation]
-| | ... | Display result of NDR+PDR search, both quantities, both bounds,
-| | ... | aggregate in packet per seconds
-| | ... | and Gbps total bandwidth (for initial packet size).
-| | ... | Througput is calculated as:
-| | ... | Sum of measured rate over streams
-| | ... | Bandwidth is calculated as:
-| | ... | (Throughput * (L2 Frame Size + IPG) * 8)
-| | ... | The given result should contain latency data as well.
-| |
-| | ... | *Test (or broader scope) variables read:*
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
-| | ... | *Arguments:*
-| | ... | - result - Measured result data per stream [pps]. Type: NdrPdrResult
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Display result of NDRPDR search \| \${result} \|
-| |
-| | [Arguments] | ${result}
-| |
-| | ${frame_size} = | Get Average Frame Size | ${frame_size}
-| | Display single bound | NDR_LOWER
-| | ... | ${result.ndr_interval.measured_low.target_tr} | ${frame_size}
-| | ... | ${result.ndr_interval.measured_low.latency}
-| | Display single bound | NDR_UPPER
-| | ... | ${result.ndr_interval.measured_high.target_tr} | ${frame_size}
-| | Display single bound | PDR_LOWER
-| | ... | ${result.pdr_interval.measured_low.target_tr} | ${frame_size}
-| | ... | ${result.pdr_interval.measured_low.latency}
-| | Display single bound | PDR_UPPER
-| | ... | ${result.pdr_interval.measured_high.target_tr} | ${frame_size}
-
-| Display result of soak search
-| | [Documentation]
-| | ... | Display result of soak search, avg+-stdev, as upper/lower bounds,
-| | ... | in aggregate packets per seconds
-| | ... | and Gbps total bandwidth (for initial packet size).
-| | ... | Througput is calculated as:
-| | ... | Sum of measured rates over streams
-| | ... | Bandwidth is calculated as:
-| | ... | (Throughput * (L2 Frame Size + IPG) * 8)
-| |
-| | ... | *Test (or broader scope) variables read:*
-| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
-| | ... | string
-| | ... | *Arguments:*
-| | ... | - avg - Estimated average critical load [pps]. Type: float
-| | ... | - stdev - Standard deviation of critical load [pps]. Type: float
-| |
-| | ... | *Returns:*
-| | ... | - Lower and upper bound of critical load [pps]. Type: 2-tuple of float
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Display result of soak search \| \${100000} \| \${100} \|
-| |
-| | [Arguments] | ${avg} | ${stdev}
-| |
-| | ${frame_size} = | Get Average Frame Size | ${frame_size}
-| | ${avg} = | Convert To Number | ${avg}
-| | ${stdev} = | Convert To Number | ${stdev}
-| | ${lower} = | Evaluate | ${avg} - ${stdev}
-| | ${upper} = | Evaluate | ${avg} + ${stdev}
-| | Display single bound | PLRsearch lower bound | ${lower} | ${frame_size}
-| | Display single bound | PLRsearch upper bound | ${upper} | ${frame_size}
-| | Return From Keyword | ${lower} | ${upper}
-
-| Check NDRPDR interval validity
+| Measure and show latency at specified rate
 | | [Documentation]
-| | ... | Extract loss ratio of lower bound of the interval.
-| | ... | Fail if it does not reach the allowed value.
+| | ... | Send traffic at specified rate, single trial.
+| | ... | Extract latency information and append it to text message.
+| | ... | The rate argument is int, so should not include "pps".
+| | ... | If the given rate is too low, a safe value is used instead.
+| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
 | |
 | | ... | *Arguments:*
-| | ... | - interval - Measured interval. Type: ReceiveRateInterval
-| | ... | - packet_loss_ratio - Accepted loss (0.0 for NDR). Type: float
+| | ... | - message_prefix - Preface to test message addition. Type: string
+| | ... | - rate - Rate [tps] for sending packets.
+| | ... | Type: float
 | |
 | | ... | *Example:*
 | |
-| | ... | \| Check NDRPDR interval validity \| \${result.pdr_interval} \
-| | ... | \| \${0.005} \|
-| |
-| | [Arguments] | ${interval} | ${packet_loss_ratio}=${0.0}
-| |
-| | ${lower_bound} = | Set Variable | ${interval.measured_low}
-| | ${lower_bound_lf} = | Set Variable | ${lower_bound.loss_fraction}
-| | Return From Keyword If | ${lower_bound_lf} <= ${packet_loss_ratio}
-| | ${message}= | Catenate | SEPARATOR=${SPACE}
-| | ... | Minimal rate loss fraction ${lower_bound_lf}
-| | ... | does not reach target ${packet_loss_ratio}.
-| | ${message_zero} = | Set Variable | Zero packets forwarded!
-| | ${message_other} = | Set Variable | ${lower_bound.loss_count} packets lost.
-| | ${message} = | Set Variable If | ${lower_bound_lf} >= 1.0
-| | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other}
-| | Fail | ${message}
+| | ... | \| Measure and show latency at specified rate \| Latency at 90% NDR \
+| | ... | \| ${10000000} \|
+| |
+| | [Arguments] | ${message_prefix} | ${rate}
+| |
+| | ${min_rate} = | Get Min Rate
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${real_rate} = | Evaluate | max(${rate}, ${min_rate})
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | Call Resetter
+| | Send traffic on tg
+| | ... | duration=${PERF_TRIAL_LATENCY_DURATION}
+| | ... | rate=${real_rate}
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | async_call=${False}
+| | ... | duration_limit=${PERF_TRIAL_LATENCY_DURATION}
+| | ... | ppta=${ppta}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_type=${transaction_type}
+| | ... | use_latency=${True}
+| | ${latency} = | Get Latency Int
+| | Set Test Message | ${\n}${message_prefix} ${latency} | append=${True}
 
-| Traffic should pass with maximum rate
+| Send ramp-up traffic
 | | [Documentation]
-| | ... | Send traffic at maximum rate.
+| | ... | Do nothing unless positive ramp-up duration is specified.
+| | ... | Else perform one trial with appropriate rate and duration.
+| | ... | This is useful for tests that set DUT state via traffic.
+| | ... | Rate has to bee low enough so packets are not lost,
+| | ... | Duration has to be long enough to set all the state.
+| | ... | The trial results are discarded.
 | |
 | | ... | *Test (or broader scope) variables read:*
 | | ... | - traffic_profile - Name of module defining traffic for measurements.
 | | ... | Type: string
 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
 | | ... | string
-| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
+| | ... | - ramp_up_duration - Suitable traffic duration [s].
 | | ... | Type: float
-| |
-| | ... | *Arguments:*
-| | ... | - trial_duration - Duration of single trial [s]. Type: float
-| | ... | - fail_no_traffic - Whether to fail on zero receive count;
-| | ... | default value: True. Type: boolean
-| | ... | - trial_multiplicity - How many trials in this measurement.
-| | ... | Type: integer
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic;
-| | ... | default value: 2. Type: integer
-| | ... | - tx_port - TX port of TG; default value: 0. Type: integer
-| | ... | - rx_port - RX port of TG; default value: 1. Type: integer
-| | ... | - latency - True to enable latency measurement; default value: False.
-| | ... | Type: boolean
+| | ... | - ramp_up_rate - Suitable transmit rate [tps].
+| | ... | Type: float
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
 | |
 | | ... | *Example:*
 | |
-| | ... | \| Traffic should pass with maximum rate \| \${1} \| \${False} \
-| | ... | \| \${10.0} \| \${2} \| \${0} \| \${1} \| \${True} \|
-| |
-| | [Arguments] | ${trial_duration}=${trial_duration}
-| | ... | ${fail_no_traffic}=${True}
-| | ... | ${trial_multiplicity}=${trial_multiplicity}
-| | ... | ${traffic_directions}=${2} | ${tx_port}=${0} | ${rx_port}=${1}
-| | ... | ${latency}=${False}
-| |
-| | ${results}= | Send traffic at specified rate
-| | ... | ${trial_duration} | ${max_rate} | ${frame_size}
-| | ... | ${traffic_profile} | ${trial_multiplicity}
-| | ... | ${traffic_directions} | ${tx_port} | ${rx_port} | latency=${latency}
-| | Set Test Message | ${\n}Maximum Receive Rate trial results
-| | Set Test Message | in packets per second: ${results}
-| | ... | append=yes
-| | Run Keyword If | ${fail_no_traffic} | Fail if no traffic forwarded
+| | ... | \| Send ramp-up traffic \|
+| |
+| | ${ramp_up_duration} = | Get Ramp Up Duration
+| | Run Keyword Unless | ${ramp_up_duration} > 0.0 | Return From Keyword
+| | ${ramp_up_rate} = | Get Ramp Up Rate
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | ${use_latency} = | Get Use Latency
+| | Send traffic on tg
+| | ... | duration=${ramp_up_duration}
+| | ... | rate=${ramp_up_rate}
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | async_call=${False}
+| | ... | duration_limit=${0.0}
+| | ... | ppta=${ppta}
+| | ... | use_latency=${use_latency}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_type=${transaction_type}
 
 | Send traffic at specified rate
 | | [Documentation]
@@ -432,178 +389,62 @@
 | | ... | Then send traffic at specified rate, possibly multiple trials.
 | | ... | Show various DUT stats, optionally also packet trace.
 | | ... | Return list of measured receive rates.
+| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
 | |
 | | ... | *Arguments:*
 | | ... | - trial_duration - Duration of single trial [s]. Type: float
-| | ... | - rate - Target aggregate transmit rate [pps] / Connections per second
-| | ... | (CPS) for UDP/TCP flows. Type: float
-| | ... | - frame_size - L2 Frame Size [B]. Type: integer or string
-| | ... | - traffic_profile - Name of module defining traffc for measurements.
+| | ... | - rate - Target transmit rate [tps]. Type: float
 | | ... | Type: string
 | | ... | - trial_multiplicity - How many trials in this measurement.
-| | ... | Type: integer
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - tx_port - TX port of TG; default value: 0. Type: integer
-| | ... | - rx_port - RX port of TG; default value: 1. Type: integer
-| | ... | - extended_debug - True to enable extended debug.
 | | ... | Type: boolean
-| | ... | - latency - True to enable latency measurement; default value: False.
+| | ... | - use_latency - Use latency stream in search; default value: False.
 | | ... | Type: boolean
+| | ... | - duration_limit - Hard limit for trial duration, overriding duration
+| | ... | computed from transaction_scale. Default 0.0 means no limit.
 | |
 | | ... | *Example:*
 | |
 | | ... | \| Send traffic at specified rate \| \${1.0} \| ${4000000.0} \
-| | ... | \| \${64} \| 3-node-IPv4 \| \${10} \| \${2} \| \${0} \| \${1} \
-| | ... | \| ${False} \| ${True} \|
+| | ... | \| \${10} \| ${False} \| ${1.0} \|
 | |
-| | [Arguments] | ${trial_duration} | ${rate} | ${frame_size}
-| | ... | ${traffic_profile} | ${trial_multiplicity}=${trial_multiplicity}
-| | ... | ${traffic_directions}=${2} | ${tx_port}=${0} | ${rx_port}=${1}
-| | ... | ${extended_debug}=${extended_debug} | ${latency}=${False}
-| |
-| | Set Test Variable | ${extended_debug}
-| | # Following setting of test variables is needed for some pre_stats actions.
-| | Set Test Variable | ${rate}
-| | Set Test Variable | ${traffic_directions}
-| | Set Test Variable | ${tx_port}
-| | Set Test Variable | ${rx_port}
+| | [Arguments] | ${trial_duration} | ${rate} | ${trial_multiplicity}
+| | ... | ${use_latency}=${False} | ${duration_limit}=${0.0}
 | |
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | Set Test Variable | \${rate_for_teardown} | ${rate}
 | | FOR | ${action} | IN | @{pre_stats}
 | | | Run Keyword | Additional Statistics Action For ${action}
 | | END
 | | ${results} = | Create List
 | | FOR | ${i} | IN RANGE | ${trial_multiplicity}
-| | | # The following line is skipping some default arguments,
-| | | # that is why subsequent arguments have to be named.
+| | | Call Resetter
 | | | Send traffic on tg
-| | | ... | ${trial_duration} | ${rate} | ${frame_size} | ${traffic_profile}
-| | | ... | warmup_time=${0} | traffic_directions=${traffic_directions}
-| | | ... | tx_port=${tx_port} | rx_port=${rx_port} | latency=${latency}
-| | | ${rx} = | Get Received
-| | | ${rr} = | Evaluate | ${rx} / ${trial_duration}
-| | | Append To List | ${results} | ${rr}
+| | | ... | duration=${trial_duration}
+| | | ... | rate=${rate}
+| | | ... | frame_size=${frame_size}
+| | | ... | traffic_profile=${traffic_profile}
+| | | ... | async_call=${False}
+| | | ... | duration_limit=${duration_limit}
+| | | ... | ppta=${ppta}
+| | | ... | traffic_directions=${traffic_directions}
+| | | ... | transaction_duration=${transaction_duration}
+| | | ... | transaction_scale=${transaction_scale}
+| | | ... | transaction_type=${transaction_type}
+| | | ... | use_latency=${use_latency}
+| | | ${result}= | Get Measurement Result
+| | | # Out of several quantities for aborted traffic (duration stretching),
+| | | # the approximated receive rate is the best estimate we have.
+| | | Append To List | ${results} | ${result.approximated_receive_rate}
 | | END
 | | FOR | ${action} | IN | @{post_stats}
 | | | Run Keyword | Additional Statistics Action For ${action}
 | | END
 | | Return From Keyword | ${results}
 
-| Measure and show latency at specified rate
-| | [Documentation]
-| | ... | Send traffic at specified rate, single trial.
-| | ... | Extract latency information and append it to text message.
-| | ... | The rate argument is int, so should not include "pps".
-| | ... | If the given rate is too low, a safe value is used instead.
-| |
-| | ... | *Arguments:*
-| | ... | - message_prefix - Preface to test message addition. Type: string
-| | ... | - trial_duration - Duration of single trial [s]. Type: float
-| | ... | - rate - Rate [pps] for sending packets in case of T-Rex stateless
-| | ... | mode or multiplier of profile CPS in case of T-Rex astf mode.
-| | ... | Type: float
-| | ... | - frame_size - L2 Frame Size [B]. Type: integer or string
-| | ... | - traffic_profile - Name of module defining traffic for measurements.
-| | ... | Type: string
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - tx_port - TX port of TG; default value: 0. Type: integer
-| | ... | - rx_port - RX port of TG; default value: 1. Type: integer
-| | ... | - safe_rate - To apply if rate is below this, as latency pps is fixed.
-| | ... | In pps. Type: integer.
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Measure and show latency at specified rate \| Latency at 90% NDR \
-| | ... | \| \${1.0} \| ${10000000} \| \${64} \| 3-node-IPv4 \| \${2} \
-| | ... | \| \${0} \| \${1} \| ${9500} \|
-| |
-| | [Arguments] | ${message_prefix} | ${trial_duration} | ${rate}
-| | ... | ${frame_size} | ${traffic_profile} | ${traffic_directions}=${2}
-| | ... | ${tx_port}=${0} | ${rx_port}=${1} | ${safe_rate}=${9001}
-| |
-| | ${real_rate} = | Evaluate | max(${rate}, ${safe_rate})
-| | # The following line is skipping some default arguments,
-| | # that is why subsequent arguments have to be named.
-| | Send traffic on tg | ${trial_duration} | ${real_rate} | ${frame_size}
-| | ... | ${traffic_profile} | warmup_time=${0}
-| | ... | traffic_directions=${traffic_directions} | tx_port=${tx_port}
-| | ... | rx_port=${rx_port} | latency=${True}
-| | ${latency} = | Get Latency Int
-| | Set Test Message | ${\n}${message_prefix} ${latency} | append=${True}
-
-| Clear and show runtime counters with running traffic
-| | [Documentation]
-| | ... | Start traffic at specified rate then clear runtime counters on all
-| | ... | DUTs. Wait for specified amount of time and capture runtime counters
-| | ... | on all DUTs. Finally stop traffic
-| |
-| | ... | *Arguments:*
-| | ... | - duration - Duration of traffic run [s]. Type: integer
-| | ... | - rate - Rate [pps] for sending packets in case of T-Rex stateless
-| | ... | mode or multiplier of profile CPS in case of T-Rex astf mode.
-| | ... | Type: float
-| | ... | - frame_size - L2 Frame Size [B] or IMIX_v4_1. Type: integer or string
-| | ... | - traffic_profile - Name of module defining traffc for measurements.
-| | ... | Type: string
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - tx_port - TX port of TG; default value: 0. Type: integer
-| | ... | - rx_port - RX port of TG, default value: 1. Type: integer
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Clear and show runtime counters with running traffic \| \${10} \
-| | ... | \| ${4000000.0} \| \${64} \| 3-node-IPv4 \| \${2} \| \${0} \| \${1} \|
-| |
-| | [Arguments] | ${duration} | ${rate} | ${frame_size} | ${traffic_profile}
-| | ... | ${traffic_directions}=${2} | ${tx_port}=${0} | ${rx_port}=${1}
-| |
-| | # Duration of -1 means we will stop traffic manually.
-| | Send traffic on tg | ${-1} | ${rate} | ${frame_size} | ${traffic_profile}
-| | ... | warmup_time=${0} | async_call=${True} | latency=${False}
-| | ... | traffic_directions=${traffic_directions} | tx_port=${tx_port}
-| | ... | rx_port=${rx_port}
-| | FOR | ${action} | IN | @{pre_run_stats}
-| | | Run Keyword | Additional Statistics Action For ${action}
-| | END
-| | Sleep | ${duration}
-| | FOR | ${action} | IN | @{post_run_stats}
-| | | Run Keyword | Additional Statistics Action For ${action}
-| | END
-| | Stop traffic on tg
-
-| Send ramp-up traffic
-| | [Documentation]
-| | ... | Start ramp-up traffic at specified rate for defined duration.
-| |
-| | ... | *Arguments:*
-| | ... | - duration - Duration of traffic run [s]. Type: integer
-| | ... | - rate - Rate [pps] for sending packets in case of T-Rex stateless
-| | ... | mode or multiplier of profile CPS in case of T-Rex astf mode.
-| | ... | Type: float
-| | ... | - frame_size - L2 Frame Size [B] or IMIX_v4_1. Type: integer or string
-| | ... | - traffic_profile - Name of module defining traffc for measurements.
-| | ... | Type: string
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - tx_port - TX port of TG; default value: 0. Type: integer
-| | ... | - rx_port - RX port of TG, default value: 1. Type: integer
-| |
-| | ... | *Example:*
-| |
-| | ... | \| Send ramp-up traffic \| \${10} \| ${400000.0} \| ${64} \
-| | ... | \| ${2} \| ${0} \| ${1} \|
-| |
-| | [Arguments] | ${duration}=${ramp_up_duration} | ${rate}=${ramp_up_rate}
-| | ... | ${frame_size}=${frame_size} | ${traffic_profile}=${traffic_profile}
-| | ... | ${traffic_directions}=${2} | ${tx_port}=${0} | ${rx_port}=${1}
-| |
-| | Send traffic on tg
-| | ... | ${duration} | ${rate} | ${frame_size} | ${traffic_profile}
-| | ... | warmup_time=${0} | traffic_directions=${traffic_directions}
-| | ... | tx_port=${tx_port} | rx_port=${rx_port} | latency=${False}
-
 | Start Traffic on Background
 | | [Documentation]
 | | ... | Start traffic at specified rate then return control to Robot.
@@ -616,27 +457,36 @@
 | | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
 | | ... | string
 | | ... | *Arguments:*
-| | ... | - rate - Rate [pps] for sending packets in case of T-Rex stateless
-| | ... | mode or multiplier of profile CPS in case of T-Rex astf mode.
+| | ... | - rate - Rate [tps] for sending packets.
 | | ... | Type: float
-| | ... | - traffic_directions - Bi- (2) or uni- (1) directional traffic.
-| | ... | Type: integer
-| | ... | - tx_port - TX port of TG; default value: 0. Type: integer
-| | ... | - rx_port - RX port of TG; default value: 1. Type: integer
 | |
 | | ... | *Example:*
 | |
-| | ... | \| Start Traffic on Background \| ${4000000.0} \| \${2} \| \${0} \
-| | ... | \| \${1} \|
+| | ... | \| Start Traffic on Background \| ${4000000.0} \|
 | |
-| | [Arguments] | ${rate} | ${traffic_directions}=${2} | ${tx_port}=${0}
-| | ... | ${rx_port}=${1}
+| | [Arguments] | ${rate}
 | |
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${traffic_directions} = | Get Traffic Directions
+| | ${transaction_duration} = | Get Transaction Duration
+| | ${transaction_scale} = | Get Transaction Scale
+| | ${transaction_type} = | Get Transaction Type
+| | ${use_latency} = | Get Use Latency
+| | Call Resetter
 | | # Duration of -1 means we will stop traffic manually.
-| | Send traffic on tg | ${-1} | ${rate} | ${frame_size} | ${traffic_profile}
-| | ... | warmup_time=${0} | async_call=${True} | latency=${False}
-| | ... | traffic_directions=${traffic_directions} | tx_port=${tx_port}
-| | ... | rx_port=${rx_port}
+| | Send traffic on tg
+| | ... | duration=${-1}
+| | ... | rate=${rate}
+| | ... | frame_size=${frame_size}
+| | ... | traffic_profile=${traffic_profile}
+| | ... | async_call=${True}
+| | ... | duration_limit=${0.0}
+| | ... | ppta=${ppta}
+| | ... | traffic_directions=${traffic_directions}
+| | ... | transaction_duration=${transaction_duration}
+| | ... | transaction_scale=${transaction_scale}
+| | ... | transaction_type=${transaction_type}
+| | ... | use_latency=${use_latency}
 
 | Stop Running Traffic
 | | [Documentation]
@@ -653,75 +503,41 @@
 | | ${result}= | Stop traffic on tg
 | | Return From Keyword | ${result}
 
-| Additional Statistics Action For vpp-clear-stats
-| | [Documentation]
-| | ... | Additional Statistics Action for clear VPP statistics.
-| |
-| | Clear Statistics On All DUTs | ${nodes}
-
-| Additional Statistics Action For vpp-show-stats
-| | [Documentation]
-| | ... | Additional Statistics Action for show VPP statistics.
-| |
-| | Show Statistics On All DUTs | ${nodes}
-
-| Additional Statistics Action For vpp-clear-runtime
-| | [Documentation]
-| | ... | Additional Statistics Action for clear VPP runtime.
-| |
-| | VPP Clear Runtime On All DUTs | ${nodes}
-
-| Additional Statistics Action For vpp-show-runtime
-| | [Documentation]
-| | ... | Additional Statistics Action for show VPP runtime.
-| |
-| | VPP Show Runtime On All DUTs | ${nodes}
-
-| Additional Statistics Action For vpp-enable-packettrace
-| | [Documentation]
-| | ... | Additional Statistics Action for enable VPP packet trace.
-| |
-| | Run Keyword If | ${extended_debug}==${True}
-| | ... | VPP Enable Traces On All DUTs | ${nodes} | fail_on_error=${False}
-
-| Additional Statistics Action For vpp-show-packettrace
-| | [Documentation]
-| | ... | Additional Statistics Action for show VPP packet trace.
-| |
-| | Run Keyword If | ${extended_debug}==${True}
-| | ... | Show Packet Trace On All Duts | ${nodes} | maximum=${100}
-
-| Additional Statistics Action For vpp-enable-elog
-| | [Documentation]
-| | ... | Additional Statistics Action for enable VPP elog trace.
-| |
-| | VPP Enable Elog Traces On All DUTs | ${nodes}
-
-| Additional Statistics Action For vpp-show-elog
-| | [Documentation]
-| | ... | Additional Statistics Action for show VPP elog trace.
-| |
-| | Show Event Logger On All DUTs | ${nodes}
-
-| Additional Statistics Action For bash-perf-stat
+| Traffic should pass with maximum rate
 | | [Documentation]
-| | ... | Additional Statistics Action for bash command "perf stat".
+| | ... | Send traffic at maximum rate.
+| | ... | Call \${resetter} (if defined) to reset DUT state before each trial.
+| | ... | Fail if no packets were forwarded.
 | |
-| | Run Keyword If | ${extended_debug}==${True}
-| | ... | Perf Stat On All DUTs | ${nodes} | cpu_list=${cpu_alloc_str}
-
-| Additional Statistics Action For clear-show-runtime-with-traffic
-| | [Documentation]
-| | ... | Additional Statistics Action for clear and show runtime counters with
-| | ... | running traffic.
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - traffic_profile - Name of module defining traffic for measurements.
+| | ... | Type: string
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
+| | ... | string
+| | ... | - max_rate - Calculated maximal transmit rate [tps].
+| | ... | Type: float
+| | ... | - transaction_type - String identifier to determine how to count
+| | ... | transactions. Default is "packet".
 | |
-| | Clear and show runtime counters with running traffic
-| | ... | ${trial_duration} | ${rate}
-| | ... | ${frame_size} | ${traffic_profile} | ${traffic_directions}
-| | ... | ${tx_port} | ${rx_port}
-
-| Additional Statistics Action For noop
-| | [Documentation]
-| | ... | Additional Statistics Action for no operation.
+| | ... | *Example:*
 | |
-| | No operation
+| | ... | \| Traffic should pass with maximum rate \|
+| |
+| | ${max_rate} = | Get Max Rate
+| | ${transaction_type} = | Get Transaction Type
+| | ${trial_duration} = | Get Mrr Trial Duration
+| | ${trial_multiplicity} = | Get Mrr Trial Multiplicity
+| | ${use_latency} = | Get Use Latency
+| | # The following also sets \${rate_for_teardown}
+| | ${results} = | Send traffic at specified rate
+| | ... | rate=${max_rate}
+| | ... | trial_duration=${trial_duration}
+| | ... | trial_multiplicity=${trial_multiplicity}
+| | ... | use_latency=${use_latency}
+| | ... | duration_limit=${0.0}
+| | ${unit} = | Set Variable If | """_cps""" in """${transaction_type}"""
+| | ... | estimated connections per second | packets per second
+| | Set Test Message | ${\n}Maximum Receive Rate trial results
+| | Set Test Message | in ${unit}: ${results}
+| | ... | append=yes
+| | Fail if no traffic forwarded
diff --git a/resources/libraries/robot/performance/performance_vars.robot b/resources/libraries/robot/performance/performance_vars.robot
new file mode 100644
index 0000000000..130237ed82
--- /dev/null
+++ b/resources/libraries/robot/performance/performance_vars.robot
@@ -0,0 +1,522 @@
+# Copyright (c) 2020 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Documentation | Performance suite keywords - Handling of various values
+| ... | to allow autodetection, computation and overriding by suite variables.
+| Library | Collections
+| Variables | ${CURDIR}/../../python/Constants.py
+
+*** Variables ***
+| ${extended_debug}= | ${EXTENDED_DEBUG}
+
+*** Keywords ***
+| Get Disable Latency
+| | [Documentation]
+| | ... | If Get Use Latency returns true, return false.
+| | ... | Otherwise return value of \${disable_latency} variable,
+| | ... | or \${False} if not defined.
+| |
+| | ... | The return value controls whether latency trials in NDRPDR tests
+| | ... | are executed. For example, ASTF tests do not support latency
+| | ... | measurements yet, so executing the trials just wastes time.
+| | ... | Return type: bool.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${disable_latency} = \| Get Disable Latency \|
+| |
+| | ${use_latency} = | Get Use Latency
+| | Return From Keyword If | ${use_latency} | ${False}
+| | ${disable_latency} = | Get Variable Value | \${disable_latency} | ${False}
+| | Return From Keyword | ${disable_latency}
+
+| Get Max Rate
+| | [Documentation]
+| | ... | Return value of \${max_rate} variable,
+| | ... | fail if it is not defined.
+| | ... | Call this just before calling a Python keyword,
+| | ... | as those have restricted access to Robot variables.
+| |
+| | ... | The return value controls the maximal unidirectional packet rate.
+| | ... | The value is also usable for minimal TPS value for ASTF tests.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${max_rate} = \| Get Max Rate \|
+| |
+| | ${max_rate} = | Get Variable Value | \${max_rate} | ${0.0}
+| | Return From Keyword If | ${max_rate} | ${max_rate}
+| | Fail | \${max_rate} is not defined. Call Set Max Rate And Jumbo keyword.
+
+| Get Min Rate
+| | [Documentation]
+| | ... | Return a hardcoded value. This is an abstraction, useful in case
+| | ... | we start allowing various other overrides or computations.
+| | ... | Call this just before calling a Python keyword,
+| | ... | as those have restricted access to Robot variables.
+| |
+| | ... | The return value controls the minimal unidirectional packet rate.
+| | ... | The value is also usable for minimal TPS value for ASTF tests.
+| | ... | The current value is the smallest one permitted
+| | ... | by STL profiles with latency streams.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${min_rate} = \| Get Min Rate \|
+| |
+| | Return From Keyword | ${9001.0}
+
+| Get Mrr Trial Duration
+| | [Documentation]
+| | ... | Return value from Constants. This is an abstraction, useful in case
+| | ... | we start allowing various other overrides or computations.
+| | ... | Call this just before calling a Python keyword,
+| | ... | as those have restricted access to Robot variables.
+| |
+| | ... | The return value controls the duration of main trial measurement
+| | ... | for MRR type tests.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${mrr_trial_duration} = \| Get Mrr Trial Duration \|
+| |
+| | Return From Keyword | ${PERF_TRIAL_DURATION}
+
+| Get Mrr Trial Multiplicity
+| | [Documentation]
+| | ... | Return value from Constants. This is an abstraction, useful in case
+| | ... | we start allowing various other overrides or computations.
+| | ... | Call this just before calling a Python keyword,
+| | ... | as those have restricted access to Robot variables.
+| |
+| | ... | The return value controls the number of main trial measurement
+| | ... | for (B)MRR type tests.
+| | ... | Return type: integer.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${mrr_trial_multiplicity} = \| Get Mrr Trial Multiplicity \|
+| |
+| | Return From Keyword | ${PERF_TRIAL_MULTIPLICITY}
+
+| Get Packet Loss Ratio
+| | [Documentation]
+| | ... | Return a hardcoded value. This is an abstraction, useful in case
+| | ... | we start allowing various other overrides or computations.
+| | ... | Call this just before calling a Python keyword,
+| | ... | as those have restricted access to Robot variables.
+| |
+| | ... | The return value controls the default packet loss ration for PDR
+| | ... | in NDRPDR tests. Some other usages of MLRsearch (e.g. reconf tests)
+| | ... | may use a different value.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${packet_loss_ratio} = \| Get Packet Loss Ratio \|
+| |
+| | Return From Keyword | ${0.005}
+
+| Get Packets Per Transaction Aggregated
+| | [Documentation]
+| | ... | Return value of \${packets_per_transaction_aggregated};
+| | ... | if not defined, assume traffic is symmetric and compute
+| | ... | from unidirectional values.
+| |
+| | ... | The return value is used when reporting PPS values from TPS found
+| | ... | by some search (e.g. NDRPDR).
+| | ... | Return type: integer.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${ppta} = \| Get Packets Per Transaction Aggregated \|
+| |
+| | ${ppta} = | Get Variable Value | \${packets_per_transaction_aggregated}
+| | ... | ${0}
+| | Return From Keyword If | "${ppta}" != "0" | ${ppta}
+| | # TODO: Insert TCP computation from packet size here.
+| | ${pptad} = | Get Packets Per Transaction And Direction
+| | ${traffic_directions} = | Get Traffic Directions
+| | # We do not support ASTF profiles with multiple transactions, yet.
+| | ${ppta} = | Evaluate | ${pptad} * ${traffic_directions}
+| | Return From Keyword | ${ppta}
+
+| Get Packets Per Transaction And Direction
+| | [Documentation]
+| | ... | Return value of \${packets_per_transaction_and_direction},
+| | ... | or ${1} if not defined.
+| |
+| | ... | The return value is used when computing max rate (TPS),
+| | ... | so for asymmetric transaction use the more numerous direction.
+| | ... | Return type: integer.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${pptad} = \| Get Packets Per Transaction And Direction \|
+| |
+| | ${pptad} = | Get Variable Value | \${packets_per_transaction_and_direction}
+| | ... | ${1}
+| | Return From Keyword | ${pptad}
+
+| Get Ramp Up Duration
+| | [Documentation]
+| | ... | Return value of \${ramp_up_duration},
+| | ... | or ${0.0} if not defined.
+| |
+| | ... | The return value determines the required duration of ramp-up phase.
+| | ... | Typically used to prepare a specific state on DUT.
+| | ... | If the value is zero, ramp-up phase is skipped.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${ramp_up_duration} = \| Get Ramp Up Duration \|
+| |
+| | ${ramp_up_duration} = | Get Variable Value | \${ramp_up_duration} | ${0.0}
+| | Return From Keyword | ${ramp_up_duration}
+
+| Get Ramp Up Rate
+| | [Documentation]
+| | ... | Return value of \${ramp_up_rate},
+| | ... | if not defined return \${max_rate}.
+| |
+| | ... | The return value determines the rate for ramp-up phase.
+| | ... | Typically used to limit the rate when max rate
+| | ... | would lose packets in the ramp up phase, thus not setting
+| | ... | the DUT state correctly.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${ramp_up_rate} = \| Get Ramp Up Rate \|
+| |
+| | ${ramp_up_rate} = | Get Variable Value | \${ramp_up_rate} | ${0.0}
+| | Return From Keyword If | ${ramp_up_rate} | ${ramp_up_rate}
+| | Run Keyword And Return | Get Max Rate
+
+| Get Rate For Teardown
+| | [Documentation]
+| | ... | Return value of \${rate_for_teardown},
+| | ... | if not defined (or zero) return the min rate.
+| |
+| | ... | The return value determines the rate for teardown trial,
+| | ... | that is executed if a perf test fails.
+| | ... | The \${rate_for_teardown} is usually not defined in suite,
+| | ... | but search keywords set it in places where failure can occur,
+| | ... | so the trial is done at the rate interesting for the failure.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${rate_for_teardown} = \| Get Rate For Teardown \|
+| |
+| | ${rate_for_teardown} = | Get Variable Value | \${rate_for_teardown} | ${0.0}
+| | Return From Keyword If | ${rate_for_teardown} | ${rate_for_teardown}
+| | Run Keyword And Return | Get Min Rate
+
+| Get Resetter
+| | [Documentation]
+| | ... | Return value of \${resetter} variable,
+| | ... | or \${None} if not defined.
+| |
+| | ... | If not \${None}, the returned value is callable.
+| | ... | Its use is to reset DUT to initial conditions,
+| | ... | for example to remove NAT sessions created in the previous trial.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${resetter} = \| Get Resetter \|
+| |
+| | ${resetter} = | Get Variable Value | \${resetter} | ${None}
+| | Return From Keyword | ${resetter}
+
+| Get Runtime Duration
+| | [Documentation]
+| | ... | Return value of \${runtime_duration} variable,
+| | ... | if not defined return ${1.0}.
+| |
+| | ... | The return value controls the duration of runtime trial,
+| | ... | which also acts as a warmup. Usually one second is enough,
+| | ... | but some suites need longer time to set up state on DUT.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${runtime_duration} = \| Get Runtime Duration \|
+| |
+| | ${runtime_duration} = | Get Variable Value | \${runtime_duration} | ${1.0}
+| | Return From Keyword | ${runtime_duration}
+
+| Get Runtime Rate
+| | [Documentation]
+| | ... | Return value of \${runtime_rate} variable,
+| | ... | if not defined return the max rate.
+| |
+| | ... | The return value controls the rate (TPS unidir) of runtime trial,
+| | ... | which also acts as a warmup. No plans to ever use a different rate,
+| | ... | but keywords look better if access to such values is uniform.
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${runtime_rate} = \| Get Runtime Rate \|
+| |
+| | ${runtime_rate} = | Get Variable Value | \${runtime_rate} | ${0.0}
+| | Return From Keyword If | ${runtime_rate} | ${runtime_rate}
+| | Run Keyword And Return | Get Max Rate
+
+| Get Traffic Directions
+| | [Documentation]
+| | ... | Return value of \${traffic_directions},
+| | ... | or ${2} if not defined.
+| |
+| | ... | The return value used when parsing for measurement results.
+| | ... | This needs to be known already in profile driver,
+| | ... | as bidirectional parsing may fail on unidirectional traffic.
+| | ... | Return type: integer.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${traffic_directions} = \| Get Traffic Directions \|
+| |
+| | ${traffic_directions} = | Get Variable Value | \${traffic_directions} | ${2}
+| | Return From Keyword | ${traffic_directions}
+
+| Get Transaction Duration
+| | [Documentation]
+| | ... | Return value of \${transaction_duration} variable,
+| | ... | or \${0.0} if not defined.
+| |
+| | ... | The return value is the expected duration of single (ASTF) transaction
+| | ... | if it is not negligible for overall trial duration computation.
+| | ... | Most tests use very short transactions (without explicit delays),
+| | ... | so the zero default works (and suite saves one line
+| | ... | of Variables table).
+| | ... | Return type: float.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${transaction_duration} = \| Get Transaction Duration \|
+| |
+| | ${transaction_duration} = | Get Variable Value | \${transaction_duration}
+| | ... | ${0.0}
+| | Return From Keyword | ${transaction_duration}
+
+| Get Transaction Scale
+| | [Documentation]
+| | ... | Return value of \${transaction_scale} variable,
+| | ... | or \${0} if not defined.
+| |
+| | ... | Zero return value means the number of transactions is not limited,
+| | ... | which is true for most STL TRex profiles (transaction is a packet).
+| | ... | Nonzero return value means the number of transactions is fixed,
+| | ... | for example in stateful NAT scale tests.
+| | ... | Return type: integer.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${transaction_scale} = \| Get Transaction Scale \|
+| |
+| | ${transaction_scale} = | Get Variable Value | \${transaction_scale} | ${0}
+| | Return From Keyword | ${transaction_scale}
+
+| Get Transaction Type
+| | [Documentation]
+| | ... | Return value of \${transaction_type} variable,
+| | ... | or "packet" if not defined.
+| |
+| | ... | The return value describes the type of transaction
+| | ... | the test is executed. For example "packet" means a transaction
+| | ... | is just a single packet. For more sophisticated transactions,
+| | ... | the logic to determine the number of passed transactions
+| | ... | is different from merely counting the packets received from DUT.
+| | ... | Return type: string.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${transaction_type} = \| Get Transaction Type \|
+| |
+| | ${transaction_type} = | Get Variable Value | \${transaction_type} | packet
+| | Return From Keyword | ${transaction_type}
+
+| Get Use Latency
+| | [Documentation]
+| | ... | Return value of \${use_latency} variable,
+| | ... | if not defined return the value from Constants.
+| |
+| | ... | The return value controls whether latency streams are active
+| | ... | during the main search.
+| | ... | Return type: bool.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| \${use_latency} = \| Get Use Latency \|
+| |
+| | ${use_latency} = | Get Variable Value | ${use_latency} | ${PERF_USE_LATENCY}
+| | Return From Keyword | ${use_latency}
+
+| Set Jumbo
+| | [Documentation]
+| | ... | For jumbo frames detection, the maximal packet size is relevant,
+| | ... | encapsulation overhead (if any) has effect.
+| |
+| | ... | This keyword computes jumbo boolean (some suites need that for
+| | ... | configuration decisions).
+| | ... | To streamline suite autogeneration, both input and output values
+| | ... | are communicated as test (or broader scope) variables,
+| | ... | instead of explicit arguments and return values.
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
+| | ... | string
+| |
+| | ... | *Test variables set:*
+| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
+| | ... | enabled. Type: boolean
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Set Jumbo \|
+| |
+| | # Already called by Set Max Rate And Jumbo, but some suites (e.g. device)
+| | # are calling this directly.
+| | Set Numeric Frame Sizes
+| | ${jumbo} = | Set Variable If | ${max_frame_size} < 1522
+| | ... | ${False} | ${True}
+| | Set Test Variable | \${jumbo}
+
+| Set Max Rate And Jumbo
+| | [Documentation]
+| | ... | Input framesize can be either integer in case of a single packet
+| | ... | in stream, or IMIX string defining mix of packets.
+| | ... | For jumbo frames detection, the maximal packet size is relevant.
+| | ... | For maximal transmit rate, the average packet size is relevant.
+| | ... | In both cases, encapsulation overhead (if any) has effect.
+| | ... | The maximal rate is computed from NIC name.
+| | ... | The implementation works by mapping from exact
+| | ... | whitelisted NIC names.
+| | ... | The mapping is hardcoded in nic_limits.yaml
+| | ... | TODO: Make the mapping from NIC names case insensistive.
+| |
+| | ... | This keyword computes maximal unidirectional transmit rate
+| | ... | and jumbo boolean (some suites need that for configuration decisions).
+| | ... | To streamline suite autogeneration, both input and output values
+| | ... | are communicated as test (or broader scope) variables,
+| | ... | instead of explicit arguments and return values.
+| |
+| | ... | If this keyword detects the test is interested in (unidirectional)
+| | ... | transactons per second maximal rate (tps), that is returned (not pps).
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - nic_name - Name of bottleneck NIC. Type: string
+| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
+| | ... | string
+| | ... | - packets_per_transaction_and_direction - Pps-tps conversion.
+| | ... | Optional, default 1.
+| |
+| | ... | *Test variables set:*
+| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
+| | ... | This never exceeds bandwidth on TG-DUT nor DUT-DUT links.
+| | ... | Type: float
+| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
+| | ... | enabled. Type: boolean
+| | ... | avg_frame_size - Average frame size including overhead. Type: float
+| | ... | max_frame_size - Maximal frame size including overhead. Type: float
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Set Max Rate And Jumbo \|
+| |
+| | # TODO: Re-check overhead values in suites with both traffics encapsulated.
+| | # TODO: Improve layered setup to detect encap/decap and update overhead.
+| | ${pps_limit} = | Get From Dictionary
+| | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name}
+| | ${bps_limit} = | Get From Dictionary
+| | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name}
+| | Set Numeric Frame Sizes
+| | ${rate} = | Evaluate | ${bps_limit} / ((${avg_frame_size} + 20.0) * 8)
+| | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit}
+| | ... | ${pps_limit} | ${rate}
+| | ${pptad} = | Get Packets Per Transaction And Direction
+| | ${max_rate} = | Evaluate | ${max_rate} / ${pptad}
+| | Set Test Variable | \${max_rate}
+| | Set Jumbo
+
+| Set Numeric Frame Sizes
+| | [Documentation]
+| | ... | Framesize can be either integer in case of a single packet
+| | ... | in stream, or set of packets in case of IMIX type or simmilar.
+| | ... | For jumbo decisions, we need a numeric size of the biggest packet.
+| | ... | For max rate decisions, we need a numeric average packet size.
+| | ... | This keyword computes both and sets them as test variables.
+| |
+| | ... | Each suite sets a value named \${overhead},
+| | ... | which describes by how many bytes the frames on DUT-DUT link
+| | ... | are larger (due to encapsulation) than those
+| | ... | on the primary TG-DUT link. But for some suites that value
+| | ... | can be negaive (if TG-DUT is encapsulated more heavily).
+| | ... | For calculations in this keyword, we need largest sizes
+| | ... | across links, so zero is used if \${overhead} is negative.
+| |
+| | ... | *Test variables read:*
+| | ... | - frame_size - Framesize. Type: integer or string
+| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
+| |
+| | ... | *Test variables set*
+| | ... | avg_frame_size - Average frame size including overhead. Type: float
+| | ... | max_frame_size - Maximal frame size including overhead. Type: float
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Set Numeric Frame Sizes \|
+| |
+| | ${max_overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
+| | ${bare_avg_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
+| | ... | Set Variable | ${353.83333}
+| | ... | ELSE
+| | ... | Convert To Number | ${frame_size}
+| | ${avg_frame_size} = | Evaluate | $bare_avg_frame_size + $max_overhead
+| | Set Test Variable | \${avg_frame_size}
+| | ${bare_max_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
+| | ... | Set Variable | ${1518}
+| | ... | ELSE
+| | ... | Convert To Number | ${frame_size}
+| | ${max_frame_size} = | Evaluate | $bare_max_frame_size + $max_overhead
+| | Set Test Variable | ${max_frame_size}
+
+| Set Rates For Policer
+| | [Documentation]
+| | ... | Policer tests need these values,
+| | ... | currently computed from \${avg_frame_size}.
+| | ... | TODO: Verify the units match and computation is correct.
+| |
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - avg_frame_size - Average L2 Frame Size [B]. Type: float
+| | ... | Set by Set Max Rate And Jumbo keyword.
+| |
+| | ... | *Test variables set:*
+| | ... | - eb - Excess burst rate for policer. Type: float
+| | ... | - cb - Committed burst rate for policer. Type: float
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Set Rates For Policer \|
+| |
+| | Set Test Variable | \${eb} | ${avg_frame_size}
+| | Set Test Variable | \${cb} | ${avg_frame_size}