aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot/performance
diff options
context:
space:
mode:
authorVratko Polak <vrpolak@cisco.com>2019-08-23 17:00:29 +0200
committerVratko Polak <vrpolak@cisco.com>2019-08-23 17:00:29 +0200
commitff820e3d53434889aacde2373314c3be8d00f765 (patch)
tree4172584444e7408af0ef47db4288cfdf4bad9ef7 /resources/libraries/robot/performance
parentfbbc47359e3f7b59bbd5a84d85c673374933a50a (diff)
Introduce reconfig suites, for dot1q+ip4+vxlan
Ticket: CSIT-1551 + Config copied from other suites. + Layer keywords prepared for additional chains. + Renamed "create" argument to "vlan_per_chain". + TrafficGenerator result extraction improvements. + Also unified TG type and subtype checking. + Throughput estimated as PDR for zero PLR. + Actual Qemu start is skipped. + Parse xstats instead of stats in trex stop scipt. - Because stats are unconditionally cleared on each script connect. + Remember start values to get better ReceiveMeasurementResult at stop. + Expose loss count and time as appropriate test message. + Autogen support. + Multiple chain amounts and core densities. - Only 1 additional chain. + Proper TAGs. Documented. + Clarified min framesize tags are for TG-DUT. + Added a TODO concerning overhead values. + Added 118B tag. + Robot performance keywords improvements. + Added missing return value documentation. + Frame size as required test variable, not an explicit argument. + MLRsearch keyword to store result (not into test message). - TODO: De-duplicate: main keyword to measure, minor to set test message. - But implementation uses the fact default arguments are different. + Keywords for starting and stopping background traffic. - No checks whether start-stop pair is complete. + Multiple minor indentation and logging improvements. - Missing / TODOs: - Support also vlan_per_chain=True? - Support dot1q without vxlan? - Support also deletion? - How to package config steps so -reconf can be generated from -ndrpdr? Change-Id: If0270697f9ffa837c34ef802917220a53f156703 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Diffstat (limited to 'resources/libraries/robot/performance')
-rw-r--r--resources/libraries/robot/performance/performance_limits.robot5
-rw-r--r--resources/libraries/robot/performance/performance_utils.robot143
2 files changed, 136 insertions, 12 deletions
diff --git a/resources/libraries/robot/performance/performance_limits.robot b/resources/libraries/robot/performance/performance_limits.robot
index 759ee5c28f..67ef3cf5b0 100644
--- a/resources/libraries/robot/performance/performance_limits.robot
+++ b/resources/libraries/robot/performance/performance_limits.robot
@@ -75,6 +75,11 @@
| | ... | \| Set test Variable \| \${frame_size} \| IMIX_v4_1 \|
| | ... | \| Set Max Rate And Jumbo \|
| | ...
+| | # Negative overhead is possible, if DUT-DUT traffic is less encapsulated
+| | # than TG-DUT traffic.
+| | # TODO: Re-check overhead values in suites with both traffics encapsulated.
+| | # TODO: Improve layered setup to detect encap/decap and update overhead.
+| | ${overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
| | ${pps_limit} = | Set Variable | ${18750000.0}
| | ${bps_limit} = | Get From Dictionary | ${NIC_NAME_TO_LIMIT} | ${nic_name}
| | ${avg_size} = | Get Average Frame Size | ${frame_size}
diff --git a/resources/libraries/robot/performance/performance_utils.robot b/resources/libraries/robot/performance/performance_utils.robot
index 54b20cba2d..d809376c0f 100644
--- a/resources/libraries/robot/performance/performance_utils.robot
+++ b/resources/libraries/robot/performance/performance_utils.robot
@@ -32,7 +32,7 @@
| | [Documentation]
| | ... | Find boundaries for RFC2544 compatible NDR and PDR values
| | ... | using an optimized search algorithm.
-| | ... | Display results as formatted test message.
+| | ... | Display findings as a formatted test message.
| | ... | Fail if a resulting lower bound has too high loss fraction.
| | ... | Input rates are understood as uni-directional,
| | ... | reported result contains bi-directional rates.
@@ -61,7 +61,7 @@
| | ...
| | ... | *Example:*
| | ...
-| | ... | \| Find NDR and PDR intervals using optimized search \| \${0.005}
+| | ... | \| Find NDR and PDR intervals using optimized search \| \${0.005} \
| | ... | \| \${0.005} \| \${30.0} \| \${1.0} \| \${2} \| ${600.0} \| ${2} \|
| | ...
| | [Arguments] | ${packet_loss_ratio}=${0.005}
@@ -76,13 +76,83 @@
| | ... | ${final_trial_duration} | ${initial_trial_duration}
| | ... | ${number_of_intermediate_phases} | timeout=${timeout}
| | ... | doublings=${doublings}
-| | Display result of NDRPDR search | ${result} | ${frame_size}
+| | Display result of NDRPDR search | ${result}
| | Check NDRPDR interval validity | ${result.pdr_interval}
| | ... | ${packet_loss_ratio}
| | Check NDRPDR interval validity | ${result.ndr_interval}
| | Perform additional measurements based on NDRPDR result
| | ... | ${result} | ${frame_size} | ${traffic_profile}
+| Display Reconfig Test Message
+| | [Documentation]
+| | ... | Display the number of packets lost (bidirectionally)
+| | ... | due to reconfiguration under traffic.
+| | ...
+| | ... | *Arguments:*
+| | ... | - result - Result of bidirectional measurtement.
+| | ... | Type: ReceiveRateMeasurement
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Display Reconfig Test Message \| \${result} \|
+| | ...
+| | [Arguments] | ${result}
+| | ...
+| | Set Test Message | Packets lost due to reconfig: ${result.loss_count}
+| | ${time_lost} = | Evaluate | ${result.loss_count} / ${result.target_tr}
+| | Set Test Message | ${\n}Implied time lost: ${time_lost} | append=yes
+
+| Find Throughput Using MLRsearch
+| | [Documentation]
+| | ... | Find and return lower bound PDR (zero PLR by default)
+| | ... | bi-directional throughput using MLRsearch algorithm.
+| | ... | Input rates are understood as uni-directional,
+| | ... | reported result contains bi-directional rates.
+| | ... | Currently, the min_rate value is hardcoded to match test teardowns.
+| | ...
+| | ... | TODO: Should the trial duration of the additional
+| | ... | measurements be configurable?
+| | ...
+| | ... | Some inputs are read from variables to streamline suites.
+| | ...
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - traffic_profile - Name of module defining traffc for measurements.
+| | ... | Type: string
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: int or str
+| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
+| | ... | Type: float
+| | ...
+| | ... | *Arguments:*
+| | ... | - packet_loss_ratio - Accepted loss during search. Type: float
+| | ... | - final_relative_width - Maximal width multiple of upper. Type: float
+| | ... | - final_trial_duration - Duration of final trials [s]. Type: float
+| | ... | - initial_trial_duration - Duration of initial trials [s]. Type: float
+| | ... | - intermediate phases - Number of intermediate phases [1]. Type: int
+| | ... | - timeout - Fail if search duration is longer [s]. Type: float
+| | ... | - doublings - How many doublings to do when expanding [1]. Type: int
+| | ...
+| | ... | *Returns:*
+| | ... | - Lower bound for bi-directional throughput at given PLR. Type: float
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| \${throughpt}= \| Find Throughput Using MLRsearch \| \${0} \
+| | ... | \| \${0.001} \| \${10.0}\| \${1.0} \| \${1} \| ${720.0} \| ${2} \|
+| | ...
+| | [Arguments] | ${packet_loss_ratio}=${0.0}
+| | ... | ${final_relative_width}=${0.001} | ${final_trial_duration}=${10.0}
+| | ... | ${initial_trial_duration}=${1.0}
+| | ... | ${number_of_intermediate_phases}=${1} | ${timeout}=${720.0}
+| | ... | ${doublings}=${2}
+| | ...
+| | ${result} = | Perform optimized ndrpdr search | ${frame_size}
+| | ... | ${traffic_profile} | ${20000} | ${max_rate*2}
+| | ... | ${packet_loss_ratio} | ${final_relative_width}
+| | ... | ${final_trial_duration} | ${initial_trial_duration}
+| | ... | ${number_of_intermediate_phases} | timeout=${timeout}
+| | ... | doublings=${doublings}
+| | Return From Keyword | ${result.pdr_interval.measured_low.transmit_rate}
+
| Find critical load using PLRsearch
| | [Documentation]
| | ... | Find boundaries for troughput (of given target loss ratio)
@@ -116,7 +186,7 @@
| | ... | ${traffic_profile} | ${min_rate} | ${max_rate*2}
| | ... | ${packet_loss_ratio} | timeout=${timeout}
| | ${lower} | ${upper} = | Display result of soak search
-| | ... | ${average} | ${stdev} | ${frame_size}
+| | ... | ${average} | ${stdev}
| | Should Not Be True | ${lower} < ${min_rate}
| | ... | Lower bound ${lower} is below bidirectional minimum ${min_rate}.
@@ -162,15 +232,16 @@
| | ... | (Throughput * (L2 Frame Size + IPG) * 8) / Max bitrate of NIC
| | ... | The given result should contain latency data as well.
| | ...
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: int or str
| | ... | *Arguments:*
| | ... | - result - Measured result data per stream [pps]. Type: NdrPdrResult
-| | ... | - frame_size - L2 Frame Size [B]. Type: integer
| | ...
| | ... | *Example:*
| | ...
-| | ... | \| Display result of NDRPDR search \| \${result} \| \${64} \|
+| | ... | \| Display result of NDRPDR search \| \${result} \|
| | ...
-| | [Arguments] | ${result} | ${frame_size}
+| | [Arguments] | ${result}
| | ...
| | ${frame_size} = | Get Average Frame Size | ${frame_size}
| | Display single bound | NDR_LOWER
@@ -196,19 +267,20 @@
| | ... | TODO: Do we want to report some latency data,
| | ... | even if not measured at the reported bounds?.
| | ...
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: int or str
| | ... | *Arguments:*
| | ... | - avg - Estimated average critical load [pps]. Type: float
| | ... | - stdev - Standard deviation of critical load [pps]. Type: float
-| | ... | - frame_size - L2 Frame Size [B]. Type: integer
| | ...
| | ... | *Returns:*
| | ... | - Lower and upper bound of critical load [pps]. Type: 2-tuple of float
| | ...
| | ... | *Example:*
| | ...
-| | ... | \| Display result of soak search \| \${100000} \| \${100} \| \${64} \|
+| | ... | \| Display result of soak search \| \${100000} \| \${100} \|
| | ...
-| | [Arguments] | ${avg} | ${stdev} | ${frame_size}
+| | [Arguments] | ${avg} | ${stdev}
| | ...
| | ${frame_size} = | Get Average Frame Size | ${frame_size}
| | ${avg} = | Convert To Number | ${avg}
@@ -321,7 +393,7 @@
| | ...
| | ... | *Example:*
| | ...
-| | ... | \| Traffic should pass with maximum rate \| ${1} \| ${10.0} \|
+| | ... | \| Traffic should pass with maximum rate \| ${1} \| ${10.0} \
| | ... | \| ${False} \| ${False} \| ${0} \| ${1} \|
| | ...
| | [Arguments] | ${trial_duration}=${perf_trial_duration}
@@ -399,7 +471,7 @@
| | ...
| | ... | *Arguments:*
| | ... | - duration - Duration of traffic run [s]. Type: integer
-| | ... | - rate - Rate for sending packets. Type: string
+| | ... | - rate - Unidirectional rate for sending packets. Type: string
| | ... | - frame_size - L2 Frame Size [B] or IMIX_v4_1. Type: integer/string
| | ... | - traffic_profile - Name of module defining traffc for measurements.
| | ... | Type: string
@@ -425,3 +497,50 @@
| | Run Keyword If | ${dut_stats}==${True}
| | ... | Show runtime counters on all DUTs | ${nodes}
| | Stop traffic on tg
+
+| Start Traffic on Background
+| | [Documentation]
+| | ... | Start traffic at specified rate then return control to Robot.
+| | ... | Useful if the test needs to do something while traffic is running.
+| | ... | Just a wrapper around L1 keyword.
+| | ... |
+| | ... | TODO: How to make sure the traffic is stopped on any failure?
+| | ...
+| | ... | *Test (or broader scope) variables read:*
+| | ... | - traffic_profile - Name of module defining traffc for measurements.
+| | ... | Type: string
+| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: int or str
+| | ... | *Arguments:*
+| | ... | - rate - Unidirectional rate for sending packets. Type: string
+| | ... | - unidirection - False if traffic is bidirectional. Type: boolean
+| | ... | - tx_port - TX port of TG, default 0. Type: integer
+| | ... | - rx_port - RX port of TG, default 1. Type: integer
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Start Traffic on Background \| 4.0mpps \| ${False} \| ${0} \
+| | ... | \| ${1} \|
+| | ...
+| | [Arguments] | ${rate} | ${unidirection}=${False} | ${tx_port}=${0}
+| | ... | ${rx_port}=${1}
+| | ...
+| | # Duration of -1 means we will stop traffic manually.
+| | Send traffic on tg | ${-1} | ${rate} | ${frame_size} | ${traffic_profile}
+| | ... | warmup_time=${0} | async_call=${True} | latency=${False}
+| | ... | unidirection=${unidirection} | tx_port=${tx_port} | rx_port=${rx_port}
+
+| Stop Running Traffic
+| | [Documentation]
+| | ... | Stop the running traffic, return measurement result.
+| | ... | For bidirectional traffic, the reported values are bi-directional.
+| | ... | Just a wrapper around L1 keyword.
+| | ... |
+| | ... | TODO: Tolerate if traffic was not started.
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \${result}= \| Stop Running Traffic \|
+| | ...
+| | Stop traffic on tg
+| | ${result}= | Get Measurement Result
+| | Return From Keyword | ${result}