diff options
author | Vratko Polak <vrpolak@cisco.com> | 2020-10-28 12:50:10 +0100 |
---|---|---|
committer | Jan Gelety <jgelety@cisco.com> | 2020-10-26 12:42:00 +0100 |
commit | de5e719fbd965e2f8e26b7f3a9cd597b572d7ab3 (patch) | |
tree | f2ee97e9d100cf20358f6c73e9b37101a3789103 /resources/libraries/robot/performance/performance_display.robot | |
parent | 90eec3d17f8a45f816b2df03dc8fed9d13ef76cd (diff) |
Support existing test types with ASTF
Manual cherry-pick from master [1],
reverting impact of [2] and [3].
[1] https://gerrit.fd.io/r/c/csit/+/28208/176
[2] https://gerrit.fd.io/r/c/csit/+/29077
[3] https://gerrit.fd.io/r/c/csit/+/29529
The heap multipliers are left in suites,
as that simplifies cherry-picking between branches.
Original [0] commit message:
Support existing test types with ASTF
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
+ Update existing cps traffic profiles.
+ Add missing traffic profiles.
+ UDP:
+ Single burst of 32 packets was confirmed as safe enough for TRex.
+ Maybe 64 could work, but not enough testing for that.
+ Multiple bursts have lead to reduced TRex performance,
as overlaping bursts (from different client instances)
tend to fill up the buffers.
+ TCP:
+ Data size set to 11111 bytes, completely arbitrarily.
+ Results look reasonable, so I have kept that.
- MSS not set at all
- No tested support for frame size other than 64B.
- Frame size does not even factor into TCP profiles.
+ So other frame sizes are skipped in autogen.
+ Update tags in related suites.
- HOSTS_{n} and SRC_USER_{n} should be unified.
- Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
+ Resetter is called (if set) before each measurement.
+ If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
+ Transaction corresponds to profile.
+ TRex multiplier argument sets target rate in transactions per second.
+ The familiar STL traffic:
+ Bidirectional is considered to be 2 packets per transaction.
+ Unidirectional is considered to be 1 packet per transaction.
+ The newer ASTF traffic:
+ 4 subtypes, each has different number of packets per transaction.
+ For max rate computation:
+ Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
+ Use the server side value.
- Client side value is higher by an overhead.
- TRex is not sending traffic during that time.
+ Remove delays from traffic profiles.
- Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
+ Only for ASTF profiles, each ASTF profile has limited scale.
+ Scale defined in suite variables.
+ For TRex to send all transactions provided duration value is ignored.
+ The appropriate value is computed in TrafficGenerator.
+ An ad-hoc time constant is added to match the TRex client side time overhead.
+ The profile driver receives the computed duration.
+ Measurement for PLRsearch add a sleep if the computed duration is smaller.
+ Alternative argument for search algos if scale is limited.
+ Both need higher timeout to accomodate big scales.
+ MLRsearch can afford fewer phases.
+ Added a parameter to optionally shorten the duration.
+ Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
- Polling could eliminate the time overhead value.
+ But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
+ The stop has several steps so that:
+ The traffic is really stopped entirely.
+ Late packets do not count (maybe as errors).
+ Stats are preserved to read for results (and cleared afterwards).
+ Several quantities added to ReceiveRateMeasurement:
+ Original target duration is preserved (algos need that).
+ Input estimate (tps) for early search iterations.
+ Output estimate (maybe pps) for MRR output.
+ Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
- TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
+ Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
+ Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
+ Rename argument names to min_rate and max_rate.
+ Use relative receive rate in initial phase.
+ PLRsearch improvements:
+ Careful computation when output (pps) does not match input (tps).
+ Use geometric distribution (instead of Poisson).
+ Helps agains math errors.
+ This should improve estimate stability.
- But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
+ send_traffic_on_tg now calls the full set_rate_provider_defaults.
+ _send_traffic_on_tg_internal for the logic without provider defaults.
+ As the internal function is re-used by measure() without affecting defaults.
+ Move _parse_traffic_results just before get_measurement_result.
+ As the latter uses fields set bu the former, it is now easier to read.
+ Multiple sources for approximate duration.
+ Tried from more precise to more available.
+ Includes logic for _pps tests (added in later change).
+ Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
+ performance_actions.robot:
+ Hosts Additional Statistics Action For * keywords.
+ performance_display.robot:
+ Hosts keyword for displaying and verifying results.
+ Change test message to use the correct unit (pps or cps).
+ performance_limits.robot renamed to performance_vars.robot
+ Added many keywords, mostly for accessing test variables.
+ Moved variables for Policer into a new keyword there.
+ Some keywords need sophisticated logic.
- Other are basically Get Variable Value.
+ But in future more logic can be added, without editing callers.
+ Documentation for the new keywords acts as a documentation for test variables.
+ performance_utils.robot has the rest.
+ Eliminated arguments if the value is in test variable.
+ Small improvements to documentation.
- Still not enough cleanup with respect to arguments and test variables.
+ Keywords are sorted alphabetically now in each one.
+ Suites:
+ Unified variables table:
+ No colons in comments.
+ ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
+ Add -cps to existing cps suite names.
+ Remove "trial data overwrite".
+ Compute max rate as in STL suites.
+ Each NAT suite has ip4base suite to compare results to.
- Those act as indirect TRex calibration.
- VPP does not lose packets in those.
+ Latency in ASTF suites is disabled hard.
- As we do not support latency in ASTF profiles yet.
+ Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
+ Prefer to use argument names.
+ In Python, also the last argument is followed by comma.
+ It makes renaming and reordering easier.
+ Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.
Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Signed-off-by: Jan Gelety <jgelety@cisco.com>
Diffstat (limited to 'resources/libraries/robot/performance/performance_display.robot')
-rw-r--r-- | resources/libraries/robot/performance/performance_display.robot | 240 |
1 files changed, 240 insertions, 0 deletions
diff --git a/resources/libraries/robot/performance/performance_display.robot b/resources/libraries/robot/performance/performance_display.robot new file mode 100644 index 0000000000..e8dfdbbfbd --- /dev/null +++ b/resources/libraries/robot/performance/performance_display.robot @@ -0,0 +1,240 @@ +# Copyright (c) 2020 Cisco and/or its affiliates. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +*** Settings *** +| Documentation +| ... | Performance suite keywords - Displaying results as test messages. +| ... | This includes checks to fail test. + +*** Keywords *** +| Check NDRPDR interval validity +| | [Documentation] +| | ... | Extract loss ratio of lower bound of the interval. +| | ... | Fail if it does not reach the allowed value. +| | +| | ... | *Arguments:* +| | ... | - interval - Measured interval. Type: ReceiveRateInterval +| | ... | - packet_loss_ratio - Accepted loss (0.0 for NDR). Type: float +| | +| | ... | *Example:* +| | +| | ... | \| Check NDRPDR interval validity \| \${result.pdr_interval} \ +| | ... | \| \${0.005} \| +| | +| | [Arguments] | ${interval} | ${packet_loss_ratio}=${0.0} +| | +| | ${lower_bound} = | Set Variable | ${interval.measured_low} +| | ${lower_bound_lf} = | Set Variable | ${lower_bound.loss_fraction} +| | Return From Keyword If | ${lower_bound_lf} <= ${packet_loss_ratio} +| | Set Test Variable | \${rate_for_teardown} | ${lower_bound_lf} +| | ${message}= | Catenate | SEPARATOR=${SPACE} +| | ... | Minimal rate loss fraction ${lower_bound_lf} +| | ... | does not reach target ${packet_loss_ratio}. +| | ${message_zero} = | Set Variable | Zero packets forwarded! +| | ${message_other} = | Set Variable | ${lower_bound.loss_count} packets lost. +| | ${message} = | Set Variable If | ${lower_bound_lf} >= 1.0 +| | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other} +| | Fail | ${message} + +| Display Reconfig Test Message +| | [Documentation] +| | ... | Display the number of packets lost (bidirectionally) +| | ... | due to reconfiguration under traffic. +| | +| | ... | *Arguments:* +| | ... | - result - Result of bidirectional measurtement. +| | ... | Type: ReceiveRateMeasurement +| | +| | ... | *Example:* +| | +| | ... | \| Display Reconfig Test Message \| \${result} \| +| | +| | [Arguments] | ${result} +| | +| | ${ppta} = | Get Packets Per Transaction Aggregated +| | ${packet_rate} = | Evaluate | ${result.target_tr} * ${ppta} +| | ${packet_loss} = | Set Variable | ${result.loss_count} +| | ${time_loss} = | Evaluate | ${packet_loss} / ${packet_rate} +| | Set Test Message | Packets lost due to reconfig: ${packet_loss} +| | Set Test Message | ${\n}Implied time lost: ${time_loss} | append=yes + +| Display result of NDRPDR search +| | [Documentation] +| | ... | Display result of NDR+PDR search, both quantities, both bounds, +| | ... | aggregate in units given by trasaction type, e.g. by default +| | ... | in packet per seconds and Gbps total bandwidth +| | ... | (for initial packet size). +| | ... | +| | ... | The bound to display is encoded as target rate, it is assumed +| | ... | it is in transactions per second. Bidirectional traffic +| | ... | transaction is understood as having 2 packets, for this purpose. +| | ... | +| | ... | Througput is calculated as: +| | ... | Sum of measured rate over streams +| | ... | Bandwidth is calculated as: +| | ... | (Throughput * (L2 Frame Size + IPG) * 8) +| | ... | If the results contain latency data, display them for lower bounds. +| | +| | ... | *Test (or broader scope) variables read:* +| | ... | - frame_size_num - L2 Frame Size [B]. Type: integer or float +| | ... | - transaction_type - String identifier to determine how to count +| | ... | transactions. Default is "packet". +| | ... | *Arguments:* +| | ... | - result - Measured result data. Aggregate rate, tps or pps. +| | ... | Type: NdrPdrResult +| | +| | ... | *Example:* +| | +| | ... | \| Display result of NDRPDR search \| \${result} \| +| | +| | [Arguments] | ${result} +| | +| | Display single bound | NDR_LOWER +| | ... | ${result.ndr_interval.measured_low.target_tr} +| | ... | ${result.ndr_interval.measured_low.latency} +| | Display single bound | NDR_UPPER +| | ... | ${result.ndr_interval.measured_high.target_tr} +| | Display single bound | PDR_LOWER +| | ... | ${result.pdr_interval.measured_low.target_tr} +| | ... | ${result.pdr_interval.measured_low.latency} +| | Display single bound | PDR_UPPER +| | ... | ${result.pdr_interval.measured_high.target_tr} + +| Display result of soak search +| | [Documentation] +| | ... | Display result of soak search, avg+-stdev, as upper/lower bounds. +| | ... | See Display single bound for units used. +| | +| | ... | *Test (or broader scope) variables read:* +| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or +| | ... | string +| | ... | - transaction_type - String identifier to determine how to count +| | ... | transactions. Default is "packet". +| | ... | *Arguments:* +| | ... | - avg - Estimated average critical load [pps]. Type: float +| | ... | - stdev - Standard deviation of critical load [pps]. Type: float +| | +| | ... | *Returns:* +| | ... | - Lower and upper bound of critical load [pps]. Type: 2-tuple of float +| | +| | ... | *Example:* +| | +| | ... | \| Display result of soak search \| \${100000} \| \${100} \| +| | +| | [Arguments] | ${avg} | ${stdev} +| | +| | ${avg} = | Convert To Number | ${avg} +| | ${stdev} = | Convert To Number | ${stdev} +| | ${lower} = | Evaluate | ${avg} - ${stdev} +| | ${upper} = | Evaluate | ${avg} + ${stdev} +| | Display single bound | PLRsearch lower bound | ${lower} +| | Display single bound | PLRsearch upper bound | ${upper} +| | Return From Keyword | ${lower} | ${upper} + +| Display single bound +| | [Documentation] +| | ... | Compute and display one bound of NDR+PDR (or soak) search result. +| | ... | If the latency string is present, it is displayed as well. +| | ... | +| | ... | The bound to display is given as target transfer rate, it is assumed +| | ... | it is in transactions per second. Bidirectional traffic +| | ... | transaction is understood as having 2 packets, for this purpose. +| | ... | +| | ... | Pps values are aggregate in packet per seconds, +| | ... | and Gbps total bandwidth (for initial packet size). +| | ... | +| | ... | Througput is calculated as: +| | ... | Sum of measured rate over streams +| | ... | Bandwidth is calculated as: +| | ... | (Throughput * (L2 Frame Size + IPG) * 8) +| | ... | If the results contain latency data, display them for lower bounds. +| | +| | ... | *Test (or broader scope) variables read:* +| | ... | - transaction_type - String identifier to determine how to count +| | ... | transactions. Default is "packet". +| | ... | *Arguments:* +| | ... | - text - Flavor text describing which bound is this. Type: string +| | ... | - tps - Transaction rate [tps]. Type: float +| | ... | - latency - Latency data to display if non-empty. Type: string +| | +| | ... | *Example:* +| | +| | ... | \| Display single bound \| NDR lower bound \| \${12345.67} \ +| | ... | \| latency=\${EMPTY} \| +| | +| | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY} +| | +| | ${transaction_type} = | Get Transaction Type +| | Run Keyword And Return If | """_cps""" in """${transaction_type}""" +| | ... | Display single cps bound | ${text} | ${tps} | ${latency} +| | Display single pps bound | ${text} | ${tps} | ${latency} + +| Display single cps bound +| | [Documentation] +| | ... | Display one bound of NDR+PDR search for CPS tests. +| | ... | The bounds are expressed as transactions per second. +| | ... | If the latency string is present, it is displayed as well. +| | +| | ... | *Arguments:* +| | ... | - text - Flavor text describing which bound is this. Type: string +| | ... | - tps - Transaction rate [tps]. Type: float +| | ... | - latency - Latency data to display if non-empty. Type: string +| | +| | ... | *Example:* +| | +| | ... | \| Display single cps bound \| NDR lower bound \| \${12345.67} \ +| | ... | \| latency=\${EMPTY} \| +| | +| | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY} +| | +| | Set Test Message | ${\n}${text}: ${tps} CPS | append=yes +| | Return From Keyword If | not """${latency}""" +| | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency} +| | ... | append=yes + +| Display single pps bound +| | [Documentation] +| | ... | Display one pps bound of NDR+PDR search, +| | ... | aggregate in packet per seconds and Gbps total bandwidth +| | ... | (for initial packet size). +| | ... | +| | ... | The bound to display is given as target transfer rate, it is assumed +| | ... | it is in transactions per second. Bidirectional traffic +| | ... | transaction is understood as having 2 packets, for this purpose. +| | ... | +| | ... | Througput is calculated as: +| | ... | Sum of measured rates over streams +| | ... | Bandwidth is calculated as: +| | ... | (Throughput * (L2 Frame Size + IPG) * 8) +| | ... | If the latency string is present, it is displayed as well. +| | +| | ... | *Arguments:* +| | ... | - text - Flavor text describing which bound is this. Type: string +| | ... | - tps - Transaction rate [tps]. Type: float +| | ... | - latency - Latency data to display if non-empty. Type: string +| | +| | ... | *Example:* +| | +| | ... | \| Display single pps bound \| NDR lower bound \| \${12345.67} \ +| | ... | \| latency=\${EMPTY} \| +| | +| | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY} +| | +| | ${ppta} = | Get Packets Per Transaction Aggregated +| | ${pps} = | Evaluate | ${tps} * ${ppta} +| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9 +| | Set Test Message | ${\n}${text}: ${pps} pps, | append=yes +| | Set Test Message | ${bandwidth} Gbps (initial) | append=yes +| | Return From Keyword If | not """${latency}""" +| | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency} +| | ... | append=yes |