diff options
author | Vratko Polak <vrpolak@cisco.com> | 2020-10-27 19:09:44 +0100 |
---|---|---|
committer | Vratko Polak <vrpolak@cisco.com> | 2020-10-29 20:25:50 +0000 |
commit | 023fa41e51c966a1956bda6b915ffd894ff10e84 (patch) | |
tree | cdb96c99a8ade4855176c43969cbd9a06adf693b /resources/libraries/robot/performance/performance_limits.robot | |
parent | e31998ea56c55879fbaae8e58b0dad0bc6549dae (diff) |
Support existing test types with ASTF
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
+ Update existing cps traffic profiles.
+ Add missing traffic profiles.
+ UDP:
+ Single burst of 32 packets was confirmed as safe enough for TRex.
+ Maybe 64 could work, but not enough testing for that.
+ Multiple bursts have lead to reduced TRex performance,
as overlaping bursts (from different client instances)
tend to fill up the buffers.
+ TCP:
+ Data size set to 11111 bytes, completely arbitrarily.
+ Results look reasonable, so I have kept that.
- MSS not set at all
- No tested support for frame size other than 64B.
- Frame size does not even factor into TCP profiles.
+ So other frame sizes are skipped in autogen.
+ Update tags in related suites.
- HOSTS_{n} and SRC_USER_{n} should be unified.
- Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
+ Resetter is called (if set) before each measurement.
+ If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
+ Transaction corresponds to profile.
+ TRex multiplier argument sets target rate in transactions per second.
+ The familiar STL traffic:
+ Bidirectional is considered to be 2 packets per transaction.
+ Unidirectional is considered to be 1 packet per transaction.
+ The newer ASTF traffic:
+ 4 subtypes, each has different number of packets per transaction.
+ For max rate computation:
+ Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
+ Use the server side value.
- Client side value is higher by an overhead.
- TRex is not sending traffic during that time.
+ Remove delays from traffic profiles.
- Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
+ Only for ASTF profiles, each ASTF profile has limited scale.
+ Scale defined in suite variables.
+ For TRex to send all transactions provided duration value is ignored.
+ The appropriate value is computed in TrafficGenerator.
+ An ad-hoc time constant is added to match the TRex client side time overhead.
+ The profile driver receives the computed duration.
+ Measurement for PLRsearch add a sleep if the computed duration is smaller.
+ Alternative argument for search algos if scale is limited.
+ Both need higher timeout to accomodate big scales.
+ MLRsearch can afford fewer phases.
+ Added a parameter to optionally shorten the duration.
+ Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
- Polling could eliminate the time overhead value.
+ But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
+ The stop has several steps so that:
+ The traffic is really stopped entirely.
+ Late packets do not count (maybe as errors).
+ Stats are preserved to read for results (and cleared afterwards).
+ Several quantities added to ReceiveRateMeasurement:
+ Original target duration is preserved (algos need that).
+ Input estimate (tps) for early search iterations.
+ Output estimate (maybe pps) for MRR output.
+ Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
- TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
+ Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
+ Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
+ Rename argument names to min_rate and max_rate.
+ Use relative receive rate in initial phase.
+ PLRsearch improvements:
+ Careful computation when output (pps) does not match input (tps).
+ Use geometric distribution (instead of Poisson).
+ Helps agains math errors.
+ This should improve estimate stability.
- But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
+ send_traffic_on_tg now calls the full set_rate_provider_defaults.
+ _send_traffic_on_tg_internal for the logic without provider defaults.
+ As the internal function is re-used by measure() without affecting defaults.
+ Move _parse_traffic_results just before get_measurement_result.
+ As the latter uses fields set bu the former, it is now easier to read.
+ Multiple sources for approximate duration.
+ Tried from more precise to more available.
+ Includes logic for _pps tests (added in later change).
+ Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
+ performance_actions.robot:
+ Hosts Additional Statistics Action For * keywords.
+ performance_display.robot:
+ Hosts keyword for displaying and verifying results.
+ Change test message to use the correct unit (pps or cps).
+ performance_limits.robot renamed to performance_vars.robot
+ Added many keywords, mostly for accessing test variables.
+ Moved variables for Policer into a new keyword there.
+ Some keywords need sophisticated logic.
- Other are basically Get Variable Value.
+ But in future more logic can be added, without editing callers.
+ Documentation for the new keywords acts as a documentation for test variables.
+ performance_utils.robot has the rest.
+ Eliminated arguments if the value is in test variable.
+ Small improvements to documentation.
- Still not enough cleanup with respect to arguments and test variables.
+ Keywords are sorted alphabetically now in each one.
+ Suites:
+ Unified variables table:
+ No colons in comments.
+ ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
+ Add -cps to existing cps suite names.
+ Remove "trial data overwrite".
+ Compute max rate as in STL suites.
+ Each NAT suite has ip4base suite to compare results to.
- Those act as indirect TRex calibration.
- VPP does not lose packets in those.
+ Latency in ASTF suites is disabled hard.
- As we do not support latency in ASTF profiles yet.
+ Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
+ Prefer to use argument names.
+ In Python, also the last argument is followed by comma.
+ It makes renaming and reordering easier.
+ Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.
Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Diffstat (limited to 'resources/libraries/robot/performance/performance_limits.robot')
-rw-r--r-- | resources/libraries/robot/performance/performance_limits.robot | 153 |
1 files changed, 0 insertions, 153 deletions
diff --git a/resources/libraries/robot/performance/performance_limits.robot b/resources/libraries/robot/performance/performance_limits.robot deleted file mode 100644 index 332ed9368b..0000000000 --- a/resources/libraries/robot/performance/performance_limits.robot +++ /dev/null @@ -1,153 +0,0 @@ -# Copyright (c) 2020 Cisco and/or its affiliates. -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at: -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -*** Settings *** -| Documentation | Performance suite keywords - Handling NIC and TG limits. -| Library | Collections -| Variables | ${CURDIR}/../../python/Constants.py - -*** Keywords *** -| Get Average Frame Size -| | [Documentation] -| | ... | Framesize can be either integer in case of a single packet -| | ... | in stream, or set of packets in case of IMIX type or simmilar. -| | -| | ... | *Arguments:* -| | ... | - frame_size - Framesize. Type: integer or string -| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer -| | -| | ... | *Returns:* -| | ... | Average frame size including overhead. Type: float -| | -| | ... | *Example:* -| | -| | ... | \| Get Average Frame Size \| IMIX_v4_1 \| -| | -| | [Arguments] | ${frame_size} | ${overhead}=${0} -| | -| | ${frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1' -| | ... | Set Variable | ${353.83333} -| | ... | ELSE -| | ... | Convert To Number | ${frame_size} -| | ${avg_frame_size} = | Evaluate | ${frame_size} + ${overhead} -| | Return From Keyword | ${avg_frame_size} - -| Get Maximum Frame Size -| | [Documentation] -| | ... | Framesize can be either integer in case of a single packet -| | ... | in stream, or set of packets in case of IMIX type or simmilar. -| | -| | ... | *Arguments:* -| | ... | - frame_size - Framesize. Type: integer or string -| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer -| | -| | ... | *Returns:* -| | ... | Maximum frame size including overhead. Type: float -| | -| | ... | *Example:* -| | -| | ... | \| Get Maximum Frame Size \| IMIX_v4_1 \| -| | -| | [Arguments] | ${frame_size} | ${overhead}=${0} -| | -| | ${frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1' -| | ... | Set Variable | ${1518} -| | ... | ELSE -| | ... | Convert To Number | ${frame_size} -| | ${max_frame_size} = | Evaluate | ${frame_size} + ${overhead} -| | Return From Keyword | ${max_frame_size} - -| Set Max Rate And Jumbo -| | [Documentation] -| | ... | Input framesize can be either integer in case of a single packet -| | ... | in stream, or IMIX string defining mix of packets. -| | ... | For jumbo frames detection, the maximal packet size is relevant. -| | ... | For maximal transmit rate, the average packet size is relevant. -| | ... | In both cases, encapsulation overhead (if any) has effect. -| | ... | The maximal rate is computed from NIC name. -| | ... | The implementation works by mapping from exact -| | ... | whitelisted NIC names. -| | ... | The mapping is hardcoded in nic_limits.yaml -| | ... | TODO: Make the mapping from NIC names case insensistive. -| | -| | ... | TODO: Make pps limit also definable per NIC. -| | -| | ... | This keyword computes maximal unidirectional transmit rate -| | ... | and jumbo boolean (some suites need that for configuration decisions). -| | ... | To streamline suite autogeneration, both input and output values -| | ... | are communicated as test (or broader scope) variables, -| | ... | instead of explicit arguments and return values. -| | -| | ... | *Test (or broader scope) variables read:* -| | ... | - nic_name - Name of bottleneck NIC. Type: string -| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer -| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or -| | ... | string -| | -| | ... | *Test variables set:* -| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps]. -| | ... | Type: float -| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be -| | ... | enabled. Type: boolean -| | -| | ... | *Example:* -| | -| | ... | \| Set Test Variable \| \${frame_size} \| IMIX_v4_1 \| -| | ... | \| Set Max Rate And Jumbo \| -| | -| | # Negative overhead is possible, if DUT-DUT traffic is less encapsulated -| | # than TG-DUT traffic. -| | # TODO: Re-check overhead values in suites with both traffics encapsulated. -| | # TODO: Improve layered setup to detect encap/decap and update overhead. -| | ${overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0} -| | ${pps_limit} = | Get From Dictionary -| | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name} -| | ${bps_limit} = | Get From Dictionary -| | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name} -| | # swo := size_with_overhead -| | ${avg_swo} = | Get Average Frame Size | ${frame_size} | ${overhead} -| | ${rate} = | Evaluate | ${bps_limit} / ((${avg_swo} + 20.0) * 8) -| | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit} -| | ... | ${pps_limit} | ${rate} -| | Set Test Variable | \${max_rate} -| | Set Jumbo - -| Set Jumbo -| | [Documentation] -| | ... | For jumbo frames detection, the maximal packet size is relevant, -| | ... | encapsulation overhead (if any) has effect. -| | -| | ... | This keyword computes jumbo boolean (some suites need that for -| | ... | configuration decisions). -| | ... | To streamline suite autogeneration, both input and output values -| | ... | are communicated as test (or broader scope) variables, -| | ... | instead of explicit arguments and return values. -| | -| | ... | *Test (or broader scope) variables read:* -| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer -| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or -| | ... | string -| | -| | ... | *Test variables set:* -| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be -| | ... | enabled. Type: boolean -| | -| | ... | *Example:* -| | -| | ... | \| Set Jumnbo \| -| | -| | ${overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0} -| | ${max_swo} = | Get Maximum Frame Size | ${frame_size} | ${overhead} -| | ${jumbo} = | Set Variable If | ${max_swo} < 1522 -| | ... | ${False} | ${True} -| | Set Test Variable | \${jumbo} |