aboutsummaryrefslogtreecommitdiffstats
path: root/tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot
diff options
context:
space:
mode:
authorVratko Polak <vrpolak@cisco.com>2020-10-27 19:09:44 +0100
committerVratko Polak <vrpolak@cisco.com>2020-10-29 20:25:50 +0000
commit023fa41e51c966a1956bda6b915ffd894ff10e84 (patch)
treecdb96c99a8ade4855176c43969cbd9a06adf693b /tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot
parente31998ea56c55879fbaae8e58b0dad0bc6549dae (diff)
Support existing test types with ASTF
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Diffstat (limited to 'tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot')
-rw-r--r--tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot150
1 files changed, 0 insertions, 150 deletions
diff --git a/tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot b/tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot
deleted file mode 100644
index 1807b880f4..0000000000
--- a/tests/vpp/perf/ip4/2n1l-10ge2p1x710-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr.robot
+++ /dev/null
@@ -1,150 +0,0 @@
-# Copyright (c) 2020 Cisco and/or its affiliates.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at:
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-*** Settings ***
-| Resource | resources/libraries/robot/shared/default.robot
-|
-| Force Tags | 2_NODE_SINGLE_LINK_TOPO | PERFTEST | HW_ENV | NDRPDR
-| ... | NIC_Intel-X710 | ETH | IP4FWD | BASE | IP4BASE | DRV_VFIO_PCI | UDP_SYN
-| ... | RXQ_SIZE_0 | TXQ_SIZE_0
-| ... | ethip4udp-ip4base-h65536-p63-s4128768
-|
-| Suite Setup | Setup suite topology interfaces | performance
-| Suite Teardown | Tear down suite | performance
-| Test Setup | Setup test | performance
-| Test Teardown | Tear down test | performance
-|
-| Test Template | Local Template
-|
-| Documentation | *RFC2544: Pkt throughput IPv4 routing test cases*
-|
-| ... | *[Top] Network Topologies:* TG-DUT1-TG 2-node circular topology
-| ... | with single links between nodes.
-| ... | *[Enc] Packet Encapsulations:* Eth-IPv4-UDP for IPv4 routing.
-| ... | *[Cfg] DUT configuration:* DUT1 is configured with IPv4
-| ... | routing and two static IPv4 /16 route entries. DUT1 tested with
-| ... | ${nic_name}.\
-| ... | *[Ver] TG verification:* TG finds and reports throughput NDR (Non Drop\
-| ... | Rate) with zero packet loss tolerance and throughput PDR (Partial Drop\
-| ... | Rate) with non-zero packet loss tolerance (LT) expressed in percentage\
-| ... | of packets transmitted. NDR and PDR are discovered for different\
-| ... | Ethernet L2 frame sizes using MLRsearch library.\
-| ... | Test packets are generated by TG on links to DUTs. TG traffic profile\
-| ... | contain L7 flows (${cps} bi-directional UDP flows) with all packets\
-| ... | containing Ethernet header, IPv4 header with UDP header and static\
-| ... | payload. MAC addresses are matching MAC addresses of the TG node\
-| ... | interfaces.
-| ... | *[Ref] Applicable standard specifications:* RFC2544.
-
-*** Variables ***
-| @{plugins_to_enable}= | dpdk_plugin.so
-| ${crypto_type}= | ${None}
-| ${nic_name}= | Intel-X710
-| ${nic_driver}= | vfio-pci
-| ${nic_rxq_size}= | 0
-| ${nic_txq_size}= | 0
-| ${nic_pfs}= | 2
-| ${nic_vfs}= | 0
-| ${osi_layer}= | L7
-| ${overhead}= | ${0}
-# Traffic profile:
-| ${traffic_profile}= | trex-astf-ethip4udp-65536h
-| ${cps}= | ${4128768}
-# Trial data overwrite
-| ${trial_duration}= | ${1.1}
-| ${trial_multiplicity}= | ${1}
-
-*** Keywords ***
-| Local Template
-| | [Documentation]
-| | ... | [Cfg] DUT runs IPv4 routing config.\
-| | ... | Each DUT uses ${phy_cores} physical core(s) for worker threads.
-| | ... | [Ver] Measure NDR and PDR values using MLRsearch algorithm.\
-| |
-| | ... | *Arguments:*
-| | ... | - frame_size - Framesize in Bytes in integer or string (IMIX_v4_1).
-| | ... | Type: integer, string
-| | ... | - phy_cores - Number of physical cores. Type: integer
-| | ... | - rxq - Number of RX queues, default value: ${None}. Type: integer
-| |
-| | [Arguments] | ${frame_size} | ${phy_cores} | ${rxq}=${None}
-| |
-| | Set Test Variable | \${frame_size}
-| | Set Test Variable | \${max_rate} | ${cps}
-| | ${pre_stats}= | Create List
-| | ... | vpp-clear-stats | vpp-enable-packettrace | vpp-enable-elog
-| | ... | vpp-clear-runtime
-| | Set Test Variable | ${pre_stats}
-| | ${post_stats}= | Create List
-| | ... | vpp-show-stats | vpp-show-packettrace | vpp-show-elog
-| | ... | vpp-show-runtime
-| | Set Test Variable | ${post_stats}
-| |
-| | Given Set Jumbo
-| | And Add worker threads to all DUTs | ${phy_cores} | ${rxq}
-| | And Pre-initialize layer driver | ${nic_driver}
-| | And Apply startup configuration on all VPP DUTs
-| | When Initialize layer driver | ${nic_driver}
-| | And Initialize layer interface
-| | And Initialize IPv4 forwarding in circular topology
-| | ... | 192.168.0.0 | 20.0.0.0 | ${16}
-| | Then Find NDR and PDR intervals using optimized search | latency=${False}
-
-*** Test Cases ***
-| 64B-1c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 64B | 1C
-| | frame_size=${64} | phy_cores=${1}
-
-| 64B-2c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 64B | 2C
-| | frame_size=${64} | phy_cores=${2}
-
-| 64B-4c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 64B | 4C
-| | frame_size=${64} | phy_cores=${4}
-
-| 1518B-1c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 1518B | 1C
-| | frame_size=${1518} | phy_cores=${1}
-
-| 1518B-2c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 1518B | 2C
-| | frame_size=${1518} | phy_cores=${2}
-
-| 1518B-4c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 1518B | 4C
-| | frame_size=${1518} | phy_cores=${4}
-
-| 9000B-1c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 9000B | 1C
-| | frame_size=${9000} | phy_cores=${1}
-
-| 9000B-2c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 9000B | 2C
-| | frame_size=${9000} | phy_cores=${2}
-
-| 9000B-4c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | 9000B | 4C
-| | frame_size=${9000} | phy_cores=${4}
-
-| IMIX-1c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | IMIX | 1C
-| | frame_size=IMIX_v4_1 | phy_cores=${1}
-
-| IMIX-2c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | IMIX | 2C
-| | frame_size=IMIX_v4_1 | phy_cores=${2}
-
-| IMIX-4c-ethip4udp-ip4base-h65536-p63-s4128768-ndrpdr
-| | [Tags] | IMIX | 4C
-| | frame_size=IMIX_v4_1 | phy_cores=${4}