From 3ccdfe158ce748d1892b66d755643fa88fe5920e Mon Sep 17 00:00:00 2001 From: Vratko Polak Date: Wed, 28 Oct 2020 12:50:10 +0100 Subject: Support existing test types with ASTF Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak --- .../libraries/python/MLRsearch/AbstractMeasurer.py | 4 +- .../python/MLRsearch/AbstractSearchAlgorithm.py | 15 +++-- .../python/MLRsearch/MultipleLossRatioSearch.py | 36 ++++++------ .../python/MLRsearch/ReceiveRateMeasurement.py | 66 ++++++++++++++++++++-- 4 files changed, 89 insertions(+), 32 deletions(-) (limited to 'resources/libraries/python/MLRsearch') diff --git a/resources/libraries/python/MLRsearch/AbstractMeasurer.py b/resources/libraries/python/MLRsearch/AbstractMeasurer.py index 622b8fdba6..82116f2e43 100644 --- a/resources/libraries/python/MLRsearch/AbstractMeasurer.py +++ b/resources/libraries/python/MLRsearch/AbstractMeasurer.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019 Cisco and/or its affiliates. +# Copyright (c) 2020 Cisco and/or its affiliates. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at: @@ -24,7 +24,7 @@ class AbstractMeasurer(metaclass=ABCMeta): """Perform trial measurement and return the result. :param duration: Trial duration [s]. - :param transmit_rate: Target transmit rate [pps]. + :param transmit_rate: Target transmit rate [tps]. :type duration: float :type transmit_rate: float :returns: Structure containing the result of the measurement. diff --git a/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py b/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py index f4f2d3f096..f2bf04e1b1 100644 --- a/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py +++ b/resources/libraries/python/MLRsearch/AbstractSearchAlgorithm.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019 Cisco and/or its affiliates. +# Copyright (c) 2020 Cisco and/or its affiliates. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at: @@ -30,16 +30,19 @@ class AbstractSearchAlgorithm(metaclass=ABCMeta): @abstractmethod def narrow_down_ndr_and_pdr( - self, fail_rate, line_rate, packet_loss_ratio): + self, min_rate, max_rate, packet_loss_ratio): """Perform measurements to narrow down intervals, return them. This will be renamed when custom loss ratio lists are supported. - :param fail_rate: Minimal target transmit rate [pps]. - :param line_rate: Maximal target transmit rate [pps]. + :param min_rate: Minimal target transmit rate [tps]. + Usually, tests are set to fail if search reaches this or below. + :param max_rate: Maximal target transmit rate [tps]. + Usually computed from line rate and various other limits, + to prevent failures or duration stretching in Traffic Generator. :param packet_loss_ratio: Fraction of packets lost, for PDR [1]. - :type fail_rate: float - :type line_rate: float + :type min_rate: float + :type max_rate: float :type packet_loss_ratio: float :returns: Structure containing narrowed down intervals and their measurements. diff --git a/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py b/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py index 2db65ef755..87dc784cbc 100644 --- a/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py +++ b/resources/libraries/python/MLRsearch/MultipleLossRatioSearch.py @@ -157,7 +157,7 @@ class MultipleLossRatioSearch(AbstractSearchAlgorithm): :returns: The relative width of double logarithmic size. :rtype: float """ - return 1.999 * relative_width - relative_width * relative_width + return 1.99999 * relative_width - relative_width * relative_width # The number should be 2.0, but we want to avoid rounding errors, # and ensure half of double is not larger than the original value. @@ -255,52 +255,50 @@ class MultipleLossRatioSearch(AbstractSearchAlgorithm): 1.0 - MultipleLossRatioSearch.half_relative_width(relative_width) ) - def narrow_down_ndr_and_pdr( - self, minimum_transmit_rate, maximum_transmit_rate, - packet_loss_ratio): + def narrow_down_ndr_and_pdr(self, min_rate, max_rate, packet_loss_ratio): """Perform initial phase, create state object, proceed with next phases. - :param minimum_transmit_rate: Minimal target transmit rate [pps]. - :param maximum_transmit_rate: Maximal target transmit rate [pps]. + :param min_rate: Minimal target transmit rate [tps]. + :param max_rate: Maximal target transmit rate [tps]. :param packet_loss_ratio: Fraction of packets lost, for PDR [1]. - :type minimum_transmit_rate: float - :type maximum_transmit_rate: float + :type min_rate: float + :type max_rate: float :type packet_loss_ratio: float :returns: Structure containing narrowed down intervals and their measurements. :rtype: NdrPdrResult.NdrPdrResult :raises RuntimeError: If total duration is larger than timeout. """ - minimum_transmit_rate = float(minimum_transmit_rate) - maximum_transmit_rate = float(maximum_transmit_rate) + minimum_transmit_rate = float(min_rate) + maximum_transmit_rate = float(max_rate) packet_loss_ratio = float(packet_loss_ratio) - line_measurement = self.measurer.measure( + max_measurement = self.measurer.measure( self.initial_trial_duration, maximum_transmit_rate) initial_width_goal = self.final_relative_width for _ in range(self.number_of_intermediate_phases): initial_width_goal = self.double_relative_width(initial_width_goal) max_lo = maximum_transmit_rate * (1.0 - initial_width_goal) - mrr = max( - minimum_transmit_rate, min(max_lo, line_measurement.receive_rate) - ) + mrr = max(minimum_transmit_rate, min( + max_lo, max_measurement.relative_receive_rate + )) mrr_measurement = self.measurer.measure( self.initial_trial_duration, mrr ) # Attempt to get narrower width. if mrr_measurement.loss_fraction > 0.0: max2_lo = mrr * (1.0 - initial_width_goal) - mrr2 = min(max2_lo, mrr_measurement.receive_rate) + mrr2 = min(max2_lo, mrr_measurement.relative_receive_rate) else: mrr2 = mrr / (1.0 - initial_width_goal) if minimum_transmit_rate < mrr2 < maximum_transmit_rate: - line_measurement = mrr_measurement + max_measurement = mrr_measurement mrr_measurement = self.measurer.measure( self.initial_trial_duration, mrr2) if mrr2 > mrr: - line_measurement, mrr_measurement = \ - (mrr_measurement, line_measurement) + max_measurement, mrr_measurement = \ + (mrr_measurement, max_measurement) starting_interval = ReceiveRateInterval( - mrr_measurement, line_measurement) + mrr_measurement, max_measurement) starting_result = NdrPdrResult(starting_interval, starting_interval) state = self.ProgressState( starting_result, self.number_of_intermediate_phases, diff --git a/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py b/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py index 31a6f8202e..c732e66026 100644 --- a/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py +++ b/resources/libraries/python/MLRsearch/ReceiveRateMeasurement.py @@ -1,4 +1,4 @@ -# Copyright (c) 2019 Cisco and/or its affiliates. +# Copyright (c) 2020 Cisco and/or its affiliates. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at: @@ -17,18 +17,39 @@ class ReceiveRateMeasurement: """Structure defining the result of single Rr measurement.""" - def __init__(self, duration, target_tr, transmit_count, loss_count): + def __init__( + self, duration, target_tr, transmit_count, loss_count, + approximated_duration=0.0, partial_transmit_count=0): """Constructor, normalize primary and compute secondary quantities. + If approximated_duration is nonzero, it is stored. + If approximated_duration is zero, duration value is stored. + Either way, additional secondary quantities are computed + from the store value. + + If there is zero transmit_count, fractions are set to zero. + + In some cases, traffic generator does not attempt all the needed + transactions. In that case, nonzero partial_transmit_count + holds (an estimate of) count of the actually attempted transactions. + This is used to populate some secondary quantities. + + TODO: Use None instead of zero? + :param duration: Measurement duration [s]. :param target_tr: Target transmit rate [pps]. If bidirectional traffic is measured, this is bidirectional rate. :param transmit_count: Number of packets transmitted [1]. :param loss_count: Number of packets transmitted but not received [1]. + :param approximated_duration: Estimate of the actual time of the trial. + :param partial_transmit_count: Estimate count of actually attempted + transactions. :type duration: float :type target_tr: float :type transmit_count: int :type loss_count: int + :type approximated_duration: float + :type partial_transmit_count: int """ self.duration = float(duration) self.target_tr = float(target_tr) @@ -38,8 +59,41 @@ class ReceiveRateMeasurement: self.transmit_rate = transmit_count / self.duration self.loss_rate = loss_count / self.duration self.receive_rate = self.receive_count / self.duration - self.loss_fraction = float(self.loss_count) / self.transmit_count - # TODO: Do we want to store also the real time (duration + overhead)? + self.loss_fraction = ( + float(self.loss_count) / self.transmit_count + if self.transmit_count > 0 else 1.0 + ) + self.receive_fraction = ( + float(self.receive_count) / self.transmit_count + if self.transmit_count > 0 else 0.0 + ) + self.approximated_duration = ( + float(approximated_duration) if approximated_duration + else self.duration + ) + self.approximated_receive_rate = ( + self.receive_count / self.approximated_duration + if self.approximated_duration > 0.0 else 0.0 + ) + # If the traffic generator is unreliable and sends less packets, + # the absolute receive rate might be too low for next target. + self.partial_transmit_count = ( + int(partial_transmit_count) if partial_transmit_count + else self.transmit_count + ) + self.partial_receive_fraction = ( + float(self.receive_count) / self.partial_transmit_count + if self.partial_transmit_count > 0 else 0.0 + ) + self.partial_receive_rate = ( + self.target_tr * self.partial_receive_fraction + ) + # We use relative packet ratios in order to support cases + # where target_tr is in transactions per second, + # but there are multiple packets per transaction. + self.relative_receive_rate = ( + self.target_tr * self.receive_count / self.transmit_count + ) def __str__(self): """Return string reporting input and loss fraction.""" @@ -51,4 +105,6 @@ class ReceiveRateMeasurement: return f"ReceiveRateMeasurement(duration={self.duration!r}," \ f"target_tr={self.target_tr!r}," \ f"transmit_count={self.transmit_count!r}," \ - f"loss_count={self.loss_count!r})" + f"loss_count={self.loss_count!r}," \ + f"approximated_duration={self.approximated_duration!r}," \ + f"partial_transmit_count={self.partial_transmit_count!r})" -- cgit 1.2.3-korg