feat(ietf): MLRsearch draft-06

Change-Id: Ia9e22ba1eca74a32ae7141b1bd111bdb9baaddf3
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

1 files changed, 0 insertions, 1460 deletions

diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-05.md b/docs/ietf/draft-ietf-bmwg-mlrsearch-05.md
deleted file mode 100644
index 937e632413..0000000000
--- a/docs/ietf/draft-ietf-bmwg-mlrsearch-05.md
+++ /dev/null
@@ -1,1460 +0,0 @@
----
-title: Multiple Loss Ratio Search
-abbrev: MLRsearch
-docname: draft-ietf-bmwg-mlrsearch-05
-date: 2023-10-23
-
-ipr: trust200902
-area: ops
-wg: Benchmarking Working Group
-kw: Internet-Draft
-cat: info
-
-coding: us-ascii
-pi:    # can use array (if all yes) or hash here
-  toc: yes
-  sortrefs:   # defaults to yes
-  symrefs: yes
-
-author:
-      -
-        ins: M. Konstantynowicz
-        name: Maciek Konstantynowicz
-        org: Cisco Systems
-        email: mkonstan@cisco.com
-      -
-        ins: V. Polak
-        name: Vratko Polak
-        org: Cisco Systems
-        email: vrpolak@cisco.com
-
-normative:
-  RFC1242:
-  RFC2285:
-  RFC2544:
-  RFC9004:
-
-informative:
-  TST009:
-    target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf
-    title: "TST 009"
-  FDio-CSIT-MLRsearch:
-    target: https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/
-    title: "FD.io CSIT Test Methodology - MLRsearch"
-    date: 2023-10
-  PyPI-MLRsearch:
-    target: https://pypi.org/project/MLRsearch/0.4.1/
-    title: "MLRsearch 0.4.1, Python Package Index"
-    date: 2021-07
-
---- abstract
-
-This document proposes extensions to [RFC2544] throughput search by
-defining a new methodology called Multiple Loss Ratio search
-(MLRsearch). The main objectives of MLRsearch are to minimize the
-total search duration, to support searching for multiple loss ratios
-and to improve results repeatability and comparability.
-
-The main motivation behind extending [RFC2544] is the new set of challenges
-and requirements posed by evaluating and testing software based networking
-systems, specifically their data planes.
-
-MLRsearch offers several ways to address these challenges, giving user
-configuration options to select their preferred way.
-
---- middle
-
-{::comment}
-    As we use kramdown to convert from markdown,
-    we use this way of marking comments not to be visible in rendered draft.
-    https://stackoverflow.com/a/42323390
-    If other engine is used, convert to this way:
-    https://stackoverflow.com/a/20885980
-{:/comment}
-
-# Purpose and Scope
-
-The purpose of this document is to describe Multiple Loss Ratio search
-(MLRsearch), a data plane throughput search methodology optimized for software
-networking DUTs.
-
-Applying vanilla [RFC2544] throughput bisection to software DUTs
-results in a number of problems:
-
-- Binary search takes too long as most of trials are done far from the
-  eventually found throughput.
-- The required final trial duration and pauses between trials
-  prolong the overall search duration.
-- Software DUTs show noisy trial results,
-  leading to a big spread of possible discovered throughput values.
-- Throughput requires loss of exactly zero frames, but the industry
-  frequently allows for small but non-zero losses.
-- The definition of throughput is not clear when trial results are inconsistent.
-
-MLRsearch library aims to address these problems by applying the following set
-of enhancements:
-
-- Allow multiple shorter trials instead of one big trial per load.
-  - Optionally, tolerate a percentage of trial result with higher loss.
-- Allow searching for multiple search goals, with differing loss ratios.
-  - Any trial result can affect each search goal in principle.
-- Insert multiple coarse targets for each search goal, earlier ones need
-  to spend less time on trials.
-  - Earlier targets also aim for lesser precision.
-  - Use Forwarding Rate (FR) at maximum offered load
-    [RFC2285] (section 3.6.2) to initialize the initial targets.
-- Take care when dealing with inconsistent trial results.
-  - Reported throughput is smaller than smallest load with high loss.
-  - Smaller load candidates are measured first.
-- Apply several load selection heuristics to save even more time
-  by trying hard to avoid unnecessarily narrow bounds.
-
-Some of these enhancements are formalized as MLRsearch specification,
-the remaining enhancements are treated as implementation details,
-thus achieving high comparability without limiting future improvements.
-
-MLRsearch configuration options are flexible enough to
-support both conservative settings and aggressive settings.
-Where the conservative settings lead to results
-unconditionally compliant with [RFC2544],
-but longer search duration and worse repeatability.
-Conversely, aggressive settings lead to shorter search duration
-and better repeatability, but the results are not compliant with [RFC2544].
-
-No part of [RFC2544] is intended to be obsoleted by this document.
-
-# Identified Problems
-
-This chapter describes the problems affecting usability
-of various preformance testing methodologies,
-mainly a binary search for [RFC2544] unconditionally compliant throughput.
-
-The last chapter will summarize how the problems are addressed,
-the middle chapters provide explanations and definitions needed for that.
-
-## Long Search Duration
-
-Emergence of software DUTs, with frequent software updates and a
-number of different frame processing modes and configurations,
-has increased both the number of performance tests requred to verify DUT update
-and the frequency of running those tests.
-This makes the overall test execution time even more important than before.
-
-In the context of characterising particular DUT's network performance,
-this calls for improving the time efficiency of throughput search.
-A vanilla bisection (at 60sec trial duration for unconditional [RFC2544]
-compliance) is slow, because most trials spend time quite far from the
-eventual throughput.
-
-[RFC2544] does not specify any stopping condition for throughput search,
-so users can trade-off between search duration and achieved precision.
-But, due to logarithmic nature of bisection, even small improvement
-in search duration needs relatively big sacrifice in the precision of the
-discovered throughput.
-
-## DUT in SUT
-
-[RFC2285] defines:
-- DUT as
-  - The network forwarding device to which stimulus is offered and
-    response measured [RFC2285] (section 3.1.1).
-- SUT as
-  - The collective set of network devices to which stimulus is offered
-    as a single entity and response measured [RFC2285] (section 3.1.2).
-
-[RFC2544] specifies a test setup with an external tester stimulating the
-networking system, treating it either as a single DUT, or as a system
-of devices, an SUT.
-
-In case of software networking, the SUT consists nt only of the DUT
-as a software program processing frames, but also of
-a server hardware and operating system functions,
-with server hardware resources shared across all programs
-and the operating system running on the same server.
-
-The DUT is effectively nested within the rest of the SUT.
-
-Due to a shared multi-tenant nature of SUT, DUT is subject to
-possible interference coming from the operating system and any other
-software running on the same server. Some sources of such interference
-can be to some degree eliminated, e.g. by pinning DUT program threads
-to specific CPU cores and isolating those cores to avoid context switching.
-But some level of adverse effects may remain even after
-all such reasonable precautions are applied.
-These effects affect DUT's network performance negatively.
-As the effects are hard to predict in general, they have impact similar to
-what other engineering disciplines define as a noise.
-Thus, all such effects are called an SUT noise.
-
-DUT can also exhibit fluctuating performance itself, for reasons
-not related to the rest of SUT, for example due to pauses in execution
-as needed for internal stateful processing. In many cases this
-may be an expected per-design behavior, as it would be observable even
-in a hypothetical scenario where all sources of SUT noise are eliminated.
-Such behavior affects trial results in a way similar to SUT noise.
-As the two phenomenons are hard to destinguish,
-this document uses the word noise as a shorthand covering both
-this internal DUT performance fluctuations and genuine SUT noise.
-
-A simple model of SUT performance consists of an idealized noiseless performance,
-and additional noise effects. The noiseless performance is assumed to be constant,
-all observed performance variations are due to noise.
-The impact of the noise can vary in time, sometimes wildly,
-even within a single trial.
-The noise can sometimes be negligible, but frequently
-it lowers the observed SUT performance as observed in trial results.
-
-In this model, SUT does not have a single performance value, it has a spectrum.
-One end of the spectrum is the idealized noiseless performance value,
-the other end can be called a noiseful performance. In practice, trial results
-close to the noiseful end of the spectrum happen only rarely.
-The worse the performance value is, the more rarely it is seen in a trial.
-Therefore, the extreme noiseful end of SUT spectrum is not really observable
-among trial results. Also, the extreme noiseless end of SUT spectrum
-is unlikely to be observable, this time because some small noise effects
-are likely to occur multiple times during a trial.
-
-Unless specified otherwise, this document talks about potentially observable
-ends of the SUT performance spectrum, not about the extreme ones.
-
-Focusing on DUT, the benchmarking effort should aim
-at eliminating only the SUT noise from SUT measurements.
-In practice that is not really possible, as based on authors experience
-and available literature, there are no realistic enough models
-able to distinguish SUT noise from DUT fluctuations.
-
-However, assuming that a well-constructed SUT has the DUT as its
-performance bottleneck, the DUT ideal noiseless performance can be defined
-as the noiseless end of SUT performance spectrum. At least for
-throughput. For other performance quantities such as latency there may be an
-additive difference.
-
-Note that by this definition, DUT noiseless performance
-also minimizes the impact of DUT fluctuations, as much as realistically possible
-for a given trial duration.
-
-In this document, we reduce the DUT in SUT problem to estimating
-the noiseless end of SUT performance spectrum from a limited number of
-trial results.
-
-Any improvements to throughput search algorithm, aimed for better
-dealing with software networking SUT and DUT setup, should employ
-strategies recognizing the presence of SUT noise, and allow discovery of
-(proxies for) DUT noiseless performance
-at different levels of sensitivity to SUT noise.
-
-## Repeatability and Comparability
-
-[RFC2544] does not suggest to repeat throughput search. And from just one
-discovered throughput value, it cannot be determined how repeatable that value is.
-In practice, poor repeatability is also the main cause of poor
-comparability, that is different benchmarking teams can test the same SUT
-but get throughput values differing more than expected from search precision.
-
-[RFC2544] throughput requirements (60 seconds trial and
-no tolerance of a single frame loss) affect the throughput results
-in the following way.
-The SUT behavior close to the noiseful end of its performance spectrum
-consists of rare occasions of significantly low performance,
-but the long trial duration makes those occasions not so rare on the trial level.
-Therefore, the binary search results tend to wander away from the noiseless end
-of SUT performance spectrum, more frequently and more widely than shorter
-trials would, thus resulting in poor throughput repeatability.
-
-The repeatability problem can be addressed by defining a search procedure
-which reports more stable results,
-even if they can no longer be called throughput in [RFC2544] sense.
-According to the SUT performance spectrum model, better repeatability
-will be at the noiseless end of the spectrum.
-Therefore, solutions to the DUT in SUT problem
-will help also with the repeatability problem.
-
-Conversely, any alteration to [RFC2544] throughput search
-that improves repeatability should be considered
-as less dependent on the SUT noise.
-
-An alternative option is to simply run a search multiple times, and report some
-statistics (e.g. average and standard deviation). This can be used
-for a subset of tests deemed more important,
-but it makes the search duration problem even more pronounced.
-
-## Throughput with Non-Zero Loss
-
-[RFC1242] (section 3.17) defines throughput as:
-    The maximum rate at which none of the offered frames
-    are dropped by the device.
-
-Then, it says:
-    Since even the loss of one frame in a
-    data stream can cause significant delays while
-    waiting for the higher level protocols to time out,
-    it is useful to know the actual maximum data
-    rate that the device can support.
-
-Contrary to that, many benchmarking teams settle with small, non-zero
-loss ratio as the goal for a their load search.
-
-Motivations are many:
-
-- Modern protocols tolerate frame loss better,
-  compared to the time when [RFC1242] and [RFC2544] were specified.
-
-- Trials nowadays send way more frames within the same duration,
-  increasing the chance small SUT performance fluctuatios
-  is enough to cause frame loss.
-
-- Small bursts of frame loss caused by noise have otherwise smaller impact
-  on the average frame loss ratio ovserved in the trial,
-  as during other parts of the same trial the SUT may work mroe closely
-  to its noiseless performance, thus perhaps lowering the trial loss ratio
-  below the goal loss ratio value.
-
-- If an approximation of the SUT noise impact on the trial loss ratio is known,
-  it can be set as the goal loss ratio.
-
-Regardless of validity of any and all similar motivations,
-support for non-zero loss goals makes any search algorithm more user friendly.
-[RFC2544] throughput is not user friendly in this regard.
-
-Assuming users are allowed to specify the goal loss ratio value,
-the usefulness is enhanced even more if users can specify multiple
-loss ratio values, especially when a single search can find all relevant bounds.
-
-Searching for multiple search goals also helps to describe the SUT performance
-spectrum better than a single search goal result.
-For example, repeated wide gap between zero and non-zero loss loads
-indicates the noise has a large impact on the observed performance,
-which is not evident from a single goal load search procedure result.
-
-It is easy to modify the vanilla bisection to find a lower bound
-for intended load that satisfies a non-zero goal loss ratio.
-But it is not that obvious how to search for multiple goals at once,
-hence the support for multiple search goals remains a problem.
-
-## Inconsistent Trial Results
-
-While performing throughput search by executing a sequence of
-measurement trials, there is a risk of encountering inconsistencies
-between trial results.
-
-The plain bisection never encounters inconsistent trials.
-But [RFC2544] hints about possibility of inconsistent trial results,
-in two places in its text.
-The first place is section 24, where full trial durations are required,
-presumably because they can be inconsistent with results
-from shorter trial durations.
-The second place is section 26.3, where two successive zero-loss trials
-are recommended, presumably because after one zero-loss trial
-there can be subsequent inconsistent non-zero-loss trial.
-
-Examples include:
-
-- A trial at the same load (same or different trial duration) results
-  in a different trial loss ratio.
-- A trial at higher load (same or different trial duration) results
-  in a smaller trial loss ratio.
-
-Any robust throughput search algorithm needs to decide how to continue
-the search in presence of such inconsistencies.
-Definitions of throughput in [RFC1242] and [RFC2544] are not specific enough
-to imply a unique way of handling such inconsistencies.
-
-Ideally, there will be a definition of a new quantity which both generalizes
-throughput for non-zero-loss (and other possible repeatibility enhancements),
-while being precise enough to force a specific way to resolve trial result
-inconsistencies.
-But until such definition is agreed upon, the correct way to handle
-inconsistent trial results remains an open problem.
-
-# MLRsearch Specification
-
-This chapter focuses on technical definitions needed for evaluating
-whether a particular test procedure adheres to MLRsearch specification.
-
-For motivations, explanations, and other comments see other chapters.
-
-## MLRsearch Architecture
-
-MLRsearch architecture consists of three main components:
-the manager, the controller and the measurer.
-Presence of other components (mainly the SUT) is also implied.
-
-While the manager and the measurer can be seen a abstractions
-present in any testing procedure, the behavior of the controller
-is what distinguishes MLRsearch algorithms from other search procedures.
-
-### Measurer
-
-The measurer is the component which performs one trial
-as described in [RFC2544] section 23, when requested by the controller.
-
-Specifically, one call to the measurer accepts a trial load value
-and trial duration value, performs the trial, and returns
-the measured trial loss ratio, and optionally a different duration value.
-
-It is responsibility of the measurer to uphold any requirements
-and assumptions present in MLRsearch specification
-(e.g. trial forwarding ratio not being larger than one).
-Implementers have some freedom, for example in the way they deal with
-duplicated frames, or what to return if tester sent zero frames towards SUT.
-Implementations are RECOMMENDED to document their behavior
-related to such freedoms in as detailed way as possible
-
-Implemenations MUST document any deviations from RFC documents,
-for example if the wait time around traffic
-is shorter than what [RFC2544] section 23 specifies.
-
-### Controller
-
-The controller is the component of MLRsearch architecture
-that is called by the manager (just once), calls the measurer
-(usually multiple times in succession),
-and returns the result of the search to the manager.
-
-The only required argument in the call to the controller
-is a list of search goals. For the structure of the search result,
-see subsection Search Result.
-
-### Manager
-
-The manager is the component that initializes SUT, traffic generator
-(tester in [RFC2544]), the measurer and the controller
-with intended configurations. It then hands over the execution
-to the controller and receives its result.
-
-Creation of reports of appropriate format can also be understood
-as the responsibility of the manager.
-
-## Units
-
-The specification deals with physical quantities, so it is assumed
-each numeric value is accompanied by an appropriate physical unit.
-
-The specification does not state which unit is appropriate,
-but implementations MUST make it explicit which unit is used
-for each value provided or received by the user.
-
-For example, load quantities (including the conditional throughput)
-returned by the controller are defined to be based on single-interface
-(unidirectional) loads. For bidirectional traffic, users are likely
-to expect bidirectional throughput quantities, so the manager is responsible
-for making its report clear.
-
-## SUT
-
-As defined in [RFC2285]:
-The collective set of network devices to which stimulus is offered
-as a single entity and response measured.
-
-## Trial
-
-A trial is the part of test described in [RFC2544] section 23.
-
-### Trial Load
-
-Trial load is the intended constant load for a trial.
-
-Load is the quantity implied by Constant Load of [RFC1242],
-Data Rate of [RFC2544] and Intended Load of [RFC2285].
-All three specify this value applies to one (input or output) interface.
-
-### Trial Duration
-
-Trial duration is the intended duration of the traffic for a trial.
-
-In general, this general quantity does not include any preparation nor waiting
-described in section 23 of [RFC2544].
-
-However, the measurer MAY return a duration value different
-from the intended duration. This may be useful for users
-who want to control the overal search duration, not just the traffic part of it.
-The manager MUST report how does the measurer computes the returned duration
-values in that case.
-
-### Trial Forwarding Ratio
-
-Trial forwarding ratio is dimensionless floating point value,
-assumed to be between 0.0 and 1.0, both including.
-It is computed as the number of frames forwarded by SUT, divided by
-the number of frames that should have been forwarded during the trial.
-
-Note that, contrary to load, frame counts used to compute
-trial forwarding ratio are aggregates over all SUT ports.
-
-Questions around what is the correct number of frames
-that should have been forwarded it outside of the scope of this document.
-E.g. what should the measurer return when it detects
-that the offered load differs significantly from the intended load.
-
-It is RECOMMENDED implementations return an irregular goal result
-if they detect questionable (in comparability sense) trial results
-affecting their goal result.
-
-### Trial Loss Ratio
-
-Trial loss ratio is equal to one minus the trial forwarding ratio.
-
-### Trial Forwarding Rate
-
-The trial forwarding rate is the trial load multiplied by
-the trial forwarding ratio.
-
-Note that this is very similar, but not identical to Forwarding Rate
-as defined in [RFC2285] section 3.6.1, as that definition
-is specific to one output interface, while trial forwarding ratio
-is based on frame counts aggregated over all SUT interfaces.
-
-## Traffic profile
-
-Any other specifics (besides trial load and trial duration)
-the measurer needs to perform the trial are understood as a composite
-called the traffic profile.
-All its attributes are assumed to be constant during the search,
-and the composite is configured on the measurer by the manager
-before the search starts.
-
-Traffic profile is REQUIRED by [RFC2544] to contain some specific quantities,
-for example frame size.
-Several more specific quantities may be RECOMMENDED.
-
-Depending on SUT configuration, e.g. when testing specific protocols,
-additional values need to be included in the traffic profile
-and in the test report. See other IETF documents.
-
-## Search Goal
-
-A search goal is one item of the list required as an argument
-when the manager calls the controller.
-
-Each search goal is composite consisting of several attributes,
-some of them are required.
-Implementations are free to add their own attributes.
-
-Subsections list all required attributes and one recommended attribute.
-
-The meaning of the attributes is formally given only by their effect
-on the computation of the attributes of the goal result.
-The subsections do contain a short informal description,
-but see other chapters for more in-depth explanations.
-
-### Goal Final Trial Duration
-
-A threshold value for trial durations.
-REQUIRED attribute, MUST be positive.
-
-Informally, the conditional throughput for this goal will be computed
-only from trial results from trials as long as this.
-
-### Goal Duration Sum
-
-A threshold value for a particular sum of trial durations.
-REQUIRED attribute, MUST be positive.
-
-This uses the duration values returned by the measurer,
-in case it returns something else than the intended durations
-for the traffic part of the search.
-
-Informally, even at looking only at trials done at this goal's
-final trial duration, MLRsearch may spend up to this time measuring
-the same load value.
-
-### Goal Loss Ratio
-
-A threshold value for trial loss ratios.
-REQUIRED attribute, MUST be non-negative and smaller than one.
-
-Informally, if a load causes too many trials with trial results larger than this,
-the conditional throughput for this goal will be smaller than that load.
-
-### Goal Exceed Ratio
-
-A threshold value for particular ratio of duration sums.
-REQUIRED attribute, MUST be non-negative and smaller than one.
-
-This uses the duration values returned by the measurer,
-in case it returns something else than the intended durations
-for the traffic part of the search.
-
-Informally, this acts as the q-value for a quantile when selecting
-the forwarding rate of which trial result becomes the conditional throughput.
-For example, when the goal exceed ratio is 0.5 and MLRsearch
-happened to use the whole goal duration sum when determining conditional
-throughput, it means the conditional throughput is the median
-of trial forwarding rates.
-In practice, MLRsearch may stop measuring a load before the goal duration sum
-is reached, and the conditional throughput in that case frequently
-is the worst trial still not exceeding the goal loss ratio.
-If the goal duration sum is no larger than the goal fina trial duration,
-MLRsearch performs only one trial per load (unless other goals need more)
-and the goal exceed ratio has no effect on the search result.
-
-### Goal Width
-
-A value used as a threshold for telling when two trial load values
-are close enough.
-
-RECOMMENDED attribute, positive. Implementations without this attribute
-MUST give the manager other ways to control the search exit condition.
-
-Absolute load difference and relative load difference are two popular choices,
-but implementations may choose a different way to specify width.
-
-Informally, this acts as a stopping condition, controling the precision
-of the search. The search stops if every goal has reached its precision.
-
-## Search Result
-
-Search result is a single composite object returned from the controller
-to the manager.
-It is a mapping from the search goals (see section Search Goal) into goal results
-(see section Goal Result).
-The mapping MUST map from all the search goals present in the controller input.
-
-Each search goal instance is mapped to a goal result instance.
-Multiple search goal instances may map to the same goal result instance.
-
-## Goal Result
-
-Goal result is a composite object consisting of several attributes.
-All attributes are related to the same search goal, the one search goal instance
-the Search Result is mapping into this instance of the Goal Result.
-
-Some of the attributes are required, some are recommended,
-implementations are free to add their own.
-
-The subsections define attributes for regular goal result.
-Implementations are free to define their own irregular goal results,
-but the manager MUST report them clearly as not regular according to this section.
-
-A typical irregular result is when all trials at maximum offered load
-have zero loss, as the relevant upper bound does not exist in that case.
-
-### Relevant Upper Bound
-
-Relevant upper bound is the intended load value that is classified
-at the end of the search as the relevant upper bound (see Appendix A)
-for this goal.
-This is a REQUIRED attribute.
-
-Informally, this is the smallest intended load that failed to uphold
-all the requirements of this search goal, mainly the goal loss ratio
-in combination with the goal exceed ratio.
-
-### Relevant Lower Bound
-
-Relevant lower bound is the intended load value that got classified
-(after all trials) as the relevant lower bound (see Appendix A) for this goal.
-This is a REQUIRED attribute.
-
-The distance between the relevant lower bound and the relevant upper bound
-MUST NOT be larger than the goal width, for a regular goal result,
-if the implementation offers width as a goal attribute.
-
-Informally, this is the smallest intended load that managed to uphold
-all the requirements of this search goal, mainly the goal loss ratio
-in combination with the goal exceed ratio, while not being larger
-than the relevant upper bound.
-
-### Conditional Throughput
-
-The conditional throughput (see Appendix B) as evaluated
-at the relevant lower bound.
-This is a RECOMMENDED attribute.
-
-Informally, a typical forwarding rate expected to be seen
-at the relevant lower bound. But frequently just a conservative estimate thereof,
-as MLRsearch implementations tend to stop gathering more data
-as soon as they confirm this estimate cannot get worse within
-the goal duration sum.
-
-# MLRsearch Explanations
-
-This chapter focuses on intuitions and motivations
-and skips over some important details.
-
-Familiarity with the MLRsearch specification is not required here,
-so this chapter can act as an introduction.
-For example, this chapter start talking about tightest lower bounds
-before it is ready to talk about the relevant lower bound from the specification.
-
-## MLRsearch Versions
-
-The MLRsearch algorithm has been developed in a code-first approach,
-a Python library has been created, debugged and used in production
-before first descriptions (even informal) were published.
-In fact, multiple versions of the library were used in production
-over past few years, and later code was usually not compatible
-with earlier descriptions.
-
-The code in (any version of) MLRsearch library fully determines
-the search process (for given configuration parameters),
-leaving no space for deviations.
-MLRsearch as a name for a broad class of possible algorithms
-leaves plenty of space for future improvements, at the cost
-of poor comparability of results of different MLRsearch implementations.
-
-This document aspires to prescribe a MLRsearch specification
-in a way that restricts the important parts related to comparability,
-while leaving other parts vague enough so implementations can improve freely.
-
-## Exit Condition
-
-[RFC2544] prescribes that after performing one trial at a specific offered load,
-the next offered load should be larger or smaller, based on frame loss.
-
-The usual implementation uses binary search. Here a lossy trial becomes
-a new upper bound, a lossless trial becomes a new lower bound.
-The span of values between the tightest lower bound and the tightest upper bound
-forms an interval of possible results,
-and after each trial the width of that interval halves.
-
-Usually the binary search implementation tracks only the two tightest bounds,
-simply calling them bounds, but the old values still remain valid bounds,
-just not as tight as the new ones.
-
-After some number of trials, the tightest lower bound becomes the throughput.
-[RFC2544] does not specify when (if ever) should the search stop.
-
-MLRsearch library introduces a concept of goal width. The search stops
-when the distance between the tightest upper bound and the tightest lower bound
-is smaller than a user-configured value called goal width from now on.
-In other words, interval width has to be smaller than goal width
-at the end of the search.
-
-This goal width value therefore determines the precision of the result.
-As MLRsearch specification requires a particular structure of the result,
-the result itself does contain enough information to determine its precision,
-thus it is not required to report the goal width value.
-
-This allows MLRsearch implementations to use exit conditions
-different from goal width.
-The MLRsearch specification only REQUIRES the search procedure
-to always finish in a finite time, regardless of possible trial results.
-
-## Load Classification
-
-MLRsearch keeps the basic logic of binary search (tracking tightest bounds,
-measuring at the middle), perhaps with minor technical clarifications.
-The algorithm chooses an intended load (as opposed to offered load),
-the interval between bounds does not need to be split exactly in two equal halves,
-and the final reported structure specifies both bounds
-(optionally also the conditional throughput at the lower bound, defined later).
-
-The biggest difference is that in order to classify a load
-as an upper or lower bound, MLRsearch may need more than one trial
-(depending on configuration options) to be performed at the same intended load.
-
-As a consequence, even if a load already does have few trial results,
-it still may be classified as undecided, neither a lower bound nor an upper bound.
-
-Explanation of the classification logic is given in the next chatper,
-as it relies heavily on other sections of this chapter.
-
-For repeatability and comparability reasons, it is important that
-given a set of trial results, all implementations of MLRsearch
-classify the load in an equivalent way.
-
-## Loss Ratios
-
-Next difference is in goals of the search. [RFC2544] has a single goal,
-based on classifying full-length trials as either loss-less or lossy.
-
-As the name suggests, MLRsearch can seach for multiple goals, differing in their
-loss ratios. Precise definition of goal loss ratio will be given later.
-The [RFC2544] throughput goal then simply becomes a zero goal loss ratio.
-Different goals also may have different goal width.
-
-A set of trial results for one specific intended load value
-can classify the load as an upper bound for some goals, but a lower bound
-for some other goals, and undecided for the rest of the goals.
-
-Therefore, the load classification depends not only on ttrial results,
-but also on the goal. The overall search procedure becomes more complicated
-(compared to binary search with a single goal),
-but most of the complications do not affect the final result,
-except for one phenomenon, loss inversion.
-
-## Loss Inversion
-
-In [RFC2544] throuhput search using bisection, any load with lossy trial
-becomes a hard upper bound, meaning every subsequent trial has smaller
-intended load.
-
-But in MLRsearch, a load that is classified as an upper bound for one goal
-may still be a lower bound for another goal, and due to that other goal
-MLRsearch will probably perform trials at even higher loads.
-What to do when all such higher load trials happen to have zero loss?
-Does it mean the earlier upper bound was not real?
-Does it mean the later lossless trials are not considered a lower bound?
-Surely we do not want to have an upper bound at a load smaller than a lower bound.
-
-MLRsearch is conservative in these situations.
-The upper bound is considered real, and the lossless trials at higher loads
-are considered to be a coincidence, at least when computing the final result.
-
-This is formalized using new notions, the relevant upper bound and
-the relevant lower bound.
-Load classification is still based just on the set of trial results
-at a given intended load (trials at other loads are ignored),
-making it possible to have a lower load classified as an upper bound
-and a higher load classified as a lower bound (for the same goal).
-The relevant upper bound (for a goal) is the smallest load classified
-as an upper bound. But the relevant lower bound is not simply
-the largest among lower bounds. It is the largest load among loads
-that are lower bounds while also being smaller than the relevant upper bound.
-
-With these definitions, the relevant lower bound is always smaller
-than the relevant upper bound (if both exist), and the two relevant bounds
-are used analogously as the two tightest bounds in the binary search.
-When they are less than goal width apart, the relevant bounds are used in output.
-
-One consequence is that every trial result can have an impact on the search result.
-That means if your SUT (or your traffic generator) needs a warmup,
-be sure to warm it up before starting the search.
-
-## Exceed Ratio
-
-The idea of performing multiple trials at the same load comes from
-a model where some trial results (those with high loss) are affected
-by infrequent effects, causing poor repeatability of [RFC2544] throughput results.
-See the discussion about noiseful and noiseless ends of SUT performance spectrum.
-Stable results are closer to the noiseless end of SUT preformance spectrum,
-so MLRsearch may need to allow some frequency of high-loss trials
-to ignore the reare but big effects near the noisefull end.
-
-MLRsearch is able to do such trial result filtering, but it needs
-a configuration option to tell it how much frequent can the infrequent big loss be.
-This option is called exceed ratio. It tells MLRsearch what ratio of trials
-(more exactly what ratio of trial seconds) can have trial loss ratio
-larger than goal loss ratio and still be classified as a lower bound.
-Zero exceed ratio means all trials have to have trial loss ratio
-equal to or smaller than the goal loss ratio.
-
-For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5,
-as it simplifies some later concepts by relating them to the concept of median.
-
-## Duration Sum
-
-When more than one trial is needed to classify a load,
-MLRsearch also needs something that controlls the number of trials needed.
-Therefore, each goal also has an attribute called duration sum.
-
-The meaning of a goal duration sum is that when a load has trials
-(at full trial duration, details later)
-whose trial durations when summed up give a value at least this,
-the load is guaranteed to be classified as an upper bound or a lower bound
-for the goal.
-
-As the duration sum has a big impact on the overall search duration,
-and [RFC2544] prescibes wait intervals around trial traffic,
-the MLRsearch algorithm may sum durations that are different
-from the actual trial traffic durations.
-
-## Short Trials
-
-Section 24 of [RFC2544] already anticipates possible time savings
-when short trials (shorter than full length trials) are used.
-
-MLRsearch requires each goal to specify its final trial duration.
-Full-length trial is the short name for a trial whose intended trial duration
-is equal to the goal final trial duration.
-
-Any MLRsearch implementation may include its own configuration options
-which control when and how MLRsearch chooses to use shorter trial durations.
-
-For explainability reasons, when exceed ratio of 0.5 is used,
-it is recommended for the goal duration sum to be an odd multiple
-of the full trial durations, so conditional throughput becomes identical to
-a median of a particular set of forwarding rates.
-
-Presence of shorter trial results complicates the load classification logic.
-Full details are given later. In short, results from short trials
-may cause a load to be classified as an upper bound.
-This may cause loss inversion, and thus lower the relevant lower bound
-(below what would classification say when considering full-length trials only).
-
-For explainability reasons, it is RECOMMENDED users use such configurations
-that guarantee all trials have the same length.
-Alas, such configurations are usually not compliant with [RFC2544] requirements,
-or not time-saving enough.
-
-## Conditional Throughput
-
-As testing equipment takes intended load as input parameter
-for a trial measurement, any load search algorithm needs to deal
-with intended load values internally.
-
-But in presence of goals with non-zero loss ratio, the intended load
-usually does not match the user intuition of what a throughput is.
-The forwarding rate (as defined in [RFC2285] section 3.6.1) is better,
-but it is not obvious how to generalize it
-for loads with multiple trial results,
-especially with non-zero goal exceed ratio.
-
-MLRsearch defines one such generalization, called the conditional throughput.
-It is the forwarding rate from one of the trials performed at the load
-in question. Specification of which trial exactly is quite technical.
-More detailed explanations are given in the next chapter.
-
-Conditional throughput is partially related to load classification.
-If a load is classified as a lower bound for a goal,
-the conditional throughpt can be calculated,
-and guaranteed to show effective loss ratio no larger than goal loss ratio.
-
-While the conditional throughput gives more intuitive-looking values
-than the relevant lower bound, especially for non-zero goal loss ratio values,
-the actual definition is more complicated than the definition of the relevant
-lower bound. In future, other intuitive values may become popular,
-but they are unlikely to supersede the definition of the relevant lower bound
-as the most fitting value for comparability purposes,
-therefore the relevant lower bound remains a required attribute
-of the goal result structure.
-
-Note that comparing best and worst case, the same relevant lower bound value
-may result in the conditional throughput differing up to the goal loss ratio.
-Therefore it is rarely needed to set the goal width (if expressed
-as relative difference of loads) below the goal loss ratio.
-In other words, setting the goal width below the goal loss ratio
-may cause the conditional throughput for a larger loss ratio to became smaller
-than a conditional throughput for a goal with a smaller goal loss ratio,
-which is counter-intuitive, considering they come from the same search.
-Therefore it is RECOMMENDED to set the goal width to a value no smaller
-than the goal loss ratio.
-
-## Search Time
-
-The main motivation for MLRsearch was to have an algorithm that spends less time
-finding a throughput, either the [RFC2544] compliant one,
-or some generalization thereof. The art of achieving short search times
-is mainly in smart selection of intended loads (and intended durations)
-for the next trial to perform.
-
-While there is an indirect impact of the load selection on the reported values,
-in practice such impact tends to be small,
-even for SUTs with quite broad performance spectrum.
-
-A typical example of two approaches to load selection leading to different
-relevant lower bounds is when the interval is split in a very uneven way.
-An implementation chosing loads very close to the current relevant lower bound
-are quite likely to eventually stumble upon a trial result
-with poor performance (due to SUT noise).
-For an implementation chosing load very close to the current relevant upper bound
-this is unlikely, as it examines more loads that can see a performance
-close to the noiseless end of the SUT performance spectrum.
-The reason why it is unlikely to have two MLRsearch implementation showing
-this kind of preference in load selection is precisely
-in the desire to have short searches.
-Even splits are the best way to achive the desired precision,
-so the more optimized a search algorithm is for the overall search duration,
-the better the repeatability and comparability
-of its results will be, assuming the user configuration remains the same.
-
-Therefore, this document remains quite vague on load selection
-and other optimisation details, and configuration attributes related to them.
-Assuming users prefer libreries that achieve short overall search time,
-the definition of the relevant lower bound
-should be strict enough to ensure result repeatability
-and comparability between different implementations,
-while not restricting future implementations much.
-
-Sadly, different implementations may exhibit their sweet spot of
-best repeatability at given search duration at different goals attribute values,
-especially with respect to optional goal attributes
-such as initial trial duration.
-Thus, this document does not comment much on which configurations
-are good for comparability between different implementations.
-For comparability between different SUTs using the same implementation,
-refer to configurations recommended by that particular implementation.
-
-## [RFC2544] compliance
-
-The following search goal ensures unconditional compliance with
-[RFC2544] throughput search procedure:
-
-- Goal loss ratio: zero.
-
-- Goal final trial duration: 60 seconds.
-
-- Goal duration sum: 60 seconds.
-
-- Goal exceed ratio: zero.
-
-Presence of other search goals does not affect compliance of this goal result.
-The relevant lower bound and the conditional throughput are in this case
-equal to each other, and the value is the [RFC2544] throughput.
-
-If the 60 second quantity is replaced by a smaller quantity in both attributes,
-the conditional throughput is still conditionally compliant with
-[RFC2544] throughput.
-
-# Selected Functional Details
-
-This chapter continues with explanations,
-but this time more precise definitions are needed
-for readers to follow the explanations.
-The definitions here are wordy, implementers can look into the next chapter
-for more concise definitions.
-
-The two areas of focus in this chapter are the load classification
-and the conditional throughput, starting with the latter.
-
-## Performance Spectrum
-
-There are several equivalent ways to define the conditional throughput computation.
-One of the ways relies on an object called the performance spectrum.
-First, two heavy definitions are needed.
-
-Take an intended load value, and a finite set of trial results, all trials
-measured at that load value. The performance spectrum is the function that maps
-any non-negative real number into a sum of trial durations among all trials
-in the set that have that number as their forwarding rate,
-e.g. map to zero if no trial has that particular forwarding rate.
-
-A related function, defined if there is at least one trial in the set,
-is the performance spectrum divided by sum of durations of all trials in the set.
-That function is called the performance probability function, as it satisfies
-all the requirements for probability mass function function
-of a discrete probability distribution,
-the one-dimensional random variable being the trial forwarding rate.
-
-These functions are related to the SUT performance spectrum,
-as sampled by the trials in the set.
-
-As for any other probability function, we can talk about percentiles,
-of the performance probability function, and bout other quantiles
-such as the median. The conditional throughput will be
-one such quantile value for a specifically chosen set of trials.
-
-Take a set of all full-length trials performed at the load in question.
-The sum of durations of those trials may be less than goal duration sum, or not.
-If it is less, add an imaginary trial result with zero forwarding rate
-such that the new sum of durations is equal to the goal duration sum.
-This is the set of trials to use. The q-value for the quantile
-is the goal exceed ratio. If the quantile touches two trials,
-the larger forwarding rate is used.
-
-First example. For zero exceed ratio when goal duration sum
-has been reached. The conditional throughput is the smallest forwarding
-rate among the trials.
-
-Second example. For zero exceed ratio when goal duration sum
-has not been reached yet. Due to the missing duration sum,
-the worst case may still happen, so the conditional througput
-is zero. This is not reported to the user, as this load
-cannot become the relevant lower bound yet.
-
-Third example. Exceed ratio 50%, goal duration sum two seconds,
-one trial present with duration one second and zero loss.
-An imaginary trial is added with duration one second and zero forwarding rate.
-Median would touch both trials, so the conditional throughput
-is the forwarding rate of the one non-imaginary trial.
-As that had zero loss, the value is equal to the offered load.
-
-The classification does not need the whole performance spectrum,
-only few duration sums.
-
-A trial is called bad (according to a goal) if its trial loss ratio
-is larger than the goal loss ratio. Trial that is not bad is called good.
-
-## Single Trial Duration
-
-When goal attributes are chosen in such a way that every trial has the same
-intended duration, the load classification is sipler.
-
-The following description looks technical, but it follows the motivation
-of goal loss ratio, goal exceed ratio and goal duration sum.
-If sum of durations of all trials (at given load) is less than the goal
-duration sum, imagine best case scenario (all subsequent trials having zero loss)
-and worst case scenario (all subsequent trials having 100% loss).
-Here we assume there is as many subsequent trials as needed
-to make the sum of all trials to become equal to the goal duration sum.
-As the exceed ratio is defined just using sums of durations
-(number of trials does not matter), it does not matter whether
-the "subsequent trials" can consist of integer number of full-length trials.
-
-If even in the best case scenario the load exceed ratio would be larger
-than the goal exceed ratio, the load is an upper bound.
-If even in the worst case scenario the load exceed ratio would not be larger
-than the goal exceed ratio, the load is a lower bound.
-
-Even more specifically.
-Take all trials measured at a given load.
-Sum of durations of all bad full-length trials is called the bad sum.
-Sum of durations of all good full-length trials is called the good sum.
-The result of adding bad sum plus the good sum is called the measured sum.
-The larger of the measured sum and the goal duration sum is called the whole sum.
-The whole sum minus the measured sum is called the missing sum.
-Optimistic exceed ratio is the bad sum divided by the whole sum.
-Pessimistic exceed ratio is the bad sum plus the missing sum, that divided by
-the whole sum.
-If optimistic exceed ratio is larger than the goal exceed ratio,
-the load is classified as an upper bound.
-If pessimistic exceed ratio is not larger than the goal exceed ratio,
-the load is classified as a lower bound.
-Else, the load is classified as undecided.
-
-The definition of pessimistic exceed ratio matches the logic in
-the conditional throughput computation, so a load is a lower bound
-if and only if the conditional throughput effective loss ratio
-is not larger than the goal loss ratio.
-If it is larger, the load is either an upper bound or undecided.
-
-## Short Trial Scenarios
-
-Trials with intended duration smaller than the goal final trial duration
-are called short trials.
-The motivation for load classification logic in presence of short trials
-is based around a counter-factual case: What would the trial result be
-if a short trial has been measured as a full-length trial instead?
-
-There are three main scenarios where human intuition guides
-the intended behavior of load classification.
-
-Scenario one. The user had their reason for not configuring shorter goal
-final trial duration. Perhaps SUT has buffers that may get full at longer
-trial durations. Perhaps SUT shows periodic decreases of performance
-the user does not want to treat as noise. In any case, many good short trials
-may became bad full-length trial in the counter-factual case.
-In extreme case, there are no bad short trials.
-In this scenario, we want the load classification NOT to classify the load
-as a lower bound, despite the abundance of good short trials.
-Effectively, we want the good short trials to be ignored, so they
-do not contribute to comparisons with the goal duration sum.
-
-Scenario two. When there is a frame loss in a short trial,
-the counter-factual full-length trial is expected to lose at least as many
-frames. And in practice, bad short trials are rarely turning into
-good full-length trials. In extreme case, there are no good short trials.
-In this scenario, we want the load classification
-to classify the load as an upper bound just based on abundance
-of short bad trials. Effectively we want the bad short trials
-to contribute to comparisons with the goal duration sum,
-so the load can be classified sooner.
-
-Scenario three. Some SUTs are quite indifferent to trial duration.
-Performance probability function constructed from short trial results
-is likely to be similar to performance probability function constructed
-from full-length trial results (perhaps with smaller dispersion,
-but overall without big impact on the median quantiles).
-For moderate goal exceed ratio values, this may mean there are both
-good short trials and bad short trials.
-This scenario is there just to invalidate a simple heuristic
-of always ignoring good short trials and never ignoring bad short trials.
-That simple heuristic would be too biased. Yes, the short bad trials
-are likely to turn into full-length bad trials in the counter-factual case,
-but there is no information on what would the good short trials turn into.
-The only way to decide is to do more trials at full length,
-the same as in scenario one.
-
-## Short Trial Logic
-
-MLRsearch picks a particular logic for load classification
-in presence of short trials, but it is still RECOMMENDED to use configurations
-that imply no short trials, so the possible inefficiencies in that logic
-do not affect the result, and the result has better explainability.
-
-With thas said, the logic differs from the single trial duration case
-only in different definition of bad sum.
-Good sum is still the sum across all good full-length trials.
-
-Few more notions are needed for definig the new bad sum.
-Sum of durations of all bad full-length trials is called the bad long sum.
-Sum of durations of all bad short trials is called the bad short sum.
-Sum of durations of all good short trials is called the good short sum.
-One minus the goal exceed ratio is called the inceed ratio.
-The goal exceed ratio divided by the inceed ratio is called the exceed coefficient.
-The good short sum multiplied by the exceed coefficient is called the balancing sum.
-The bad short sum minus the balancing sum is called the excess sum.
-If the excess sum is negative, the bad sum is equal to the bad long sum.
-Else, the bad sum is equal to the bad long sum plus the excess sum.
-
-Here is how the new definition of the bad sum fares in the three scenarios,
-where the load is close to what would relevant bounds be
-if only full-length trials were used for the search.
-
-Scenario one. If duration is too short, we expect to see higher frequency
-of good short trials. This could lead to negative excess sum,
-which has no impact, hence the load classification is given just by
-full-length trials.
-Thus, MLRsearch using too short trials has no detrimental effect
-on result comparability in this scenario.
-But also using short trials does not help with overall search duration,
-proably making it worse.
-
-Scenario two. Settings with small exceed ratio have small exceed coefficient,
-so the impact of good short sum is small and the bad short sum
-is almost wholly converted into excess sum, thus bad short trials
-have almost as big impact as full-length bad trials.
-The same conclusion applies for moderate exceed ratio values
-when the good short sum is small.
-Thus, short trials can cause a load to get classified as an upper bound earlier
-bringing time savings (while not affecting comparability).
-
-Scenario three. Here excess sum is small in absolute value, as balancing sum
-is expected to be be similar to the bad short sum.
-Once again, full-length trials are needed for final load classification,
-but usage of short trials probably means MLRsearch needed shorter search time
-before selecting this load for measurement, bringing time savings
-(while not affecting comparability).
-
-## Longer Trial Durations
-
-If there are trial results with intended duration larger
-than the goal trial duration, the classification logic is intentionally undefined.
-
-The implementations MAY treat such longer trials as if they were full-length.
-In any case, presence of such longer trials in either the relevant lower bound
-or the relevant upper bound SHOULD be mentioned, as for sume SUTs
-it is likely to affect comparability.
-
-
-TODO: Here will be a chapter summarizing how MLRsearch library
-adresses the problems from the Identified Problems chapter.
-
-{::comment}
-
-    # Problems after MLRsearch
-
-    Now when MLRsearch is clearly specified and explained,
-    it is possible to summarize how does MLRsearch specification help with problems.
-
-    Here, "multiple trials" is a shorthand for having the goal final trial duration
-    significantly smaller than the goal duration sum.
-    This results in MLRsearch performing multiple trials at the same load,
-    which may not be the case with other configurations.
-
-    ## Long Test Duration
-
-    As shortening the overall search duration is the main motivation
-    of MLRsearch library development, the library implements
-    multiple improvements on this front, both big and small.
-    Most of implementation details are not part of the MLRsearch specification,
-    so that future implementations may keep shortening the search duration even more.
-
-    TODO: The rest is about attributes.
-
-    From the required goal attributes, the goal duration sum
-    remains the best way to get even shorter searches.
-
-    Usage of multiple trials can also save time,
-    depending on wait times around trial traffic.
-
-    The farther the goal exceed ratio is from 0.5 towards either side,
-    the less predictable the overal search duration becomes in practice.
-
-    Width parameter does not change search duration much in practice
-    (compared to other, mainly optional goal attributes).
-
-    ## DUT in SUT
-
-    Practice shows big improvements when multiple trials
-    and moderate exceed ratios are used. Mainly when it comes to result
-    repeatability, as sometimes it is not easy to distinguish
-    SUT noise from DUT instability.
-
-    Conditional throughput has intuitive meaning when described
-    using the performance spectrum, so this is an improvement,
-    especially when compared to search procedures which use non-zero
-    goal loss ratio but return only the intended load at a lower bound.
-
-    Multiple trials can save time also when the noisy end of
-    the preformance spectrum needs to be examined, e.g. for [RFC9004].
-
-    Under some circumstances, testing the same DUT and SUT setup with different
-    DUT configurations can give some hints on what part of noise us SUT noise
-    and what part is DUT performance fluctuations.
-    In practice, both types of noise tend to be too complicated for that analysis.
-    MLRsearch does not offer additional tools in this regard,
-    apart of giving users ability to search for more goals,
-    hoping to get more insight at the cost of longer overall search time.
-
-    ## Repeatability and Comparability
-
-    Multiple trials improve repeatability, depending on exceed ratio.
-
-    In practice, 1s goal final trial duration with exceed ratio 0.5
-    is good enough for modern SUTs (but that usually requires
-    smaller wait times around the traffic part of the trial,
-    otherwise too much search time is wasted waiting).
-
-    It is not clear whether exceed ratios higher than 0.5 are better
-    for repeatability.
-    The 0.5 value is still preferred due to explainability using median.
-    TODO: Stress single value is for comparability, which one is due explainability.
-
-    It is possible that the conditional throughput values (with non-zero
-    goal loss ratio) are better for repeatability than the relevant
-    lower bound values, especially for implementations
-    which pick load from a small set of discrete values.
-
-    Implementations focusing on shortening the overall search time
-    are automatically forced to avoid comparability issues
-    due to load selection, as they must prefer even splits wherever possible.
-    But this conclusion only holds when the same goals are used.
-    Larger adoption is needed before any further claims on comparability
-    between MLRsearch implementations can be made.
-
-    ## Throughput with Non-Zero Loss
-
-    Suported by the goal loss ratio attribute.
-    Improves repeatability as expected.
-
-    ## Inconsistent Trial Results
-
-    MLRsearch is conservative wherever possible,
-    this is built into the definition of conditional throughput,
-    and into the treatment of short trial results for load classification.
-
-    This is consistent with [RFC2544] zero loss tolerance motivation.
-
-    If the very best (noiseless) part of the SUT performance spectrum
-    is of interest, it should be enough to set small value for
-    the goal final trial duration, and perhaps also a large value
-    for the goal exceed ratio.
-
-    Implementations may offer other (optional) configuration attributes
-    (and optional goal result attributes)
-    to become less conservative, but currently it is not clear
-    what impact would that have on repeatability.
-
-{:/comment}
-
-# IANA Considerations
-
-No requests of IANA.
-
-# Security Considerations
-
-Benchmarking activities as described in this memo are limited to
-technology characterization of a DUT/SUT using controlled stimuli in a
-laboratory environment, with dedicated address space and the constraints
-specified in the sections above.
-
-The benchmarking network topology will be an independent test setup and
-MUST NOT be connected to devices that may forward the test traffic into
-a production network or misroute traffic to the test management network.
-
-Further, benchmarking is performed on a "black-box" basis, relying
-solely on measurements observable external to the DUT/SUT.
-
-Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
-benchmarking purposes. Any implications for network security arising
-from the DUT/SUT SHOULD be identical in the lab and in production
-networks.
-
-# Acknowledgements
-
-Many thanks to Alec Hothan of OPNFV NFVbench project for thorough
-review and numerous useful comments and suggestions.
-
-Special wholehearted gratitude and thanks to late Al Morton for his
-thorough reviews filled with very specific feedback and constructive
-guidelines. Thank you Al for the close collaboration over the years,
-for your continuous unwavering encouragements full of empathy and
-positive attitude.
-Al, you are dearly missed.
-
-# Appendix A
-
-This is a specification of load classification.
-
-The block at the end of this appendix holds pseudocode
-which computes two values, stored in variables named optimistic and pessimistic.
-The pseudocode happens to be a valid Python code.
-
-If both values are computed to be true, the load in question
-is classified as a lower bound according to the goal in question.
-If both values are false, the load is classified as an upper bound.
-Otherwise, the load is classifies as undecided.
-
-The pseudocole expects the following variables hold values as follows:
-
-- goal_duration_sum: The goal duration sum value.
-
-- goal_exceed_ratio: The goal exceed ratio value.
-
-- good_long_sum: Sum of durations across trials with trial duration
-  at least equal to the goal final trial duration and with trial loss ratio
-  not higher than the goal loss ratio.
-
-- bad_long_sum: Sum of durations across trials with trial duration
-  at least equal to the goal final trial duration and with trial loss ratio
-  higher than the goal loss ratio.
-
-- good_short_sum: Sum of durations across trials with trial duration
-  shorter than the goal final trial duration and with trial loss ratio
-  not higher than the goal loss ratio.
-
-- bad_short_sum: Sum of durations across trials with trial duration
-  shorter than the goal final trial duration and with trial loss ratio
-  higher than the goal loss ratio.
-
-Here the implicit set of available trial results consists of all trials
-measured at given intended load at the end of search.
-
-The code works correctly also when there are no trial results at given load.
-
-```
-balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio)
-effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum)
-effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum)
-quantile_duration_sum = effective_whole_sum * goal_exceed_ratio
-optimistic = effective_bad_sum <= quantile_duration_sum
-pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum
-```
-
-# Appendix B
-
-This is a specification of conditional throughput.
-
-The block at the end of this appendix holds pseudocode
-which computes a value stored as variable conditional_throughput.
-The pseudocode happens to be a valid Python code.
-
-The pseudocole expects the following variables hold values as follows:
-
-- goal_duration_sum: The goal duration sum value.
-
-- goal_exceed_ratio: The goal exceed ratio value.
-
-- good_long_sum: Sum of durations across trials with trial duration
-  at least equal to the goal final trial duration and with trial loss ratio
-  not higher than the goal loss ratio.
-
-- bad_long_sum: Sum of durations across trials with trial duration
-  at least equal to the goal final trial duration and with trial loss ratio
-  higher than the goal loss ratio.
-
-- long_trials: An iterable of all trial results from trials with trial duration
-  at least equal to the goal final trial duration,
-  sorted by increasing trial loss ratio.
-  A trial result is a composite with the following two attributes available:
-
-  - trial.loss_ratio: The trial loss ratio as measured for this trial.
-
-  - trial.duration: The trial duration of this trial.
-
-Here the implicit set of available trial results consists of all trials
-measured at given intended load at the end of search.
-
-The code works correctly only when there if there is at leas one
-trial result measured at given load.
-
-```
-all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum)
-remaining = all_long_sum * (1.0 - goal_exceed_ratio)
-quantile_loss_ratio = None
-for trial in long_trials:
-    if quantile_loss_ratio is None or remaining > 0.0:
-        quantile_loss_ratio = trial.loss_ratio
-        remaining -= trial.duration
-    else:
-        break
-else:
-    if remaining > 0.0:
-        quantile_loss_ratio = 1.0
-
-conditional_throughput = intended_load * (1.0 - quantile_loss_ratio)
-```
-
---- back