diff options
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-06.md | 1634 | ||||
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-07.md | 3123 | ||||
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt | 2800 | ||||
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml | 3136 | ||||
| -rw-r--r-- | docs/ietf/process.txt | 2 |
5 files changed, 9060 insertions, 1635 deletions
diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-06.md b/docs/ietf/draft-ietf-bmwg-mlrsearch-06.md deleted file mode 100644 index 27d65e2690..0000000000 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-06.md +++ /dev/null @@ -1,1634 +0,0 @@ ---- - -title: Multiple Loss Ratio Search -abbrev: MLRsearch -docname: draft-ietf-bmwg-mlrsearch-06 -date: 2024-03-04 - -ipr: trust200902 -area: ops -wg: Benchmarking Working Group -kw: Internet-Draft -cat: info - -coding: us-ascii -pi: # can use array (if all yes) or hash here - toc: yes - sortrefs: # defaults to yes - symrefs: yes - -author: - - - ins: M. Konstantynowicz - name: Maciek Konstantynowicz - org: Cisco Systems - email: mkonstan@cisco.com - - - ins: V. Polak - name: Vratko Polak - org: Cisco Systems - email: vrpolak@cisco.com - -normative: - RFC1242: - RFC2285: - RFC2544: - RFC9004: - -informative: - TST009: - target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf - title: "TST 009" - FDio-CSIT-MLRsearch: - target: https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/ - title: "FD.io CSIT Test Methodology - MLRsearch" - date: 2023-10 - PyPI-MLRsearch: - target: https://pypi.org/project/MLRsearch/1.2.1/ - title: "MLRsearch 1.2.1, Python Package Index" - date: 2023-10 - ---- abstract - -This document proposes extensions to [RFC2544] throughput search by -defining a new methodology called Multiple Loss Ratio search -(MLRsearch). MLRsearch aims to minimize search duration, -support multiple loss ratio searches, -and enhance result repeatability and comparability. - -The primary reason for extending [RFC2544] is to address the challenges -and requirements presented by the evaluation and testing -of software-based networking systems' data planes. - -To give users more freedom, MLRsearch provides additional configuration options -such as allowing multiple shorter trials per load instead of one large trial, -tolerating a certain percentage of trial results with higher loss, -and supporting the search for multiple goals with varying loss ratios. - ---- middle - -{::comment} - As we use Kramdown to convert from Markdown, - we use this way of marking comments not to be visible in the rendered draft. - https://stackoverflow.com/a/42323390 - If another engine is used, convert to this way: - https://stackoverflow.com/a/20885980 -{:/comment} - -# Purpose and Scope - -The purpose of this document is to describe Multiple Loss Ratio search -(MLRsearch), a data plane throughput search methodology optimized for software -networking DUTs. - -Applying vanilla [RFC2544] throughput bisection to software DUTs -results in several problems: - -- Binary search takes too long as most trials are done far from the - eventually found throughput. -- The required final trial duration and pauses between trials - prolong the overall search duration. -- Software DUTs show noisy trial results, - leading to a big spread of possible discovered throughput values. -- Throughput requires a loss of exactly zero frames, but the industry - frequently allows for small but non-zero losses. -- The definition of throughput is not clear when trial results are inconsistent. - - -To address the problems mentioned above, -the MLRsearch library employs the following enhancements: - -- Allow multiple shorter trials instead of one big trial per load. - - Optionally, tolerate a percentage of trial results with higher loss. -- Allow searching for multiple search goals, with differing loss ratios. - - Any trial result can affect each search goal in principle. -- Insert multiple coarse targets for each search goal, earlier ones need - to spend less time on trials. - - Earlier targets also aim for lesser precision. - - Use Forwarding Rate (FR) at maximum offered load - [RFC2285] (section 3.6.2) to initialize the initial targets. -- Take care when dealing with inconsistent trial results. - - Reported throughput is smaller than the smallest load with high loss. - - Smaller load candidates are measured first. -- Apply several load selection heuristics to save even more time - by trying hard to avoid unnecessarily narrow bounds. - -Some of these enhancements are formalized as MLRsearch specification, -the remaining enhancements are treated as implementation details, -thus achieving high comparability without limiting future improvements. - -MLRsearch configuration options are flexible enough to -support both conservative settings and aggressive settings. -Where the conservative settings lead to results -unconditionally compliant with [RFC2544], -but longer search duration and worse repeatability. -Conversely, aggressive settings lead to shorter search duration -and better repeatability, but the results are not compliant with [RFC2544]. - -No part of [RFC2544] is intended to be obsoleted by this document. - -# Identified Problems - -This chapter describes the problems affecting usability -of various performance testing methodologies, -mainly a binary search for [RFC2544] unconditionally compliant throughput. - -## Long Search Duration - -The emergence of software DUTs, with frequent software updates and a -number of different frame processing modes and configurations, -has increased both the number of performance tests -required to verify the DUT update and the frequency of running those tests. -This makes the overall test execution time even more important than before. - -The current [RFC2544] throughput definition restricts the potential -for time-efficiency improvements. -A more generalized throughput concept could enable further enhancements -while maintaining the precision of simpler methods. - -The bisection method, when unconditionally compliant with [RFC2544], -is excessively slow. -This is because a significant amount of time is spent on trials -with loads that, in retrospect, are far from the final determined throughput. - -[RFC2544] does not specify any stopping condition for throughput search, -so users already have an access to a limited trade-off -between search duration and achieved precision. -However, each full 60-second trials doubles the precision, -so not many trials can be removed without a substantial loss of precision. - -## DUT in SUT - -[RFC2285] defines: -- DUT as - - The network forwarding device to which stimulus is offered and - response measured [RFC2285] (section 3.1.1). -- SUT as - - The collective set of network devices to which stimulus is offered - as a single entity and response measured [RFC2285] (section 3.1.2). - -[RFC2544] specifies a test setup with an external tester stimulating the -networking system, treating it either as a single DUT, or as a system -of devices, an SUT. - -In the case of software networking, the SUT consists of not only the DUT -as a software program processing frames, but also of -a server hardware and operating system functions, -with server hardware resources shared across all programs -and the operating system running on the same server. - -Given that the SUT is a shared multi-tenant environment -encompassing the DUT and other components, the DUT might inadvertently -experience interference from the operating system -or other software operating on the same server. - -Some of this interference can be mitigated. -For instance, -pinning DUT program threads to specific CPU cores -and isolating those cores can prevent context switching. - -Despite taking all feasible precautions, some adverse effects may still impact -the DUT's network performance. -In this document, these effects are collectively -referred to as SUT noise, even if the effects are not as unpredictable -as what other engineering disciplines call noise. - -DUT can also exhibit fluctuating performance itself, for reasons -not related to the rest of SUT; for example due to pauses in execution -as needed for internal stateful processing. -In many cases this -may be an expected per-design behavior, as it would be observable even -in a hypothetical scenario where all sources of SUT noise are eliminated. -Such behavior affects trial results in a way similar to SUT noise. -As the two phenomenons are hard to distinguish, -in this document the term 'noise' is used to encompass -both the internal performance fluctuations of the DUT -and the genuine noise of the SUT. - -A simple model of SUT performance consists of an idealized noiseless performance, -and additional noise effects. -For a specific SUT, the noiseless performance is assumed to be constant, -with all observed performance variations being attributed to noise. -The impact of the noise can vary in time, sometimes wildly, -even within a single trial. -The noise can sometimes be negligible, but frequently -it lowers the observed SUT performance as observed in trial results. - -In this model, SUT does not have a single performance value, it has a spectrum. -One end of the spectrum is the idealized noiseless performance value, -the other end can be called a noiseful performance. -In practice, trial result -close to the noiseful end of the spectrum happens only rarely. -The worse the performance value is, the more rarely it is seen in a trial. -Therefore, the extreme noiseful end of the SUT spectrum is not observable -among trial results. -Also, the extreme noiseless end of the SUT spectrum -is unlikely to be observable, this time because some small noise effects -are likely to occur multiple times during a trial. - -Unless specified otherwise, this document's focus is -on the potentially observable ends of the SUT performance spectrum, -as opposed to the extreme ones. - -When focusing on the DUT, the benchmarking effort should ideally aim -to eliminate only the SUT noise from SUT measurements. -However, -this is currently not feasible in practice, as there are no realistic enough -models available to distinguish SUT noise from DUT fluctuations, -based on the author's experience and available literature. - -Assuming a well-constructed SUT, the DUT is likely its -primary performance bottleneck. -In this case, we can define the DUT's -ideal noiseless performance as the noiseless end of the SUT performance spectrum, -especially for throughput. -However, other performance metrics, such as latency, -may require additional considerations. - -Note that by this definition, DUT noiseless performance -also minimizes the impact of DUT fluctuations, as much as realistically possible -for a given trial duration. - -This document aims to solve the DUT in SUT problem -by estimating the noiseless end of the SUT performance spectrum -using a limited number of trial results. - -Any improvements to the throughput search algorithm, aimed at better -dealing with software networking SUT and DUT setup, should employ -strategies recognizing the presence of SUT noise, allowing the discovery of -(proxies for) DUT noiseless performance -at different levels of sensitivity to SUT noise. - -## Repeatability and Comparability - -[RFC2544] does not suggest to repeat throughput search. -And from just one -discovered throughput value, it cannot be determined how repeatable that value is. -Poor repeatability then leads to poor comparability, -as different benchmarking teams may obtain varying throughput values -for the same SUT, exceeding the expected differences from search precision. - -[RFC2544] throughput requirements (60 seconds trial and -no tolerance of a single frame loss) affect the throughput results -in the following way. -The SUT behavior close to the noiseful end of its performance spectrum -consists of rare occasions of significantly low performance, -but the long trial duration makes those occasions not so rare on the trial level. -Therefore, the binary search results tend to wander away from the noiseless end -of SUT performance spectrum, more frequently and more widely than shorter -trials would, thus causing poor throughput repeatability. - -The repeatability problem can be addressed by defining a search procedure -that identifies a consistent level of performance, -even if it does not meet the strict definition of throughput in [RFC2544]. - -According to the SUT performance spectrum model, better repeatability -will be at the noiseless end of the spectrum. -Therefore, solutions to the DUT in SUT problem -will help also with the repeatability problem. - -Conversely, any alteration to [RFC2544] throughput search -that improves repeatability should be considered -as less dependent on the SUT noise. - -An alternative option is to simply run a search multiple times, and report some -statistics (e.g. average and standard deviation). -This can be used -for a subset of tests deemed more important, -but it makes the search duration problem even more pronounced. - -## Throughput with Non-Zero Loss - -[RFC1242] (section 3.17) defines throughput as: - The maximum rate at which none of the offered frames - are dropped by the device. - -Then, it says: - Since even the loss of one frame in a - data stream can cause significant delays while - waiting for the higher level protocols to time out, - it is useful to know the actual maximum data - rate that the device can support. - -However, many benchmarking teams accept a small, -non-zero loss ratio as the goal for their load search. - -Motivations are many: - -- Modern protocols tolerate frame loss better, - compared to the time when [RFC1242] and [RFC2544] were specified. - -- Trials nowadays send way more frames within the same duration, - increasing the chance of a small SUT performance fluctuation - being enough to cause frame loss. - -- Small bursts of frame loss caused by noise have otherwise smaller impact - on the average frame loss ratio observed in the trial, - as during other parts of the same trial the SUT may work more closely - to its noiseless performance, thus perhaps lowering the trial loss ratio - below the goal loss ratio value. - -- If an approximation of the SUT noise impact on the trial loss ratio is known, - it can be set as the goal loss ratio. - -Regardless of the validity of all similar motivations, -support for non-zero loss goals makes any search algorithm more user-friendly. -[RFC2544] throughput is not user-friendly in this regard. - -Furthermore, allowing users to specify multiple loss ratio values, -and enabling a single search to find all relevant bounds, -significantly enhances the usefulness of the search algorithm. - -Searching for multiple search goals also helps to describe the SUT performance -spectrum better than the result of a single search goal. -For example, the repeated wide gap between zero and non-zero loss loads -indicates the noise has a large impact on the observed performance, -which is not evident from a single goal load search procedure result. - -It is easy to modify the vanilla bisection to find a lower bound -for the intended load that satisfies a non-zero goal loss ratio. -But it is not that obvious how to search for multiple goals at once, -hence the support for multiple search goals remains a problem. - -## Inconsistent Trial Results - -While performing throughput search by executing a sequence of -measurement trials, there is a risk of encountering inconsistencies -between trial results. - -The plain bisection never encounters inconsistent trials. -But [RFC2544] hints about the possibility of inconsistent trial results, -in two places in its text. -The first place is section 24, where full trial durations are required, -presumably because they can be inconsistent with the results -from shorter trial durations. -The second place is section 26.3, where two successive zero-loss trials -are recommended, presumably because after one zero-loss trial -there can be a subsequent inconsistent non-zero-loss trial. - -Examples include: - -- A trial at the same load (same or different trial duration) results - in a different trial loss ratio. -- A trial at a higher load (same or different trial duration) results - in a smaller trial loss ratio. - -Any robust throughput search algorithm needs to decide how to continue -the search in the presence of such inconsistencies. -Definitions of throughput in [RFC1242] and [RFC2544] are not specific enough -to imply a unique way of handling such inconsistencies. - -Ideally, there will be a definition of a new quantity which both generalizes -throughput for non-zero-loss (and other possible repeatability enhancements), -while being precise enough to force a specific way to resolve trial result -inconsistencies. -But until such a definition is agreed upon, the correct way to handle -inconsistent trial results remains an open problem. - -# MLRsearch Specification - -This chapter focuses on technical definitions needed for evaluating -whether a particular test procedure adheres to MLRsearch specification. - -For motivations, explanations, and other comments see other chapters. - -## MLRsearch Architecture - -MLRsearch architecture consists of three main components: -the manager, the controller, and the measurer. -For definitions of the components, see the following sections. - -The architecture also implies the presence of other components, such as the SUT. - -These components can be seen as abstractions present in any testing procedure. - -### Measurer - -The measurer is the component that performs one trial -as described in [RFC2544] section 23. - -Specifically, one call to the measurer accepts a trial load value -and trial duration value, performs the trial, and returns -the measured trial loss ratio, and optionally a different duration value. - -It is the responsibility of the measurer to uphold any requirements -and assumptions present in MLRsearch specification -(e.g. trial forwarding ratio not being larger than one). -Implementers have some freedom, for example in the way they deal with -duplicated frames, or what to return if the tester sent zero frames towards SUT. -Implementations are RECOMMENDED to document their behavior -related to such freedoms in as detailed a way as possible. - -Implementations MUST document any deviations from RFC documents, -for example if the wait time around traffic -is shorter than what [RFC2544] section 23 specifies. - -### Controller - -The controller selects trial load and duration values -to achieve the search goals in the shortest expected time. - -The controller calls the measurer multiple times, -receiving the trial result from each call. -After exit condition is met, the controller returns -the overall search results. - -The controller's role in optimizing trial load and duration selection -distinguishes MLRsearch algorithms from simpler search procedures. - -For controller inputs, see later section Controller Inputs. -For controller outputs, see later section Controller Outputs. - -### Manager - -The controller gets initiated by the manager once, and subsequently calls - -The manager is the component that initializes SUT, the traffic generator -(tester in [RFC2544] terminology), the measurer and the controller -with intended configurations. -It then calls the controller once, and receives its outputs. - -The manager is also responsible for creating reports in the appropriate format, -based on information in controller outputs. - -## Units - -The specification deals with physical quantities, so it is assumed -each numeric value is accompanied by an appropriate physical unit. - -The specification does not state which unit is appropriate, -but implementations MUST make it explicit which unit is used -for each value provided or received by the user. - -For example, load quantities (including the conditional throughput) -returned by the controller are defined to be based on a single-interface -(unidirectional) loads. -For bidirectional traffic, users are likely -to expect bidirectional throughput quantities, so the manager is responsible -for making its report clear. - -## SUT - -As defined in [RFC2285]: -The collective set of network devices to which stimulus is offered -as a single entity and response measured. - -## Trial - -A trial is the part of the test described in [RFC2544] section 23. - -### Trial Load - -The trial load is the intended constant load for a trial. - -Load is the quantity implied by Constant Load of [RFC1242], -Data Rate of [RFC2544] and Intended Load of [RFC2285]. -All three specify this value applies to one (input or output) interface. - -### Trial Duration - -Trial duration is the intended duration of the traffic for a trial. - -In general, this quantity does not include any preparation nor waiting -described in section 23 of [RFC2544]. - -However, the measurer MAY return a duration value that deviates -from the intended duration. -This feature can be beneficial for users -who wish to manage the overall search duration, -rather than solely the traffic portion of it. -The manager MUST report -how the measurer computes the returned duration values in that case. - -### Trial Forwarding Ratio - -The trial forwarding ratio is a dimensionless floating point value -that ranges from 0.0 to 1.0, inclusive. -It is calculated by dividing the number of frames -successfully forwarded by the SUT -by the total number of frames expected to be forwarded during the trial. - -Note that, contrary to loads, frame counts used to compute -trial forwarding ratio are aggregates over all SUT output ports. - -Questions around what is the correct number of frames -that should have been forwarded is outside of the scope of this document. -E.g. what should the measurer return when it detects -that the offered load differs significantly from the intended load. - -### Trial Loss Ratio - -The trial loss ratio is equal to one minus the trial forwarding ratio. - -### Trial Forwarding Rate - -The trial forwarding rate is a derived quantity, calculated by -multiplying the trial load by the trial forwarding ratio. - -It is important to note that while similar, this quantity is not identical -to the Forwarding Rate as defined in [RFC2285] section 3.6.1, -as the latter is specific to one output interface, -whereas the trial forwarding ratio is based -on frame counts aggregated over all SUT output interfaces. - -## Traffic profile - -Any other specifics (besides trial load and trial duration) -the measurer needs in order to perform the trial -are understood as a composite called the traffic profile. -All its attributes are assumed to be constant during the search, -and the composite is configured on the measurer by the manager -before the search starts. - -The traffic profile is REQUIRED by [RFC2544] -to contain some specific quantities, for example frame size. -Several more specific quantities may be RECOMMENDED. - -Depending on SUT configuration, e.g. when testing specific protocols, -additional values need to be included in the traffic profile -and in the test report. -See other IETF documents. - -## Search Goal - -The search goal is a composite consisting of several attributes, -some of them are required. -Implementations are free to add their own attributes. - -A particular set of attribute values is called a search goal instance. - -Subsections list all required attributes and one recommended attribute. -Each subsection contains a short informal description, -but see other chapters for more in-depth explanations. - -The meaning of the attributes is formally given only by their effect -on the controller output attributes (defined in later in section Search Result). - -Informally, later chapters give additional intuitions and examples -to the search goal attribute values. -Later chapters also give motivation to formulas of computation of the outputs. - -### Goal Final Trial Duration - -A threshold value for trial durations. -This attribute is REQUIRED, and the value MUST be positive. - -Informally, while MLRsearch is allowed to perform trials shorter than this, -but results from such short trials have only limited impact on search results. - -The full relation needs definitions is later subsections. -But for example, the conditional throughput -(definition in subsection Conditional Throughput) -for this goal will be computed only from trial results -from trials at least as long as this. - -### Goal Duration Sum - -A threshold value for a particular sum of trial durations. -This attribute is REQUIRED, and the value MUST be positive. - -This uses the duration values returned by the measurer. - -Informally, even when looking only at trials done at this goal's -final trial duration, MLRsearch may spend up to this time measuring -the same load value. -If the goal duration sum is larger than -the goal final trial duration, it means multiple trials need to be measured -at the same load. - -### Goal Loss Ratio - -A threshold value for trial loss ratios. -REQUIRED attribute, MUST be non-negative and smaller than one. - -Informally, if a load causes too many trials with trial loss ratios -larger than this, the conditional throughput for this goal -will be smaller than that load. - -### Goal Exceed Ratio - -A threshold value for a particular ratio of duration sums. -REQUIRED attribute, MUST be non-negative and smaller than one. - -The duration sum values come from the duration values returned by the measurer. - -Informally, the impact of lossy trials is controlled by this value. -The full relation needs definitions is later subsections. - -But for example, the definition of the conditional throughput -(given later in subsection Conditional Throughput) -refers to a q-value for a quantile when selecting -which trial result gives the conditional throughput. -The goal exceed ratio acts as the q-value to use there. - -Specifically, when the goal exceed ratio is 0.5 and MLRsearch happened -to use the whole goal duration sum (using full-length trials), -it means the conditional throughput is the median of trial forwarding rates. - -### Goal Width - -A value used as a threshold for telling when two trial load values -are close enough. - -RECOMMENDED attribute, positive. -Implementations without this attribute -MUST give the manager other ways to control the search exit condition. - -Absolute load difference and relative load difference are two popular choices, -but implementations may choose a different way to specify width. - -Informally, this acts as a stopping condition, controlling the precision -of the search. -The search stops if every goal has reached its precision. - -## Controller Inputs - -The only REQUIRED input for controller is a set of search goal instances. -MLRsearch implementations MAY use additional input parameters for the controller. - -The order of instances SHOULD NOT have a big impact on controller outputs, -but MLRsearch implementations MAY base their behavior on the order -of search goal instances. - -The search goal instances SHOULD NOT be identical. -MLRsearch implementation MAY allow identical instances. - -## Goal Result - -Before defining the output of the controller, -it is useful to define what the goal result is. - -The goal result is a composite object consisting of several attributes. -A particular set of attribute values is called a goal result instance. - -Any goal result instance can be either regular or irregular. -MLRsearch specification puts requirements on regular goal result instances. -Any instance that does not meet the requirements is deemed irregular. - -Implementations are free to define their own irregular goal results, -but the manager MUST report them clearly as not regular according to this section. - -All attribute values in one goal result instance -are related to a single search goal instance, -referred to as the given search goal. - -Some of the attributes of a regular goal result instance are required, -some are recommended, implementations are free to add their own. - -The subsections define two required and one optional attribute -for a regular goal result. - -A typical irregular result is when all trials at the maximal offered load -have zero loss, as the relevant upper bound does not exist in that case. - -### Relevant Upper Bound - -The relevant upper bound is the smallest intended load value that is classified -at the end of the search as an upper bound (see Appendix A) -for the given search goal. -This is a REQUIRED attribute. - -Informally, this is the smallest intended load that failed to uphold -all the requirements of the given search goal, mainly the goal loss ratio -in combination with the goal exceed ratio. - -### Relevant Lower Bound - -The relevant lower bound is the largest intended load value -among those smaller than the relevant upper bound -that got classified at the end of the search -as a lower bound (see Appendix A) for the given search goal. -This is a REQUIRED attribute. - -For a regular goal result, the distance between the relevant lower bound -and the relevant upper bound MUST NOT be larger than the goal width, -if the implementation offers width as a goal attribute. - -Informally, this is the largest intended load that managed to uphold -all the requirements of the given search goal, mainly the goal loss ratio -in combination with the goal exceed ratio, while not being larger -than the relevant upper bound. - -### Conditional Throughput - -The conditional throughput (see Appendix B) -as evaluated at the relevant lower bound of the given search goal -at the end of the search. -This is a RECOMMENDED attribute. - -Informally, this is a typical forwarding rate expected to be seen -at the relevant lower bound of the given search goal. -But frequently just a conservative estimate thereof, -as MLRsearch implementations tend to stop gathering more data -as soon as they confirm the result cannot get worse than this estimate -within the goal duration sum. - -## Search Result - -The search result is a single composite object -that maps each search goal to a corresponding goal result. - -In other words, search result is an unordered list of key-value pairs, -where no two pairs contain equal keys. -The key is a search goal instance, acting as the given search goal -for the goal result instance in the value portion of the key-value pair. - -The search result (as a mapping) -MUST map from all the search goals present in the controller input. - -## Controller Outputs - -The search result is the only REQUIRED output -returned from the controller to the manager. - -MLRsearch implementation MAY return additional data in the controller output. - -# Further Explanations - -This chapter focuses on intuitions and motivations -and skips over some important details. - -Familiarity with the MLRsearch specification is not required here, -so this chapter can act as an introduction. -For example, this chapter starts talking about the tightest lower bounds -before it is ready to talk about the relevant lower bound from the specification. - -## MLRsearch Versions - -The MLRsearch algorithm has been developed in a code-first approach, -a Python library has been created, debugged, and used in production -before the first descriptions (even informal) were published. -In fact, multiple versions of the library were used in the production -over the past few years, and later code was usually not compatible -with earlier descriptions. - -The code in (any version of) MLRsearch library fully determines -the search process (for given configuration parameters), -leaving no space for deviations. -MLRsearch, as a name for a broad class of possible algorithms, -leaves plenty of space for future improvements, at the cost -of poor comparability of results of different MLRsearch implementations. - -There are two competing needs. -There is the need for standardization in areas critical to comparability. -There is also the need to allow flexibility for implementations -to innovate and improve in other areas. -This document defines the MLRsearch specification -in a manner that aims to fairly balances both needs. - -## Exit Condition - -[RFC2544] prescribes that after performing one trial at a specific offered load, -the next offered load should be larger or smaller, based on frame loss. - -The usual implementation uses binary search. -Here a lossy trial becomes -a new upper bound, a lossless trial becomes a new lower bound. -The span of values between (including both) the tightest lower bound -and the tightest upper bound forms an interval of possible results, -and after each trial the width of that interval halves. - -Usually the binary search implementation tracks only the two tightest bounds, -simply calling them bounds. -But the old values still B remain valid bounds, -just not as tight as the new ones. - -After some number of trials, the tightest lower bound becomes the throughput. -[RFC2544] does not specify when (if ever) should the search stop. - -MLRsearch library introduces a concept of goal width. -The search stops -when the distance between the tightest upper bound and the tightest lower bound -is smaller than a user-configured value, called goal width from now on. -In other words, the interval width at the end of the search -has to be no larger than the goal width. - -This goal width value therefore determines the precision of the result. -As MLRsearch specification requires a particular structure of the result, -the result itself does contain enough information to determine its precision, -thus it is not required to report the goal width value. - -This allows MLRsearch implementations to use exit conditions -different from goal width. - -## Load Classification - -MLRsearch keeps the basic logic of binary search (tracking tightest bounds, -measuring at the middle), perhaps with minor technical clarifications. -The algorithm chooses an intended load (as opposed to the offered load), -the interval between bounds does not need to be split -exactly into two equal halves, -and the final reported structure specifies both bounds. - -The biggest difference is that to classify a load -as an upper or lower bound, MLRsearch may need more than one trial -(depending on configuration options) to be performed at the same intended load. - -As a consequence, even if a load already does have few trial results, -it still may be classified as undecided, neither a lower bound nor an upper bound. - -An explanation of the classification logic is given in the next chapter, -as it relies heavily on other sections of this chapter. - -For repeatability and comparability reasons, it is important that -given a set of trial results, all implementations of MLRsearch -classify the load equivalently. - -## Loss Ratios - -The next difference is in the goals of the search. -[RFC2544] has a single goal, -based on classifying full-length trials as either lossless or lossy. - -As the name suggests, MLRsearch can search for multiple goals, -differing in their loss ratios. -The precise definition of the goal loss ratio will be given later. -The [RFC2544] throughput goal then simply becomes a zero goal loss ratio. -Different goals also may have different goal widths. - -A set of trial results for one specific intended load value -can classify the load as an upper bound for some goals, but a lower bound -for some other goals, and undecided for the rest of the goals. - -Therefore, the load classification depends not only on trial results, -but also on the goal. -The overall search procedure becomes more complicated -(compared to binary search with a single goal), -but most of the complications do not affect the final result, -except for one phenomenon, loss inversion. - -## Loss Inversion - -In [RFC2544] throughput search using bisection, any load with a lossy trial -becomes a hard upper bound, meaning every subsequent trial has a smaller -intended load. - -But in MLRsearch, a load that is classified as an upper bound for one goal -may still be a lower bound for another goal, and due to the other goal -MLRsearch will probably perform trials at even higher loads. -What to do when all such higher load trials happen to have zero loss? -Does it mean the earlier upper bound was not real? -Does it mean the later lossless trials are not considered a lower bound? -Surely we do not want to have an upper bound at a load smaller than a lower bound. - -MLRsearch is conservative in these situations. -The upper bound is considered real, and the lossless trials at higher loads -are considered to be a coincidence, at least when computing the final result. - -This is formalized using new notions, the relevant upper bound and -the relevant lower bound. -Load classification is still based just on the set of trial results -at a given intended load (trials at other loads are ignored), -making it possible to have a lower load classified as an upper bound, -and a higher load classified as a lower bound (for the same goal). -The relevant upper bound (for a goal) is the smallest load classified -as an upper bound. -But the relevant lower bound is not simply -the largest among lower bounds. -It is the largest load among loads -that are lower bounds while also being smaller than the relevant upper bound. - -With these definitions, the relevant lower bound is always smaller -than the relevant upper bound (if both exist), and the two relevant bounds -are used analogously as the two tightest bounds in the binary search. -When they are less than the goal width apart, -the relevant bounds are used in the output. - -One consequence is that every trial result can have an impact on the search result. -That means if your SUT (or your traffic generator) needs a warmup, -be sure to warm it up before starting the search. - -## Exceed Ratio - -The idea of performing multiple trials at the same load comes from -a model where some trial results (those with high loss) are affected -by infrequent effects, causing poor repeatability of [RFC2544] throughput results. -See the discussion about noiseful and noiseless ends -of the SUT performance spectrum. -Stable results are closer to the noiseless end of the SUT performance spectrum, -so MLRsearch may need to allow some frequency of high-loss trials -to ignore the rare but big effects near the noiseful end. - -MLRsearch can do such trial result filtering, but it needs -a configuration option to tell it how frequent can the infrequent big loss be. -This option is called the exceed ratio. -It tells MLRsearch what ratio of trials -(more exactly what ratio of trial seconds) can have a trial loss ratio -larger than the goal loss ratio and still be classified as a lower bound. -Zero exceed ratio means all trials have to have a trial loss ratio -equal to or smaller than the goal loss ratio. - -For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5, -as it simplifies some later concepts by relating them to the concept of median. - -## Duration Sum - -When more than one trial is needed to classify a load, -MLRsearch also needs something that controls the number of trials needed. -Therefore, each goal also has an attribute called duration sum. - -The meaning of a goal duration sum is that when a load has trials -(at full trial duration, details later) -whose trial durations when summed up give a value at least this long, -the load is guaranteed to be classified as an upper bound or a lower bound -for the goal. - -As the duration sum has a big impact on the overall search duration, -and [RFC2544] prescribes wait intervals around trial traffic, -the MLRsearch algorithm is allowed to sum durations that are different -from the actual trial traffic durations. - -## Short Trials - -MLRsearch requires each goal to specify its final trial duration. -Full-length trial is a shorter name for a trial whose intended trial duration -is equal to (or longer than) the goal final trial duration. - -Section 24 of [RFC2544] already anticipates possible time savings -when short trials (shorter than full-length trials) are used. -Full-length trials are the opposite of short trials, -so they may also be called long trials. - -Any MLRsearch implementation may include its own configuration options -which control when and how MLRsearch chooses to use shorter trial durations. - -For explainability reasons, when exceed ratio of 0.5 is used, -it is recommended for the goal duration sum to be an odd multiple -of the full trial durations, so conditional throughput becomes identical to -a median of a particular set of forwarding rates. - -The presence of shorter trial results complicates the load classification logic. -Full details are given later. -In short, results from short trials -may cause a load to be classified as an upper bound. -This may cause loss inversion, and thus lower the relevant lower bound -(below what would classification say when considering full-length trials only). - -For explainability reasons, it is RECOMMENDED users use such configurations -that guarantee all trials have the same length. -Alas, such configurations are usually not compliant with [RFC2544] requirements, -or not time-saving enough. - -## Conditional Throughput - -As testing equipment takes the intended load as an input parameter -for a trial measurement, any load search algorithm needs to deal -with intended load values internally. - -But in the presence of goals with a non-zero loss ratio, the intended load -usually does not match the user's intuition of what a throughput is. -The forwarding rate (as defined in [RFC2285] section 3.6.1) is better, -but it is not obvious how to generalize it -for loads with multiple trial results and a non-zero goal loss ratio. - -MLRsearch defines one such generalization, called the conditional throughput. -It is the forwarding rate from one of the trials performed at the load -in question. -Specification of which trial exactly is quite technical, -see the specification and Appendix B. - -Conditional throughput is partially related to load classification. -If a load is classified as a lower bound for a goal, -the conditional throughput can be calculated, -and guaranteed to show an effective loss ratio -no larger than the goal loss ratio. - -While the conditional throughput gives more intuitive-looking values -than the relevant lower bound, especially for non-zero goal loss ratio values, -the actual definition is more complicated than the definition of the relevant -lower bound. -In the future, other intuitive values may become popular, -but they are unlikely to supersede the definition of the relevant lower bound -as the most fitting value for comparability purposes, -therefore the relevant lower bound remains a required attribute -of the goal result structure, while the conditional throughput is only optional. - -Note that comparing the best and worst case, the same relevant lower bound value -may result in the conditional throughput differing up to the goal loss ratio. -Therefore it is rarely needed to set the goal width (if expressed -as the relative difference of loads) below the goal loss ratio. -In other words, setting the goal width below the goal loss ratio -may cause the conditional throughput for a larger loss ratio to become smaller -than a conditional throughput for a goal with a smaller goal loss ratio, -which is counter-intuitive, considering they come from the same search. -Therefore it is RECOMMENDED to set the goal width to a value no smaller -than the goal loss ratio. - -## Search Time - -MLRsearch was primarily developed to reduce the time -required to determine a throughput, either the [RFC2544] compliant one, -or some generalization thereof. -The art of achieving short search times -is mainly in the smart selection of intended loads (and intended durations) -for the next trial to perform. - -While there is an indirect impact of the load selection on the reported values, -in practice such impact tends to be small, -even for SUTs with quite a broad performance spectrum. - -A typical example of two approaches to load selection leading to different -relevant lower bounds is when the interval is split in a very uneven way. -Any implementation choosing loads very close to the current relevant lower bound -is quite likely to eventually stumble upon a trial result -with poor performance (due to SUT noise). -For an implementation choosing loads very close -to the current relevant upper bound, this is unlikely, -as it examines more loads that can see a performance -close to the noiseless end of the SUT performance spectrum. - -However, as even splits optimize search duration at give precision, -MLRsearch implementations that prioritize minimizing search time -are unlikely to suffer from any such bias. - -Therefore, this document remains quite vague on load selection -and other optimization details, and configuration attributes related to them. -Assuming users prefer libraries that achieve short overall search time, -the definition of the relevant lower bound -should be strict enough to ensure result repeatability -and comparability between different implementations, -while not restricting future implementations much. - -Sadly, different implementations may exhibit their sweet spot of -the best repeatability for a given search duration -at different goals attribute values, especially concerning -any optional goal attributes such as the initial trial duration. -Thus, this document does not comment much on which configurations -are good for comparability between different implementations. -For comparability between different SUTs using the same implementation, -refer to configurations recommended by that particular implementation. - -## [RFC2544] compliance - -The following search goal ensures unconditional compliance with -[RFC2544] throughput search procedure: - -- Goal loss ratio: zero. - -- Goal final trial duration: 60 seconds. - -- Goal duration sum: 60 seconds. - -- Goal exceed ratio: zero. - -The presence of other search goals does not affect the compliance -of this goal result. -The relevant lower bound and the conditional throughput are in this case -equal to each other, and the value is the [RFC2544] throughput. - -If the 60 second quantity is replaced by a smaller quantity in both attributes, -the conditional throughput is still conditionally compliant with -[RFC2544] throughput. - -# Logic of Load Classification - -This chapter continues with explanations, -but this time more precise definitions are needed -for readers to follow the explanations. -The definitions here are wordy, implementers should read the specification -chapter and appendices for more concise definitions. - -The two related areas of focus in this chapter are load classification -and the conditional throughput, starting with the latter. - -The section Performance Spectrum contains definitions -needed to gain insight into what conditional throughput means. -The rest of the subsections discuss load classification, -they do not refer to Performance Spectrum, only to a few duration sums. - -For load classification, it is useful to define good and bad trials. -A trial is called bad (according to a goal) if its trial loss ratio -is larger than the goal loss ratio. -The trial that is not bad is called good. - -## Performance Spectrum - -There are several equivalent ways to explain -the conditional throughput computation. -One of the ways relies on an object called the performance spectrum. -First, two heavy definitions are needed. - -Take an intended load value, a trial duration value, and a finite set -of trial results, all trials measured at that load value and duration value. -The performance spectrum is the function that maps -any non-negative real number into a sum of trial durations among all trials -in the set that has that number as their forwarding rate, -e.g. map to zero if no trial has that particular forwarding rate. - -A related function, defined if there is at least one trial in the set, -is the performance spectrum divided by the sum of the durations -of all trials in the set. -That function is called the performance probability function, as it satisfies -all the requirements for probability mass function function -of a discrete probability distribution, -the one-dimensional random variable being the trial forwarding rate. - -These functions are related to the SUT performance spectrum, -as sampled by the trials in the set. - -As for any other probability function, we can talk about percentiles -of the performance probability function, including the median. -The conditional throughput will be one such quantile value -for a specifically chosen set of trials. - -Take a set of all full-length trials performed at the relevant lower bound, -sorted by decreasing forwarding rate. -The sum of the durations of those trials -may be less than the goal duration sum, or not. -If it is less, add an imaginary trial result with zero forwarding rate, -such that the new sum of durations is equal to the goal duration sum. -This is the set of trials to use. -The q-value for the quantile -is the goal exceed ratio. -If the quantile touches two trials, -the larger forwarding rate (from the trial result sorted earlier) is used. -The resulting quantity is the conditional throughput of the goal in question. - -First example. -For zero exceed ratio, when goal duration sum has been reached. -The conditional throughput is the smallest forwarding rate among the trials. - -Second example. -For zero exceed ratio, when goal duration sum has not been reached yet. -Due to the missing duration sum, the worst case may still happen, -so the conditional throughput is zero. -This is not reported to the user, -as this load cannot become the relevant lower bound yet. - -Third example. -Exceed ratio 50%, goal duration sum two seconds, -one trial present with the duration of one second and zero loss. -The imaginary trial is added with the duration -of one second and zero forwarding rate. -The median would touch both trials, so the conditional throughput -is the forwarding rate of the one non-imaginary trial. -As that had zero loss, the value is equal to the offered load. - -Note that Appendix B does not take into account short trial results. - -### Summary - -While the conditional throughput is a generalization of the forwarding rate, -its definition is not an obvious one. - -Other than the forwarding rate, the other source of intuition -is the quantile in general, and the median the the recommended case. - -In future, different quantities may prove more useful, -especially when applying to specific problems, -but currently the conditional throughput is the recommended compromise, -especially for repeatability and comparability reasons. - -## Single Trial Duration - -When goal attributes are chosen in such a way that every trial has the same -intended duration, the load classification is simpler. - -The following description looks technical, but it follows the motivation -of goal loss ratio, goal exceed ratio, and goal duration sum. -If the sum of the durations of all trials (at the given load) -is less than the goal duration sum, imagine best case scenario -(all subsequent trials having zero loss) and worst case scenario -(all subsequent trials having 100% loss). -Here we assume there are as many subsequent trials as needed -to make the sum of all trials equal to the goal duration sum. -As the exceed ratio is defined just using sums of durations -(number of trials does not matter), it does not matter whether -the "subsequent trials" can consist of an integer number of full-length trials. - -In any of the two scenarios, we can compute the load exceed ratio, -As the duration sum of good trials divided by the duration sum of all trials, -in both cases including the assumed trials. - -If even in the best case scenario the load exceed ratio would be larger -than the goal exceed ratio, the load is an upper bound. -If even in the worst case scenario the load exceed ratio would not be larger -than the goal exceed ratio, the load is a lower bound. - -Even more specifically. -Take all trials measured at a given load. -The sum of the durations of all bad full-length trials is called the bad sum. -The sum of the durations of all good full-length trials is called the good sum. -The result of adding the bad sum plus the good sum is called the measured sum. -The larger of the measured sum and the goal duration sum is called the whole sum. -The whole sum minus the measured sum is called the missing sum. -The optimistic exceed ratio is the bad sum divided by the whole sum. -The pessimistic exceed ratio is the bad sum plus the missing sum, -that divided by the whole sum. -If the optimistic exceed ratio is larger than the goal exceed ratio, -the load is classified as an upper bound. -If the pessimistic exceed ratio is not larger than the goal exceed ratio, -the load is classified as a lower bound. -Else, the load is classified as undecided. - -The definition of pessimistic exceed ratio is compatible with the logic in -the conditional throughput computation, so in this single trial duration case, -a load is a lower bound if and only if the conditional throughput -effective loss ratio is not larger than the goal loss ratio. -If it is larger, the load is either an upper bound or undecided. - -## Short Trial Scenarios - -Trials with intended duration smaller than the goal final trial duration -are called short trials. -The motivation for load classification logic in the presence of short trials -is based around a counter-factual case: What would the trial result be -if a short trial has been measured as a full-length trial instead? - -There are three main scenarios where human intuition guides -the intended behavior of load classification. - -False good scenario. -The user had their reason for not configuring a shorter goal -final trial duration. -Perhaps SUT has buffers that may get full at longer -trial durations. -Perhaps SUT shows periodic decreases in performance -the user does not want to be treated as noise. -In any case, many good short trials may become bad full-length trials -in the counter-factual case. -In extreme cases, there are plenty of good short trials and no bad short trials. -In this scenario, we want the load classification NOT to classify the load -as a lower bound, despite the abundance of good short trials. -Effectively, we want the good short trials to be ignored, so they -do not contribute to comparisons with the goal duration sum. - -True bad scenario. -When there is a frame loss in a short trial, -the counter-factual full-length trial is expected to lose at least as many -frames. -And in practice, bad short trials are rarely turning into -good full-length trials. -In extreme cases, there are no good short trials. -In this scenario, we want the load classification -to classify the load as an upper bound just based on the abundance -of short bad trials. -Effectively, we want the bad short trials -to contribute to comparisons with the goal duration sum, -so the load can be classified sooner. - -Balanced scenario. -Some SUTs are quite indifferent to trial duration. -Performance probability function constructed from short trial results -is likely to be similar to the performance probability function constructed -from full-length trial results (perhaps with larger dispersion, -but without a big impact on the median quantiles overall). -For a moderate goal exceed ratio value, this may mean there are both -good short trials and bad short trials. -This scenario is there just to invalidate a simple heuristic -of always ignoring good short trials and never ignoring bad short trials. -That simple heuristic would be too biased. -Yes, the short bad trials -are likely to turn into full-length bad trials in the counter-factual case, -but there is no information on what would the good short trials turn into. -The only way to decide safely is to do more trials at full length, -the same as in scenario one. - -## Short Trial Logic - -MLRsearch picks a particular logic for load classification -in the presence of short trials, but it is still RECOMMENDED -to use configurations that imply no short trials, -so the possible inefficiencies in that logic -do not affect the result, and the result has better explainability. - -With that said, the logic differs from the single trial duration case -only in different definition of the bad sum. -The good sum is still the sum across all good full-length trials. - -Few more notions are needed for defining the new bad sum. -The sum of durations of all bad full-length trials is called the bad long sum. -The sum of durations of all bad short trials is called the bad short sum. -The sum of durations of all good short trials is called the good short sum. -One minus the goal exceed ratio is called the inceed ratio. -The goal exceed ratio divided by the inceed ratio is called the exceed coefficient. -The good short sum multiplied by the exceed coefficient is called the balancing sum. -The bad short sum minus the balancing sum is called the excess sum. -If the excess sum is negative, the bad sum is equal to the bad long sum. -Otherwise, the bad sum is equal to the bad long sum plus the excess sum. - -Here is how the new definition of the bad sum fares in the three scenarios, -where the load is close to what would the relevant bounds be -if only full-length trials were used for the search. - -False good scenario. -If the duration is too short, we expect to see a higher frequency -of good short trials. -This could lead to a negative excess sum, -which has no impact, hence the load classification is given just by -full-length trials. -Thus, MLRsearch using too short trials has no detrimental effect -on result comparability in this scenario. -But also using short trials does not help with overall search duration, -probably making it worse. - -True bad cenario. -Settings with a small exceed ratio -have a small exceed coefficient, so the impact of the good short sum is small, -and the bad short sum is almost wholly converted into excess sum, -thus bad short trials have almost as big an impact as full-length bad trials. -The same conclusion applies to moderate exceed ratio values -when the good short sum is small. -Thus, short trials can cause a load to get classified as an upper bound earlier, -bringing time savings (while not affecting comparability). - -Balanced scenario. -Here excess sum is small in absolute value, as the balancing sum -is expected to be similar to the bad short sum. -Once again, full-length trials are needed for final load classification; -but usage of short trials probably means MLRsearch needed -a shorter overall search time before selecting this load for measurement, -thus bringing time savings (while not affecting comparability). - -Note that in presence of short trial results, -the comparibility between the load classification -and the conditional throughput is only partial. -The conditional throughput still comes from a good long trial, -but a load higher than the relevant lower bound may also compute to a good value. - -## Longer Trial Durations - -If there are trial results with an intended duration larger -than the goal trial duration, the precise definitions -in Appendix A and Appendix B treat them in exactly the same way -as trials with duration equal to the goal trial duration. - -But in configurations with moderate (including 0.5) or small -goal exceed ratio and small goal loss ratio (especially zero), -bad trials with longer than goal durations may bias the search -towards the lower load values, as the noiseful end of the spectrum -gets a larger probability of causing the loss within the longer trials. - -For some users, this is an acceptable price -for increased configuration flexibility -(perhaps saving time for the related goals), -so implementations SHOULD allow such configurations. -Still, users are encouraged to avoid such configurations -by making all goals use the same final trial duration, -so their results remain comparable across implementations. - -# Addressed Problems - -Now when MLRsearch is clearly specified and explained, -it is possible to summarize how does MLRsearch specification help with problems. - -Here, "multiple trials" is a shorthand for having the goal final trial duration -significantly smaller than the goal duration sum. -This results in MLRsearch performing multiple trials at the same load, -which may not be the case with other configurations. - -## Long Test Duration - -As shortening the overall search duration is the main motivation -of MLRsearch library development, the library implements -multiple improvements on this front, both big and small. - -Most of implementation details are not constrained by the MLRsearch specification, -so that future implementations may keep shortening the search duration even more. - -One exception is the impact of short trial results on the relevant lower bound. -While motivated by human intuition, the logic is not straightforward. -In practice, configurations with only one common trial duration value -are capable of achieving good overal search time and result repeatability -without the need to consider short trials. - -### Impact of goal attribute values - -From the required goal attributes, the goal duration sum -remains the best way to get even shorter searches. - -Usage of multiple trials can also save time, -depending on wait times around trial traffic. - -The farther the goal exceed ratio is from 0.5 (towards zero or one), -the less predictable the overal search duration becomes in practice. - -Width parameter does not change search duration much in practice -(compared to other, mainly optional goal attributes). - -## DUT in SUT - -In practice, using multiple trials and moderate exceed ratios -often improves result repeatability without increasing the overall search time, -depending on the specific SUT and DUT characteristics. -Benefits for separating SUT noise are less clear though, -as it is not easy to distinguish SUT noise from DUT instability in general. - -Conditional throughput has an intuitive meaning when described -using the performance spectrum, so this is an improvement -over existing simple (less configurable) search procedures. - -Multiple trials can save time also when the noisy end of -the preformance spectrum needs to be examined, e.g. for [RFC9004]. - -Under some circumstances, testing the same DUT and SUT setup with different -DUT configurations can give some hints on what part of noise is SUT noise -and what part is DUT performance fluctuations. -In practice, both types of noise tend to be too complicated for that analysis. - -MLRsearch enables users to search for multiple goals, -potentially providing more insight at the cost of a longer overall search time. -However, for a thorough and reliable examination of DUT-SUT interactions, -it is necessary to employ additional methods beyond black-box benchmarking, -such as collecting and analyzing DUT and SUT telemetry. - -## Repeatability and Comparability - -Multiple trials improve repeatability, depending on exceed ratio. - -In practice, one-second goal final trial duration with exceed ratio 0.5 -is good enough for modern SUTs. -However, unless smaller wait times around the traffic part of the trial -are allowed, too much of overal search time would be wasted on waiting. - -It is not clear whether exceed ratios higher than 0.5 are better -for repeatability. -The 0.5 value is still preferred due to explainability using median. - -It is possible that the conditional throughput values (with non-zero goal -loss ratio) are better for repeatability than the relevant lower bound values. -This is especially for implementations -which pick load from a small set of discrete values, -as that hides small variances in relevant lower bound values -other implementations may find. - -Implementations focusing on shortening the overall search time -are automatically forced to avoid comparability issues due to load selection, -as they must prefer even splits wherever possible. -But this conclusion only holds when the same goals are used. -Larger adoption is needed before any further claims on comparability -between MLRsearch implementations can be made. - -## Throughput with Non-Zero Loss - -Trivially suported by the goal loss ratio attribute. - -In practice, usage of non-zero loss ratio values -improves the result repeatability -(exactly as expected based on results from simpler search methods). - -## Inconsistent Trial Results - -MLRsearch is conservative wherever possible. -This is built into the definition of conditional throughput, -and into the treatment of short trial results for load classification. - -This is consistent with [RFC2544] zero loss tolerance motivation. - -If the noiseless part of the SUT performance spectrum is of interest, -it should be enough to set small value for the goal final trial duration, -and perhaps also a large value for the goal exceed ratio. - -Implementations may offer other (optional) configuration attributes -to become less conservative, but currently it is not clear -what impact would that have on repeatability. - -# IANA Considerations - -No requests of IANA. - -# Security Considerations - -Benchmarking activities as described in this memo are limited to -technology characterization of a DUT/SUT using controlled stimuli in a -laboratory environment, with dedicated address space and the constraints -specified in the sections above. - -The benchmarking network topology will be an independent test setup and -MUST NOT be connected to devices that may forward the test traffic into -a production network or misroute traffic to the test management network. - -Further, benchmarking is performed on a "black-box" basis, relying -solely on measurements observable external to the DUT/SUT. - -Special capabilities SHOULD NOT exist in the DUT/SUT specifically for -benchmarking purposes. Any implications for network security arising -from the DUT/SUT SHOULD be identical in the lab and in production -networks. - -# Acknowledgements - -Some phrases and statements in this document were created -with help of Mistral AI (mistral.ai). - -Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough -review and numerous useful comments and suggestions. - -Special wholehearted gratitude and thanks to the late Al Morton for his -thorough reviews filled with very specific feedback and constructive -guidelines. Thank you Al for the close collaboration over the years, -for your continuous unwavering encouragement full of empathy and -positive attitude. -Al, you are dearly missed. - -# Appendix A: Load Classification - -This is the specification of how to perform the load classification. - -Any intended load value can be classified, according to the given search goal. - -The algorithm uses (some subsets of) the set of all available trial results -from trials measured at a given intended load at the end of the search. -All durations are those returned by the measurer. - -The block at the end of this appendix holds pseudocode -which computes two values, stored in variables named optimistic and pessimistic. -The pseudocode happens to be a valid Python code. - -If both values are computed to be true, the load in question -is classified as a lower bound according to the given search goal. -If both values are false, the load is classified as an upper bound. -Otherwise, the load is classified as undecided. - -The pseudocode expects the following variables to hold values as follows: - -- goal_duration_sum: The duration sum value of the given search goal. - -- goal_exceed_ratio: The exceed ratio value of the given search goal. - -- good_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a trial loss ratio - not higher than the goal loss ratio. - -- bad_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a trial loss ratio - higher than the goal loss ratio. - -- good_short_sum: Sum of durations across trials with trial duration - shorter than the goal final trial duration and with a trial loss ratio - not higher than the goal loss ratio. - -- bad_short_sum: Sum of durations across trials with trial duration - shorter than the goal final trial duration and with a trial loss ratio - higher than the goal loss ratio. - -The code works correctly also when there are no trial results at the given load. - -~~~ python -balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio) -effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum) -effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum) -quantile_duration_sum = effective_whole_sum * goal_exceed_ratio -optimistic = effective_bad_sum <= quantile_duration_sum -pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum -~~~ - -# Appendix B: Conditional Throughput - -This is the specification of how to compute conditional throughput. - -Any intended load value can be used as the basis for the following computation, -but only the relevant lower bound (at the end of the search) -leads to the value called the conditional throughput for a given search goal. - -The algorithm uses (some subsets of) the set of all available trial results -from trials measured at a given intended load at the end of the search. -All durations are those returned by the measurer. - -The block at the end of this appendix holds pseudocode -which computes a value stored as variable conditional_throughput. -The pseudocode happens to be a valid Python code. - -The pseudocode expects the following variables to hold values as follows: - -- goal_duration_sum: The duration sum value of the given search goal. - -- goal_exceed_ratio: The exceed ratio value of the given search goal. - -- good_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a trial loss ratio - not higher than the goal loss ratio. - -- bad_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a trial loss ratio - higher than the goal loss ratio. - -- long_trials: An iterable of all trial results from trials with trial duration - at least equal to the goal final trial duration, - sorted by increasing the trial loss ratio. - A trial result is a composite with the following two attributes available: - - - trial.loss_ratio: The trial loss ratio as measured for this trial. - - - trial.duration: The trial duration of this trial. - -The code works correctly only when there if there is at least one -trial result measured at a given load. - -~~~ python -all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum) -remaining = all_long_sum * (1.0 - goal_exceed_ratio) -quantile_loss_ratio = None -for trial in long_trials: - if quantile_loss_ratio is None or remaining > 0.0: - quantile_loss_ratio = trial.loss_ratio - remaining -= trial.duration - else: - break -else: - if remaining > 0.0: - quantile_loss_ratio = 1.0 -conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) -~~~ - ---- back diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.md b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.md new file mode 100644 index 0000000000..eb2a218bb8 --- /dev/null +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.md @@ -0,0 +1,3123 @@ +--- + +title: Multiple Loss Ratio Search +abbrev: MLRsearch +docname: draft-ietf-bmwg-mlrsearch-07 +date: 2024-07-18 + +ipr: trust200902 +area: ops +wg: Benchmarking Working Group +kw: Internet-Draft +cat: info + +coding: us-ascii +pi: # can use array (if all yes) or hash here + toc: yes + sortrefs: # defaults to yes + symrefs: yes + +author: + - + ins: M. Konstantynowicz + name: Maciek Konstantynowicz + org: Cisco Systems + email: mkonstan@cisco.com + - + ins: V. Polak + name: Vratko Polak + org: Cisco Systems + email: vrpolak@cisco.com + +normative: + RFC1242: + RFC2285: + RFC2544: + RFC8219: + RFC9004: + +informative: + TST009: + target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf + title: "TST 009" + FDio-CSIT-MLRsearch: + target: https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/ + title: "FD.io CSIT Test Methodology - MLRsearch" + date: 2023-10 + PyPI-MLRsearch: + target: https://pypi.org/project/MLRsearch/1.2.1/ + title: "MLRsearch 1.2.1, Python Package Index" + date: 2023-10 + +--- abstract + +This document proposes extensions to [RFC2544] throughput search by +defining a new methodology called Multiple Loss Ratio search +(MLRsearch). MLRsearch aims to minimize search duration, +support multiple loss ratio searches, +and enhance result repeatability and comparability. + +The primary reason for extending [RFC2544] is to address the challenges +and requirements presented by the evaluation and testing +of software-based networking systems' data planes. + +To give users more freedom, MLRsearch provides additional configuration options +such as allowing multiple short trials per load instead of one large trial, +tolerating a certain percentage of trial results with higher loss, +and supporting the search for multiple goals with varying loss ratios. + +--- middle + +{::comment} + + As we use Kramdown to convert from Markdown, + we use this way of marking comments not to be visible in the rendered draft. + https://stackoverflow.com/a/42323390 + If another engine is used, convert to this way: + https://stackoverflow.com/a/20885980 + + [toc] + +{:/comment} + + +# Purpose and Scope + +The purpose of this document is to describe Multiple Loss Ratio search +(MLRsearch), a data plane throughput search methodology optimized for software +networking DUTs. + +Applying vanilla [RFC2544] throughput bisection to software DUTs +results in several problems: + +- Binary search takes too long as most trials are done far from the + eventually found throughput. +- The required final trial duration and pauses between trials + prolong the overall search duration. +- Software DUTs show noisy trial results, + leading to a big spread of possible discovered throughput values. +- Throughput requires a loss of exactly zero frames, but the industry + frequently allows for small but non-zero losses. +- The definition of throughput is not clear when trial results are inconsistent. + +To address the problems mentioned above, +the MLRsearch test methodology specification employs the following enhancements: + +- Allow multiple short trials instead of one big trial per load. + - Optionally, tolerate a percentage of trial results with higher loss. +- Allow searching for multiple Search Goals, with differing loss ratios. + - Any trial result can affect each Search Goal in principle. +- Insert multiple coarse targets for each Search Goal, earlier ones need + to spend less time on trials. + - Earlier targets also aim for lesser precision. + - Use Forwarding Rate (FR) at maximum offered load + [RFC2285] (section 3.6.2) to initialize the initial targets. +- Take care when dealing with inconsistent trial results. + - Reported throughput is smaller than the smallest load with high loss. + - Smaller load candidates are measured first. +- Apply several load selection heuristics to save even more time + by trying hard to avoid unnecessarily narrow bounds. + +Some of these enhancements are formalized as MLRsearch specification, +the remaining enhancements are treated as implementation details, +thus achieving high comparability without limiting future improvements. + +MLRsearch configuration options are flexible enough to +support both conservative settings and aggressive settings. +The conservative settings lead to results +unconditionally compliant with [RFC2544], +but longer search duration and worse repeatability. +Conversely, aggressive settings lead to shorter search duration +and better repeatability, but the results are not compliant with [RFC2544]. + +No part of [RFC2544] is intended to be obsoleted by this document. + +# Identified Problems + +This chapter describes the problems affecting usability +of various performance testing methodologies, +mainly a binary search for [RFC2544] unconditionally compliant throughput. + +## Long Search Duration + +{::comment} + [Low priority] + + <mark>MKP2 [VP] TODO: Look for mentions of search duration in existing RFCs.</mark> + + <mark>MKP2 [VP] TODO: If not found, define right after defining "the search".</mark> + +{:/comment} + +The emergence of software DUTs, with frequent software updates and a +number of different frame processing modes and configurations, +has increased both the number of performance tests +required to verify the DUT update and the frequency of running those tests. +This makes the overall test execution time even more important than before. + +The current [RFC2544] throughput definition restricts the potential +for time-efficiency improvements. +A more generalized throughput concept could enable further enhancements +while maintaining the precision of simpler methods. + +The bisection method, when unconditionally compliant with [RFC2544], +is excessively slow. +This is because a significant amount of time is spent on trials +with loads that, in retrospect, are far from the final determined throughput. + +[RFC2544] does not specify any stopping condition for throughput search, +so users already have an access to a limited trade-off +between search duration and achieved precision. +However, each full 60-second trials doubles the precision, +so not many trials can be removed without a substantial loss of precision. + +## DUT in SUT + +[RFC2285] defines: +- DUT as + - The network forwarding device to which stimulus is offered and + response measured [RFC2285] (section 3.1.1). +- SUT as + - The collective set of network devices to which stimulus is offered + as a single entity and response measured [RFC2285] (section 3.1.2). + +[RFC2544] specifies a test setup with an external tester stimulating the +networking system, treating it either as a single DUT, or as a system +of devices, an SUT. + +In the case of software networking, the SUT consists of not only the DUT +as a software program processing frames, but also of +server hardware and operating system functions, +with that server hardware resources shared across all programs including +the operating system. + +Given that the SUT is a shared multi-tenant environment +encompassing the DUT and other components, the DUT might inadvertently +experience interference from the operating system +or other software operating on the same server. + +Some of this interference can be mitigated. +For instance, +pinning DUT program threads to specific CPU cores +and isolating those cores can prevent context switching. + +Despite taking all feasible precautions, some adverse effects may still impact +the DUT's network performance. +In this document, these effects are collectively +referred to as SUT noise, even if the effects are not as unpredictable +as what other engineering disciplines call noise. + +DUT can also exhibit fluctuating performance itself, for reasons +not related to the rest of SUT. For example due to pauses in execution +as needed for internal stateful processing. +In many cases this +may be an expected per-design behavior, as it would be observable even +in a hypothetical scenario where all sources of SUT noise are eliminated. +Such behavior affects trial results in a way similar to SUT noise. +As the two phenomenons are hard to distinguish, +in this document the term 'noise' is used to encompass +both the internal performance fluctuations of the DUT +and the genuine noise of the SUT. + +A simple model of SUT performance consists of an idealized noiseless performance, +and additional noise effects. +For a specific SUT, the noiseless performance is assumed to be constant, +with all observed performance variations being attributed to noise. +The impact of the noise can vary in time, sometimes wildly, +even within a single trial. +The noise can sometimes be negligible, but frequently +it lowers the observed SUT performance as observed in trial results. + +In this model, SUT does not have a single performance value, it has a spectrum. +One end of the spectrum is the idealized noiseless performance value, +the other end can be called a noiseful performance. +In practice, trial result +close to the noiseful end of the spectrum happens only rarely. +The worse the performance value is, the more rarely it is seen in a trial. +Therefore, the extreme noiseful end of the SUT spectrum is not observable +among trial results. +Also, the extreme noiseless end of the SUT spectrum +is unlikely to be observable, this time because some small noise effects +are likely to occur multiple times during a trial. + +Unless specified otherwise, this document's focus is +on the potentially observable ends of the SUT performance spectrum, +as opposed to the extreme ones. + +When focusing on the DUT, the benchmarking effort should ideally aim +to eliminate only the SUT noise from SUT measurements. +However, +this is currently not feasible in practice, as there are no realistic enough +models available to distinguish SUT noise from DUT fluctuations, +based on authors' experience and available literature. + +Assuming a well-constructed SUT, the DUT is likely its +primary performance bottleneck. +In this case, we can define the DUT's +ideal noiseless performance as the noiseless end of the SUT performance spectrum, +especially for throughput. +However, other performance metrics, such as latency, +may require additional considerations. + +Note that by this definition, DUT noiseless performance +also minimizes the impact of DUT fluctuations, as much as realistically possible +for a given trial duration. + +MLRsearch methodology aims to solve the DUT in SUT problem +by estimating the noiseless end of the SUT performance spectrum +using a limited number of trial results. + +Any improvements to the throughput search algorithm, aimed at better +dealing with software networking SUT and DUT setup, should employ +strategies recognizing the presence of SUT noise, allowing the discovery of +(proxies for) DUT noiseless performance +at different levels of sensitivity to SUT noise. + +## Repeatability and Comparability + +[RFC2544] does not suggest to repeat throughput search. +And from just one +discovered throughput value, it cannot be determined how repeatable that value is. +Poor repeatability then leads to poor comparability, +as different benchmarking teams may obtain varying throughput values +for the same SUT, exceeding the expected differences from search precision. + +[RFC2544] throughput requirements (60 seconds trial and +no tolerance of a single frame loss) affect the throughput results +in the following way. +The SUT behavior close to the noiseful end of its performance spectrum +consists of rare occasions of significantly low performance, +but the long trial duration makes those occasions not so rare on the trial level. +Therefore, the binary search results tend to wander away from the noiseless end +of SUT performance spectrum, more frequently and more widely than short +trials would, thus causing poor throughput repeatability. + +The repeatability problem can be addressed by defining a search procedure +that identifies a consistent level of performance, +even if it does not meet the strict definition of throughput in [RFC2544]. + +According to the SUT performance spectrum model, better repeatability +will be at the noiseless end of the spectrum. +Therefore, solutions to the DUT in SUT problem +will help also with the repeatability problem. + +Conversely, any alteration to [RFC2544] throughput search +that improves repeatability should be considered +as less dependent on the SUT noise. + +An alternative option is to simply run a search multiple times, and report some +statistics (e.g. average and standard deviation). +This can be used +for a subset of tests deemed more important, +but it makes the search duration problem even more pronounced. + +## Throughput with Non-Zero Loss + +[RFC1242] (section 3.17 Throughput) defines throughput as: + The maximum rate at which none of the offered frames + are dropped by the device. + +Then, it says: + Since even the loss of one frame in a + data stream can cause significant delays while + waiting for the higher level protocols to time out, + it is useful to know the actual maximum data + rate that the device can support. + +However, many benchmarking teams accept a small, +non-zero loss ratio as the goal for their load search. + +Motivations are many: + +- Modern protocols tolerate frame loss better, + compared to the time when [RFC1242] and [RFC2544] were specified. + +- Trials nowadays send way more frames within the same duration, + increasing the chance of a small SUT performance fluctuation + being enough to cause frame loss. + +- Small bursts of frame loss caused by noise have otherwise smaller impact + on the average frame loss ratio observed in the trial, + as during other parts of the same trial the SUT may work more closely + to its noiseless performance, thus perhaps lowering the Trial Loss Ratio + below the Goal Loss Ratio value. + +- If an approximation of the SUT noise impact on the Trial Loss Ratio is known, + it can be set as the Goal Loss Ratio. + +Regardless of the validity of all similar motivations, +support for non-zero loss goals makes any search algorithm more user-friendly. +[RFC2544] throughput is not user-friendly in this regard. + +Furthermore, allowing users to specify multiple loss ratio values, +and enabling a single search to find all relevant bounds, +significantly enhances the usefulness of the search algorithm. + +Searching for multiple Search Goals also helps to describe the SUT performance +spectrum better than the result of a single Search Goal. +For example, the repeated wide gap between zero and non-zero loss loads +indicates the noise has a large impact on the observed performance, +which is not evident from a single goal load search procedure result. + +It is easy to modify the vanilla bisection to find a lower bound +for the intended load that satisfies a non-zero Goal Loss Ratio. +But it is not that obvious how to search for multiple goals at once, +hence the support for multiple Search Goals remains a problem. + +## Inconsistent Trial Results + +While performing throughput search by executing a sequence of +measurement trials, there is a risk of encountering inconsistencies +between trial results. + +The plain bisection never encounters inconsistent trials. +But [RFC2544] hints about the possibility of inconsistent trial results, +in two places in its text. +The first place is section 24, where full trial durations are required, +presumably because they can be inconsistent with the results +from short trial durations. +The second place is section 26.3, where two successive zero-loss trials +are recommended, presumably because after one zero-loss trial +there can be a subsequent inconsistent non-zero-loss trial. + +Examples include: + +- A trial at the same load (same or different trial duration) results + in a different Trial Loss Ratio. +- A trial at a higher load (same or different trial duration) results + in a smaller Trial Loss Ratio. + +Any robust throughput search algorithm needs to decide how to continue +the search in the presence of such inconsistencies. +Definitions of throughput in [RFC1242] and [RFC2544] are not specific enough +to imply a unique way of handling such inconsistencies. + +Ideally, there will be a definition of a new quantity which both generalizes +throughput for non-zero-loss (and other possible repeatability enhancements), +while being precise enough to force a specific way to resolve trial result +inconsistencies. +But until such a definition is agreed upon, the correct way to handle +inconsistent trial results remains an open problem. + +# MLRsearch Specification + +This section describes MLRsearch specification including all technical +definitions needed for evaluating whether a particular test procedure +complies with MLRsearch specification. + +{::comment} + [Good idea for 08, maybe ask BMWG first?] + + <mark>TODO VP: Separate Requirements and Recommendations/Suggestions + paragraphs? (currently requirements are in discussion subsections - + discussion should only clarify things without adding new + requirements)</mark> + +{:/comment} + +## Overview + +MLRsearch specification describes a set of abstract system components, +acting as functions with specified inputs and outputs. + +A test procedure is said to comply with MLRsearch specification +if it can be conceptually divided into analogous components, +each satisfying requirements for the corresponding MLRsearch component. + +The Measurer component is tasked to perform trials, +the Controller component is tasked to select trial loads and durations, +the Manager component is tasked to pre-configure everything +and to produce the test report. +The test report explicitly states Search Goals (as the Controller Inputs) +and corresponding Goal Results (Controller Outputs). + +{::comment} + [Low priority] + + <mark>MKP2 TODO: Find a good reference for the test report, seems only implicit in RFC2544.</mark> + +{:/comment} + +The Manager calls the Controller once, +the Controller keeps calling the Measurer +until all stopping conditions are met. + +The part where Controller calls the Measurer is called the search. +Any activity done by the Manager before it calls the Controller +(or after Controller returns) is not considered to be part of the search. + +MLRsearch specification prescribes regular search results and recommends +their stopping conditions. Irregular search results are also allowed, +they may have different requirements and stopping conditions. + +Search results are based on load classification. +When measured enough, any chosen load either achieves of fails each search goal, +thus becoming a lower or an upper bound for that goal. +When the relevant bounds are at loads that are close enough +(according to goal precision), the regular result is found. +Search stops when all regular results are found +(or if some goals are proven to have only irregular results). + +## Measurement Quantities + +MLRsearch specification uses a number of measurement quantities. + +In general, MLRsearch specification does not require particular units to be used, +but it is REQUIRED for the test report to state all the units. +For example, ratio quantities can be dimensionless numbers between zero and one, +but may be expressed as percentages instead. + +For convenience, a group of quantities can be treated as a composite quantity, +One constituent of a composite quantity is called an attribute, +and a group of attribute values is called an instance of that composite quantity. + +Some attributes are not independent from others, +and they can be calculated from other attributes. +Such quantites are called derived quantities. + +## Existing Terms + +RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" +contains basic definitions, and +RFC 2544 "Benchmarking Methodology for Network Interconnect Devices" +contains discussions of a number of terms and additional methodology requirements. +RFC 2285 adds more terms and discussions, describing some known situations +in more precise way. + +All three documents should be consulted +before attempting to make use of this document. + +Definitions of some central terms are copied and discussed in subsections. + +{::comment} + [Good idea for 08, but needs more work. Ask BMWG?] + + Alternatively, quick list of all (existing and new here) terms, + with links (external or internal respectively) to definitions. + + <mark>MKP3 [VP] TODO: Even if the following list will not be in final draft, + it is useful to keep it around (maybe commented-out) while editing.</mark> + + <mark>MKP3 VP note: rough list of all RFC references: + - [RFC1242] (section 3.17 Throughput) ... definition + - [RFC2544] (section 26.1 Throughput) ... methodology + - [RFC2544] (section 24. Trial duration): + - full trial durations (implies short trials) + - Also 60s for unconditional compliance is here. + - Also "the search" (without quotes) appears there. + - Also "binary search" (with quotes) appears there. + - [RFC2544] (section 26.3 Frame loss rate): + - two successive zero-loss trials are recommended (hints about loss inversion) + - un/conditionally compliant with [RFC2544] + - [RFC2544] (section 26. Benchmarking tests:) + - all its "dot sections" have "Reporting format:" paragraphs + - (implies test report) + - [RFC2544] (section 26.1 Throughput) wants graph, frame size on X axis. + - [RFC2544] (section 23. Trial description) trial + - general description of trial + - wait times specifically, maybe also learning frames? + - Constant Load of [RFC1242] (section 3.4 Constant Load) + - Data Rate of [RFC2544] (section 14. Bidirectional traffic) + - seems equal to input frame rate [RFC2544] (23. Trial description). + - [RFC2544] (section 21. Bursty traffic) suggests non-constant loads? + - Intended Load of [RFC2285] (section 3.5.1 Intended load (Iload)) + - [RFC2285] (Section 3.5.2 Offered load (Oload)) + - Frame Loss Rate of [RFC1242] (section 3.6 Frame Loss Rate) + - Forwarding Rate as defined in [RFC2285] (section 3.6.1 Forwarding rate (FR)) + - [RFC2544] (section 20. Maximum frame rate) + - [RFC2285] (3.5.3 Maximum offered load (MOL)) + - reordered frames [RFC2544] (section 10. Verifying received frames) + - For example, [RFC2544] (Appendix C) lists frame formats and protocol addresses, + as recommended from [RFC2544] (section 8. Frame formats) + and [RFC2544] (section 12. Protocol addresses). + - [RFC8219] (section 5.3. Traffic Setup) introduces traffic setups consisting of a mix of IPv4 and IPv6 traffic + - [RFC2544] (section 9. Frame sizes) + - [RFC1242] (section 3.5 Data link frame size) + - [RFC2285] (section 3.6.2) FRMOL + - [RFC2285] (section 3.1.1) DUT + - [RFC2285] (section 3.1.2) SUT + - [RFC2544] (section 6. Test set up) test setup with (an external) tester + - [RFC9004] B2B + - [RFC8219] (section 5.3. Traffic Setup) for an example of ip4+ip6 mixed traffic + </mark> + + <mark>MKP3 [VP] TODO: Do not mention those that do not need discussion here.</mark> + +{:/comment} + + +{::comment} + [Low priority] + + <mark>MKP3 [VP] TODO: Do we even need RFC9004?</mark> + +{:/comment} + +{::comment} + [I do not understand what I meant. Typos? Probably not important overall.] + + <mark>MKP2 [VP] TODO: Even terms that are discussed in this memo, + they perhaps do not need a separate list (just free paragraphs), + in a chapter after MLRsearch specification.</mark> + +{:/comment} + +{::comment} + [Important, just not enough time in 07.] + + <mark>MKP3 [VP] TODO: Verify that MLRsearch specification does not discuss + meaning of existing terms without quoting their original definition.</mark> + +{:/comment} + +### SUT + +Defined in [RFC2285] (section 3.1.2 System Under Test (SUT)) as follows. + +Definition: + +The collective set of network devices to which stimulus is offered +as a single entity and response measured. + +Discussion: + +An SUT consisting of a single network device is also allowed. + +### DUT + +Defined in [RFC2285] (section 3.1.1 Device Under Test (DUT)) as follows. + +Definition: + +The network forwarding device to which stimulus is offered and +response measured. + +Discussion: + +DUT, as a sub-component of SUT, is only indirectly mentioned +in MLRsearch specification, but is of key relevance for its motivation. + +{::comment} + [Could be useful, but not high priority.] + + ### Tester + + <mark>MKP3 TODO: Add Definition and Discusion paragraphs</mark> + + <mark>MKP3 MK note: Bizarre ... i can't find tester definition in + rfc1242, rfc2288 or rfc2544, but will keep looking. If there isn't one, + we need to define one :).</mark> + + <mark>[VP] TODO: There were some documents distinguishing TG and TA.</mark> + +{:/comment} + +### Trial + +A trial is the part of the test described in [RFC2544] (section 23. Trial description). + +Definition: + + A particular test consists of multiple trials. Each trial returns + one piece of information, for example the loss rate at a particular + input frame rate. Each trial consists of a number of phases: + + a) If the DUT is a router, send the routing update to the "input" + port and pause two seconds to be sure that the routing has settled. + + b) Send the "learning frames" to the "output" port and wait 2 + seconds to be sure that the learning has settled. Bridge learning + frames are frames with source addresses that are the same as the + destination addresses used by the test frames. Learning frames for + other protocols are used to prime the address resolution tables in + the DUT. The formats of the learning frame that should be used are + shown in the Test Frame Formats document. + + c) Run the test trial. + + d) Wait for two seconds for any residual frames to be received. + + e) Wait for at least five seconds for the DUT to restabilize. + +Discussion: + +The definition describes some traits, it is not clear whether all of them +are REQUIRED, or some of them are only RECOMMENDED. + +{::comment} + [Useful if possible.] + + <mark>MKP2 [VP] TODO: Search RFCs for better description of "Run the test trial".</mark> + +{:/comment} + +For the purposes of the MLRsearch specification, +it is ALLOWED for the test procedure to deviate from the [RFC2544] description, +but any such deviation MUST be made explicit in the test report. + +Trials are the only stimuli the SUT is expected to experience +during the search. + +In some discussion paragraphs, it is useful to consider the traffic +as sent and received by a tester, as implicitly defined +in [RFC2544] (section 6. Test set up). + +An example of deviation from [RFC2544] is using shorter wait times. + +## Trial Terms + +This section defines new and redefine existing terms for quantities +relevant as inputs or outputs of trial, as used by the Measurer component. + +### Trial Duration + +Definition: + +Trial duration is the intended duration of the traffic for a trial. + +Discussion: + +In general, this quantity does not include any preparation nor waiting +described in section 23 of [RFC2544] (section 23. Trial description). + +While any positive real value may be provided, some Measurer implementations +MAY limit possible values, e.g. by rounding down to neared integer in seconds. +In that case, it is RECOMMENDED to give such inputs to the Controller +so the Controller only proposes the accepted values. +Alternatively, the test report MUST present the rounded values +as Search Goal attributes. + +### Trial Load + +Definition: + +The trial load is the intended load for a trial + +Discussion: + +For test report purposes, it is assumed that this is a constant load by default. +This MAY be only an average load, e.g. when the traffic is intended to be busty, +e.g. as suggested in [RFC2544] (section 21. Bursty traffic), +but the test report MUST explicitly mention how non-constant the traffic is. + +Trial load is the quantity defined as Constant Load of [RFC1242] +(section 3.4 Constant Load), Data Rate of [RFC2544] +(section 14. Bidirectional traffic) +and Intended Load of [RFC2285] (section 3.5.1 Intended load (Iload)). +All three definitions specify +that this value applies to one (input or output) interface. + +{::comment} + [Not important.] + + <mark>MKP2 [VP] TODO: Also mention input frame rate [RFC2544] (23. Trial description).</mark> + +{:/comment} + +For test report purposes, multi-interface aggregate load MAY be reported, +this is understood as the same quantity expressed using different units. +From the report it MUST be clear whether a particular trial load value +is per one interface, or an aggregate over all interfaces. + +Similarly to trial duration, some Measurers may limit the possible values +of trial load. Contrary to trial duration, the test report is NOT REQUIRED +to document such behavior. + +{::comment} + [Can of worms. Be aware, but probably do not let spill into draft.] + + <mark>MKP2 [VP] TODO: Why? In practice the difference is small, but what if it is big? + Do we need Trial Effective Load for bounds an conditional throughput purposes? + Should the Controller be recommended to chose load values that are exactly accepted? + </mark> + +{:/comment} + +It is ALLOWED to combine trial load and trial duration in a way +that would not be possible to achieve using any integer number of data frames. + +{::comment} + [I feel this is important, to be discussed separately (not in-scope).] + + <mark>MKP2 [VP] TODO: Explain why are we not using Oload. + 1. MLRsearch implementations cannot react correctly to big differences + between Iload and Oload. + 2. The media between the tested and the DUT are thus considered to be part of SUT. + If DUT causes congestion control, it is not expected to handle Iload. + </mark> + + See further discussion in [Trial Forwarding Ratio] (#Trial-Forwarding-Ratio) + and in [Measurer] (#Measurer) sections for other related issues. + + <mark>MKP2 [VP] TODO: Create a separate subsection for Oload discussion, + or clearly separate which aspects are discussed under which term.</mark> + + <mark>MKP2 [VP] TODO: New idea. Compare the tester to an ordinary router + in some datacenter. The Intended Load is not jst some abstract input. + It is the real traffic coming from routers next hop farther. + It does not matter that DUT has forwarded each frame it received, + if the tester was unable to sent all the traffic in time. + Endpoint see packet loss, they do not care about [RFC2285] + half-duplex, spanning trees, nor congestion control mechanisms. + Formally speaking, I consider even the sending interface of the sender + to be the part of SUT. + Reading [RFC2285] (section 3.5.3 Maximum offered load (MOL)) + "This will be the case when an external source lacks the resources + to transmit frames at the minimum legal inter-frame gap" + that means TRex workers are also part of SUT. If they do not have + enough CPU power to generate frames are required, those frames are lost. + </mark> + + <mark>MKP2 [VP] TODO: That new idea warants some discussion in "DUT within SUT", + as it is just another case of ther rest of SUT ruining + otherwise good DUT performance.</mark> + +{:/comment} + +### Trial Input + +Definition: + +Trial Input is a composite quantity, consisting of two attributes: +trial duration and trial load. + +Discussion: + +When talking about multiple trials, it is common to say "Trial Inputs" +to denote all corresponding Trial Input instances. + +A Trial Input instance acts as the input for one call of the Measurer component. + +Contrary to other composite quantities, MLRsearch implementations +are NOT ALLOWED to add optional attributes here. +This improves interoperability between various implementations of +the Controller and the Measurer. + +### Traffic Profile + +Definition: + +Traffic profile is a composite quantity +containing attributes other than trial load and trial duration, +needed for unique determination of the trial to be performed. + +Discussion: + +All its attributes are assumed to be constant during the search, +and the composite is configured on the Measurer by the Manager +before the search starts. +This is why the traffic profile is not part of the Trial Input. + +As a consequence, implementations of the Manager and the Measurer +must be aware of their common set of capabilities, so that the traffic profile +uniquely defines the traffic during the search. +The important fact is that none of those capabilities +have to be known by the Controller implementations. + +The traffic profile SHOULD contain some specific quantities, +for example [RFC2544] (section 9. Frame sizes) governs +data link frame size as defined in [RFC1242] (section 3.5 Data link frame size). + +Several more specific quantities may be RECOMMENDED, depending on media type. +For example, [RFC2544] (Appendix C) lists frame formats and protocol addresses, +as recommended from [RFC2544] (section 8. Frame formats) +and [RFC2544] (section 12. Protocol addresses). + +Depending on SUT configuration, e.g. when testing specific protocols, +additional attributes MUST be included in the traffic profile +and in the test report. + +Example: [RFC8219] (section 5.3. Traffic Setup) introduces traffic setups +consisting of a mix of IPv4 and IPv6 traffic - the implied traffic profile +therefore must include an attribute for their percentage. + +Other traffic properties that need to be somehow specified +in Traffic Profile include: +[RFC2544] (section 14. Bidirectional traffic), +[RFC2285] (section 3.3.3 Fully meshed traffic), +and [RFC2544] (section 11. Modifiers). + +### Trial Forwarding Ratio + +Definition: + +The trial forwarding ratio is a dimensionless floating point value. +It MUST range between 0.0 and 1.0, both inclusive. +It is calculated by dividing the number of frames +successfully forwarded by the SUT +by the total number of frames expected to be forwarded during the trial + +Discussion: + +For most traffic profiles, "expected to be forwarded" means +"intended to get transmitted from Tester towards SUT". + +Trial forwarding ratio MAY be expressed in other units +(e.g. as a percentage) in the test report. + +Note that, contrary to loads, frame counts used to compute +trial forwarding ratio are aggregates over all SUT output interfaces. + +Questions around what is the correct number of frames +that should have been forwarded +is generally outside of the scope of this document. + +{::comment} + [Part two of iload/oload discussion.] + + See discussion in [Measurer] (#Measurer) section + for more details about calibrating test equipment. + + <mark>MKP2 [VP] TODO: Define unsent frames?</mark> + + <mark>MKP2 [VP] TODO: If Oload is fairly below Iload, the unsent frames + should be counted as lost, otherwise search outputs are misleading. + But what is "fairly"? CSIT tolerates 10 microseconds worth of unsent frames.</mark> + +{:/comment} + +{::comment} + [Low priority, but maybe useful for somebody?] + + <mark>MKP2 [VP] TODO: Mention traffic profiles with uneven frame counts? + E.g. when SUT is expected to perform IP packet fragmentation or reassembly. + </mark> + +{:/comment} + +### Trial Loss Ratio + +Definition: + +The Trial Loss Ratio is equal to one minus the trial forwarding ratio. + +Discussion: + +100% minus the trial forwarding ratio, when expressed as a percentage. + +This is almost identical to Frame Loss Rate of [RFC1242] +(section 3.6 Frame Loss Rate), +the only minor difference is that Trial Loss Ratio +does not need to be expressed as a percentage. + +### Trial Forwarding Rate + +Definition: + +The trial forwarding rate is a derived quantity, calculated by +multiplying the trial load by the trial forwarding ratio. + +Discussion: + +It is important to note that while similar, this quantity is not identical +to the Forwarding Rate as defined in [RFC2285] +(section 3.6.1 Forwarding rate (FR)). +The latter is specific to one output interface only, +whereas the trial forwarding ratio is based +on frame counts aggregated over all SUT output interfaces. + +{::comment} + [Part 3 of iload/oload discussion.] + + <mark>MKP2 [VP] TODO: If some unsent frames were tolerated (not counted as lost), + this value is actually higher than the real fps output of the SUT. + Should we use the real FR as the basis for Conditional Throughput + (instead of this TFR)? That would require additional Trial Output attribute. + </mark> + + <mark>MKP2 [VP] TODO: What about duration stretching? + This also causes difference between Iload and Oload, + but in an invisible way.</mark> + + <mark>MKP2 [VP] TODO: Recommend start+sleep+stop? + How long wait for late frames? RFC2544 2s is too much even at 30s trial.</mark> + +{:/comment} + +### Trial Effective Duration + +Definition: + +Trial effective duration is a time quantity related to the trial, +by default equal to the trial duration. + +Discussion: + +This is an optional feature. +If the Measurer does not return any trial effective duration value, +the Controller MUST use the trial duration value instead. + +Trial effective duration may be any time quantity chosen by the Measurer +to be used for time-based decisions in the Controller. + +The test report MUST explain how the Measurer computes the returned +trial effective duration values, if they are not always +equal to the trial duration. + +This feature can be beneficial for users +who wish to manage the overall search duration, +rather than solely the traffic portion of it. +Simply measure the duration of the whole trial (waits including) +and use that as the trial effective duration. + +Also, this is a way for the Measurer to inform the Controller about +its surprising behavior, for example when rounding the trial duration value. + +{::comment} + [Not very important, but easy and nice recommendation.] + + <mark>MKP2 [VP] TODO: Recommend for Measurer to return all trials at relevant bounds, + as that may better inform users when surprisingly small amount of trials + was performed, just because the the trial effective duration values were big.</mark> + + <mark>MKP2 [VP] TODO: Repeat that this is not here to deal with duration stretching.</mark> + +{:/comment} + +### Trial Output + +Definition: + +Trial Output is a composite quantity. The REQUIRED attributes are +Trial Loss Ratio, trial effective duration and trial forwarding rate. + +Discussion: + +When talking about multiple trials, it is common to say "Trial Outputs" +to denote all corresponding Trial Output instances. + +Implementations may provide additional (optional) attributes. +The Controller implementations MUST ignore values of any optional attribute +they are not familiar with, +except when passing Trial Output instance to the Manager. + +Example of an optional attribute: +The aggregate number of frames expected to be forwarded during the trial, +especially if it is not just (a rounded-up value) +implied by trial load and trial duration. + +While [RFC2285] (Section 3.5.2 Offered load (Oload)) +requires the offered load value to be reported for forwarding rate measurements, +it is NOT REQUIRED in MLRsearch specification. + +{::comment} + [Side tangent from iload/oload discussion. Stilll recommendation is not obvious.] + + <mark>MKP2 TODO: Why? Just because bound trial results are optional in Controller Output?</mark> + + <mark>MKP2 mk edit note: we need to more explicitly address + the relevance or irrelevance of [RFC2285] (Section 3.5.2 Offered load (Oload)). + Current text in [Trial Load] (#Trial-Load) is ambiguous - quoted below.</mark> + + <mark>MKP2 "Questions around what is the correct number of frames that should + have been forwarded is generally outside of the scope of this document. + See discussion in [Measurer] (#Measurer) section for more details about + calibrating test equipment."</mark> + +{:/comment} + +### Trial Result + +Definition: + +Trial result is a composite quantity, +consisting of the Trial Input and the Trial Output. + +Discussion: + +When talking about multiple trials, it is common to say "trial results" +to denote all corresponding trial result instances. + +While implementations SHOULD NOT include additional attributes +with independent values, they MAY include derived quantities. + +## Goal Terms + +This section defines new and redefine existing terms for quantities +indirectly relevant for inputs or outputs of the Controller component. + +Several goal attributes are defined before introducing +the main component quantity: the Search Goal. + +### Goal Final Trial Duration + +Definition: + +A threshold value for trial durations. + +Discussion: + +This attribute value MUST be positive. + +A trial with Trial Duration at least as long as the Goal Final Trial Duration +is called a full-length trial (with respect to the given Search Goal). + +A trial that is not full-length is called a short trial. + +Informally, while MLRsearch is allowed to perform short trials, +the results from such short trials have only limited impact on search results. + +One trial may be full-length for some Search Goals, but not for others. + +The full relation of this goal to Controller Output is defined later in +this document in subsections of [Goal Result] (#Goal-Result). +For example, the Conditional Throughput for this goal is computed only from +full-length trial results. + +### Goal Duration Sum + +Definition: + +A threshold value for a particular sum of trial effective durations. + +Discussion: + +This attribute value MUST be positive. + +Informally, even when looking only at full-length trials, +MLRsearch may spend up to this time measuring the same load value. + +If the Goal Duration Sum is larger than the Goal Final Trial Duration, +multiple full-length trials may need to be performed at the same load. + +See [TST009 Example] (#TST009-Example) for an example where possibility +of multiple full-length trials at the same load is intended. + +A Goal Duration Sum value lower than the Goal Final Trial Duration +(of the same goal) could save some search time, but is NOT RECOMMENDED. +See [Relevant Upper Bound] (#Relevant-Upper-Bound) for partial explanation. + +### Goal Loss Ratio + +Definition: + +A threshold value for Trial Loss Ratios. + +Discussion: + +Attribute value MUST be non-negative and smaller than one. + +A trial with Trial Loss Ratio larger than a Goal Loss Ratio value +is called a lossy trial, with respect to given Search Goal. + +Informally, if a load causes too many lossy trials, +the Relevant Lower Bound for this goal will be smaller than that load. + +If a trial is not lossy, it is called a low-loss trial, +or (specifically for zero Goal Loss Ratio value) zero-loss trial. + +### Goal Exceed Ratio + +Definition: + +A threshold value for a particular ratio of sums of Trial Effective Durations. + +Discussion: + +Attribute value MUST be non-negative and smaller than one. + +See later sections for details on which sums. +Specifically, the direct usage is only in +[Appendix A: Load Classification] (#Appendix-A\:-Load-Classification) +and [Appendix B: Conditional Throughput] (#Appendix-B\:-Conditional-Throughput). +The impact of that usage is discussed in subsections leading to +[Goal Result] (#Goal-Result). + +Informally, the impact of lossy trials is controlled by this value. +Effectively, Goal Exceed Ratio is a percentage of full-length trials +that may be lossy without the load being classified +as the [Relevant Upper Bound] (#Relevant-Upper-Bound). + +### Goal Width + +Definition: + +A value used as a threshold for deciding +whether two trial load values are close enough. + +Discussion: + +If present, the value MUST be positive. + +Informally, this acts as a stopping condition, +controlling the precision of the search. +The search stops if every goal has reached its precision. + +Implementations without this attribute +MUST give the Controller other ways to control the search stopping conditions. + +Absolute load difference and relative load difference are two popular choices, +but implementations may choose a different way to specify width. + +The test report MUST make it clear what specific quantity is used as Goal Width. + +It is RECOMMENDED to set the Goal Width (as relative difference) value +to a value no smaller than the Goal Loss Ratio. +(The reason is not obvious, see [Throughput] (#Throughput) if interested.) + +### Search Goal + +Definition: + +The Search Goal is a composite quantity consisting of several attributes, +some of them are required. + +Required attributes: +- Goal Final Trial Duration +- Goal Duration Sum +- Goal Loss Ratio +- Goal Exceed Ratio + +Optional attribute: +- Goal Width + +Discussion: + +Implementations MAY add their own attributes. +Those additional attributes may be required by the implementation +even if they are not required by MLRsearch specification. +But it is RECOMMENDED for those implementations +to support missing values by computing reasonable defaults. + +The meaning of listed attributes is formally given only by their indirect effect +on the search results. + +Informally, later sections provide additional intuitions and examples +of the Search Goal attribute values. + +An example of additional attributes required by some implementations +is Goal Initial Trial Duration, together with another attribute +that controls possible intermediate Trial Duration values. +The reasonable default in this case is using the Goal Final Trial Duration +and no intermediate values. + +### Controller Input + +Definition: + +Controller Input is a composite quantity +required as an input for the Controller. +The only REQUIRED attribute is a list of Search Goal instances. + +Discussion: + +MLRsearch implementations MAY use additional attributes. +Those additional attributes may be required by the implementation +even if they are not required by MLRsearch specification. + +Formally, the Manager does not apply any Controller configuration +apart from one Controller Input instance. + +For example, Traffic Profile is configured on the Measurer by the Manager +(without explicit assistance of the Controller). + +The order of Search Goal instances in a list SHOULD NOT +have a big impact on Controller Output (see section [Controller Output] (#Controller-Output) , +but MLRsearch implementations MAY base their behavior on the order +of Search Goal instances in a list. + +An example of an optional attribute (outside the list of Search Goals) +required by some implementations is Max Load. +While this is a frequently used configuration parameter, +already governed by [RFC2544] (section 20. Maximum frame rate) +and [RFC2285] (3.5.3 Maximum offered load (MOL)), +some implementations may detect or discover it instead. + +{::comment} + [Not important directly, may matter for iload/oload.] + + <mark>MKP2 [VP] TODO: 2544 and 2285 care about half-duplex media. Should we?</mark> + +{:/comment} + +{::comment} + [Maybe obvious but I think useful. RFC2544 talks about header compression in WANs.] + + <mark>MKP2 [VP] TODO: Mention that Max Load should care about all media within SUT, + including DUT-DUT links. Important when that link carries encapsulated traffic, + as bandwidth limit there implies lower max rate + (than implied by tester-SUT links).</mark> + +{:/comment} + +In MLRsearch specification, the [Relevant Upper Bound] (#Relevant-Upper-Bound) +is added as a required attribute precisely because it makes the search result +independent of Max Load value. + +{::comment} + [User recommendation, we should have separate section summarizing those.] + + <mark>[VP] TODO for MK: The rest of this subsection is new, review?</mark> + + It is RECOMMENDED to use the same Goal Final Trial Duration value across all goals. + Otherwise, some goals may be measured at Trial Durations longer than needed, + with possibly unexpected impacts on repeatability and comparability. + + For example when Goal Loss Ratio is zero, any increase in Trial Duration + also increases the likelihood of the trial to become lossy, + similar to usage of lower Goal Exceed Ratio or larger Goal Duration Sum, + both of which tend to lower the search results, towards noisy end + of performance spectrum. + + Also, it is recommended to avoid "incomparable" goals, e.g. one with + lower loss ratio but higher exceed ratio, and other with higher loss ratio + but lower loss ratio. In worst case, this can make the search to last too long. + Implementations are RECOMMENDED to sort the goals and start with + stricter ones first, as bounds for those will not get invalidated + byt measureing for less trict goal later in the search. + +{:/comment} + +## Search Goal Examples + +### RFC2544 Goal + +The following set of values makes the search result unconditionally compliant +with [RFC2544] (section 24 Trial duration) + +- Goal Final Trial Duration = 60 seconds +- Goal Duration Sum = 60 seconds +- Goal Loss Ratio = 0% +- Goal Exceed Ratio = 0% + +The latter two attributes are enough to make the search goal +conditionally compliant, adding the first attribute +makes it unconditionally compliant. + +The second attribute (Goal Duration Sum) only prevents MLRsearch +from repeating zero-loss full-length trials. + +Non-zero exceed ratio could prolong the search and allow loss inversion +between lower-load lossy short trial and higher-load full-length zero-loss trial. +From [RFC2544] alone, it is not clear whether that higher load +could be considered as compliant throughput. + +### TST009 Goal + +One of the alternatives to RFC2544 is described in +[TST009] (section 12.3.3 Binary search with loss verification). +The idea there is to repeat lossy trials, hoping for zero loss on second try, +so the results are closer to the noiseless end of performance sprectum, +and more repeatable and comparable. + +Only the variant with "z = infinity" is achievable with MLRsearch. + +{::comment} + [Low priority, unless a short sentence is found.] + + <mark>MKP2 MK note: Shouldn't we add a note about how MLRsearch goes about + addressing the TST009 point related to z, that is "z is threshold of + Lord(r) to override Loss Verification when the count of lost frames is + very high and unnecessary verification trials."? i.e. by have Goal Loss + Ratio. Thoughts?</mark> + +{:/comment} + +For example, for "r = 2" variant, the following search goal should be used: + +- Goal Final Trial Duration = 60 seconds +- Goal Duration Sum = 120 seconds +- Goal Loss Ratio = 0% +- Goal Exceed Ratio = 50% + +If the first 60s trial has zero loss, it is enough for MLRsearch to stop +measuring at that load, as even a second lossy trial +would still fit within the exceed ratio. + +But if the first trial is lossy, MLRsearch needs to perform also +the second trial to classify that load. +As Goal Duration Sum is twice as long as Goal Final Trial Duration, +third full-length trial is never needed. + +## Result Terms + +Before defining the output of the Controller, +it is useful to define what the Goal Result is. + +The Goal Result is a composite quantity. + +Following subsections define its attribute first, before describing the Goal Result quantity. + +There is a correspondence between Search Goals and Goal Results. +Most of the following subsections refer to a given Search Goal, +when defining attributes of the Goal Result. +Conversely, at the end of the search, each Search Goal +has its corresponding Goal Result. + +Conceptually, the search can be seen as a process of load classification, +where the Controller attempts to classify some loads as an Upper Bound +or a Lower Bound with respect to some Search Goal. + +Before defining real attributes of the goal result, +it is useful to define bounds in general. + +### Relevant Upper Bound + +Definition: + +The Relevant Upper Bound is the smallest trial load value that is classified +at the end of the search as an upper bound +(see [Appendix A: Load Classification] (#Appendix-A\:-Load-Classification)) +for the given Search Goal. + +Discussion: + +One search goal can have many different load classified as an upper bound. +At the end of the search, one of those loads will be the smallest, +becoming the relevant upper bound for that goal. + +In more detail, the set of all trial outputs (both short and full-length, +enough of them according to Goal Duration Sum) +performed at that smallest load failed to uphold all the requirements +of the given Search Goal, mainly the Goal Loss Ratio +in combination with the Goal Exceed Ratio. + +{::comment} + [Recheck. Move to end?] + + <mark>[VP] TODO: Is the above a good summary of Appendix A inputs?</mark> + +{:/comment} + +If Max Load does not cause enough lossy trials, +the Relevant Upper Bound does not exist. +Conversely, if Relevant Upper Bound exists, +it is not affected by Max Load value. + +{::comment} + [Medium priority, depends on how many user recommendations we have.] + + With non-zero exceed ratio values, a lossy short trial may not be enough + to classify a load as the relevant upper bound. + Users MAY apply Goal Duration Sum value lower than Goal Final Trial Duration + to force such classification in hope to save time, + but it is RECOMMENDED not to do so, as in practice + it hurts comparability and repeatability. + +{:/comment} + +{::comment} + [Probably too technical, unless relation to repeatability is found.] + + In general, a load starts as as undecided, then maybe flips to become + an upper bound. MLRsearch stops measuring at that load for this goal, + but it may be forced to measure more for some other search goals, + in which case the load may flip to a lower bound (and back and forth). + + <mark>[VP] TODO: Confirm the load can never flip back to being undecided.</mark> + + Even though the load classification may change during the search, + the goal results are established at the end of the search. + + If the exceed ratio is zero, an upper bound can never flip; + one lossy trial (even short) is enough to pin the classification. + +{:/comment} + +### Relevant Lower Bound + +Definition: + +The Relevant Lower Bound is the largest trial load value +among those smaller than the Relevant Upper Bound, +that got classified at the end of the search as a lower bound (see +[Appendix A: Load Classification] (#Appendix-A\:-Load-Classification)) +for the given Search Goal. + +Discussion: + +Only among loads smaller that the relevant upper bound, +the largest load becomes the relevant lower bound. +With loss inversion, stricter upper bound matters. + +In more detail, the set of all trial outputs (both short and full-length, +enough of them according to Goal Duration Sum) +performed at that largest load managed to uphold all the requirements +of the given Search Goal, mainly the Goal Loss Ratio +in combination with the Goal Exceed Ratio. + +Is no load had enough low-loss trials, the relevant lower bound +MAY not exist. + +{::comment} + [Min Load us useful for detecting broken SUTs (and latency).] + + <mark>[VP] TODO: Mention min load here?</mark> + + <mark>[VP] TODO: Allow zero as implicit lower bound that needs no trials? + If yes, then probably way earlier than here.</mark> + +{:/comment} + +Strictly speaking, if the Relevant Upper Bound does not exist, +the Relevant Lower Bound also does not exist. +In that case, Max Load is classified as a lower bound, +but it is not clear whether a higher lower bound +would be found if the search used a higher Max Load value. + +For a regular Goal Result, the distance between the Relevant Lower Bound +and the Relevant Upper Bound MUST NOT be larger than the Goal Width, +if the implementation offers width as a goal attribute. + +{::comment} + [True but no time to fix properly.] + + <mark>mk note: Seemingly broken grammar, + "managed to uphold all requirements", should be followed + by stating what it means.</mark> + +{:/comment} + +Searching for anther search goal may cause a loss inversion phenomenon, +where a lower load is classified as an upper bound, +but also a higher load is classified as a lower bound for the same search goal. +The definition of the Relevant Lower Bound ignores such high lower bounds. + +{::comment} + [Compare to similar block in upper bound.] + + In general, a load starts as as undecided, then maybe flips to become + a lower bound. MLRsearch stops measuring at that load for this goal, + but it may be forced to measure more for some other search goals, + in which case the load may flip to an upper bound (and back and forth). + + <mark>[VP] TODO: Confirm the load can never flip back to being undecided.</mark> + + Even though the load classification may change during the search, + the goal results are established at the end of the search. + + No valid exceed ratio value pins the classification as a lower bound. + +{:/comment} + +### Conditional Throughput + +Definition: + +The Conditional Throughput (see section [Appendix B: Conditional Throughput] (#Appendix-B\:-Conditional-Throughput)) +as evaluated at the Relevant Lower Bound of the given Search Goal +at the end of the search. + +Discussion: + +Informally, this is a typical trial forwarding rate, expected to be seen +at the Relevant Lower Bound of the given Search Goal. + +But frequently it is only a conservative estimate thereof, +as MLRsearch implementations tend to stop gathering more data +as soon as they confirm the value cannot get worse than this estimate +within the Goal Duration Sum. + +This value is RECOMMENDED to be used when evaluating repeatability +and comparability if different MLRsearch implementations. + +{::comment} + [Low priority but useful for comparabuility.] + + <mark>[VP] TODO: Add subsection for Trial Results At Relevant Bounds + as an optional attribute of Goal Result.</mark> + +{:/comment} + +### Goal Result + +Definition: + +The Goal Result is a composite quantity consisting of several attributes. +Relevant Upper Bound and Relevant Lower Bound are REQUIRED attributes, +Conditional Throughput is a RECOMMENDED attribute. + +Discussion: + +Depending on SUT behavior, it is possible that one or both relevant bounds +do not exist. The goal result instance where the required attribute values exist +is informally called a Regular Goal Result instance, +so we can say some goals reached Irregular Goal Results. + +{::comment} + [Probably delete after last edits re irregular results.] + + <mark>MKP2 [VP] TODO: Additional attributes should not be required by the Manager? + Explicitly mention that irregular goal result may support different attributes. + </mark> + + <mark>MKP2 Implementations are free to define their own specific subtypes + of irregular Goal Results, but the test report MUST mark them clearly + as irregular according to this section.</mark> + +{:/comment} + +A typical Irregular Goal Result is when all trials at the Max Load +have zero loss, as the Relevant Upper Bound does not exist in that case. + +It is RECOMMENDED that the test report will display such results appropriately, +although MLRsearch specification does not prescibe how. + +{::comment} + [Useful.] + + <mark>MKP2 [VP] TODO: Also allways-fail. Link to bounds to avoid duplication.</mark> + +{:/comment} + +Anything else regarging Irregular Goal Results, +including their role in stopping conditions of the search +is outside the scope of this document. + +### Search Result + +Definition: + +The Search Result is a single composite object +that maps each Search Goal instance to a corresponding Goal Result instance. + +Discussion: + +Alternatively, the Search Result can be implemented as an ordered list +of the Goal Result instances, matching the order of Search Goal instances. + +{::comment} + [Low priority, as there is no obvious harm.] + + <mark>MKP1 [VP] TODO: Disallow any additional attributes?</mark> + +{:/comment} + +The Search Result (as a mapping) +MUST map from all the Search Goal instances present in the Controller Input. + +{::comment} + [Not important.] + + <mark>[VP] Postponed: API independence, modularity.</mark> + +{:/comment} + +{::comment} + [Not needed?] + + <mark>MKP1 [VP] TODO: Short sentence on what to do on irregular goal result.</mark> + +{:/comment} + +### Controller Output + +Definition: + +The Controller Output is a composite quantity returned from the Controller +to the Manager at the end of the search. +The Search Result instance is its only REQUIRED attribute. + +Discussion: + +MLRsearch implementation MAY return additional data in the Controller Output. + +{::comment} + [Not needed?] + + <mark>MKP1 [VP] TODO: Short sentence on what to do on irregular goal result.</mark> + + <mark>MKP1 [VP] TODO: Irregular output, e.g. with "max search time exceeded" flag?</mark> + +{:/comment} + +## MLRsearch Architecture + +{::comment} + [Meta and irrelevant. Delete after verifying other text is good.] + + <mark>MKP2 [VP] TODO: Review the folowing: + This section is about division into components, + so it fits this definition: + "The software architecture of a system represents the design decisions + related to overall system structure and behavior." + Saying "MLRsearch Design" does not make it clear if it is + Vratko designing the MLRsearch specification, + or some other person designing a new MLRsearch implementation using that spec. + </mark> + +{:/comment} + +MLRsearch architecture consists of three main system components: +the Manager, the Controller, and the Measurer. + +The architecture also implies the presence of other components, +such as the SUT and the Tester (as a sub-component of the Measurer). + +Protocols of communication between components are generally left unspecified. +For example, when MLRsearch specification mentions "Controller calls Measurer", +it is possible that the Controller notifies the Manager +to call the Measurer indirectly instead. This way the Measurer implementations +can be fully independent from the Controller implementations, +e.g. programmed in different programming languages. + +### Measurer + +Definition: + +The Measurer is an abstract system component +that when called with a [Trial Input] (#Trial-Input) instance, +performs one [Trial] (#Trial), +and returns a [Trial Output] (#Trial-Output) instance. + +Discussion: + +This definition assumes the Measurer is already initialized. +In practice, there may be additional steps before the search, +e.g. when the Manager configures the traffic profile +(either on the Measurer or on its tester sub-component directly) +and performs a warmup (if the tester requires one). + +It is the responsibility of the Measurer implementation to uphold +any requirements and assumptions present in MLRsearch specification, +e.g. trial forwarding ratio not being larger than one. + +Implementers have some freedom. +For example [RFC2544] (section 10. Verifying received frames) +gives some suggestions (but not requirements) related to +duplicated or reordered frames. +Implementations are RECOMMENDED to document their behavior +related to such freedoms in as detailed a way as possible. + +It is RECOMMENDED to benchmark the test equipment first, +e.g. connect sender and receiver directly (without any SUT in the path), +find a load value that guarantees the offered load is not too far +from the intended load, and use that value as the Max Load value. +When testing the real SUT, it is RECOMMENDED to turn any big difference +between the intended load and the offered load into increased Trial Loss Ratio. + +Neither of the two recommendations are made into requirements, +because it is not easy to tell when the difference is big enough, +in a way thay would be dis-entangled from other Measurer freedoms. + +### Controller + +Definition: + +The Controller is an abstract system component +that when called with a Controller Input instance +repeatedly computes Trial Input instance for the Measurer, +obtains corresponding Trial Output instances, +and eventually returns a Controller Output instance. + +Discussion: + +Informally, the Controller has big freedom in selection of Trial Inputs, +and the implementations want to achieve the Search Goals +in the shortest expected time. + +The Controller's role in optimizing the overall search time +distinguishes MLRsearch algorithms from simpler search procedures. + +Informally, each implementation can have different stopping conditions. +Goal Width is only one example. +In practice, implementation details do not matter, +as long as Goal Results are regular. + +### Manager + +Definition: + +The Manager is an abstract system component that is reponsible for +configuring other components, calling the Controller component once, +and for creating the test report following the reporting format as +defined in [RFC2544] (section 26. Benchmarking tests). + +Discussion: + +The Manager initializes the SUT, the Measurer (and the Tester if independent) +with their intended configurations before calling the Controller. + +The Manager does not need to be able to tweak any Search Goal attributes, +but it MUST report all applied attribute values even if not tweaked. + +{::comment} + [Not very important but also should be easy to add.] + + <mark>MKP2 [VP] TODO: Is saying "RFC2544" indirectly reporting RFC2544 Goal values?</mark> + +{:/comment} + +In principle, there should be a "user" (human or CI) +that "starts" or "calls" the Manager and receives the report. +The Manager MAY be able to be called more than once whis way. + +{::comment} + [Not important, unless anybody else asks.] + + <mark>MKP2 The Manager may use the Measurer or other system components + to perform other tests, e.g. back-to-back frames, + as the Controller is only replacing the search from + [RFC2544] (section 26.1 Throughput).</mark> + +{:/comment} + +## Implementation Compliance + +Any networking measurement setup where there can be logically delineated system components +and there are components satisfying requirements for the Measurer, +the Controller and the Manager, is considered to be compliant with MLRsearch design. + +These components can be seen as abstractions present in any testing procedure. +For example, there can be a single component acting both +as the Manager and the Controller, but as long as values of required attributes +of Search Goals and Goal Results are visible in the test report, +the Controller Input instance and output instance are implied. + +For example, any setup for conditionally (or unconditionally) +compliant [RFC2544] throughput testing +can be understood as a MLRsearch architecture, +assuming there is enough data to reconstruct the Relevant Upper Bound. + +See [RFC2544 Goal] (#RFC2544-Goal) subsection for equivalent Search Goal. + +Any test procedure that can be understood as (one call to the Manager of) +MLRsearch architecture is said to be compliant with MLRsearch specification. + +# Additional Considerations + +This section focuses on additional considerations, intuitions and motivations +pertaining to MLRsearch methodology. + +{::comment} + [Meta, redundant.] + + <mark>MKP2 [VP] TODO: Review the following: + If MLRsearch specification is a product design specification + for MLRsearch implementation, then this chapter talks about + my goals and early attempts at designing the MLRsearch specification. + </mark> + +{:/comment} + +## MLRsearch Versions + +The MLRsearch algorithm has been developed in a code-first approach, +a Python library has been created, debugged, used in production +and published in PyPI before the first descriptions +(even informal) were published. + +But the code (and hence the description) was evolving over time. +Multiple versions of the library were used over past several years, +and later code was usually not compatible with earlier descriptions. + +The code in (some version of) MLRsearch library fully determines +the search process (for a given set of configuration parameters), +leaving no space for deviations. + +{::comment} + [Different type of external link, should be in 08.] + + <mark>MKP2 mk3 note: any references to library + should have specific reference link. + We have FDio-CSIT-MLRsearch in informative: at the start. Link it. + </mark> + +{:/comment} + +{::comment} + [Lesson learned is important, but maybe does not need version history?] + + <mark>MKP2 mk edit note: Suggest to remove crossed-out text, as it is + distracting, doesn't bring any value, and recalls multiple versions of + MLRsearch library, without any references. A much more appropriate + approach would be to provide a pointer to MLRsearch code versions in + FD.io that evolved over the years, as an example implementation. But I + would question the value of referring to old previous versions in this + document. It's okay for the blog, but not for IETF specification, + unless there are specific lessons learned that need to be highlighted + to support the specification.</mark> + +{:/comment} + +This historic meaning of MLRsearch, as a family +of search algorithm implementations, +leaves plenty of space for future improvements, at the cost +of poor comparability of results of search algoritm implementations. + +{::comment} + [Reckeck after clarifying library/algorithm/implementation/specification mess.] + + <mark>mk edit note: If the aim of this sentence is to state that there + could be possibly other approaches to address this problem space, then + I think we are already addressing it in the opening sections discussing + problems, and referring to ETSi TST.009 and opnfv work. If the aim is + to define "MLRsearch" as a completely new class of algorithms for + software network benchmarking, of which this spec is just one example, + then i have a problem with it. This specification is very prescriptive + in the main functional areas to address the problem identified, but + still leaving space for further exploration and innovation as noted + elsewhere in this document. It is not a new class of algorithms. It is + a newly defined methodology to amend RFC2544, to specifically address + identified problems.</mark> + +{:/comment} + +There are two competing needs. +There is the need for standardization in areas critical to comparability. +There is also the need to allow flexibility for implementations +to innovate and improve in other areas. +This document defines MLRsearch as a new specification +in a manner that aims to fairly balance both needs. + +## Stopping Conditions + +[RFC2544] prescribes that after performing one trial at a specific offered load, +the next offered load should be larger or smaller, based on frame loss. + +The usual implementation uses binary search. +Here a lossy trial becomes +a new upper bound, a lossless trial becomes a new lower bound. +The span of values between the tightest lower bound +and the tightest upper bound (including both values) forms an interval of possible results, +and after each trial the width of that interval halves. + +Usually the binary search implementation tracks only the two tightest bounds, +simply calling them bounds. +But the old values still remain valid bounds, +just not as tight as the new ones. + +After some number of trials, the tightest lower bound becomes the throughput. +[RFC2544] does not specify when, if ever, should the search stop. + +MLRsearch introduces a concept of [Goal Width] (#Goal-Width). + +The search stops +when the distance between the tightest upper bound and the tightest lower bound +is smaller than a user-configured value, called Goal Width from now on. +In other words, the interval width at the end of the search +has to be no larger than the Goal Width. + +This Goal Width value therefore determines the precision of the result. +Due to the fact that MLRsearch specification requires a particular +structure of the result (see [Trial Result] (#Trial-Result) section), +the result itself does contain enough information to determine its +precision, thus it is not required to report the Goal Width value. + +This allows MLRsearch implementations to use stopping conditions +different from Goal Width. + +## Load Classification + +MLRsearch keeps the basic logic of binary search (tracking tightest bounds, +measuring at the middle), perhaps with minor technical differences. + +MLRsearch algorithm chooses an intended load (as opposed to the offered load), +the interval between bounds does not need to be split +exactly into two equal halves, +and the final reported structure specifies both bounds. + +The biggest difference is that to classify a load +as an upper or lower bound, MLRsearch may need more than one trial +(depending on configuration options) to be performed at the same intended load. + +In consequence, even if a load already does have few trial results, +it still may be classified as undecided, neither a lower bound nor an upper bound. + +An explanation of the classification logic is given in the next section [Logic of Load Classification] (#Logic-of-Load-Classification), +as it heavily relies on other subsections of this section. + +For repeatability and comparability reasons, it is important that +given a set of trial results, all implementations of MLRsearch +classify the load equivalently. + +## Loss Ratios + +Another difference between MLRsearch and [RFC2544] binary search is in the goals of the search. +[RFC2544] has a single goal, +based on classifying full-length trials as either lossless or lossy. + +MLRsearch, as the name suggests, can search for multiple goals, +differing in their loss ratios. +The precise definition of the Goal Loss Ratio will be given later. +The [RFC2544] throughput goal then simply becomes a zero Goal Loss Ratio. +Different goals also may have different Goal Widths. + +A set of trial results for one specific intended load value +can classify the load as an upper bound for some goals, but a lower bound +for some other goals, and undecided for the rest of the goals. + +Therefore, the load classification depends not only on trial results, +but also on the goal. +The overall search procedure becomes more complicated, when +compared to binary search with a single goal, +but most of the complications do not affect the final result, +except for one phenomenon, loss inversion. + +## Loss Inversion + +In [RFC2544] throughput search using bisection, any load with a lossy trial +becomes a hard upper bound, meaning every subsequent trial has a smaller +intended load. + +But in MLRsearch, a load that is classified as an upper bound for one goal +may still be a lower bound for another goal, and due to the other goal +MLRsearch will probably perform trials at even higher loads. +What to do when all such higher load trials happen to have zero loss? +Does it mean the earlier upper bound was not real? +Does it mean the later lossless trials are not considered a lower bound? +Surely we do not want to have an upper bound at a load smaller than a lower bound. + +MLRsearch is conservative in these situations. +The upper bound is considered real, and the lossless trials at higher loads +are considered to be a coincidence, at least when computing the final result. + +This is formalized using new notions, the [Relevant Upper Bound] (#Relevant-Upper-Bound) and +the [Relevant Lower Bound] (#Relevant-Lower-Bound). +Load classification is still based just on the set of trial results +at a given intended load (trials at other loads are ignored), +making it possible to have a lower load classified as an upper bound, +and a higher load classified as a lower bound (for the same goal). +The Relevant Upper Bound (for a goal) is the smallest load classified +as an upper bound. +But the Relevant Lower Bound is not simply +the largest among lower bounds. +It is the largest load among loads +that are lower bounds while also being smaller than the Relevant Upper Bound. + +With these definitions, the Relevant Lower Bound is always smaller +than the Relevant Upper Bound (if both exist), and the two relevant bounds +are used analogously as the two tightest bounds in the binary search. +When they are less than the Goal Width apart, +the relevant bounds are used in the output. + +One consequence is that every trial result can have an impact on the search result. +That means if your SUT (or your traffic generator) needs a warmup, +be sure to warm it up before starting the search. + +## Exceed Ratio + +The idea of performing multiple trials at the same load comes from +a model where some trial results (those with high loss) are affected +by infrequent effects, causing poor repeatability of [RFC2544] throughput results. +See the discussion about noiseful and noiseless ends +of the SUT performance spectrum in section [DUT in SUT] (#DUT-in-SUT). +Stable results are closer to the noiseless end of the SUT performance spectrum, +so MLRsearch may need to allow some frequency of high-loss trials +to ignore the rare but big effects near the noiseful end. + +MLRsearch can do such trial result filtering, but it needs +a configuration option to tell it how frequent can the infrequent big loss be. +This option is called the exceed ratio. +It tells MLRsearch what ratio of trials +(more exactly what ratio of trial seconds) can have a [Trial Loss Ratio] (#Trial-Loss-Ratio) +larger than the Goal Loss Ratio and still be classified as a lower bound. +Zero exceed ratio means all trials have to have a Trial Loss Ratio +equal to or smaller than the Goal Loss Ratio. + +For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5, +as it simplifies some later concepts by relating them to the concept of median. + +## Duration Sum + +When more than one trial is intended to classify a load, +MLRsearch also needs something that controls the number of trials needed. +Therefore, each goal also has an attribute called duration sum. + +The meaning of a [Goal Duration Sum] (#Goal-Duration-Sum) is that +when a load has (full-length) trials +whose trial durations when summed up give a value at least as big +as the Goal Duration Sum value, +the load is guaranteed to be classified either as an upper bound +or a lower bound for that goal. + +Due to the fact that the duration sum has a big impact +on the overall search duration, and [RFC2544] prescribes +wait intervals around trial traffic, +the MLRsearch algorithm is allowed to sum durations that are different +from the actual trial traffic durations. + +In the MLRsearch specification, the different duration values are called +[Trial Effective Duration] (#Trial-Effective-Duration). + +## Short Trials + +MLRsearch requires each goal to specify its final trial duration. +Full-length trial is a shorter name for a trial whose intended trial duration +is equal to (or longer than) the goal final trial duration. + +Section 24 of [RFC2544] already anticipates possible time savings +when short trials (shorter than full-length trials) are used. +Full-length trials are the opposite of short trials, +so they may also be called long trials. + +Any MLRsearch implementation may include its own configuration options +which control when and how MLRsearch chooses to use short trial durations. + +For explainability reasons, when exceed ratio of 0.5 is used, +it is recommended for the Goal Duration Sum to be an odd multiple +of the full trial durations, so Conditional Throughput becomes identical to +a median of a particular set of trial forwarding rates. + +The presence of short trial results complicates the load classification logic. + +Full details are given later in section [Logic of Load Classification] (#Logic-of-Load-Classification). +In a nutshell, results from short trials +may cause a load to be classified as an upper bound. +This may cause loss inversion, and thus lower the Relevant Lower Bound, +below what would classification say when considering full-length trials only. + +{::comment} + [I still think this is important, revisit after explanations re quantiles.] + + <mark>For explainability reasons, it is RECOMMENDED users use such configurations + that guarantee all trials have the same length.</mark> + + <mark>mk edit note: Using RFC2119 keyword here does not seem to be + appropriate. Moreover, I do not get the meaning nor the logic behind + this statement. It seems to say that in order for users to understand + the workings of MLRsearch, they should use simplified configuration, + otherwise they won't get it. Illogical it seems to me. Suggest to + remove it.</mark> + +{:/comment} + +{::comment} + [Important. Keeping compatibility slows search considerably.] + + <mark>Alas, such configurations are usually not compliant with [RFC2544] requirements, + or not time-saving enough.</mark> + + <mark>mk edit note: This statement does not make sense to me. Suggest to remove it.</mark> + +{:/comment} + +## Throughput + +{::comment} + [Important, we need better title.] + + <mark>[VP] TODO: Was named Conditional Troughput, but spec chapter already has one.</mark> + +{:/comment} + +Due to the fact that testing equipment takes the intended load as an input parameter +for a trial measurement, any load search algorithm needs to deal +with intended load values internally. + +But in the presence of goals with a non-zero loss ratio, the intended load +usually does not match the user's intuition of what a throughput is. +The forwarding rate (as defined in [RFC2285] section 3.6.1) is better, +but it is not obvious how to generalize it +for loads with multiple trial results and a non-zero +[Goal Loss Ratio] (#Goal-Loss-Ratio). + +The best example is also the main motivation: hard limit performance. +Even if the medium allows higher performance, +the SUT interfaces may have their additional own limitations, +e.g. a specific fps limit on the NIC (a very common occurance). + +Ideally, those should be known and used when computing Max Load. +But if Max Load is higher that what interface can receive or transmit, +there will be a "hard limit" observed in trial results. +Imagine the hard limit is at 100 Mfps, Max Load is higher, +and the goal loss ratio is 0.5%. If DUT has no additional losses, +0.5% loss ratio will be achieved at 100.5025 Mfps (the relevant lower bound). +But it is not intuitive to report SUT performance as a value that is +larger than known hard limit. +We need a generalization of RFC2544 throughput, +different from just the relevant lower bound. + +MLRsearch defines one such generalization, called the Conditional Throughput. +It is the trial forwarding rate from one of the trials +performed at the load in question. +Determining which trial exactly is defined in +[MLRsearch Specification] (#MLRsearch-Specification), +and in [Appendix B: Conditional Throughput] (#Appendix-B\:-Conditional-Throughput). + +In the hard limit example, 100.5 Mfps load will still have +only 100.0 Mfps forwarding rate, nicely confirming the known limitation. + +Conditional Throughput is partially related to load classification. +If a load is classified as a lower bound for a goal, +the Conditional Throughput can be calculated from trial results, +and guaranteed to show an loss ratio +no larger than the Goal Loss Ratio. + +{::comment} + [Revisit after other edits, may be addressed elsewhere.] + + <mark>While the Conditional Throughput gives more intuitive-looking + values than the Relevant Lower Bound (for non-zero Goal Loss Ratio + values), the actual definition is more complicated than the definition + of the Relevant Lower Bound.</mark> + + <mark>mk edit note: Looking at this again, and per improved text, I + don't think it is that complicated. (BTW saying it is more complicated + and not addressing it, and leaving it open ended is not + good.) "Conditional throughput" intuitively is really throughput under + certain conditions, these being offered load determined by Relevant + Lower Bound and actual loss. For comparability, and taking into account + multiple trial samples, per MLRsearch definition, this is + mathematically expressed as `conditional_throughput = intended_load * + (1.0 - quantile_loss_ratio)`.</mark> + + <mark>DONE VP to MK: Hmm. Frequently, Conditional Throughput comes + from the worst among low-loss full-length trials. + But if two disparate goals are interested at the same load, + things get complicated (does not happen in CSIT production, + but I found few bugs when testing in simulator). + Computation in load classification is also not trivial, + but at least it only needs two "duration sum" values, + no need to sort all trial results.</mark> + + <mark>MKP2 [VP] TODO: Still not sure what to do with this subsection. + Possibly a bigger rewrite once VP and MK agree on what is (or is not) + complicated. :)</mark> + +{:/comment} + +{::comment} + [Important only for "design principles" chapter we may never have.] + + <mark>In the future, other intuitive values may become popular, + but they are unlikely to supersede the definition of the Relevant Lower Bound + as the most fitting value for comparability purposes, + therefore the Relevant Lower Bound remains a required attribute + of the Goal Result structure, while the Conditional Throughput is only optional.</mark> + + <mark>mk edit note: This paragraph adds to the confusion. I would remove + this paragraph, as with the new text above it doesn't seem to add any + value.</mark> + + <mark>[VP] TODO: This is an example of MLRsearch design principles.</mark> + +{:/comment} + +{::comment} + [Useful.] + + <mark>[VP] TODO: Mention somewhere that trending is a specific case + of repeatability/comparability.</mark> + +{:/comment} + +Note that when comparing the best (all zero loss) and worst case (all loss +just below Goal Loss Ratio), the same Relevant Lower Bound value +may result in the Conditional Throughput differing up to the Goal Loss Ratio. + +Therefore it is rarely needed to set the Goal Width (if expressed +as the relative difference of loads) below the Goal Loss Ratio. +In other words, setting the Goal Width below the Goal Loss Ratio +may cause the Conditional Throughput for a larger loss ratio to become smaller +than a Conditional Throughput for a goal with a smaller Goal Loss Ratio, +which is counter-intuitive, considering they come from the same search. +Therefore it is RECOMMENDED to set the Goal Width to a value no smaller +than the Goal Loss Ratio. + +Overall, this Conditional Throughput does behave well for comparability purposes. + +## Search Time + +MLRsearch was primarily developed to reduce the time +required to determine a throughput, either the [RFC2544] compliant one, +or some generalization thereof. +The art of achieving short search times +is mainly in the smart selection of intended loads (and intended durations) +for the next trial to perform. + +While there is an indirect impact of the load selection on the reported values, +in practice such impact tends to be small, +even for SUTs with quite a broad performance spectrum. + +A typical example of two approaches to load selection leading to different +Relevant Lower Bounds is when the interval is split in a very uneven way. +Any implementation choosing loads very close to the current Relevant Lower Bound +is quite likely to eventually stumble upon a trial result +with poor performance (due to SUT noise). +For an implementation choosing loads very close +to the current Relevant Upper Bound, this is unlikely, +as it examines more loads that can see a performance +close to the noiseless end of the SUT performance spectrum. + +However, as even splits optimize search duration at give precision, +MLRsearch implementations that prioritize minimizing search time +are unlikely to suffer from any such bias. + +Therefore, this document remains quite vague on load selection +and other optimization details, and configuration attributes related to them. +Assuming users prefer libraries that achieve short overall search time, +the definition of the Relevant Lower Bound +should be strict enough to ensure result repeatability +and comparability between different implementations, +while not restricting future implementations much. + +{::comment} + [Important for BMWG. Configurability is bad for comparability.] + + <mark>MKP2 Sadly, different implementations may exhibit their sweet spot of</mark> + <mark>the best repeatability for a given search duration</mark> + <mark>at different goals attribute values, especially concerning</mark> + <mark>any optional goal attributes such as the initial trial duration.</mark> + <mark>Thus, this document does not comment much on which configurations</mark> + <mark>are good for comparability between different implementations.</mark> + <mark>For comparability between different SUTs using the same implementation,</mark> + <mark>refer to configurations recommended by that particular implementation.</mark> + + <mark>MKP2 mk edit note: Isn't this going off on a tangent, hypothesising and + second guessing about different possible implementations. What is the + value of this content to this document? Suggest to remove it.</mark> + +{:/comment} + +## [RFC2544] Compliance + +Some Search Goal instances lead to results compliant with RFC2544. +See [RFC2544 Goal] (#RFC2544-Goal) for more details +regarding both conditional and unconditional compliance. + +The presence of other Search Goals does not affect the compliance +of this Goal Result. +The Relevant Lower Bound and the Conditional Throughput are in this case +equal to each other, and the value is the [RFC2544] throughput. + +# Logic of Load Classification + +## Introductory Remarks + +This chapter continues with explanations, +but this time more precise definitions are needed +for readers to follow the explanations. + +Descriptions in this section are wordy and implementers should read +[MLRsearch Specification] (#MLRsearch-Specification) section +and Appendices for more concise definitions. + +The two areas of focus here are load classification +and the Conditional Throughput. + +To start with [Performance Spectrum] (#Performance-Spectrum) +subsection contains definitions needed to gain insight +into what Conditional Throughput means. +Remaining subsections discuss load classification. + +For load classification, it is useful to define **good trials** and **bad trials**: + +- **Bad trial**: Trial is called bad (according to a goal) + if its [Trial Loss Ratio] (#Trial-Loss-Ratio) + is larger than the [Goal Loss Ratio] (#Goal-Loss-Ratio). + +- **Good trial**: Trial that is not bad is called good. + +## Performance Spectrum +### Description + +There are several equivalent ways to explain the Conditional Throughput +computation. One of the ways relies on performance +spectrum. + +Take an intended load value, a trial duration value, and a finite set +of trial results, with all trials measured at that load value and duration value. + +The performance spectrum is the function that maps +any non-negative real number into a sum of trial durations among all trials +in the set, that has that number, as their trial forwarding rate, +e.g. map to zero if no trial has that particular forwarding rate. + +A related function, defined if there is at least one trial in the set, +is the performance spectrum divided by the sum of the durations +of all trials in the set. + +That function is called the performance probability function, as it satisfies +all the requirements for probability mass function +of a discrete probability distribution, +the one-dimensional random variable being the trial forwarding rate. + +These functions are related to the SUT performance spectrum, +as sampled by the trials in the set. + +{::comment} + [Middle of rewrite?] + + <mark>MKP1 The performance spectrum is the function that maps + any non-negative real number into a sum of trial durations among all trials + in the set, that has that number, as their trial forwarding rate, + e.g. map to zero if no trial has that particular forwarding rate.</mark> + + <mark>MKP1 A related function, defined if there is at least one trial in the set, + is the performance spectrum divided by the sum of the durations + of all trials in the set.</mark> + + <mark>MKP1 That function is called the performance probability function, as it satisfies + all the requirements for probability mass function + of a discrete probability distribution, + the one-dimensional random variable being the trial forwarding rate.</mark> + + <mark>MKP1 These functions are related to the SUT performance spectrum, + as sampled by the trials in the set.</mark> + + <mark>MKP1 [VP] TODO: Introduce quantiles properly by incorporating the below.</mark> + + <mark>MKP1 [VP] TODO: "q-quantile" is plainly wrong. I meant the "p" in "p-quantile". + + - wikipedia: The 100-quantiles are called percentiles + - also wiki: If, instead of using integers k and q, the "p-quantile" is based on a real number p with 0 < p < 1 then... + - https://en.wikipedia.org/wiki/Quantile_function + - exceed ratio is an input to a quantile function: percentage? + </mark> + + <mark>MKP1 mk2 TODO for VP: Above is not making it clearer at all. Can't we really not explain the spectrum and exceed ratio with just percentiles and quantiles?</mark> + + As for any other probability function, we can talk about percentiles + of the performance probability function, including the median. + The Conditional Throughput will be one such quantile value + for a specifically chosen set of trials. + + <mark>MKP2 As for any other probability function, we can talk about percentiles + of the performance probability function, including the median. + The Conditional Throughput will be one such quantile value + for a specifically chosen set of trials.</mark> + +{:/comment} + +Take a set of all full-length trials performed at the Relevant Lower Bound, +sorted by decreasing trial forwarding rate. +The sum of the durations of those trials +may be less than the Goal Duration Sum, or not. +If it is less, add an imaginary trial result with zero trial forwarding rate, +such that the new sum of durations is equal to the Goal Duration Sum. +This is the set of trials to use. + +If the quantile touches two trials, + +{::comment} + [Clarity.] + + <mark>mk edit note: What does it mean "quantile touches two trials"? + Do you mean two trials are within specific quantile or percentile?</mark> + +{:/comment} + +the larger trial forwarding rate (from the trial result sorted earlier) is used. + +{::comment} + [Oh, unspecified exceed ratio?] + + <mark>the larger trial forwarding rate (from the trial result sorted earlier) is used.</mark> + + <mark>mk edit note: Why is that? Is it because you silently assumed that + quantile here is median or 50th percentile?</mark> + +{:/comment} + +The resulting quantity is the Conditional Throughput of the goal in question. + +{::comment} + [Motivation has lead to code. Now code is definition, this should be equivalent.] + + <mark>The resulting quantity is the Conditional Throughput of the goal in question.</mark> + + <mark>mk edit note: Is this is supposed to be another definition of + Conditional Throughput? If so, how does this relate to Performance + Spectrum? I suggest to either remove these unclear paragraphs above and + rely on examples below that are clear, or rework above so it fits the + flow. Cause right now it's confusion. Even more so, that + [Conditional Throughput] (#Conditional-Throughput) has been already + defined elsewhere in the document.</mark> + +{:/comment} + +A set of examples follows. + +### First Example + +- [Goal Exceed Ratio] (#Goal-Exceed-Ratio) = 0 and [Goal Duration Sum] (#Goal-Duration-Sum) has been reached. +- Conditional Throughput is the smallest trial forwarding rate among the trials. + +### Second Example + +- Goal Exceed Ratio = 0 and Goal Duration Sum has not been reached yet. +- Due to the missing duration sum, the worst case may still happen, so the Conditional Throughput is zero. +- This is not reported to the user, as this load cannot become the Relevant Lower Bound yet. + +### Third Example + +- Goal Exceed Ratio = 50% and Goal Duration Sum is two seconds. +- One trial is present with the duration of one second and zero loss. +- The imaginary trial is added with the duration of one second and zero trial forwarding rate. +- The median would touch both trials, so the Conditional Throughput is the trial forwarding rate of the one non-imaginary trial. +- As that had zero loss, the value is equal to the offered load. + +{::comment} + [Middle of rewrite?] + + <mark>MKP2 mk edit note: how is the median "touching" both trials? + Isn't median of even set of data samples + the average of the two middle data points, + in this case the non-imaginary trial and the imaginary one?</mark> + + <mark>MKP2 Note that Appendix B does not take into account short trial results.</mark> + + <mark>MKP2 mk edit note: Whis is this relevant here? Appendix B has not been mentioned in this section.</mark> + +{:/comment} + +### Summary + +While the Conditional Throughput is a generalization of the trial forwarding rate, +its definition is not an obvious one. + +Other than the trial forwarding rate, the other source of intuition +is the quantile in general, and the median the recommended case. + +{::comment} + [Next version of MLRsearch library may invent new quantity that is more stable.] + + <mark>In future, different quantities may prove more useful, + especially when applying to specific problems, + but currently the Conditional Throughput is the recommended compromise, + especially for repeatability and comparability reasons.</mark> + + <mark>MKP2 mk edit note: This is future looking and hand wavy without + specifics. What are the "specific problems" that are referred here? + Networking, else?Some specific behaviours, if so, what sort? If + something is classified as future work, it needs to be better defined. + The same applies to any out of scope statements.</mark> + +{:/comment} + +## Trials with Single Duration + +{::comment} + [Clarity.] + + <mark>MKP2 mk edit note: Need to improve explanations in this subsection.</mark> + +{:/comment} + +When goal attributes are chosen in such a way that every trial has the same +intended duration, the load classification is simpler. + +The following description follows the motivation +of Goal Loss Ratio, Goal Exceed Ratio, and Goal Duration Sum. + +If the sum of the durations of all trials (at the given load) +is less than the Goal Duration Sum, imagine two scenarios: + +- **best case scenario**: all subsequent trials having zero loss, and +- **worst case scenario**: all subsequent trials having 100% loss. + +Here we assume there are as many subsequent trials as needed +to make the sum of all trials equal to the Goal Duration Sum. + +The exceed ratio is defined using sums of durations +(and number of trials does not matter), so it does not matter whether +the "subsequent trials" can consist of an integer number of full-length trials. + +In any of the two scenarios, best case and worst case, we can compute the load exceed ratio, +as the duration sum of good trials divided by the duration sum of all trials, +in both cases including the assumed trials. + +Even if, in the best case scenario, the load exceed ratio is larger +than the Goal Exceed Ratio, the load is an upper bound. + +MKP2 Even if, in the worst case scenario, the load exceed ratio is not larger +than the Goal Exceed Ratio, the load is a lower bound. + +{::comment} + [Middle of rewrite?] + + <mark>Even if</mark>, in the best case scenario, the load exceed ratio is larger + than the Goal Exceed Ratio, the load is an upper bound. + + <mark>MKP2 Even if</mark>, in the worst case scenario, the load exceed ratio is not larger + than the Goal Exceed Ratio, the load is a lower bound. + + <mark>MKP2 mk edit note: I am confused by "Even if" prefixing + each of the above statements. And even more so by your version + with "If even".</mark> + + <mark>mk edit note: I do not get how this statements are true, as they + are counter-intuitive. For the best case scenario, if load exceed ratio + is larger than the goal exceed ratio, I expect the load to be lower + bound. Need more examples.</mark> + +{:/comment} + +More specifically: + +- Take all trials measured at a given load. +- The sum of the durations of all bad full-length trials is called the bad sum. +- The sum of the durations of all good full-length trials is called the good sum. +- The result of adding the bad sum plus the good sum is called the measured sum. +- The larger of the measured sum and the Goal Duration Sum is called the whole sum. +- The whole sum minus the measured sum is called the missing sum. +- The optimistic exceed ratio is the bad sum divided by the whole sum. +- The pessimistic exceed ratio is the bad sum plus the missing sum, that divided by the whole sum. +- If the optimistic exceed ratio is larger than the Goal Exceed Ratio, the load is classified as an upper bound. +- If the pessimistic exceed ratio is not larger than the Goal Exceed Ratio, the load is classified as a lower bound. +- Else, the load is classified as undecided. + +The definition of pessimistic exceed ratio is compatible with the logic in +the Conditional Throughput computation, so in this single trial duration case, +a load is a lower bound if and only if the Conditional Throughput +loss ratio is not larger than the Goal Loss Ratio. + +{::comment} + [Useful (depends on the whole chapter).] + + <mark>MKP2 mk edit note: I do not get the defintion of optimistic and + pessmistic exceed ratios. Please define or describe what they + are.</mark> + +{:/comment} + +If it is larger, the load is either an upper bound or undecided. + +## Trials with Short Duration + +### Scenarios + +Trials with intended duration smaller than the goal final trial duration +are called short trials. +The motivation for load classification logic in the presence of short trials +is based around a counter-factual case: What would the trial result be +if a short trial has been measured as a full-length trial instead? + +There are three main scenarios where human intuition guides +the intended behavior of load classification. + +#### False Good Scenario + +The user had their reason for not configuring a shorter goal +final trial duration. +Perhaps SUT has buffers that may get full at longer +trial durations. +Perhaps SUT shows periodic decreases in performance +the user does not want to be treated as noise. + +In any case, many good short trials may become bad full-length trials +in the counter-factual case. + +In extreme cases, there are plenty of good short trials and no bad short trials. + +In this scenario, we want the load classification NOT to classify the load +as a lower bound, despite the abundance of good short trials. + +{::comment} + [I agree.] + + <mark>MKP2 mk edit note: It may be worth adding why that is. i.e. because + there is a risk that at longer trial this could turn into a bad + trial.</mark> + +{:/comment} + +Effectively, we want the good short trials to be ignored, so they +do not contribute to comparisons with the Goal Duration Sum. + +#### True Bad Scenario + +When there is a frame loss in a short trial, +the counter-factual full-length trial is expected to lose at least as many +frames. + +In practice, bad short trials are rarely turning into +good full-length trials. + +In extreme cases, there are no good short trials. + +In this scenario, we want the load classification +to classify the load as an upper bound just based on the abundance +of short bad trials. + +Effectively, we want the bad short trials +to contribute to comparisons with the Goal Duration Sum, +so the load can be classified sooner. + +#### Balanced Scenario + +Some SUTs are quite indifferent to trial duration. +Performance probability function constructed from short trial results +is likely to be similar to the performance probability function constructed +from full-length trial results (perhaps with larger dispersion, +but without a big impact on the median quantiles overall). + +{::comment} + [Recheck after edits earlier.] + + <mark>MKP1 mk edit note: "Performance probability function" is this function + defined anywhere? Mention in [Performance Spectrum] (#Performance Spectrum) + is not a complete definition.</mark> + +{:/comment} + +For a moderate Goal Exceed Ratio value, this may mean there are both +good short trials and bad short trials. + +This scenario is there just to invalidate a simple heuristic +of always ignoring good short trials and never ignoring bad short trials, +as that simple heuristic would be too biased. + +Yes, the short bad trials +are likely to turn into full-length bad trials in the counter-factual case, +but there is no information on what would the good short trials turn into. + +The only way to decide safely is to do more trials at full length, +the same as in False Good Scenario. + +### Classification Logic + +MLRsearch picks a particular logic for load classification +in the presence of short trials, but it is still RECOMMENDED +to use configurations that imply no short trials, +so the possible inefficiencies in that logic +do not affect the result, and the result has better explainability. + +With that said, the logic differs from the single trial duration case +only in different definition of the bad sum. +The good sum is still the sum across all good full-length trials. + +Few more notions are needed for defining the new bad sum: + +- The sum of durations of all bad full-length trials is called the bad long sum. +- The sum of durations of all bad short trials is called the bad short sum. +- The sum of durations of all good short trials is called the good short sum. +- One minus the Goal Exceed Ratio is called the subceed ratio. +- The Goal Exceed Ratio divided by the subceed ratio is called the exceed coefficient. +- The good short sum multiplied by the exceed coefficient is called the balancing sum. +- The bad short sum minus the balancing sum is called the excess sum. +- If the excess sum is negative, the bad sum is equal to the bad long sum. +- Otherwise, the bad sum is equal to the bad long sum plus the excess sum. + +Here is how the new definition of the bad sum fares in the three scenarios, +where the load is close to what would the relevant bounds be +if only full-length trials were used for the search. + +#### False Good Scenario + +If the duration is too short, we expect to see a higher frequency +of good short trials. +This could lead to a negative excess sum, +which has no impact, hence the load classification is given just by +full-length trials. +Thus, MLRsearch using too short trials has no detrimental effect +on result comparability in this scenario. +But also using short trials does not help with overall search duration, +probably making it worse. + +#### True Bad Scenario + +Settings with a small exceed ratio +have a small exceed coefficient, so the impact of the good short sum is small, +and the bad short sum is almost wholly converted into excess sum, +thus bad short trials have almost as big an impact as full-length bad trials. +The same conclusion applies to moderate exceed ratio values +when the good short sum is small. +Thus, short trials can cause a load to get classified as an upper bound earlier, +bringing time savings (while not affecting comparability). + +#### Balanced Scenario + +Here excess sum is small in absolute value, as the balancing sum +is expected to be similar to the bad short sum. +Once again, full-length trials are needed for final load classification; +but usage of short trials probably means MLRsearch needed +a shorter overall search time before selecting this load for measurement, +thus bringing time savings (while not affecting comparability). + +Note that in presence of short trial results, +the comparibility between the load classification +and the Conditional Throughput is only partial. +The Conditional Throughput still comes from a good long trial, +but a load higher than the Relevant Lower Bound may also compute to a good value. + +## Trials with Longer Duration + +If there are trial results with an intended duration larger +than the goal trial duration, the precise definitions +in Appendix A and Appendix B treat them in exactly the same way +as trials with duration equal to the goal trial duration. + +But in configurations with moderate (including 0.5) or small +Goal Exceed Ratio and small Goal Loss Ratio (especially zero), +bad trials with longer than goal durations may bias the search +towards the lower load values, as the noiseful end of the spectrum +gets a larger probability of causing the loss within the longer trials. + +{::comment} + [Use single goal when testing externaly, deviate freely in internal tests.] + + <mark>For some users, this is an acceptable price</mark> + <mark>for increased configuration flexibility</mark> + <mark>(perhaps saving time for the related goals),</mark> + <mark>so implementations SHOULD allow such configurations.</mark> + <mark>Still, users are encouraged to avoid such configurations</mark> + <mark>by making all goals use the same final trial duration,</mark> + <mark>so their results remain comparable across implementations.</mark> + + <mark>MKP2 mk edit note: This paragraph has no value in my view. + Statements like "For some users, this is an acceptable price + for increased configuration flexibility" do not make sense. + Configuration flexibility for flexibility sake is not a valid argument + in the specification that aims at standardising benchmarking methodologies. + If one wants to test with longer durations, + then one should configure these as Goal Final Trial Duration. + Simple, no? Or am I reading this point wrong?</mark> + +{:/comment} + +{::comment} + [MKP4 Out of scope here, subject for future work] + + # Current practices? + + <mark>MKP2 [VP] TODO: Even if not mentioned in spec (not even recommended), + some tricks from CSIT code may be worth mentioning? Not sure.</mark> + + <mark>MKP2 [VP] TODO: Tricks with big impact on search time + can be mentioned so that Addressed Problems : Long Test Duration + has something specific to refer to.</mark> + + <mark>MKP2 [VP] TODO: It is important to mention trick that have impact + on repeatability and comparability.</mark> + + <mark>MKP2 [VP] TODO: CSIT computes a discrete "grid" of load values to use.</mark> + + <mark>MKP2 [VP] TODO: + If all Goal Widths are aligned, there is one common coarse grid. + In that case, NDR (and even PDR conditional throughput + for tests with zer-or-big losses) values are identical in trending, + hiding the real performance variance, and causing fake anomaly + when the performance shifts just one gridpoint. + </mark> + + <mark>MKP2 [VP] TODO: Conversely, when Goal Width do not match well, + CSIT needs to compute a fine-grained grid to match them all. + In this case, similar performances can be "rounded differently", + mostly based on specific loss that happened at Max Load, + where SUT may be less stable than around PDR. + This way trending sees higher variance (still within corresponding Goal Width), + but at least there are no fake anomalies. + </mark> + + <mark>MKP2 [VP] TODO: In general, do not trust stdev if not larged than width.</mark> + + <mark>MKP2 [VP] TODO: De we have a chapter section fosucing on design principles? + - Make Controller API independent from Measurer API. + - The "allowed if makes worse" principle: + - RFC1242 specmanship happens when testing own DUTs. + - Shortening trial wait times only risks making goal results lower. + - So it is fine to save time aggressively when testing own DUTs. + </mark> + +{:/comment} + + +{::comment} + [Will be nice if made substantial.] + + # Addressed Problems + + <mark>MKP1 all of this section requires updating based on the updated content. + And it is for information only anyways. In fact not sure it's needed. + Maybe in appendix for posterity.</mark> + + Now when MLRsearch is clearly specified and explained, + it is possible to summarize how does MLRsearch specification help with problems. + + Here, "multiple trials" is a shorthand for having the goal final trial duration + significantly smaller than the Goal Duration Sum. + This results in MLRsearch performing multiple trials at the same load, + which may not be the case with other configurations. + + ## Long Test Duration + + As shortening the overall search duration is the main motivation + of MLRsearch library development, the library implements + multiple improvements on this front, both big and small. + + Most of implementation details are not constrained by MLRsearch specification, + so that future implementations may keep shortening the search duration even more. + + One exception is the impact of short trial results on the Relevant Lower Bound. + While motivated by human intuition, the logic is not straightforward. + In practice, configurations with only one common trial duration value + are capable of achieving good overal search time and result repeatability + without the need to consider short trials. + + ### Impact of goal attribute values + + From the required goal attributes, the Goal Duration Sum + remains the best way to get even shorter searches. + + Usage of multiple trials can also save time, + depending on wait times around trial traffic. + + The farther the Goal Exceed Ratio is from 0.5 (towards zero or one), + the less predictable the overal search duration becomes in practice. + + Width parameter does not change search duration much in practice + (compared to other, mainly optional goal attributes). + + ## DUT in SUT + + In practice, using multiple trials and moderate exceed ratios + often improves result repeatability without increasing the overall search time, + depending on the specific SUT and DUT characteristics. + Benefits for separating SUT noise are less clear though, + as it is not easy to distinguish SUT noise from DUT instability in general. + + Conditional Throughput has an intuitive meaning when described + using the performance spectrum, so this is an improvement + over existing simple (less configurable) search procedures. + + Multiple trials can save time also when the noisy end of + the preformance spectrum needs to be examined, e.g. for [RFC9004]. + + Under some circumstances, testing the same DUT and SUT setup with different + DUT configurations can give some hints on what part of noise is SUT noise + and what part is DUT performance fluctuations. + In practice, both types of noise tend to be too complicated for that analysis. + + MLRsearch enables users to search for multiple goals, + potentially providing more insight at the cost of a longer overall search time. + However, for a thorough and reliable examination of DUT-SUT interactions, + it is necessary to employ additional methods beyond black-box benchmarking, + such as collecting and analyzing DUT and SUT telemetry. + + ## Repeatability and Comparability + + Multiple trials improve repeatability, depending on exceed ratio. + + In practice, one-second goal final trial duration with exceed ratio 0.5 + is good enough for modern SUTs. + However, unless smaller wait times around the traffic part of the trial + are allowed, too much of overal search time would be wasted on waiting. + + It is not clear whether exceed ratios higher than 0.5 are better + for repeatability. + The 0.5 value is still preferred due to explainability using median. + + It is possible that the Conditional Throughput values (with non-zero goal + loss ratio) are better for repeatability than the Relevant Lower Bound values. + This is especially for implementations + which pick load from a small set of discrete values, + as that hides small variances in Relevant Lower Bound values + other implementations may find. + + Implementations focusing on shortening the overall search time + are automatically forced to avoid comparability issues due to load selection, + as they must prefer even splits wherever possible. + But this conclusion only holds when the same goals are used. + Larger adoption is needed before any further claims on comparability + between MLRsearch implementations can be made. + + ## Throughput with Non-Zero Loss + + Trivially suported by the Goal Loss Ratio attribute. + + In practice, usage of non-zero loss ratio values + improves the result repeatability + (exactly as expected based on results from simpler search methods). + + ## Inconsistent Trial Results + + MLRsearch is conservative wherever possible. + This is built into the definition of Conditional Throughput, + and into the treatment of short trial results for load classification. + + This is consistent with [RFC2544] zero loss tolerance motivation. + + If the noiseless part of the SUT performance spectrum is of interest, + it should be enough to set small value for the goal final trial duration, + and perhaps also a large value for the Goal Exceed Ratio. + + Implementations may offer other (optional) configuration attributes + to become less conservative, but currently it is not clear + what impact would that have on repeatability. + +{:/comment} + +# IANA Considerations + +No requests of IANA. + +# Security Considerations + +Benchmarking activities as described in this memo are limited to +technology characterization of a DUT/SUT using controlled stimuli in a +laboratory environment, with dedicated address space and the constraints +specified in the sections above. + +The benchmarking network topology will be an independent test setup and +MUST NOT be connected to devices that may forward the test traffic into +a production network or misroute traffic to the test management network. + +Further, benchmarking is performed on a "black-box" basis, relying +solely on measurements observable external to the DUT/SUT. + +Special capabilities SHOULD NOT exist in the DUT/SUT specifically for +benchmarking purposes. Any implications for network security arising +from the DUT/SUT SHOULD be identical in the lab and in production +networks. + +# Acknowledgements + +Some phrases and statements in this document were created +with help of Mistral AI (mistral.ai). + +Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough +review and numerous useful comments and suggestions in the earlier versions of this document. + +Special wholehearted gratitude and thanks to the late Al Morton for his +thorough reviews filled with very specific feedback and constructive +guidelines. Thank you Al for the close collaboration over the years, +for your continuous unwavering encouragement full of empathy and +positive attitude. Al, you are dearly missed. + +# Appendix A: Load Classification + +This section specifies how to perform the load classification. + +Any intended load value can be classified, according to a given [Search Goal] (#Search-Goal). + +The algorithm uses (some subsets of) the set of all available trial results +from trials measured at a given intended load at the end of the search. +All durations are those returned by the Measurer. + +The block at the end of this appendix holds pseudocode +which computes two values, stored in variables named +`optimistic` and `pessimistic`. + +{::comment} + [We have other section re optimistic. Not going to talk about variable naming here.] + + <mark>MKP2 mk edit note: Need to add the description of what + the `optimistic` and `pessimistic` variables represent. + Or a reference to where this is described + e.g. in [Single Trial Duration] (#Single-Trial-Duration) section.</mark> + +{:/comment} + +The pseudocode happens to be a valid Python code. + +If values of both variables are computed to be true, the load in question +is classified as a lower bound according to the given Search Goal. +If values of both variables are false, the load is classified as an upper bound. +Otherwise, the load is classified as undecided. + +The pseudocode expects the following variables to hold values as follows: + +- `goal_duration_sum`: The duration sum value of the given Search Goal. + +- `goal_exceed_ratio`: The exceed ratio value of the given Search Goal. + +- `good_long_sum`: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial Loss Ratio + not higher than the Goal Loss Ratio. + +- `bad_long_sum`: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial Loss Ratio + higher than the Goal Loss Ratio. + +- `good_short_sum`: Sum of durations across trials with trial duration + shorter than the goal final trial duration and with a Trial Loss Ratio + not higher than the Goal Loss Ratio. + +- `bad_short_sum`: Sum of durations across trials with trial duration + shorter than the goal final trial duration and with a Trial Loss Ratio + higher than the Goal Loss Ratio. + +The code works correctly also when there are no trial results at a given load. + +~~~ python +balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio) +effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum) +effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum) +quantile_duration_sum = effective_whole_sum * goal_exceed_ratio +optimistic = effective_bad_sum <= quantile_duration_sum +pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum +~~~ + +# Appendix B: Conditional Throughput + +This section specifies how to compute Conditional Throughput, as referred to in section [Conditional Throughput] (#Conditional-Throughput). + +Any intended load value can be used as the basis for the following computation, +but only the Relevant Lower Bound (at the end of the search) +leads to the value called the Conditional Throughput for a given Search Goal. + +The algorithm uses (some subsets of) the set of all available trial results +from trials measured at a given intended load at the end of the search. +All durations are those returned by the Measurer. + +The block at the end of this appendix holds pseudocode +which computes a value stored as variable `conditional_throughput`. + +{::comment} + [CT is CT. But text could make more obvious.] + + <mark>MKP2 mk edit note: Need to add the description of what does + the `conditional_throughput` variable represent. + Or a reference to where this is described + e.g. in [Conditional Throughput] (#Conditional-Throughput) section.</mark> + +{:/comment} + +The pseudocode happens to be a valid Python code. + +The pseudocode expects the following variables to hold values as follows: + +- `goal_duration_sum`: The duration sum value of the given Search Goal. + +- `goal_exceed_ratio`: The exceed ratio value of the given Search Goal. + +- `good_long_sum`: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial Loss Ratio + not higher than the Goal Loss Ratio. + +- `bad_long_sum`: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial Loss Ratio + higher than the Goal Loss Ratio. + +- `long_trials`: An iterable of all trial results from trials with trial duration + at least equal to the goal final trial duration, + sorted by increasing the Trial Loss Ratio. + A trial result is a composite with the following two attributes available: + + - `trial.loss_ratio`: The Trial Loss Ratio as measured for this trial. + + - `trial.duration`: The trial duration of this trial. + +The code works correctly only when there if there is at least one +trial result measured at a given load. + +~~~ python +all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum) +remaining = all_long_sum * (1.0 - goal_exceed_ratio) +quantile_loss_ratio = None +for trial in long_trials: + if quantile_loss_ratio is None or remaining > 0.0: + quantile_loss_ratio = trial.loss_ratio + remaining -= trial.duration + else: + break +else: + if remaining > 0.0: + quantile_loss_ratio = 1.0 +conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) +~~~ + +--- back + +{::comment} + [Final checklist.] + + <mark>[VP] Final Checks. Only mark as done when there are no active todos above.</mark> + + <mark>[VP] Rename chapter/sub-/section to better match their content.</mark> + + <mark>MKP3 [VP] TODO: Recheck the definition dependencies go bottom-up.</mark> + + <mark>[VP] TODO: Unify external reference style (brackets, spaces, section numbers and names).</mark> + + <mark>[VP] TODO: Add internal links wherever Captialized Term is mentioned.</mark> + + <mark>MKP2 [VP] TODO: Capitalization of New Terms: useful when editing and reviewing, + but I still vote to remove capitalization before final submit, + because all other RFCs I see only capitalize due to being section title.</mark> + + <mark>[VP] TODO: If time permits, keep improving formal style (e.g. using AI).</mark> + +{:/comment} diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt new file mode 100644 index 0000000000..c5c94410a8 --- /dev/null +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt @@ -0,0 +1,2800 @@ + + + + +Benchmarking Working Group M. Konstantynowicz +Internet-Draft V. Polak +Intended status: Informational Cisco Systems +Expires: 18 January 2025 18 July 2024 + + + Multiple Loss Ratio Search + draft-ietf-bmwg-mlrsearch-07 + +Abstract + + This document proposes extensions to [RFC2544] throughput search by + defining a new methodology called Multiple Loss Ratio search + (MLRsearch). MLRsearch aims to minimize search duration, support + multiple loss ratio searches, and enhance result repeatability and + comparability. + + The primary reason for extending [RFC2544] is to address the + challenges and requirements presented by the evaluation and testing + of software-based networking systems' data planes. + + To give users more freedom, MLRsearch provides additional + configuration options such as allowing multiple short trials per load + instead of one large trial, tolerating a certain percentage of trial + results with higher loss, and supporting the search for multiple + goals with varying loss ratios. + +Status of This Memo + + This Internet-Draft is submitted in full conformance with the + provisions of BCP 78 and BCP 79. + + Internet-Drafts are working documents of the Internet Engineering + Task Force (IETF). Note that other groups may also distribute + working documents as Internet-Drafts. The list of current Internet- + Drafts is at https://datatracker.ietf.org/drafts/current/. + + Internet-Drafts are draft documents valid for a maximum of six months + and may be updated, replaced, or obsoleted by other documents at any + time. It is inappropriate to use Internet-Drafts as reference + material or to cite them other than as "work in progress." + + This Internet-Draft will expire on 18 January 2025. + +Copyright Notice + + Copyright (c) 2024 IETF Trust and the persons identified as the + document authors. All rights reserved. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 1] + +Internet-Draft MLRsearch July 2024 + + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents (https://trustee.ietf.org/ + license-info) in effect on the date of publication of this document. + Please review these documents carefully, as they describe your rights + and restrictions with respect to this document. Code Components + extracted from this document must include Revised BSD License text as + described in Section 4.e of the Trust Legal Provisions and are + provided without warranty as described in the Revised BSD License. + +Table of Contents + + 1. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4 + 2. Identified Problems . . . . . . . . . . . . . . . . . . . . . 5 + 2.1. Long Search Duration . . . . . . . . . . . . . . . . . . 5 + 2.2. DUT in SUT . . . . . . . . . . . . . . . . . . . . . . . 6 + 2.3. Repeatability and Comparability . . . . . . . . . . . . . 8 + 2.4. Throughput with Non-Zero Loss . . . . . . . . . . . . . . 8 + 2.5. Inconsistent Trial Results . . . . . . . . . . . . . . . 9 + 3. MLRsearch Specification . . . . . . . . . . . . . . . . . . . 10 + 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 10 + 3.2. Measurement Quantities . . . . . . . . . . . . . . . . . 11 + 3.3. Existing Terms . . . . . . . . . . . . . . . . . . . . . 12 + 3.3.1. SUT . . . . . . . . . . . . . . . . . . . . . . . . . 12 + 3.3.2. DUT . . . . . . . . . . . . . . . . . . . . . . . . . 12 + 3.3.3. Trial . . . . . . . . . . . . . . . . . . . . . . . . 12 + 3.4. Trial Terms . . . . . . . . . . . . . . . . . . . . . . . 13 + 3.4.1. Trial Duration . . . . . . . . . . . . . . . . . . . 14 + 3.4.2. Trial Load . . . . . . . . . . . . . . . . . . . . . 14 + 3.4.3. Trial Input . . . . . . . . . . . . . . . . . . . . . 15 + 3.4.4. Traffic Profile . . . . . . . . . . . . . . . . . . . 15 + 3.4.5. Trial Forwarding Ratio . . . . . . . . . . . . . . . 16 + 3.4.6. Trial Loss Ratio . . . . . . . . . . . . . . . . . . 16 + 3.4.7. Trial Forwarding Rate . . . . . . . . . . . . . . . . 17 + 3.4.8. Trial Effective Duration . . . . . . . . . . . . . . 17 + 3.4.9. Trial Output . . . . . . . . . . . . . . . . . . . . 18 + 3.4.10. Trial Result . . . . . . . . . . . . . . . . . . . . 18 + 3.5. Goal Terms . . . . . . . . . . . . . . . . . . . . . . . 19 + 3.5.1. Goal Final Trial Duration . . . . . . . . . . . . . . 19 + 3.5.2. Goal Duration Sum . . . . . . . . . . . . . . . . . . 19 + 3.5.3. Goal Loss Ratio . . . . . . . . . . . . . . . . . . . 20 + 3.5.4. Goal Exceed Ratio . . . . . . . . . . . . . . . . . . 20 + 3.5.5. Goal Width . . . . . . . . . . . . . . . . . . . . . 21 + 3.5.6. Search Goal . . . . . . . . . . . . . . . . . . . . . 21 + 3.5.7. Controller Input . . . . . . . . . . . . . . . . . . 22 + 3.6. Search Goal Examples . . . . . . . . . . . . . . . . . . 23 + 3.6.1. RFC2544 Goal . . . . . . . . . . . . . . . . . . . . 23 + 3.6.2. TST009 Goal . . . . . . . . . . . . . . . . . . . . . 24 + 3.7. Result Terms . . . . . . . . . . . . . . . . . . . . . . 24 + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 2] + +Internet-Draft MLRsearch July 2024 + + + 3.7.1. Relevant Upper Bound . . . . . . . . . . . . . . . . 25 + 3.7.2. Relevant Lower Bound . . . . . . . . . . . . . . . . 25 + 3.7.3. Conditional Throughput . . . . . . . . . . . . . . . 26 + 3.7.4. Goal Result . . . . . . . . . . . . . . . . . . . . . 26 + 3.7.5. Search Result . . . . . . . . . . . . . . . . . . . . 27 + 3.7.6. Controller Output . . . . . . . . . . . . . . . . . . 27 + 3.8. MLRsearch Architecture . . . . . . . . . . . . . . . . . 28 + 3.8.1. Measurer . . . . . . . . . . . . . . . . . . . . . . 28 + 3.8.2. Controller . . . . . . . . . . . . . . . . . . . . . 29 + 3.8.3. Manager . . . . . . . . . . . . . . . . . . . . . . . 29 + 3.9. Implementation Compliance . . . . . . . . . . . . . . . . 30 + 4. Additional Considerations . . . . . . . . . . . . . . . . . . 30 + 4.1. MLRsearch Versions . . . . . . . . . . . . . . . . . . . 31 + 4.2. Stopping Conditions . . . . . . . . . . . . . . . . . . . 31 + 4.3. Load Classification . . . . . . . . . . . . . . . . . . . 32 + 4.4. Loss Ratios . . . . . . . . . . . . . . . . . . . . . . . 32 + 4.5. Loss Inversion . . . . . . . . . . . . . . . . . . . . . 33 + 4.6. Exceed Ratio . . . . . . . . . . . . . . . . . . . . . . 34 + 4.7. Duration Sum . . . . . . . . . . . . . . . . . . . . . . 34 + 4.8. Short Trials . . . . . . . . . . . . . . . . . . . . . . 35 + 4.9. Throughput . . . . . . . . . . . . . . . . . . . . . . . 35 + 4.10. Search Time . . . . . . . . . . . . . . . . . . . . . . . 37 + 4.11. RFC2544 Compliance . . . . . . . . . . . . . . . . . . . 38 + 5. Logic of Load Classification . . . . . . . . . . . . . . . . 38 + 5.1. Introductory Remarks . . . . . . . . . . . . . . . . . . 38 + 5.2. Performance Spectrum . . . . . . . . . . . . . . . . . . 38 + 5.2.1. First Example . . . . . . . . . . . . . . . . . . . . 39 + 5.2.2. Second Example . . . . . . . . . . . . . . . . . . . 40 + 5.2.3. Third Example . . . . . . . . . . . . . . . . . . . . 40 + 5.2.4. Summary . . . . . . . . . . . . . . . . . . . . . . . 40 + 5.3. Trials with Single Duration . . . . . . . . . . . . . . . 40 + 5.4. Trials with Short Duration . . . . . . . . . . . . . . . 42 + 5.4.1. Scenarios . . . . . . . . . . . . . . . . . . . . . . 42 + 5.4.2. Classification Logic . . . . . . . . . . . . . . . . 43 + 5.5. Trials with Longer Duration . . . . . . . . . . . . . . . 45 + 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 45 + 7. Security Considerations . . . . . . . . . . . . . . . . . . . 45 + 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46 + 9. Appendix A: Load Classification . . . . . . . . . . . . . . . 46 + 10. Appendix B: Conditional Throughput . . . . . . . . . . . . . 47 + 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 49 + 11.1. Normative References . . . . . . . . . . . . . . . . . . 49 + 11.2. Informative References . . . . . . . . . . . . . . . . . 49 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 49 + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 3] + +Internet-Draft MLRsearch July 2024 + + +1. Purpose and Scope + + The purpose of this document is to describe Multiple Loss Ratio + search (MLRsearch), a data plane throughput search methodology + optimized for software networking DUTs. + + Applying vanilla [RFC2544] throughput bisection to software DUTs + results in several problems: + + * Binary search takes too long as most trials are done far from the + eventually found throughput. + + * The required final trial duration and pauses between trials + prolong the overall search duration. + + * Software DUTs show noisy trial results, leading to a big spread of + possible discovered throughput values. + + * Throughput requires a loss of exactly zero frames, but the + industry frequently allows for small but non-zero losses. + + * The definition of throughput is not clear when trial results are + inconsistent. + + To address the problems mentioned above, the MLRsearch test + methodology specification employs the following enhancements: + + * Allow multiple short trials instead of one big trial per load. + + - Optionally, tolerate a percentage of trial results with higher + loss. + + * Allow searching for multiple Search Goals, with differing loss + ratios. + + - Any trial result can affect each Search Goal in principle. + + * Insert multiple coarse targets for each Search Goal, earlier ones + need to spend less time on trials. + + - Earlier targets also aim for lesser precision. + + - Use Forwarding Rate (FR) at maximum offered load [RFC2285] + (section 3.6.2) to initialize the initial targets. + + * Take care when dealing with inconsistent trial results. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 4] + +Internet-Draft MLRsearch July 2024 + + + - Reported throughput is smaller than the smallest load with high + loss. + + - Smaller load candidates are measured first. + + * Apply several load selection heuristics to save even more time by + trying hard to avoid unnecessarily narrow bounds. + + Some of these enhancements are formalized as MLRsearch specification, + the remaining enhancements are treated as implementation details, + thus achieving high comparability without limiting future + improvements. + + MLRsearch configuration options are flexible enough to support both + conservative settings and aggressive settings. The conservative + settings lead to results unconditionally compliant with [RFC2544], + but longer search duration and worse repeatability. Conversely, + aggressive settings lead to shorter search duration and better + repeatability, but the results are not compliant with [RFC2544]. + + No part of [RFC2544] is intended to be obsoleted by this document. + +2. Identified Problems + + This chapter describes the problems affecting usability of various + performance testing methodologies, mainly a binary search for + [RFC2544] unconditionally compliant throughput. + +2.1. Long Search Duration + + The emergence of software DUTs, with frequent software updates and a + number of different frame processing modes and configurations, has + increased both the number of performance tests required to verify the + DUT update and the frequency of running those tests. This makes the + overall test execution time even more important than before. + + The current [RFC2544] throughput definition restricts the potential + for time-efficiency improvements. A more generalized throughput + concept could enable further enhancements while maintaining the + precision of simpler methods. + + The bisection method, when unconditionally compliant with [RFC2544], + is excessively slow. This is because a significant amount of time is + spent on trials with loads that, in retrospect, are far from the + final determined throughput. + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 5] + +Internet-Draft MLRsearch July 2024 + + + [RFC2544] does not specify any stopping condition for throughput + search, so users already have an access to a limited trade-off + between search duration and achieved precision. However, each full + 60-second trials doubles the precision, so not many trials can be + removed without a substantial loss of precision. + +2.2. DUT in SUT + + [RFC2285] defines: - DUT as - The network forwarding device to which + stimulus is offered and response measured [RFC2285] (section 3.1.1). + - SUT as - The collective set of network devices to which stimulus is + offered as a single entity and response measured [RFC2285] (section + 3.1.2). + + [RFC2544] specifies a test setup with an external tester stimulating + the networking system, treating it either as a single DUT, or as a + system of devices, an SUT. + + In the case of software networking, the SUT consists of not only the + DUT as a software program processing frames, but also of server + hardware and operating system functions, with that server hardware + resources shared across all programs including the operating system. + + Given that the SUT is a shared multi-tenant environment encompassing + the DUT and other components, the DUT might inadvertently experience + interference from the operating system or other software operating on + the same server. + + Some of this interference can be mitigated. For instance, pinning + DUT program threads to specific CPU cores and isolating those cores + can prevent context switching. + + Despite taking all feasible precautions, some adverse effects may + still impact the DUT's network performance. In this document, these + effects are collectively referred to as SUT noise, even if the + effects are not as unpredictable as what other engineering + disciplines call noise. + + DUT can also exhibit fluctuating performance itself, for reasons not + related to the rest of SUT. For example due to pauses in execution + as needed for internal stateful processing. In many cases this may + be an expected per-design behavior, as it would be observable even in + a hypothetical scenario where all sources of SUT noise are + eliminated. Such behavior affects trial results in a way similar to + SUT noise. As the two phenomenons are hard to distinguish, in this + document the term 'noise' is used to encompass both the internal + performance fluctuations of the DUT and the genuine noise of the SUT. + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 6] + +Internet-Draft MLRsearch July 2024 + + + A simple model of SUT performance consists of an idealized noiseless + performance, and additional noise effects. For a specific SUT, the + noiseless performance is assumed to be constant, with all observed + performance variations being attributed to noise. The impact of the + noise can vary in time, sometimes wildly, even within a single trial. + The noise can sometimes be negligible, but frequently it lowers the + observed SUT performance as observed in trial results. + + In this model, SUT does not have a single performance value, it has a + spectrum. One end of the spectrum is the idealized noiseless + performance value, the other end can be called a noiseful + performance. In practice, trial result close to the noiseful end of + the spectrum happens only rarely. The worse the performance value + is, the more rarely it is seen in a trial. Therefore, the extreme + noiseful end of the SUT spectrum is not observable among trial + results. Also, the extreme noiseless end of the SUT spectrum is + unlikely to be observable, this time because some small noise effects + are likely to occur multiple times during a trial. + + Unless specified otherwise, this document's focus is on the + potentially observable ends of the SUT performance spectrum, as + opposed to the extreme ones. + + When focusing on the DUT, the benchmarking effort should ideally aim + to eliminate only the SUT noise from SUT measurements. However, this + is currently not feasible in practice, as there are no realistic + enough models available to distinguish SUT noise from DUT + fluctuations, based on authors' experience and available literature. + + Assuming a well-constructed SUT, the DUT is likely its primary + performance bottleneck. In this case, we can define the DUT's ideal + noiseless performance as the noiseless end of the SUT performance + spectrum, especially for throughput. However, other performance + metrics, such as latency, may require additional considerations. + + Note that by this definition, DUT noiseless performance also + minimizes the impact of DUT fluctuations, as much as realistically + possible for a given trial duration. + + MLRsearch methodology aims to solve the DUT in SUT problem by + estimating the noiseless end of the SUT performance spectrum using a + limited number of trial results. + + Any improvements to the throughput search algorithm, aimed at better + dealing with software networking SUT and DUT setup, should employ + strategies recognizing the presence of SUT noise, allowing the + discovery of (proxies for) DUT noiseless performance at different + levels of sensitivity to SUT noise. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 7] + +Internet-Draft MLRsearch July 2024 + + +2.3. Repeatability and Comparability + + [RFC2544] does not suggest to repeat throughput search. And from + just one discovered throughput value, it cannot be determined how + repeatable that value is. Poor repeatability then leads to poor + comparability, as different benchmarking teams may obtain varying + throughput values for the same SUT, exceeding the expected + differences from search precision. + + [RFC2544] throughput requirements (60 seconds trial and no tolerance + of a single frame loss) affect the throughput results in the + following way. The SUT behavior close to the noiseful end of its + performance spectrum consists of rare occasions of significantly low + performance, but the long trial duration makes those occasions not so + rare on the trial level. Therefore, the binary search results tend + to wander away from the noiseless end of SUT performance spectrum, + more frequently and more widely than short trials would, thus causing + poor throughput repeatability. + + The repeatability problem can be addressed by defining a search + procedure that identifies a consistent level of performance, even if + it does not meet the strict definition of throughput in [RFC2544]. + + According to the SUT performance spectrum model, better repeatability + will be at the noiseless end of the spectrum. Therefore, solutions + to the DUT in SUT problem will help also with the repeatability + problem. + + Conversely, any alteration to [RFC2544] throughput search that + improves repeatability should be considered as less dependent on the + SUT noise. + + An alternative option is to simply run a search multiple times, and + report some statistics (e.g. average and standard deviation). This + can be used for a subset of tests deemed more important, but it makes + the search duration problem even more pronounced. + +2.4. Throughput with Non-Zero Loss + + [RFC1242] (section 3.17 Throughput) defines throughput as: The + maximum rate at which none of the offered frames are dropped by the + device. + + Then, it says: Since even the loss of one frame in a data stream can + cause significant delays while waiting for the higher level protocols + to time out, it is useful to know the actual maximum data rate that + the device can support. + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 8] + +Internet-Draft MLRsearch July 2024 + + + However, many benchmarking teams accept a small, non-zero loss ratio + as the goal for their load search. + + Motivations are many: + + * Modern protocols tolerate frame loss better, compared to the time + when [RFC1242] and [RFC2544] were specified. + + * Trials nowadays send way more frames within the same duration, + increasing the chance of a small SUT performance fluctuation being + enough to cause frame loss. + + * Small bursts of frame loss caused by noise have otherwise smaller + impact on the average frame loss ratio observed in the trial, as + during other parts of the same trial the SUT may work more closely + to its noiseless performance, thus perhaps lowering the Trial Loss + Ratio below the Goal Loss Ratio value. + + * If an approximation of the SUT noise impact on the Trial Loss + Ratio is known, it can be set as the Goal Loss Ratio. + + Regardless of the validity of all similar motivations, support for + non-zero loss goals makes any search algorithm more user-friendly. + [RFC2544] throughput is not user-friendly in this regard. + + Furthermore, allowing users to specify multiple loss ratio values, + and enabling a single search to find all relevant bounds, + significantly enhances the usefulness of the search algorithm. + + Searching for multiple Search Goals also helps to describe the SUT + performance spectrum better than the result of a single Search Goal. + For example, the repeated wide gap between zero and non-zero loss + loads indicates the noise has a large impact on the observed + performance, which is not evident from a single goal load search + procedure result. + + It is easy to modify the vanilla bisection to find a lower bound for + the intended load that satisfies a non-zero Goal Loss Ratio. But it + is not that obvious how to search for multiple goals at once, hence + the support for multiple Search Goals remains a problem. + +2.5. Inconsistent Trial Results + + While performing throughput search by executing a sequence of + measurement trials, there is a risk of encountering inconsistencies + between trial results. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 9] + +Internet-Draft MLRsearch July 2024 + + + The plain bisection never encounters inconsistent trials. But + [RFC2544] hints about the possibility of inconsistent trial results, + in two places in its text. The first place is section 24, where full + trial durations are required, presumably because they can be + inconsistent with the results from short trial durations. The second + place is section 26.3, where two successive zero-loss trials are + recommended, presumably because after one zero-loss trial there can + be a subsequent inconsistent non-zero-loss trial. + + Examples include: + + * A trial at the same load (same or different trial duration) + results in a different Trial Loss Ratio. + + * A trial at a higher load (same or different trial duration) + results in a smaller Trial Loss Ratio. + + Any robust throughput search algorithm needs to decide how to + continue the search in the presence of such inconsistencies. + Definitions of throughput in [RFC1242] and [RFC2544] are not specific + enough to imply a unique way of handling such inconsistencies. + + Ideally, there will be a definition of a new quantity which both + generalizes throughput for non-zero-loss (and other possible + repeatability enhancements), while being precise enough to force a + specific way to resolve trial result inconsistencies. But until such + a definition is agreed upon, the correct way to handle inconsistent + trial results remains an open problem. + +3. MLRsearch Specification + + This section describes MLRsearch specification including all + technical definitions needed for evaluating whether a particular test + procedure complies with MLRsearch specification. + +3.1. Overview + + MLRsearch specification describes a set of abstract system + components, acting as functions with specified inputs and outputs. + + A test procedure is said to comply with MLRsearch specification if it + can be conceptually divided into analogous components, each + satisfying requirements for the corresponding MLRsearch component. + + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 10] + +Internet-Draft MLRsearch July 2024 + + + The Measurer component is tasked to perform trials, the Controller + component is tasked to select trial loads and durations, the Manager + component is tasked to pre-configure everything and to produce the + test report. The test report explicitly states Search Goals (as the + Controller Inputs) and corresponding Goal Results (Controller + Outputs). + + The Manager calls the Controller once, the Controller keeps calling + the Measurer until all stopping conditions are met. + + The part where Controller calls the Measurer is called the search. + Any activity done by the Manager before it calls the Controller (or + after Controller returns) is not considered to be part of the search. + + MLRsearch specification prescribes regular search results and + recommends their stopping conditions. Irregular search results are + also allowed, they may have different requirements and stopping + conditions. + + Search results are based on load classification. When measured + enough, any chosen load either achieves of fails each search goal, + thus becoming a lower or an upper bound for that goal. When the + relevant bounds are at loads that are close enough (according to goal + precision), the regular result is found. Search stops when all + regular results are found (or if some goals are proven to have only + irregular results). + +3.2. Measurement Quantities + + MLRsearch specification uses a number of measurement quantities. + + In general, MLRsearch specification does not require particular units + to be used, but it is REQUIRED for the test report to state all the + units. For example, ratio quantities can be dimensionless numbers + between zero and one, but may be expressed as percentages instead. + + For convenience, a group of quantities can be treated as a composite + quantity, One constituent of a composite quantity is called an + attribute, and a group of attribute values is called an instance of + that composite quantity. + + Some attributes are not independent from others, and they can be + calculated from other attributes. Such quantites are called derived + quantities. + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 11] + +Internet-Draft MLRsearch July 2024 + + +3.3. Existing Terms + + RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" + contains basic definitions, and RFC 2544 "Benchmarking Methodology + for Network Interconnect Devices" contains discussions of a number of + terms and additional methodology requirements. RFC 2285 adds more + terms and discussions, describing some known situations in more + precise way. + + All three documents should be consulted before attempting to make use + of this document. + + Definitions of some central terms are copied and discussed in + subsections. + +3.3.1. SUT + + Defined in [RFC2285] (section 3.1.2 System Under Test (SUT)) as + follows. + + Definition: + + The collective set of network devices to which stimulus is offered as + a single entity and response measured. + + Discussion: + + An SUT consisting of a single network device is also allowed. + +3.3.2. DUT + + Defined in [RFC2285] (section 3.1.1 Device Under Test (DUT)) as + follows. + + Definition: + + The network forwarding device to which stimulus is offered and + response measured. + + Discussion: + + DUT, as a sub-component of SUT, is only indirectly mentioned in + MLRsearch specification, but is of key relevance for its motivation. + +3.3.3. Trial + + A trial is the part of the test described in [RFC2544] (section 23. + Trial description). + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 12] + +Internet-Draft MLRsearch July 2024 + + + Definition: + + A particular test consists of multiple trials. Each trial returns + one piece of information, for example the loss rate at a particular + input frame rate. Each trial consists of a number of phases: + + a) If the DUT is a router, send the routing update to the "input" + port and pause two seconds to be sure that the routing has settled. + + b) Send the "learning frames" to the "output" port and wait 2 seconds + to be sure that the learning has settled. Bridge learning frames are + frames with source addresses that are the same as the destination + addresses used by the test frames. Learning frames for other + protocols are used to prime the address resolution tables in the DUT. + The formats of the learning frame that should be used are shown in + the Test Frame Formats document. + + c) Run the test trial. + + d) Wait for two seconds for any residual frames to be received. + + e) Wait for at least five seconds for the DUT to restabilize. + + Discussion: + + The definition describes some traits, it is not clear whether all of + them are REQUIRED, or some of them are only RECOMMENDED. + + For the purposes of the MLRsearch specification, it is ALLOWED for + the test procedure to deviate from the [RFC2544] description, but any + such deviation MUST be made explicit in the test report. + + Trials are the only stimuli the SUT is expected to experience during + the search. + + In some discussion paragraphs, it is useful to consider the traffic + as sent and received by a tester, as implicitly defined in [RFC2544] + (section 6. Test set up). + + An example of deviation from [RFC2544] is using shorter wait times. + +3.4. Trial Terms + + This section defines new and redefine existing terms for quantities + relevant as inputs or outputs of trial, as used by the Measurer + component. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 13] + +Internet-Draft MLRsearch July 2024 + + +3.4.1. Trial Duration + + Definition: + + Trial duration is the intended duration of the traffic for a trial. + + Discussion: + + In general, this quantity does not include any preparation nor + waiting described in section 23 of [RFC2544] (section 23. Trial + description). + + While any positive real value may be provided, some Measurer + implementations MAY limit possible values, e.g. by rounding down to + neared integer in seconds. In that case, it is RECOMMENDED to give + such inputs to the Controller so the Controller only proposes the + accepted values. Alternatively, the test report MUST present the + rounded values as Search Goal attributes. + +3.4.2. Trial Load + + Definition: + + The trial load is the intended load for a trial + + Discussion: + + For test report purposes, it is assumed that this is a constant load + by default. This MAY be only an average load, e.g. when the traffic + is intended to be busty, e.g. as suggested in [RFC2544] (section 21. + Bursty traffic), but the test report MUST explicitly mention how non- + constant the traffic is. + + Trial load is the quantity defined as Constant Load of [RFC1242] + (section 3.4 Constant Load), Data Rate of [RFC2544] (section 14. + Bidirectional traffic) and Intended Load of [RFC2285] (section 3.5.1 + Intended load (Iload)). All three definitions specify that this + value applies to one (input or output) interface. + + For test report purposes, multi-interface aggregate load MAY be + reported, this is understood as the same quantity expressed using + different units. From the report it MUST be clear whether a + particular trial load value is per one interface, or an aggregate + over all interfaces. + + Similarly to trial duration, some Measurers may limit the possible + values of trial load. Contrary to trial duration, the test report is + NOT REQUIRED to document such behavior. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 14] + +Internet-Draft MLRsearch July 2024 + + + It is ALLOWED to combine trial load and trial duration in a way that + would not be possible to achieve using any integer number of data + frames. + +3.4.3. Trial Input + + Definition: + + Trial Input is a composite quantity, consisting of two attributes: + trial duration and trial load. + + Discussion: + + When talking about multiple trials, it is common to say "Trial + Inputs" to denote all corresponding Trial Input instances. + + A Trial Input instance acts as the input for one call of the Measurer + component. + + Contrary to other composite quantities, MLRsearch implementations are + NOT ALLOWED to add optional attributes here. This improves + interoperability between various implementations of the Controller + and the Measurer. + +3.4.4. Traffic Profile + + Definition: + + Traffic profile is a composite quantity containing attributes other + than trial load and trial duration, needed for unique determination + of the trial to be performed. + + Discussion: + + All its attributes are assumed to be constant during the search, and + the composite is configured on the Measurer by the Manager before the + search starts. This is why the traffic profile is not part of the + Trial Input. + + As a consequence, implementations of the Manager and the Measurer + must be aware of their common set of capabilities, so that the + traffic profile uniquely defines the traffic during the search. The + important fact is that none of those capabilities have to be known by + the Controller implementations. + + The traffic profile SHOULD contain some specific quantities, for + example [RFC2544] (section 9. Frame sizes) governs data link frame + size as defined in [RFC1242] (section 3.5 Data link frame size). + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 15] + +Internet-Draft MLRsearch July 2024 + + + Several more specific quantities may be RECOMMENDED, depending on + media type. For example, [RFC2544] (Appendix C) lists frame formats + and protocol addresses, as recommended from [RFC2544] (section 8. + Frame formats) and [RFC2544] (section 12. Protocol addresses). + + Depending on SUT configuration, e.g. when testing specific protocols, + additional attributes MUST be included in the traffic profile and in + the test report. + + Example: [RFC8219] (section 5.3. Traffic Setup) introduces traffic + setups consisting of a mix of IPv4 and IPv6 traffic - the implied + traffic profile therefore must include an attribute for their + percentage. + + Other traffic properties that need to be somehow specified in Traffic + Profile include: [RFC2544] (section 14. Bidirectional traffic), + [RFC2285] (section 3.3.3 Fully meshed traffic), and [RFC2544] + (section 11. Modifiers). + +3.4.5. Trial Forwarding Ratio + + Definition: + + The trial forwarding ratio is a dimensionless floating point value. + It MUST range between 0.0 and 1.0, both inclusive. It is calculated + by dividing the number of frames successfully forwarded by the SUT by + the total number of frames expected to be forwarded during the trial + + Discussion: + + For most traffic profiles, "expected to be forwarded" means "intended + to get transmitted from Tester towards SUT". + + Trial forwarding ratio MAY be expressed in other units (e.g. as a + percentage) in the test report. + + Note that, contrary to loads, frame counts used to compute trial + forwarding ratio are aggregates over all SUT output interfaces. + + Questions around what is the correct number of frames that should + have been forwarded is generally outside of the scope of this + document. + +3.4.6. Trial Loss Ratio + + Definition: + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 16] + +Internet-Draft MLRsearch July 2024 + + + The Trial Loss Ratio is equal to one minus the trial forwarding + ratio. + + Discussion: + + 100% minus the trial forwarding ratio, when expressed as a + percentage. + + This is almost identical to Frame Loss Rate of [RFC1242] (section 3.6 + Frame Loss Rate), the only minor difference is that Trial Loss Ratio + does not need to be expressed as a percentage. + +3.4.7. Trial Forwarding Rate + + Definition: + + The trial forwarding rate is a derived quantity, calculated by + multiplying the trial load by the trial forwarding ratio. + + Discussion: + + It is important to note that while similar, this quantity is not + identical to the Forwarding Rate as defined in [RFC2285] (section + 3.6.1 Forwarding rate (FR)). The latter is specific to one output + interface only, whereas the trial forwarding ratio is based on frame + counts aggregated over all SUT output interfaces. + +3.4.8. Trial Effective Duration + + Definition: + + Trial effective duration is a time quantity related to the trial, by + default equal to the trial duration. + + Discussion: + + This is an optional feature. If the Measurer does not return any + trial effective duration value, the Controller MUST use the trial + duration value instead. + + Trial effective duration may be any time quantity chosen by the + Measurer to be used for time-based decisions in the Controller. + + The test report MUST explain how the Measurer computes the returned + trial effective duration values, if they are not always equal to the + trial duration. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 17] + +Internet-Draft MLRsearch July 2024 + + + This feature can be beneficial for users who wish to manage the + overall search duration, rather than solely the traffic portion of + it. Simply measure the duration of the whole trial (waits including) + and use that as the trial effective duration. + + Also, this is a way for the Measurer to inform the Controller about + its surprising behavior, for example when rounding the trial duration + value. + +3.4.9. Trial Output + + Definition: + + Trial Output is a composite quantity. The REQUIRED attributes are + Trial Loss Ratio, trial effective duration and trial forwarding rate. + + Discussion: + + When talking about multiple trials, it is common to say "Trial + Outputs" to denote all corresponding Trial Output instances. + + Implementations may provide additional (optional) attributes. The + Controller implementations MUST ignore values of any optional + attribute they are not familiar with, except when passing Trial + Output instance to the Manager. + + Example of an optional attribute: The aggregate number of frames + expected to be forwarded during the trial, especially if it is not + just (a rounded-up value) implied by trial load and trial duration. + + While [RFC2285] (Section 3.5.2 Offered load (Oload)) requires the + offered load value to be reported for forwarding rate measurements, + it is NOT REQUIRED in MLRsearch specification. + +3.4.10. Trial Result + + Definition: + + Trial result is a composite quantity, consisting of the Trial Input + and the Trial Output. + + Discussion: + + When talking about multiple trials, it is common to say "trial + results" to denote all corresponding trial result instances. + + While implementations SHOULD NOT include additional attributes with + independent values, they MAY include derived quantities. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 18] + +Internet-Draft MLRsearch July 2024 + + +3.5. Goal Terms + + This section defines new and redefine existing terms for quantities + indirectly relevant for inputs or outputs of the Controller + component. + + Several goal attributes are defined before introducing the main + component quantity: the Search Goal. + +3.5.1. Goal Final Trial Duration + + Definition: + + A threshold value for trial durations. + + Discussion: + + This attribute value MUST be positive. + + A trial with Trial Duration at least as long as the Goal Final Trial + Duration is called a full-length trial (with respect to the given + Search Goal). + + A trial that is not full-length is called a short trial. + + Informally, while MLRsearch is allowed to perform short trials, the + results from such short trials have only limited impact on search + results. + + One trial may be full-length for some Search Goals, but not for + others. + + The full relation of this goal to Controller Output is defined later + in this document in subsections of [Goal Result] (#Goal-Result). For + example, the Conditional Throughput for this goal is computed only + from full-length trial results. + +3.5.2. Goal Duration Sum + + Definition: + + A threshold value for a particular sum of trial effective durations. + + Discussion: + + This attribute value MUST be positive. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 19] + +Internet-Draft MLRsearch July 2024 + + + Informally, even when looking only at full-length trials, MLRsearch + may spend up to this time measuring the same load value. + + If the Goal Duration Sum is larger than the Goal Final Trial + Duration, multiple full-length trials may need to be performed at the + same load. + + See [TST009 Example] (#TST009-Example) for an example where + possibility of multiple full-length trials at the same load is + intended. + + A Goal Duration Sum value lower than the Goal Final Trial Duration + (of the same goal) could save some search time, but is NOT + RECOMMENDED. See [Relevant Upper Bound] (#Relevant-Upper-Bound) for + partial explanation. + +3.5.3. Goal Loss Ratio + + Definition: + + A threshold value for Trial Loss Ratios. + + Discussion: + + Attribute value MUST be non-negative and smaller than one. + + A trial with Trial Loss Ratio larger than a Goal Loss Ratio value is + called a lossy trial, with respect to given Search Goal. + + Informally, if a load causes too many lossy trials, the Relevant + Lower Bound for this goal will be smaller than that load. + + If a trial is not lossy, it is called a low-loss trial, or + (specifically for zero Goal Loss Ratio value) zero-loss trial. + +3.5.4. Goal Exceed Ratio + + Definition: + + A threshold value for a particular ratio of sums of Trial Effective + Durations. + + Discussion: + + Attribute value MUST be non-negative and smaller than one. + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 20] + +Internet-Draft MLRsearch July 2024 + + + See later sections for details on which sums. Specifically, the + direct usage is only in [Appendix A: Load Classification] (#Appendix- + A:-Load-Classification) and [Appendix B: Conditional Throughput] + (#Appendix-B:-Conditional-Throughput). The impact of that usage is + discussed in subsections leading to [Goal Result] (#Goal-Result). + + Informally, the impact of lossy trials is controlled by this value. + Effectively, Goal Exceed Ratio is a percentage of full-length trials + that may be lossy without the load being classified as the [Relevant + Upper Bound] (#Relevant-Upper-Bound). + +3.5.5. Goal Width + + Definition: + + A value used as a threshold for deciding whether two trial load + values are close enough. + + Discussion: + + If present, the value MUST be positive. + + Informally, this acts as a stopping condition, controlling the + precision of the search. The search stops if every goal has reached + its precision. + + Implementations without this attribute MUST give the Controller other + ways to control the search stopping conditions. + + Absolute load difference and relative load difference are two popular + choices, but implementations may choose a different way to specify + width. + + The test report MUST make it clear what specific quantity is used as + Goal Width. + + It is RECOMMENDED to set the Goal Width (as relative difference) + value to a value no smaller than the Goal Loss Ratio. (The reason is + not obvious, see [Throughput] (#Throughput) if interested.) + +3.5.6. Search Goal + + Definition: + + The Search Goal is a composite quantity consisting of several + attributes, some of them are required. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 21] + +Internet-Draft MLRsearch July 2024 + + + Required attributes: - Goal Final Trial Duration - Goal Duration Sum + - Goal Loss Ratio - Goal Exceed Ratio + + Optional attribute: - Goal Width + + Discussion: + + Implementations MAY add their own attributes. Those additional + attributes may be required by the implementation even if they are not + required by MLRsearch specification. But it is RECOMMENDED for those + implementations to support missing values by computing reasonable + defaults. + + The meaning of listed attributes is formally given only by their + indirect effect on the search results. + + Informally, later sections provide additional intuitions and examples + of the Search Goal attribute values. + + An example of additional attributes required by some implementations + is Goal Initial Trial Duration, together with another attribute that + controls possible intermediate Trial Duration values. The reasonable + default in this case is using the Goal Final Trial Duration and no + intermediate values. + +3.5.7. Controller Input + + Definition: + + Controller Input is a composite quantity required as an input for the + Controller. The only REQUIRED attribute is a list of Search Goal + instances. + + Discussion: + + MLRsearch implementations MAY use additional attributes. Those + additional attributes may be required by the implementation even if + they are not required by MLRsearch specification. + + Formally, the Manager does not apply any Controller configuration + apart from one Controller Input instance. + + For example, Traffic Profile is configured on the Measurer by the + Manager (without explicit assistance of the Controller). + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 22] + +Internet-Draft MLRsearch July 2024 + + + The order of Search Goal instances in a list SHOULD NOT have a big + impact on Controller Output (see section [Controller Output] + (#Controller-Output) , but MLRsearch implementations MAY base their + behavior on the order of Search Goal instances in a list. + + An example of an optional attribute (outside the list of Search + Goals) required by some implementations is Max Load. While this is a + frequently used configuration parameter, already governed by + [RFC2544] (section 20. Maximum frame rate) and [RFC2285] (3.5.3 + Maximum offered load (MOL)), some implementations may detect or + discover it instead. + + In MLRsearch specification, the [Relevant Upper Bound] (#Relevant- + Upper-Bound) is added as a required attribute precisely because it + makes the search result independent of Max Load value. + +3.6. Search Goal Examples + +3.6.1. RFC2544 Goal + + The following set of values makes the search result unconditionally + compliant with [RFC2544] (section 24 Trial duration) + + * Goal Final Trial Duration = 60 seconds + + * Goal Duration Sum = 60 seconds + + * Goal Loss Ratio = 0% + + * Goal Exceed Ratio = 0% + + The latter two attributes are enough to make the search goal + conditionally compliant, adding the first attribute makes it + unconditionally compliant. + + The second attribute (Goal Duration Sum) only prevents MLRsearch from + repeating zero-loss full-length trials. + + Non-zero exceed ratio could prolong the search and allow loss + inversion between lower-load lossy short trial and higher-load full- + length zero-loss trial. From [RFC2544] alone, it is not clear + whether that higher load could be considered as compliant throughput. + + + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 23] + +Internet-Draft MLRsearch July 2024 + + +3.6.2. TST009 Goal + + One of the alternatives to RFC2544 is described in [TST009] (section + 12.3.3 Binary search with loss verification). The idea there is to + repeat lossy trials, hoping for zero loss on second try, so the + results are closer to the noiseless end of performance sprectum, and + more repeatable and comparable. + + Only the variant with "z = infinity" is achievable with MLRsearch. + + For example, for "r = 2" variant, the following search goal should be + used: + + * Goal Final Trial Duration = 60 seconds + + * Goal Duration Sum = 120 seconds + + * Goal Loss Ratio = 0% + + * Goal Exceed Ratio = 50% + + If the first 60s trial has zero loss, it is enough for MLRsearch to + stop measuring at that load, as even a second lossy trial would still + fit within the exceed ratio. + + But if the first trial is lossy, MLRsearch needs to perform also the + second trial to classify that load. As Goal Duration Sum is twice as + long as Goal Final Trial Duration, third full-length trial is never + needed. + +3.7. Result Terms + + Before defining the output of the Controller, it is useful to define + what the Goal Result is. + + The Goal Result is a composite quantity. + + Following subsections define its attribute first, before describing + the Goal Result quantity. + + There is a correspondence between Search Goals and Goal Results. + Most of the following subsections refer to a given Search Goal, when + defining attributes of the Goal Result. Conversely, at the end of + the search, each Search Goal has its corresponding Goal Result. + + Conceptually, the search can be seen as a process of load + classification, where the Controller attempts to classify some loads + as an Upper Bound or a Lower Bound with respect to some Search Goal. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 24] + +Internet-Draft MLRsearch July 2024 + + + Before defining real attributes of the goal result, it is useful to + define bounds in general. + +3.7.1. Relevant Upper Bound + + Definition: + + The Relevant Upper Bound is the smallest trial load value that is + classified at the end of the search as an upper bound (see + [Appendix A: Load Classification] (#Appendix-A:-Load-Classification)) + for the given Search Goal. + + Discussion: + + One search goal can have many different load classified as an upper + bound. At the end of the search, one of those loads will be the + smallest, becoming the relevant upper bound for that goal. + + In more detail, the set of all trial outputs (both short and full- + length, enough of them according to Goal Duration Sum) performed at + that smallest load failed to uphold all the requirements of the given + Search Goal, mainly the Goal Loss Ratio in combination with the Goal + Exceed Ratio. + + If Max Load does not cause enough lossy trials, the Relevant Upper + Bound does not exist. Conversely, if Relevant Upper Bound exists, it + is not affected by Max Load value. + +3.7.2. Relevant Lower Bound + + Definition: + + The Relevant Lower Bound is the largest trial load value among those + smaller than the Relevant Upper Bound, that got classified at the end + of the search as a lower bound (see [Appendix A: Load Classification] + (#Appendix-A:-Load-Classification)) for the given Search Goal. + + Discussion: + + Only among loads smaller that the relevant upper bound, the largest + load becomes the relevant lower bound. With loss inversion, stricter + upper bound matters. + + In more detail, the set of all trial outputs (both short and full- + length, enough of them according to Goal Duration Sum) performed at + that largest load managed to uphold all the requirements of the given + Search Goal, mainly the Goal Loss Ratio in combination with the Goal + Exceed Ratio. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 25] + +Internet-Draft MLRsearch July 2024 + + + Is no load had enough low-loss trials, the relevant lower bound MAY + not exist. + + Strictly speaking, if the Relevant Upper Bound does not exist, the + Relevant Lower Bound also does not exist. In that case, Max Load is + classified as a lower bound, but it is not clear whether a higher + lower bound would be found if the search used a higher Max Load + value. + + For a regular Goal Result, the distance between the Relevant Lower + Bound and the Relevant Upper Bound MUST NOT be larger than the Goal + Width, if the implementation offers width as a goal attribute. + + Searching for anther search goal may cause a loss inversion + phenomenon, where a lower load is classified as an upper bound, but + also a higher load is classified as a lower bound for the same search + goal. The definition of the Relevant Lower Bound ignores such high + lower bounds. + +3.7.3. Conditional Throughput + + Definition: + + The Conditional Throughput (see section [Appendix B: Conditional + Throughput] (#Appendix-B:-Conditional-Throughput)) as evaluated at + the Relevant Lower Bound of the given Search Goal at the end of the + search. + + Discussion: + + Informally, this is a typical trial forwarding rate, expected to be + seen at the Relevant Lower Bound of the given Search Goal. + + But frequently it is only a conservative estimate thereof, as + MLRsearch implementations tend to stop gathering more data as soon as + they confirm the value cannot get worse than this estimate within the + Goal Duration Sum. + + This value is RECOMMENDED to be used when evaluating repeatability + and comparability if different MLRsearch implementations. + +3.7.4. Goal Result + + Definition: + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 26] + +Internet-Draft MLRsearch July 2024 + + + The Goal Result is a composite quantity consisting of several + attributes. Relevant Upper Bound and Relevant Lower Bound are + REQUIRED attributes, Conditional Throughput is a RECOMMENDED + attribute. + + Discussion: + + Depending on SUT behavior, it is possible that one or both relevant + bounds do not exist. The goal result instance where the required + attribute values exist is informally called a Regular Goal Result + instance, so we can say some goals reached Irregular Goal Results. + + A typical Irregular Goal Result is when all trials at the Max Load + have zero loss, as the Relevant Upper Bound does not exist in that + case. + + It is RECOMMENDED that the test report will display such results + appropriately, although MLRsearch specification does not prescibe + how. + + Anything else regarging Irregular Goal Results, including their role + in stopping conditions of the search is outside the scope of this + document. + +3.7.5. Search Result + + Definition: + + The Search Result is a single composite object that maps each Search + Goal instance to a corresponding Goal Result instance. + + Discussion: + + Alternatively, the Search Result can be implemented as an ordered + list of the Goal Result instances, matching the order of Search Goal + instances. + + The Search Result (as a mapping) MUST map from all the Search Goal + instances present in the Controller Input. + +3.7.6. Controller Output + + Definition: + + The Controller Output is a composite quantity returned from the + Controller to the Manager at the end of the search. The Search + Result instance is its only REQUIRED attribute. + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 27] + +Internet-Draft MLRsearch July 2024 + + + Discussion: + + MLRsearch implementation MAY return additional data in the Controller + Output. + +3.8. MLRsearch Architecture + + MLRsearch architecture consists of three main system components: the + Manager, the Controller, and the Measurer. + + The architecture also implies the presence of other components, such + as the SUT and the Tester (as a sub-component of the Measurer). + + Protocols of communication between components are generally left + unspecified. For example, when MLRsearch specification mentions + "Controller calls Measurer", it is possible that the Controller + notifies the Manager to call the Measurer indirectly instead. This + way the Measurer implementations can be fully independent from the + Controller implementations, e.g. programmed in different programming + languages. + +3.8.1. Measurer + + Definition: + + The Measurer is an abstract system component that when called with a + [Trial Input] (#Trial-Input) instance, performs one [Trial] (#Trial), + and returns a [Trial Output] (#Trial-Output) instance. + + Discussion: + + This definition assumes the Measurer is already initialized. In + practice, there may be additional steps before the search, e.g. when + the Manager configures the traffic profile (either on the Measurer or + on its tester sub-component directly) and performs a warmup (if the + tester requires one). + + It is the responsibility of the Measurer implementation to uphold any + requirements and assumptions present in MLRsearch specification, e.g. + trial forwarding ratio not being larger than one. + + Implementers have some freedom. For example [RFC2544] (section 10. + Verifying received frames) gives some suggestions (but not + requirements) related to duplicated or reordered frames. + Implementations are RECOMMENDED to document their behavior related to + such freedoms in as detailed a way as possible. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 28] + +Internet-Draft MLRsearch July 2024 + + + It is RECOMMENDED to benchmark the test equipment first, e.g. connect + sender and receiver directly (without any SUT in the path), find a + load value that guarantees the offered load is not too far from the + intended load, and use that value as the Max Load value. When + testing the real SUT, it is RECOMMENDED to turn any big difference + between the intended load and the offered load into increased Trial + Loss Ratio. + + Neither of the two recommendations are made into requirements, + because it is not easy to tell when the difference is big enough, in + a way thay would be dis-entangled from other Measurer freedoms. + +3.8.2. Controller + + Definition: + + The Controller is an abstract system component that when called with + a Controller Input instance repeatedly computes Trial Input instance + for the Measurer, obtains corresponding Trial Output instances, and + eventually returns a Controller Output instance. + + Discussion: + + Informally, the Controller has big freedom in selection of Trial + Inputs, and the implementations want to achieve the Search Goals in + the shortest expected time. + + The Controller's role in optimizing the overall search time + distinguishes MLRsearch algorithms from simpler search procedures. + + Informally, each implementation can have different stopping + conditions. Goal Width is only one example. In practice, + implementation details do not matter, as long as Goal Results are + regular. + +3.8.3. Manager + + Definition: + + The Manager is an abstract system component that is reponsible for + configuring other components, calling the Controller component once, + and for creating the test report following the reporting format as + defined in [RFC2544] (section 26. Benchmarking tests). + + Discussion: + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 29] + +Internet-Draft MLRsearch July 2024 + + + The Manager initializes the SUT, the Measurer (and the Tester if + independent) with their intended configurations before calling the + Controller. + + The Manager does not need to be able to tweak any Search Goal + attributes, but it MUST report all applied attribute values even if + not tweaked. + + In principle, there should be a "user" (human or CI) that "starts" or + "calls" the Manager and receives the report. The Manager MAY be able + to be called more than once whis way. + +3.9. Implementation Compliance + + Any networking measurement setup where there can be logically + delineated system components and there are components satisfying + requirements for the Measurer, the Controller and the Manager, is + considered to be compliant with MLRsearch design. + + These components can be seen as abstractions present in any testing + procedure. For example, there can be a single component acting both + as the Manager and the Controller, but as long as values of required + attributes of Search Goals and Goal Results are visible in the test + report, the Controller Input instance and output instance are + implied. + + For example, any setup for conditionally (or unconditionally) + compliant [RFC2544] throughput testing can be understood as a + MLRsearch architecture, assuming there is enough data to reconstruct + the Relevant Upper Bound. + + See [RFC2544 Goal] (#RFC2544-Goal) subsection for equivalent Search + Goal. + + Any test procedure that can be understood as (one call to the Manager + of) MLRsearch architecture is said to be compliant with MLRsearch + specification. + +4. Additional Considerations + + This section focuses on additional considerations, intuitions and + motivations pertaining to MLRsearch methodology. + + + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 30] + +Internet-Draft MLRsearch July 2024 + + +4.1. MLRsearch Versions + + The MLRsearch algorithm has been developed in a code-first approach, + a Python library has been created, debugged, used in production and + published in PyPI before the first descriptions (even informal) were + published. + + But the code (and hence the description) was evolving over time. + Multiple versions of the library were used over past several years, + and later code was usually not compatible with earlier descriptions. + + The code in (some version of) MLRsearch library fully determines the + search process (for a given set of configuration parameters), leaving + no space for deviations. + + This historic meaning of MLRsearch, as a family of search algorithm + implementations, leaves plenty of space for future improvements, at + the cost of poor comparability of results of search algoritm + implementations. + + There are two competing needs. There is the need for standardization + in areas critical to comparability. There is also the need to allow + flexibility for implementations to innovate and improve in other + areas. This document defines MLRsearch as a new specification in a + manner that aims to fairly balance both needs. + +4.2. Stopping Conditions + + [RFC2544] prescribes that after performing one trial at a specific + offered load, the next offered load should be larger or smaller, + based on frame loss. + + The usual implementation uses binary search. Here a lossy trial + becomes a new upper bound, a lossless trial becomes a new lower + bound. The span of values between the tightest lower bound and the + tightest upper bound (including both values) forms an interval of + possible results, and after each trial the width of that interval + halves. + + Usually the binary search implementation tracks only the two tightest + bounds, simply calling them bounds. But the old values still remain + valid bounds, just not as tight as the new ones. + + After some number of trials, the tightest lower bound becomes the + throughput. [RFC2544] does not specify when, if ever, should the + search stop. + + MLRsearch introduces a concept of [Goal Width] (#Goal-Width). + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 31] + +Internet-Draft MLRsearch July 2024 + + + The search stops when the distance between the tightest upper bound + and the tightest lower bound is smaller than a user-configured value, + called Goal Width from now on. In other words, the interval width at + the end of the search has to be no larger than the Goal Width. + + This Goal Width value therefore determines the precision of the + result. Due to the fact that MLRsearch specification requires a + particular structure of the result (see [Trial Result] (#Trial- + Result) section), the result itself does contain enough information + to determine its precision, thus it is not required to report the + Goal Width value. + + This allows MLRsearch implementations to use stopping conditions + different from Goal Width. + +4.3. Load Classification + + MLRsearch keeps the basic logic of binary search (tracking tightest + bounds, measuring at the middle), perhaps with minor technical + differences. + + MLRsearch algorithm chooses an intended load (as opposed to the + offered load), the interval between bounds does not need to be split + exactly into two equal halves, and the final reported structure + specifies both bounds. + + The biggest difference is that to classify a load as an upper or + lower bound, MLRsearch may need more than one trial (depending on + configuration options) to be performed at the same intended load. + + In consequence, even if a load already does have few trial results, + it still may be classified as undecided, neither a lower bound nor an + upper bound. + + An explanation of the classification logic is given in the next + section [Logic of Load Classification] (#Logic-of-Load- + Classification), as it heavily relies on other subsections of this + section. + + For repeatability and comparability reasons, it is important that + given a set of trial results, all implementations of MLRsearch + classify the load equivalently. + +4.4. Loss Ratios + + Another difference between MLRsearch and [RFC2544] binary search is + in the goals of the search. [RFC2544] has a single goal, based on + classifying full-length trials as either lossless or lossy. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 32] + +Internet-Draft MLRsearch July 2024 + + + MLRsearch, as the name suggests, can search for multiple goals, + differing in their loss ratios. The precise definition of the Goal + Loss Ratio will be given later. The [RFC2544] throughput goal then + simply becomes a zero Goal Loss Ratio. Different goals also may have + different Goal Widths. + + A set of trial results for one specific intended load value can + classify the load as an upper bound for some goals, but a lower bound + for some other goals, and undecided for the rest of the goals. + + Therefore, the load classification depends not only on trial results, + but also on the goal. The overall search procedure becomes more + complicated, when compared to binary search with a single goal, but + most of the complications do not affect the final result, except for + one phenomenon, loss inversion. + +4.5. Loss Inversion + + In [RFC2544] throughput search using bisection, any load with a lossy + trial becomes a hard upper bound, meaning every subsequent trial has + a smaller intended load. + + But in MLRsearch, a load that is classified as an upper bound for one + goal may still be a lower bound for another goal, and due to the + other goal MLRsearch will probably perform trials at even higher + loads. What to do when all such higher load trials happen to have + zero loss? Does it mean the earlier upper bound was not real? Does + it mean the later lossless trials are not considered a lower bound? + Surely we do not want to have an upper bound at a load smaller than a + lower bound. + + MLRsearch is conservative in these situations. The upper bound is + considered real, and the lossless trials at higher loads are + considered to be a coincidence, at least when computing the final + result. + + This is formalized using new notions, the [Relevant Upper Bound] + (#Relevant-Upper-Bound) and the [Relevant Lower Bound] (#Relevant- + Lower-Bound). Load classification is still based just on the set of + trial results at a given intended load (trials at other loads are + ignored), making it possible to have a lower load classified as an + upper bound, and a higher load classified as a lower bound (for the + same goal). The Relevant Upper Bound (for a goal) is the smallest + load classified as an upper bound. But the Relevant Lower Bound is + not simply the largest among lower bounds. It is the largest load + among loads that are lower bounds while also being smaller than the + Relevant Upper Bound. + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 33] + +Internet-Draft MLRsearch July 2024 + + + With these definitions, the Relevant Lower Bound is always smaller + than the Relevant Upper Bound (if both exist), and the two relevant + bounds are used analogously as the two tightest bounds in the binary + search. When they are less than the Goal Width apart, the relevant + bounds are used in the output. + + One consequence is that every trial result can have an impact on the + search result. That means if your SUT (or your traffic generator) + needs a warmup, be sure to warm it up before starting the search. + +4.6. Exceed Ratio + + The idea of performing multiple trials at the same load comes from a + model where some trial results (those with high loss) are affected by + infrequent effects, causing poor repeatability of [RFC2544] + throughput results. See the discussion about noiseful and noiseless + ends of the SUT performance spectrum in section [DUT in SUT] (#DUT- + in-SUT). Stable results are closer to the noiseless end of the SUT + performance spectrum, so MLRsearch may need to allow some frequency + of high-loss trials to ignore the rare but big effects near the + noiseful end. + + MLRsearch can do such trial result filtering, but it needs a + configuration option to tell it how frequent can the infrequent big + loss be. This option is called the exceed ratio. It tells MLRsearch + what ratio of trials (more exactly what ratio of trial seconds) can + have a [Trial Loss Ratio] (#Trial-Loss-Ratio) larger than the Goal + Loss Ratio and still be classified as a lower bound. Zero exceed + ratio means all trials have to have a Trial Loss Ratio equal to or + smaller than the Goal Loss Ratio. + + For explainability reasons, the RECOMMENDED value for exceed ratio is + 0.5, as it simplifies some later concepts by relating them to the + concept of median. + +4.7. Duration Sum + + When more than one trial is intended to classify a load, MLRsearch + also needs something that controls the number of trials needed. + Therefore, each goal also has an attribute called duration sum. + + The meaning of a [Goal Duration Sum] (#Goal-Duration-Sum) is that + when a load has (full-length) trials whose trial durations when + summed up give a value at least as big as the Goal Duration Sum + value, the load is guaranteed to be classified either as an upper + bound or a lower bound for that goal. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 34] + +Internet-Draft MLRsearch July 2024 + + + Due to the fact that the duration sum has a big impact on the overall + search duration, and [RFC2544] prescribes wait intervals around trial + traffic, the MLRsearch algorithm is allowed to sum durations that are + different from the actual trial traffic durations. + + In the MLRsearch specification, the different duration values are + called [Trial Effective Duration] (#Trial-Effective-Duration). + +4.8. Short Trials + + MLRsearch requires each goal to specify its final trial duration. + Full-length trial is a shorter name for a trial whose intended trial + duration is equal to (or longer than) the goal final trial duration. + + Section 24 of [RFC2544] already anticipates possible time savings + when short trials (shorter than full-length trials) are used. Full- + length trials are the opposite of short trials, so they may also be + called long trials. + + Any MLRsearch implementation may include its own configuration + options which control when and how MLRsearch chooses to use short + trial durations. + + For explainability reasons, when exceed ratio of 0.5 is used, it is + recommended for the Goal Duration Sum to be an odd multiple of the + full trial durations, so Conditional Throughput becomes identical to + a median of a particular set of trial forwarding rates. + + The presence of short trial results complicates the load + classification logic. + + Full details are given later in section [Logic of Load + Classification] (#Logic-of-Load-Classification). In a nutshell, + results from short trials may cause a load to be classified as an + upper bound. This may cause loss inversion, and thus lower the + Relevant Lower Bound, below what would classification say when + considering full-length trials only. + +4.9. Throughput + + Due to the fact that testing equipment takes the intended load as an + input parameter for a trial measurement, any load search algorithm + needs to deal with intended load values internally. + + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 35] + +Internet-Draft MLRsearch July 2024 + + + But in the presence of goals with a non-zero loss ratio, the intended + load usually does not match the user's intuition of what a throughput + is. The forwarding rate (as defined in [RFC2285] section 3.6.1) is + better, but it is not obvious how to generalize it for loads with + multiple trial results and a non-zero [Goal Loss Ratio] (#Goal-Loss- + Ratio). + + The best example is also the main motivation: hard limit performance. + Even if the medium allows higher performance, the SUT interfaces may + have their additional own limitations, e.g. a specific fps limit on + the NIC (a very common occurance). + + Ideally, those should be known and used when computing Max Load. But + if Max Load is higher that what interface can receive or transmit, + there will be a "hard limit" observed in trial results. Imagine the + hard limit is at 100 Mfps, Max Load is higher, and the goal loss + ratio is 0.5%. If DUT has no additional losses, 0.5% loss ratio will + be achieved at 100.5025 Mfps (the relevant lower bound). But it is + not intuitive to report SUT performance as a value that is larger + than known hard limit. We need a generalization of RFC2544 + throughput, different from just the relevant lower bound. + + MLRsearch defines one such generalization, called the Conditional + Throughput. It is the trial forwarding rate from one of the trials + performed at the load in question. Determining which trial exactly + is defined in [MLRsearch Specification] (#MLRsearch-Specification), + and in [Appendix B: Conditional Throughput] (#Appendix-B:- + Conditional-Throughput). + + In the hard limit example, 100.5 Mfps load will still have only 100.0 + Mfps forwarding rate, nicely confirming the known limitation. + + Conditional Throughput is partially related to load classification. + If a load is classified as a lower bound for a goal, the Conditional + Throughput can be calculated from trial results, and guaranteed to + show an loss ratio no larger than the Goal Loss Ratio. + + Note that when comparing the best (all zero loss) and worst case (all + loss just below Goal Loss Ratio), the same Relevant Lower Bound value + may result in the Conditional Throughput differing up to the Goal + Loss Ratio. + + + + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 36] + +Internet-Draft MLRsearch July 2024 + + + Therefore it is rarely needed to set the Goal Width (if expressed as + the relative difference of loads) below the Goal Loss Ratio. In + other words, setting the Goal Width below the Goal Loss Ratio may + cause the Conditional Throughput for a larger loss ratio to become + smaller than a Conditional Throughput for a goal with a smaller Goal + Loss Ratio, which is counter-intuitive, considering they come from + the same search. Therefore it is RECOMMENDED to set the Goal Width + to a value no smaller than the Goal Loss Ratio. + + Overall, this Conditional Throughput does behave well for + comparability purposes. + +4.10. Search Time + + MLRsearch was primarily developed to reduce the time required to + determine a throughput, either the [RFC2544] compliant one, or some + generalization thereof. The art of achieving short search times is + mainly in the smart selection of intended loads (and intended + durations) for the next trial to perform. + + While there is an indirect impact of the load selection on the + reported values, in practice such impact tends to be small, even for + SUTs with quite a broad performance spectrum. + + A typical example of two approaches to load selection leading to + different Relevant Lower Bounds is when the interval is split in a + very uneven way. Any implementation choosing loads very close to the + current Relevant Lower Bound is quite likely to eventually stumble + upon a trial result with poor performance (due to SUT noise). For an + implementation choosing loads very close to the current Relevant + Upper Bound, this is unlikely, as it examines more loads that can see + a performance close to the noiseless end of the SUT performance + spectrum. + + However, as even splits optimize search duration at give precision, + MLRsearch implementations that prioritize minimizing search time are + unlikely to suffer from any such bias. + + Therefore, this document remains quite vague on load selection and + other optimization details, and configuration attributes related to + them. Assuming users prefer libraries that achieve short overall + search time, the definition of the Relevant Lower Bound should be + strict enough to ensure result repeatability and comparability + between different implementations, while not restricting future + implementations much. + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 37] + +Internet-Draft MLRsearch July 2024 + + +4.11. [RFC2544] Compliance + + Some Search Goal instances lead to results compliant with RFC2544. + See [RFC2544 Goal] (#RFC2544-Goal) for more details regarding both + conditional and unconditional compliance. + + The presence of other Search Goals does not affect the compliance of + this Goal Result. The Relevant Lower Bound and the Conditional + Throughput are in this case equal to each other, and the value is the + [RFC2544] throughput. + +5. Logic of Load Classification + +5.1. Introductory Remarks + + This chapter continues with explanations, but this time more precise + definitions are needed for readers to follow the explanations. + + Descriptions in this section are wordy and implementers should read + [MLRsearch Specification] (#MLRsearch-Specification) section and + Appendices for more concise definitions. + + The two areas of focus here are load classification and the + Conditional Throughput. + + To start with [Performance Spectrum] (#Performance-Spectrum) + subsection contains definitions needed to gain insight into what + Conditional Throughput means. Remaining subsections discuss load + classification. + + For load classification, it is useful to define *good trials* and + *bad trials*: + + * *Bad trial*: Trial is called bad (according to a goal) if its + [Trial Loss Ratio] (#Trial-Loss-Ratio) is larger than the [Goal + Loss Ratio] (#Goal-Loss-Ratio). + + * *Good trial*: Trial that is not bad is called good. + +5.2. Performance Spectrum + + ### Description + + There are several equivalent ways to explain the Conditional + Throughput computation. One of the ways relies on performance + spectrum. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 38] + +Internet-Draft MLRsearch July 2024 + + + Take an intended load value, a trial duration value, and a finite set + of trial results, with all trials measured at that load value and + duration value. + + The performance spectrum is the function that maps any non-negative + real number into a sum of trial durations among all trials in the + set, that has that number, as their trial forwarding rate, e.g. map + to zero if no trial has that particular forwarding rate. + + A related function, defined if there is at least one trial in the + set, is the performance spectrum divided by the sum of the durations + of all trials in the set. + + That function is called the performance probability function, as it + satisfies all the requirements for probability mass function of a + discrete probability distribution, the one-dimensional random + variable being the trial forwarding rate. + + These functions are related to the SUT performance spectrum, as + sampled by the trials in the set. + + Take a set of all full-length trials performed at the Relevant Lower + Bound, sorted by decreasing trial forwarding rate. The sum of the + durations of those trials may be less than the Goal Duration Sum, or + not. If it is less, add an imaginary trial result with zero trial + forwarding rate, such that the new sum of durations is equal to the + Goal Duration Sum. This is the set of trials to use. + + If the quantile touches two trials, + + the larger trial forwarding rate (from the trial result sorted + earlier) is used. + + The resulting quantity is the Conditional Throughput of the goal in + question. + + A set of examples follows. + +5.2.1. First Example + + * [Goal Exceed Ratio] (#Goal-Exceed-Ratio) = 0 and [Goal Duration + Sum] (#Goal-Duration-Sum) has been reached. + + * Conditional Throughput is the smallest trial forwarding rate among + the trials. + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 39] + +Internet-Draft MLRsearch July 2024 + + +5.2.2. Second Example + + * Goal Exceed Ratio = 0 and Goal Duration Sum has not been reached + yet. + + * Due to the missing duration sum, the worst case may still happen, + so the Conditional Throughput is zero. + + * This is not reported to the user, as this load cannot become the + Relevant Lower Bound yet. + +5.2.3. Third Example + + * Goal Exceed Ratio = 50% and Goal Duration Sum is two seconds. + + * One trial is present with the duration of one second and zero + loss. + + * The imaginary trial is added with the duration of one second and + zero trial forwarding rate. + + * The median would touch both trials, so the Conditional Throughput + is the trial forwarding rate of the one non-imaginary trial. + + * As that had zero loss, the value is equal to the offered load. + +5.2.4. Summary + + While the Conditional Throughput is a generalization of the trial + forwarding rate, its definition is not an obvious one. + + Other than the trial forwarding rate, the other source of intuition + is the quantile in general, and the median the recommended case. + +5.3. Trials with Single Duration + + When goal attributes are chosen in such a way that every trial has + the same intended duration, the load classification is simpler. + + The following description follows the motivation of Goal Loss Ratio, + Goal Exceed Ratio, and Goal Duration Sum. + + If the sum of the durations of all trials (at the given load) is less + than the Goal Duration Sum, imagine two scenarios: + + * *best case scenario*: all subsequent trials having zero loss, and + + * *worst case scenario*: all subsequent trials having 100% loss. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 40] + +Internet-Draft MLRsearch July 2024 + + + Here we assume there are as many subsequent trials as needed to make + the sum of all trials equal to the Goal Duration Sum. + + The exceed ratio is defined using sums of durations (and number of + trials does not matter), so it does not matter whether the + "subsequent trials" can consist of an integer number of full-length + trials. + + In any of the two scenarios, best case and worst case, we can compute + the load exceed ratio, as the duration sum of good trials divided by + the duration sum of all trials, in both cases including the assumed + trials. + + Even if, in the best case scenario, the load exceed ratio is larger + than the Goal Exceed Ratio, the load is an upper bound. + + MKP2 Even if, in the worst case scenario, the load exceed ratio is + not larger than the Goal Exceed Ratio, the load is a lower bound. + + More specifically: + + * Take all trials measured at a given load. + + * The sum of the durations of all bad full-length trials is called + the bad sum. + + * The sum of the durations of all good full-length trials is called + the good sum. + + * The result of adding the bad sum plus the good sum is called the + measured sum. + + * The larger of the measured sum and the Goal Duration Sum is called + the whole sum. + + * The whole sum minus the measured sum is called the missing sum. + + * The optimistic exceed ratio is the bad sum divided by the whole + sum. + + * The pessimistic exceed ratio is the bad sum plus the missing sum, + that divided by the whole sum. + + * If the optimistic exceed ratio is larger than the Goal Exceed + Ratio, the load is classified as an upper bound. + + * If the pessimistic exceed ratio is not larger than the Goal Exceed + Ratio, the load is classified as a lower bound. + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 41] + +Internet-Draft MLRsearch July 2024 + + + * Else, the load is classified as undecided. + + The definition of pessimistic exceed ratio is compatible with the + logic in the Conditional Throughput computation, so in this single + trial duration case, a load is a lower bound if and only if the + Conditional Throughput loss ratio is not larger than the Goal Loss + Ratio. + + If it is larger, the load is either an upper bound or undecided. + +5.4. Trials with Short Duration + +5.4.1. Scenarios + + Trials with intended duration smaller than the goal final trial + duration are called short trials. The motivation for load + classification logic in the presence of short trials is based around + a counter-factual case: What would the trial result be if a short + trial has been measured as a full-length trial instead? + + There are three main scenarios where human intuition guides the + intended behavior of load classification. + +5.4.1.1. False Good Scenario + + The user had their reason for not configuring a shorter goal final + trial duration. Perhaps SUT has buffers that may get full at longer + trial durations. Perhaps SUT shows periodic decreases in performance + the user does not want to be treated as noise. + + In any case, many good short trials may become bad full-length trials + in the counter-factual case. + + In extreme cases, there are plenty of good short trials and no bad + short trials. + + In this scenario, we want the load classification NOT to classify the + load as a lower bound, despite the abundance of good short trials. + + Effectively, we want the good short trials to be ignored, so they do + not contribute to comparisons with the Goal Duration Sum. + +5.4.1.2. True Bad Scenario + + When there is a frame loss in a short trial, the counter-factual + full-length trial is expected to lose at least as many frames. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 42] + +Internet-Draft MLRsearch July 2024 + + + In practice, bad short trials are rarely turning into good full- + length trials. + + In extreme cases, there are no good short trials. + + In this scenario, we want the load classification to classify the + load as an upper bound just based on the abundance of short bad + trials. + + Effectively, we want the bad short trials to contribute to + comparisons with the Goal Duration Sum, so the load can be classified + sooner. + +5.4.1.3. Balanced Scenario + + Some SUTs are quite indifferent to trial duration. Performance + probability function constructed from short trial results is likely + to be similar to the performance probability function constructed + from full-length trial results (perhaps with larger dispersion, but + without a big impact on the median quantiles overall). + + For a moderate Goal Exceed Ratio value, this may mean there are both + good short trials and bad short trials. + + This scenario is there just to invalidate a simple heuristic of + always ignoring good short trials and never ignoring bad short + trials, as that simple heuristic would be too biased. + + Yes, the short bad trials are likely to turn into full-length bad + trials in the counter-factual case, but there is no information on + what would the good short trials turn into. + + The only way to decide safely is to do more trials at full length, + the same as in False Good Scenario. + +5.4.2. Classification Logic + + MLRsearch picks a particular logic for load classification in the + presence of short trials, but it is still RECOMMENDED to use + configurations that imply no short trials, so the possible + inefficiencies in that logic do not affect the result, and the result + has better explainability. + + With that said, the logic differs from the single trial duration case + only in different definition of the bad sum. The good sum is still + the sum across all good full-length trials. + + Few more notions are needed for defining the new bad sum: + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 43] + +Internet-Draft MLRsearch July 2024 + + + * The sum of durations of all bad full-length trials is called the + bad long sum. + + * The sum of durations of all bad short trials is called the bad + short sum. + + * The sum of durations of all good short trials is called the good + short sum. + + * One minus the Goal Exceed Ratio is called the subceed ratio. + + * The Goal Exceed Ratio divided by the subceed ratio is called the + exceed coefficient. + + * The good short sum multiplied by the exceed coefficient is called + the balancing sum. + + * The bad short sum minus the balancing sum is called the excess + sum. + + * If the excess sum is negative, the bad sum is equal to the bad + long sum. + + * Otherwise, the bad sum is equal to the bad long sum plus the + excess sum. + + Here is how the new definition of the bad sum fares in the three + scenarios, where the load is close to what would the relevant bounds + be if only full-length trials were used for the search. + +5.4.2.1. False Good Scenario + + If the duration is too short, we expect to see a higher frequency of + good short trials. This could lead to a negative excess sum, which + has no impact, hence the load classification is given just by full- + length trials. Thus, MLRsearch using too short trials has no + detrimental effect on result comparability in this scenario. But + also using short trials does not help with overall search duration, + probably making it worse. + +5.4.2.2. True Bad Scenario + + Settings with a small exceed ratio have a small exceed coefficient, + so the impact of the good short sum is small, and the bad short sum + is almost wholly converted into excess sum, thus bad short trials + have almost as big an impact as full-length bad trials. The same + conclusion applies to moderate exceed ratio values when the good + short sum is small. Thus, short trials can cause a load to get + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 44] + +Internet-Draft MLRsearch July 2024 + + + classified as an upper bound earlier, bringing time savings (while + not affecting comparability). + +5.4.2.3. Balanced Scenario + + Here excess sum is small in absolute value, as the balancing sum is + expected to be similar to the bad short sum. Once again, full-length + trials are needed for final load classification; but usage of short + trials probably means MLRsearch needed a shorter overall search time + before selecting this load for measurement, thus bringing time + savings (while not affecting comparability). + + Note that in presence of short trial results, the comparibility + between the load classification and the Conditional Throughput is + only partial. The Conditional Throughput still comes from a good + long trial, but a load higher than the Relevant Lower Bound may also + compute to a good value. + +5.5. Trials with Longer Duration + + If there are trial results with an intended duration larger than the + goal trial duration, the precise definitions in Appendix A and + Appendix B treat them in exactly the same way as trials with duration + equal to the goal trial duration. + + But in configurations with moderate (including 0.5) or small Goal + Exceed Ratio and small Goal Loss Ratio (especially zero), bad trials + with longer than goal durations may bias the search towards the lower + load values, as the noiseful end of the spectrum gets a larger + probability of causing the loss within the longer trials. + +6. IANA Considerations + + No requests of IANA. + +7. Security Considerations + + Benchmarking activities as described in this memo are limited to + technology characterization of a DUT/SUT using controlled stimuli in + a laboratory environment, with dedicated address space and the + constraints specified in the sections above. + + The benchmarking network topology will be an independent test setup + and MUST NOT be connected to devices that may forward the test + traffic into a production network or misroute traffic to the test + management network. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 45] + +Internet-Draft MLRsearch July 2024 + + + Further, benchmarking is performed on a "black-box" basis, relying + solely on measurements observable external to the DUT/SUT. + + Special capabilities SHOULD NOT exist in the DUT/SUT specifically for + benchmarking purposes. Any implications for network security arising + from the DUT/SUT SHOULD be identical in the lab and in production + networks. + +8. Acknowledgements + + Some phrases and statements in this document were created with help + of Mistral AI (mistral.ai). + + Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough + review and numerous useful comments and suggestions in the earlier + versions of this document. + + Special wholehearted gratitude and thanks to the late Al Morton for + his thorough reviews filled with very specific feedback and + constructive guidelines. Thank you Al for the close collaboration + over the years, for your continuous unwavering encouragement full of + empathy and positive attitude. Al, you are dearly missed. + +9. Appendix A: Load Classification + + This section specifies how to perform the load classification. + + Any intended load value can be classified, according to a given + [Search Goal] (#Search-Goal). + + The algorithm uses (some subsets of) the set of all available trial + results from trials measured at a given intended load at the end of + the search. All durations are those returned by the Measurer. + + The block at the end of this appendix holds pseudocode which computes + two values, stored in variables named optimistic and pessimistic. + + The pseudocode happens to be a valid Python code. + + If values of both variables are computed to be true, the load in + question is classified as a lower bound according to the given Search + Goal. If values of both variables are false, the load is classified + as an upper bound. Otherwise, the load is classified as undecided. + + The pseudocode expects the following variables to hold values as + follows: + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 46] + +Internet-Draft MLRsearch July 2024 + + + * goal_duration_sum: The duration sum value of the given Search + Goal. + + * goal_exceed_ratio: The exceed ratio value of the given Search + Goal. + + * good_long_sum: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial + Loss Ratio not higher than the Goal Loss Ratio. + + * bad_long_sum: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial + Loss Ratio higher than the Goal Loss Ratio. + + * good_short_sum: Sum of durations across trials with trial duration + shorter than the goal final trial duration and with a Trial Loss + Ratio not higher than the Goal Loss Ratio. + + * bad_short_sum: Sum of durations across trials with trial duration + shorter than the goal final trial duration and with a Trial Loss + Ratio higher than the Goal Loss Ratio. + + The code works correctly also when there are no trial results at a + given load. + + balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio) + effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum) + effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum) + quantile_duration_sum = effective_whole_sum * goal_exceed_ratio + optimistic = effective_bad_sum <= quantile_duration_sum + pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum + +10. Appendix B: Conditional Throughput + + This section specifies how to compute Conditional Throughput, as + referred to in section [Conditional Throughput] (#Conditional- + Throughput). + + Any intended load value can be used as the basis for the following + computation, but only the Relevant Lower Bound (at the end of the + search) leads to the value called the Conditional Throughput for a + given Search Goal. + + The algorithm uses (some subsets of) the set of all available trial + results from trials measured at a given intended load at the end of + the search. All durations are those returned by the Measurer. + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 47] + +Internet-Draft MLRsearch July 2024 + + + The block at the end of this appendix holds pseudocode which computes + a value stored as variable conditional_throughput. + + The pseudocode happens to be a valid Python code. + + The pseudocode expects the following variables to hold values as + follows: + + * goal_duration_sum: The duration sum value of the given Search + Goal. + + * goal_exceed_ratio: The exceed ratio value of the given Search + Goal. + + * good_long_sum: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial + Loss Ratio not higher than the Goal Loss Ratio. + + * bad_long_sum: Sum of durations across trials with trial duration + at least equal to the goal final trial duration and with a Trial + Loss Ratio higher than the Goal Loss Ratio. + + * long_trials: An iterable of all trial results from trials with + trial duration at least equal to the goal final trial duration, + sorted by increasing the Trial Loss Ratio. A trial result is a + composite with the following two attributes available: + + - trial.loss_ratio: The Trial Loss Ratio as measured for this + trial. + + - trial.duration: The trial duration of this trial. + + The code works correctly only when there if there is at least one + trial result measured at a given load. + + all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum) + remaining = all_long_sum * (1.0 - goal_exceed_ratio) + quantile_loss_ratio = None + for trial in long_trials: + if quantile_loss_ratio is None or remaining > 0.0: + quantile_loss_ratio = trial.loss_ratio + remaining -= trial.duration + else: + break + else: + if remaining > 0.0: + quantile_loss_ratio = 1.0 + conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 48] + +Internet-Draft MLRsearch July 2024 + + +11. References + +11.1. Normative References + + [RFC1242] Bradner, S., "Benchmarking Terminology for Network + Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, + July 1991, <https://www.rfc-editor.org/info/rfc1242>. + + [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN + Switching Devices", RFC 2285, DOI 10.17487/RFC2285, + February 1998, <https://www.rfc-editor.org/info/rfc2285>. + + [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for + Network Interconnect Devices", RFC 2544, + DOI 10.17487/RFC2544, March 1999, + <https://www.rfc-editor.org/info/rfc2544>. + + [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking + Methodology for IPv6 Transition Technologies", RFC 8219, + DOI 10.17487/RFC8219, August 2017, + <https://www.rfc-editor.org/info/rfc8219>. + + [RFC9004] Morton, A., "Updates for the Back-to-Back Frame Benchmark + in RFC 2544", RFC 9004, DOI 10.17487/RFC9004, May 2021, + <https://www.rfc-editor.org/info/rfc9004>. + +11.2. Informative References + + [FDio-CSIT-MLRsearch] + "FD.io CSIT Test Methodology - MLRsearch", October 2023, + <https://csit.fd.io/cdocs/methodology/measurements/ + data_plane_throughput/mlr_search/>. + + [PyPI-MLRsearch] + "MLRsearch 1.2.1, Python Package Index", October 2023, + <https://pypi.org/project/MLRsearch/1.2.1/>. + + [TST009] "TST 009", n.d., <https://www.etsi.org/deliver/etsi_gs/ + NFV-TST/001_099/009/03.04.01_60/gs_NFV- + TST009v030401p.pdf>. + +Authors' Addresses + + Maciek Konstantynowicz + Cisco Systems + Email: mkonstan@cisco.com + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 49] + +Internet-Draft MLRsearch July 2024 + + + Vratko Polak + Cisco Systems + Email: vrpolak@cisco.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Konstantynowicz & Polak Expires 18 January 2025 [Page 50] diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml new file mode 100644 index 0000000000..c3aede3d3b --- /dev/null +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml @@ -0,0 +1,3136 @@ +<?xml version="1.0" encoding="us-ascii"?> + <?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?> + <!-- generated by https://github.com/cabo/kramdown-rfc version 1.7.18 (Ruby 3.1.2) --> + + +<!DOCTYPE rfc [ + <!ENTITY nbsp " "> + <!ENTITY zwsp "​"> + <!ENTITY nbhy "‑"> + <!ENTITY wj "⁠"> + +<!ENTITY RFC1242 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.1242.xml"> +<!ENTITY RFC2285 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2285.xml"> +<!ENTITY RFC2544 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2544.xml"> +<!ENTITY RFC8219 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8219.xml"> +<!ENTITY RFC9004 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9004.xml"> +]> + + +<rfc ipr="trust200902" docName="draft-ietf-bmwg-mlrsearch-07" category="info" tocInclude="true" sortRefs="true" symRefs="true"> + <front> + <title abbrev="MLRsearch">Multiple Loss Ratio Search</title> + + <author initials="M." surname="Konstantynowicz" fullname="Maciek Konstantynowicz"> + <organization>Cisco Systems</organization> + <address> + <email>mkonstan@cisco.com</email> + </address> + </author> + <author initials="V." surname="Polak" fullname="Vratko Polak"> + <organization>Cisco Systems</organization> + <address> + <email>vrpolak@cisco.com</email> + </address> + </author> + + <date year="2024" month="July" day="18"/> + + <area>ops</area> + <workgroup>Benchmarking Working Group</workgroup> + <keyword>Internet-Draft</keyword> + + <abstract> + + +<?line 52?> + +<t>This document proposes extensions to <xref target="RFC2544"></xref> throughput search by +defining a new methodology called Multiple Loss Ratio search +(MLRsearch). MLRsearch aims to minimize search duration, +support multiple loss ratio searches, +and enhance result repeatability and comparability.</t> + +<t>The primary reason for extending <xref target="RFC2544"></xref> is to address the challenges +and requirements presented by the evaluation and testing +of software-based networking systems' data planes.</t> + +<t>To give users more freedom, MLRsearch provides additional configuration options +such as allowing multiple short trials per load instead of one large trial, +tolerating a certain percentage of trial results with higher loss, +and supporting the search for multiple goals with varying loss ratios.</t> + + + + </abstract> + + + + </front> + + <middle> + + +<?line 69?> + + +<section anchor="purpose-and-scope"><name>Purpose and Scope</name> + +<t>The purpose of this document is to describe Multiple Loss Ratio search +(MLRsearch), a data plane throughput search methodology optimized for software +networking DUTs.</t> + +<t>Applying vanilla <xref target="RFC2544"></xref> throughput bisection to software DUTs +results in several problems:</t> + +<t><list style="symbols"> + <t>Binary search takes too long as most trials are done far from the +eventually found throughput.</t> + <t>The required final trial duration and pauses between trials +prolong the overall search duration.</t> + <t>Software DUTs show noisy trial results, +leading to a big spread of possible discovered throughput values.</t> + <t>Throughput requires a loss of exactly zero frames, but the industry +frequently allows for small but non-zero losses.</t> + <t>The definition of throughput is not clear when trial results are inconsistent.</t> +</list></t> + +<t>To address the problems mentioned above, +the MLRsearch test methodology specification employs the following enhancements:</t> + +<t><list style="symbols"> + <t>Allow multiple short trials instead of one big trial per load. + <list style="symbols"> + <t>Optionally, tolerate a percentage of trial results with higher loss.</t> + </list></t> + <t>Allow searching for multiple Search Goals, with differing loss ratios. + <list style="symbols"> + <t>Any trial result can affect each Search Goal in principle.</t> + </list></t> + <t>Insert multiple coarse targets for each Search Goal, earlier ones need +to spend less time on trials. + <list style="symbols"> + <t>Earlier targets also aim for lesser precision.</t> + <t>Use Forwarding Rate (FR) at maximum offered load +<xref target="RFC2285"></xref> (section 3.6.2) to initialize the initial targets.</t> + </list></t> + <t>Take care when dealing with inconsistent trial results. + <list style="symbols"> + <t>Reported throughput is smaller than the smallest load with high loss.</t> + <t>Smaller load candidates are measured first.</t> + </list></t> + <t>Apply several load selection heuristics to save even more time +by trying hard to avoid unnecessarily narrow bounds.</t> +</list></t> + +<t>Some of these enhancements are formalized as MLRsearch specification, +the remaining enhancements are treated as implementation details, +thus achieving high comparability without limiting future improvements.</t> + +<t>MLRsearch configuration options are flexible enough to +support both conservative settings and aggressive settings. +The conservative settings lead to results +unconditionally compliant with <xref target="RFC2544"></xref>, +but longer search duration and worse repeatability. +Conversely, aggressive settings lead to shorter search duration +and better repeatability, but the results are not compliant with <xref target="RFC2544"></xref>.</t> + +<t>No part of <xref target="RFC2544"></xref> is intended to be obsoleted by this document.</t> + +</section> +<section anchor="identified-problems"><name>Identified Problems</name> + +<t>This chapter describes the problems affecting usability +of various performance testing methodologies, +mainly a binary search for <xref target="RFC2544"></xref> unconditionally compliant throughput.</t> + +<section anchor="long-search-duration"><name>Long Search Duration</name> + + +<t>The emergence of software DUTs, with frequent software updates and a +number of different frame processing modes and configurations, +has increased both the number of performance tests +required to verify the DUT update and the frequency of running those tests. +This makes the overall test execution time even more important than before.</t> + +<t>The current <xref target="RFC2544"></xref> throughput definition restricts the potential +for time-efficiency improvements. +A more generalized throughput concept could enable further enhancements +while maintaining the precision of simpler methods.</t> + +<t>The bisection method, when unconditionally compliant with <xref target="RFC2544"></xref>, +is excessively slow. +This is because a significant amount of time is spent on trials +with loads that, in retrospect, are far from the final determined throughput.</t> + +<t><xref target="RFC2544"></xref> does not specify any stopping condition for throughput search, +so users already have an access to a limited trade-off +between search duration and achieved precision. +However, each full 60-second trials doubles the precision, +so not many trials can be removed without a substantial loss of precision.</t> + +</section> +<section anchor="dut-in-sut"><name>DUT in SUT</name> + +<t><xref target="RFC2285"></xref> defines: +- DUT as + - The network forwarding device to which stimulus is offered and + response measured <xref target="RFC2285"></xref> (section 3.1.1). +- SUT as + - The collective set of network devices to which stimulus is offered + as a single entity and response measured <xref target="RFC2285"></xref> (section 3.1.2).</t> + +<t><xref target="RFC2544"></xref> specifies a test setup with an external tester stimulating the +networking system, treating it either as a single DUT, or as a system +of devices, an SUT.</t> + +<t>In the case of software networking, the SUT consists of not only the DUT +as a software program processing frames, but also of +server hardware and operating system functions, +with that server hardware resources shared across all programs including +the operating system.</t> + +<t>Given that the SUT is a shared multi-tenant environment +encompassing the DUT and other components, the DUT might inadvertently +experience interference from the operating system +or other software operating on the same server.</t> + +<t>Some of this interference can be mitigated. +For instance, +pinning DUT program threads to specific CPU cores +and isolating those cores can prevent context switching.</t> + +<t>Despite taking all feasible precautions, some adverse effects may still impact +the DUT's network performance. +In this document, these effects are collectively +referred to as SUT noise, even if the effects are not as unpredictable +as what other engineering disciplines call noise.</t> + +<t>DUT can also exhibit fluctuating performance itself, for reasons +not related to the rest of SUT. For example due to pauses in execution +as needed for internal stateful processing. +In many cases this +may be an expected per-design behavior, as it would be observable even +in a hypothetical scenario where all sources of SUT noise are eliminated. +Such behavior affects trial results in a way similar to SUT noise. +As the two phenomenons are hard to distinguish, +in this document the term 'noise' is used to encompass +both the internal performance fluctuations of the DUT +and the genuine noise of the SUT.</t> + +<t>A simple model of SUT performance consists of an idealized noiseless performance, +and additional noise effects. +For a specific SUT, the noiseless performance is assumed to be constant, +with all observed performance variations being attributed to noise. +The impact of the noise can vary in time, sometimes wildly, +even within a single trial. +The noise can sometimes be negligible, but frequently +it lowers the observed SUT performance as observed in trial results.</t> + +<t>In this model, SUT does not have a single performance value, it has a spectrum. +One end of the spectrum is the idealized noiseless performance value, +the other end can be called a noiseful performance. +In practice, trial result +close to the noiseful end of the spectrum happens only rarely. +The worse the performance value is, the more rarely it is seen in a trial. +Therefore, the extreme noiseful end of the SUT spectrum is not observable +among trial results. +Also, the extreme noiseless end of the SUT spectrum +is unlikely to be observable, this time because some small noise effects +are likely to occur multiple times during a trial.</t> + +<t>Unless specified otherwise, this document's focus is +on the potentially observable ends of the SUT performance spectrum, +as opposed to the extreme ones.</t> + +<t>When focusing on the DUT, the benchmarking effort should ideally aim +to eliminate only the SUT noise from SUT measurements. +However, +this is currently not feasible in practice, as there are no realistic enough +models available to distinguish SUT noise from DUT fluctuations, +based on authors' experience and available literature.</t> + +<t>Assuming a well-constructed SUT, the DUT is likely its +primary performance bottleneck. +In this case, we can define the DUT's +ideal noiseless performance as the noiseless end of the SUT performance spectrum, +especially for throughput. +However, other performance metrics, such as latency, +may require additional considerations.</t> + +<t>Note that by this definition, DUT noiseless performance +also minimizes the impact of DUT fluctuations, as much as realistically possible +for a given trial duration.</t> + +<t>MLRsearch methodology aims to solve the DUT in SUT problem +by estimating the noiseless end of the SUT performance spectrum +using a limited number of trial results.</t> + +<t>Any improvements to the throughput search algorithm, aimed at better +dealing with software networking SUT and DUT setup, should employ +strategies recognizing the presence of SUT noise, allowing the discovery of +(proxies for) DUT noiseless performance +at different levels of sensitivity to SUT noise.</t> + +</section> +<section anchor="repeatability-and-comparability"><name>Repeatability and Comparability</name> + +<t><xref target="RFC2544"></xref> does not suggest to repeat throughput search. +And from just one +discovered throughput value, it cannot be determined how repeatable that value is. +Poor repeatability then leads to poor comparability, +as different benchmarking teams may obtain varying throughput values +for the same SUT, exceeding the expected differences from search precision.</t> + +<t><xref target="RFC2544"></xref> throughput requirements (60 seconds trial and +no tolerance of a single frame loss) affect the throughput results +in the following way. +The SUT behavior close to the noiseful end of its performance spectrum +consists of rare occasions of significantly low performance, +but the long trial duration makes those occasions not so rare on the trial level. +Therefore, the binary search results tend to wander away from the noiseless end +of SUT performance spectrum, more frequently and more widely than short +trials would, thus causing poor throughput repeatability.</t> + +<t>The repeatability problem can be addressed by defining a search procedure +that identifies a consistent level of performance, +even if it does not meet the strict definition of throughput in <xref target="RFC2544"></xref>.</t> + +<t>According to the SUT performance spectrum model, better repeatability +will be at the noiseless end of the spectrum. +Therefore, solutions to the DUT in SUT problem +will help also with the repeatability problem.</t> + +<t>Conversely, any alteration to <xref target="RFC2544"></xref> throughput search +that improves repeatability should be considered +as less dependent on the SUT noise.</t> + +<t>An alternative option is to simply run a search multiple times, and report some +statistics (e.g. average and standard deviation). +This can be used +for a subset of tests deemed more important, +but it makes the search duration problem even more pronounced.</t> + +</section> +<section anchor="throughput-with-non-zero-loss"><name>Throughput with Non-Zero Loss</name> + +<t><xref target="RFC1242"></xref> (section 3.17 Throughput) defines throughput as: + The maximum rate at which none of the offered frames + are dropped by the device.</t> + +<t>Then, it says: + Since even the loss of one frame in a + data stream can cause significant delays while + waiting for the higher level protocols to time out, + it is useful to know the actual maximum data + rate that the device can support.</t> + +<t>However, many benchmarking teams accept a small, +non-zero loss ratio as the goal for their load search.</t> + +<t>Motivations are many:</t> + +<t><list style="symbols"> + <t>Modern protocols tolerate frame loss better, +compared to the time when <xref target="RFC1242"></xref> and <xref target="RFC2544"></xref> were specified.</t> + <t>Trials nowadays send way more frames within the same duration, +increasing the chance of a small SUT performance fluctuation +being enough to cause frame loss.</t> + <t>Small bursts of frame loss caused by noise have otherwise smaller impact +on the average frame loss ratio observed in the trial, +as during other parts of the same trial the SUT may work more closely +to its noiseless performance, thus perhaps lowering the Trial Loss Ratio +below the Goal Loss Ratio value.</t> + <t>If an approximation of the SUT noise impact on the Trial Loss Ratio is known, +it can be set as the Goal Loss Ratio.</t> +</list></t> + +<t>Regardless of the validity of all similar motivations, +support for non-zero loss goals makes any search algorithm more user-friendly. +<xref target="RFC2544"></xref> throughput is not user-friendly in this regard.</t> + +<t>Furthermore, allowing users to specify multiple loss ratio values, +and enabling a single search to find all relevant bounds, +significantly enhances the usefulness of the search algorithm.</t> + +<t>Searching for multiple Search Goals also helps to describe the SUT performance +spectrum better than the result of a single Search Goal. +For example, the repeated wide gap between zero and non-zero loss loads +indicates the noise has a large impact on the observed performance, +which is not evident from a single goal load search procedure result.</t> + +<t>It is easy to modify the vanilla bisection to find a lower bound +for the intended load that satisfies a non-zero Goal Loss Ratio. +But it is not that obvious how to search for multiple goals at once, +hence the support for multiple Search Goals remains a problem.</t> + +</section> +<section anchor="inconsistent-trial-results"><name>Inconsistent Trial Results</name> + +<t>While performing throughput search by executing a sequence of +measurement trials, there is a risk of encountering inconsistencies +between trial results.</t> + +<t>The plain bisection never encounters inconsistent trials. +But <xref target="RFC2544"></xref> hints about the possibility of inconsistent trial results, +in two places in its text. +The first place is section 24, where full trial durations are required, +presumably because they can be inconsistent with the results +from short trial durations. +The second place is section 26.3, where two successive zero-loss trials +are recommended, presumably because after one zero-loss trial +there can be a subsequent inconsistent non-zero-loss trial.</t> + +<t>Examples include:</t> + +<t><list style="symbols"> + <t>A trial at the same load (same or different trial duration) results +in a different Trial Loss Ratio.</t> + <t>A trial at a higher load (same or different trial duration) results +in a smaller Trial Loss Ratio.</t> +</list></t> + +<t>Any robust throughput search algorithm needs to decide how to continue +the search in the presence of such inconsistencies. +Definitions of throughput in <xref target="RFC1242"></xref> and <xref target="RFC2544"></xref> are not specific enough +to imply a unique way of handling such inconsistencies.</t> + +<t>Ideally, there will be a definition of a new quantity which both generalizes +throughput for non-zero-loss (and other possible repeatability enhancements), +while being precise enough to force a specific way to resolve trial result +inconsistencies. +But until such a definition is agreed upon, the correct way to handle +inconsistent trial results remains an open problem.</t> + +</section> +</section> +<section anchor="mlrsearch-specification"><name>MLRsearch Specification</name> + +<t>This section describes MLRsearch specification including all technical +definitions needed for evaluating whether a particular test procedure +complies with MLRsearch specification.</t> + + +<section anchor="overview"><name>Overview</name> + +<t>MLRsearch specification describes a set of abstract system components, +acting as functions with specified inputs and outputs.</t> + +<t>A test procedure is said to comply with MLRsearch specification +if it can be conceptually divided into analogous components, +each satisfying requirements for the corresponding MLRsearch component.</t> + +<t>The Measurer component is tasked to perform trials, +the Controller component is tasked to select trial loads and durations, +the Manager component is tasked to pre-configure everything +and to produce the test report. +The test report explicitly states Search Goals (as the Controller Inputs) +and corresponding Goal Results (Controller Outputs).</t> + + +<t>The Manager calls the Controller once, +the Controller keeps calling the Measurer +until all stopping conditions are met.</t> + +<t>The part where Controller calls the Measurer is called the search. +Any activity done by the Manager before it calls the Controller +(or after Controller returns) is not considered to be part of the search.</t> + +<t>MLRsearch specification prescribes regular search results and recommends +their stopping conditions. Irregular search results are also allowed, +they may have different requirements and stopping conditions.</t> + +<t>Search results are based on load classification. +When measured enough, any chosen load either achieves of fails each search goal, +thus becoming a lower or an upper bound for that goal. +When the relevant bounds are at loads that are close enough +(according to goal precision), the regular result is found. +Search stops when all regular results are found +(or if some goals are proven to have only irregular results).</t> + +</section> +<section anchor="measurement-quantities"><name>Measurement Quantities</name> + +<t>MLRsearch specification uses a number of measurement quantities.</t> + +<t>In general, MLRsearch specification does not require particular units to be used, +but it is REQUIRED for the test report to state all the units. +For example, ratio quantities can be dimensionless numbers between zero and one, +but may be expressed as percentages instead.</t> + +<t>For convenience, a group of quantities can be treated as a composite quantity, +One constituent of a composite quantity is called an attribute, +and a group of attribute values is called an instance of that composite quantity.</t> + +<t>Some attributes are not independent from others, +and they can be calculated from other attributes. +Such quantites are called derived quantities.</t> + +</section> +<section anchor="existing-terms"><name>Existing Terms</name> + +<t>RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" +contains basic definitions, and +RFC 2544 "Benchmarking Methodology for Network Interconnect Devices" +contains discussions of a number of terms and additional methodology requirements. +RFC 2285 adds more terms and discussions, describing some known situations +in more precise way.</t> + +<t>All three documents should be consulted +before attempting to make use of this document.</t> + +<t>Definitions of some central terms are copied and discussed in subsections.</t> + + + + + +<section anchor="sut"><name>SUT</name> + +<t>Defined in <xref target="RFC2285"></xref> (section 3.1.2 System Under Test (SUT)) as follows.</t> + +<t>Definition:</t> + +<t>The collective set of network devices to which stimulus is offered +as a single entity and response measured.</t> + +<t>Discussion:</t> + +<t>An SUT consisting of a single network device is also allowed.</t> + +</section> +<section anchor="dut"><name>DUT</name> + +<t>Defined in <xref target="RFC2285"></xref> (section 3.1.1 Device Under Test (DUT)) as follows.</t> + +<t>Definition:</t> + +<t>The network forwarding device to which stimulus is offered and +response measured.</t> + +<t>Discussion:</t> + +<t>DUT, as a sub-component of SUT, is only indirectly mentioned +in MLRsearch specification, but is of key relevance for its motivation.</t> + + +</section> +<section anchor="trial"><name>Trial</name> + +<t>A trial is the part of the test described in <xref target="RFC2544"></xref> (section 23. Trial description).</t> + +<t>Definition:</t> + +<t>A particular test consists of multiple trials. Each trial returns + one piece of information, for example the loss rate at a particular + input frame rate. Each trial consists of a number of phases:</t> + +<t>a) If the DUT is a router, send the routing update to the "input" + port and pause two seconds to be sure that the routing has settled.</t> + +<t>b) Send the "learning frames" to the "output" port and wait 2 + seconds to be sure that the learning has settled. Bridge learning + frames are frames with source addresses that are the same as the + destination addresses used by the test frames. Learning frames for + other protocols are used to prime the address resolution tables in + the DUT. The formats of the learning frame that should be used are + shown in the Test Frame Formats document.</t> + +<t>c) Run the test trial.</t> + +<t>d) Wait for two seconds for any residual frames to be received.</t> + +<t>e) Wait for at least five seconds for the DUT to restabilize.</t> + +<t>Discussion:</t> + +<t>The definition describes some traits, it is not clear whether all of them +are REQUIRED, or some of them are only RECOMMENDED.</t> + + +<t>For the purposes of the MLRsearch specification, +it is ALLOWED for the test procedure to deviate from the <xref target="RFC2544"></xref> description, +but any such deviation MUST be made explicit in the test report.</t> + +<t>Trials are the only stimuli the SUT is expected to experience +during the search.</t> + +<t>In some discussion paragraphs, it is useful to consider the traffic +as sent and received by a tester, as implicitly defined +in <xref target="RFC2544"></xref> (section 6. Test set up).</t> + +<t>An example of deviation from <xref target="RFC2544"></xref> is using shorter wait times.</t> + +</section> +</section> +<section anchor="trial-terms"><name>Trial Terms</name> + +<t>This section defines new and redefine existing terms for quantities +relevant as inputs or outputs of trial, as used by the Measurer component.</t> + +<section anchor="trial-duration"><name>Trial Duration</name> + +<t>Definition:</t> + +<t>Trial duration is the intended duration of the traffic for a trial.</t> + +<t>Discussion:</t> + +<t>In general, this quantity does not include any preparation nor waiting +described in section 23 of <xref target="RFC2544"></xref> (section 23. Trial description).</t> + +<t>While any positive real value may be provided, some Measurer implementations +MAY limit possible values, e.g. by rounding down to neared integer in seconds. +In that case, it is RECOMMENDED to give such inputs to the Controller +so the Controller only proposes the accepted values. +Alternatively, the test report MUST present the rounded values +as Search Goal attributes.</t> + +</section> +<section anchor="trial-load"><name>Trial Load</name> + +<t>Definition:</t> + +<t>The trial load is the intended load for a trial</t> + +<t>Discussion:</t> + +<t>For test report purposes, it is assumed that this is a constant load by default. +This MAY be only an average load, e.g. when the traffic is intended to be busty, +e.g. as suggested in <xref target="RFC2544"></xref> (section 21. Bursty traffic), +but the test report MUST explicitly mention how non-constant the traffic is.</t> + +<t>Trial load is the quantity defined as Constant Load of <xref target="RFC1242"></xref> +(section 3.4 Constant Load), Data Rate of <xref target="RFC2544"></xref> +(section 14. Bidirectional traffic) +and Intended Load of <xref target="RFC2285"></xref> (section 3.5.1 Intended load (Iload)). +All three definitions specify +that this value applies to one (input or output) interface.</t> + + +<t>For test report purposes, multi-interface aggregate load MAY be reported, +this is understood as the same quantity expressed using different units. +From the report it MUST be clear whether a particular trial load value +is per one interface, or an aggregate over all interfaces.</t> + +<t>Similarly to trial duration, some Measurers may limit the possible values +of trial load. Contrary to trial duration, the test report is NOT REQUIRED +to document such behavior.</t> + + +<t>It is ALLOWED to combine trial load and trial duration in a way +that would not be possible to achieve using any integer number of data frames.</t> + + +</section> +<section anchor="trial-input"><name>Trial Input</name> + +<t>Definition:</t> + +<t>Trial Input is a composite quantity, consisting of two attributes: +trial duration and trial load.</t> + +<t>Discussion:</t> + +<t>When talking about multiple trials, it is common to say "Trial Inputs" +to denote all corresponding Trial Input instances.</t> + +<t>A Trial Input instance acts as the input for one call of the Measurer component.</t> + +<t>Contrary to other composite quantities, MLRsearch implementations +are NOT ALLOWED to add optional attributes here. +This improves interoperability between various implementations of +the Controller and the Measurer.</t> + +</section> +<section anchor="traffic-profile"><name>Traffic Profile</name> + +<t>Definition:</t> + +<t>Traffic profile is a composite quantity +containing attributes other than trial load and trial duration, +needed for unique determination of the trial to be performed.</t> + +<t>Discussion:</t> + +<t>All its attributes are assumed to be constant during the search, +and the composite is configured on the Measurer by the Manager +before the search starts. +This is why the traffic profile is not part of the Trial Input.</t> + +<t>As a consequence, implementations of the Manager and the Measurer +must be aware of their common set of capabilities, so that the traffic profile +uniquely defines the traffic during the search. +The important fact is that none of those capabilities +have to be known by the Controller implementations.</t> + +<t>The traffic profile SHOULD contain some specific quantities, +for example <xref target="RFC2544"></xref> (section 9. Frame sizes) governs +data link frame size as defined in <xref target="RFC1242"></xref> (section 3.5 Data link frame size).</t> + +<t>Several more specific quantities may be RECOMMENDED, depending on media type. +For example, <xref target="RFC2544"></xref> (Appendix C) lists frame formats and protocol addresses, +as recommended from <xref target="RFC2544"></xref> (section 8. Frame formats) +and <xref target="RFC2544"></xref> (section 12. Protocol addresses).</t> + +<t>Depending on SUT configuration, e.g. when testing specific protocols, +additional attributes MUST be included in the traffic profile +and in the test report.</t> + +<t>Example: <xref target="RFC8219"></xref> (section 5.3. Traffic Setup) introduces traffic setups +consisting of a mix of IPv4 and IPv6 traffic - the implied traffic profile +therefore must include an attribute for their percentage.</t> + +<t>Other traffic properties that need to be somehow specified +in Traffic Profile include: +<xref target="RFC2544"></xref> (section 14. Bidirectional traffic), +<xref target="RFC2285"></xref> (section 3.3.3 Fully meshed traffic), +and <xref target="RFC2544"></xref> (section 11. Modifiers).</t> + +</section> +<section anchor="trial-forwarding-ratio"><name>Trial Forwarding Ratio</name> + +<t>Definition:</t> + +<t>The trial forwarding ratio is a dimensionless floating point value. +It MUST range between 0.0 and 1.0, both inclusive. +It is calculated by dividing the number of frames +successfully forwarded by the SUT +by the total number of frames expected to be forwarded during the trial</t> + +<t>Discussion:</t> + +<t>For most traffic profiles, "expected to be forwarded" means +"intended to get transmitted from Tester towards SUT".</t> + +<t>Trial forwarding ratio MAY be expressed in other units +(e.g. as a percentage) in the test report.</t> + +<t>Note that, contrary to loads, frame counts used to compute +trial forwarding ratio are aggregates over all SUT output interfaces.</t> + +<t>Questions around what is the correct number of frames +that should have been forwarded +is generally outside of the scope of this document.</t> + + + +</section> +<section anchor="trial-loss-ratio"><name>Trial Loss Ratio</name> + +<t>Definition:</t> + +<t>The Trial Loss Ratio is equal to one minus the trial forwarding ratio.</t> + +<t>Discussion:</t> + +<t>100% minus the trial forwarding ratio, when expressed as a percentage.</t> + +<t>This is almost identical to Frame Loss Rate of <xref target="RFC1242"></xref> +(section 3.6 Frame Loss Rate), +the only minor difference is that Trial Loss Ratio +does not need to be expressed as a percentage.</t> + +</section> +<section anchor="trial-forwarding-rate"><name>Trial Forwarding Rate</name> + +<t>Definition:</t> + +<t>The trial forwarding rate is a derived quantity, calculated by +multiplying the trial load by the trial forwarding ratio.</t> + +<t>Discussion:</t> + +<t>It is important to note that while similar, this quantity is not identical +to the Forwarding Rate as defined in <xref target="RFC2285"></xref> +(section 3.6.1 Forwarding rate (FR)). +The latter is specific to one output interface only, +whereas the trial forwarding ratio is based +on frame counts aggregated over all SUT output interfaces.</t> + + +</section> +<section anchor="trial-effective-duration"><name>Trial Effective Duration</name> + +<t>Definition:</t> + +<t>Trial effective duration is a time quantity related to the trial, +by default equal to the trial duration.</t> + +<t>Discussion:</t> + +<t>This is an optional feature. +If the Measurer does not return any trial effective duration value, +the Controller MUST use the trial duration value instead.</t> + +<t>Trial effective duration may be any time quantity chosen by the Measurer +to be used for time-based decisions in the Controller.</t> + +<t>The test report MUST explain how the Measurer computes the returned +trial effective duration values, if they are not always +equal to the trial duration.</t> + +<t>This feature can be beneficial for users +who wish to manage the overall search duration, +rather than solely the traffic portion of it. +Simply measure the duration of the whole trial (waits including) +and use that as the trial effective duration.</t> + +<t>Also, this is a way for the Measurer to inform the Controller about +its surprising behavior, for example when rounding the trial duration value.</t> + + +</section> +<section anchor="trial-output"><name>Trial Output</name> + +<t>Definition:</t> + +<t>Trial Output is a composite quantity. The REQUIRED attributes are +Trial Loss Ratio, trial effective duration and trial forwarding rate.</t> + +<t>Discussion:</t> + +<t>When talking about multiple trials, it is common to say "Trial Outputs" +to denote all corresponding Trial Output instances.</t> + +<t>Implementations may provide additional (optional) attributes. +The Controller implementations MUST ignore values of any optional attribute +they are not familiar with, +except when passing Trial Output instance to the Manager.</t> + +<t>Example of an optional attribute: +The aggregate number of frames expected to be forwarded during the trial, +especially if it is not just (a rounded-up value) +implied by trial load and trial duration.</t> + +<t>While <xref target="RFC2285"></xref> (Section 3.5.2 Offered load (Oload)) +requires the offered load value to be reported for forwarding rate measurements, +it is NOT REQUIRED in MLRsearch specification.</t> + + +</section> +<section anchor="trial-result"><name>Trial Result</name> + +<t>Definition:</t> + +<t>Trial result is a composite quantity, +consisting of the Trial Input and the Trial Output.</t> + +<t>Discussion:</t> + +<t>When talking about multiple trials, it is common to say "trial results" +to denote all corresponding trial result instances.</t> + +<t>While implementations SHOULD NOT include additional attributes +with independent values, they MAY include derived quantities.</t> + +</section> +</section> +<section anchor="goal-terms"><name>Goal Terms</name> + +<t>This section defines new and redefine existing terms for quantities +indirectly relevant for inputs or outputs of the Controller component.</t> + +<t>Several goal attributes are defined before introducing +the main component quantity: the Search Goal.</t> + +<section anchor="goal-final-trial-duration"><name>Goal Final Trial Duration</name> + +<t>Definition:</t> + +<t>A threshold value for trial durations.</t> + +<t>Discussion:</t> + +<t>This attribute value MUST be positive.</t> + +<t>A trial with Trial Duration at least as long as the Goal Final Trial Duration +is called a full-length trial (with respect to the given Search Goal).</t> + +<t>A trial that is not full-length is called a short trial.</t> + +<t>Informally, while MLRsearch is allowed to perform short trials, +the results from such short trials have only limited impact on search results.</t> + +<t>One trial may be full-length for some Search Goals, but not for others.</t> + +<t>The full relation of this goal to Controller Output is defined later in +this document in subsections of [Goal Result] (#Goal-Result). +For example, the Conditional Throughput for this goal is computed only from +full-length trial results.</t> + +</section> +<section anchor="goal-duration-sum"><name>Goal Duration Sum</name> + +<t>Definition:</t> + +<t>A threshold value for a particular sum of trial effective durations.</t> + +<t>Discussion:</t> + +<t>This attribute value MUST be positive.</t> + +<t>Informally, even when looking only at full-length trials, +MLRsearch may spend up to this time measuring the same load value.</t> + +<t>If the Goal Duration Sum is larger than the Goal Final Trial Duration, +multiple full-length trials may need to be performed at the same load.</t> + +<t>See [TST009 Example] (#TST009-Example) for an example where possibility +of multiple full-length trials at the same load is intended.</t> + +<t>A Goal Duration Sum value lower than the Goal Final Trial Duration +(of the same goal) could save some search time, but is NOT RECOMMENDED. +See [Relevant Upper Bound] (#Relevant-Upper-Bound) for partial explanation.</t> + +</section> +<section anchor="goal-loss-ratio"><name>Goal Loss Ratio</name> + +<t>Definition:</t> + +<t>A threshold value for Trial Loss Ratios.</t> + +<t>Discussion:</t> + +<t>Attribute value MUST be non-negative and smaller than one.</t> + +<t>A trial with Trial Loss Ratio larger than a Goal Loss Ratio value +is called a lossy trial, with respect to given Search Goal.</t> + +<t>Informally, if a load causes too many lossy trials, +the Relevant Lower Bound for this goal will be smaller than that load.</t> + +<t>If a trial is not lossy, it is called a low-loss trial, +or (specifically for zero Goal Loss Ratio value) zero-loss trial.</t> + +</section> +<section anchor="goal-exceed-ratio"><name>Goal Exceed Ratio</name> + +<t>Definition:</t> + +<t>A threshold value for a particular ratio of sums of Trial Effective Durations.</t> + +<t>Discussion:</t> + +<t>Attribute value MUST be non-negative and smaller than one.</t> + +<t>See later sections for details on which sums. +Specifically, the direct usage is only in +[Appendix A: Load Classification] (#Appendix-A:-Load-Classification) +and [Appendix B: Conditional Throughput] (#Appendix-B:-Conditional-Throughput). +The impact of that usage is discussed in subsections leading to +[Goal Result] (#Goal-Result).</t> + +<t>Informally, the impact of lossy trials is controlled by this value. +Effectively, Goal Exceed Ratio is a percentage of full-length trials +that may be lossy without the load being classified +as the [Relevant Upper Bound] (#Relevant-Upper-Bound).</t> + +</section> +<section anchor="goal-width"><name>Goal Width</name> + +<t>Definition:</t> + +<t>A value used as a threshold for deciding +whether two trial load values are close enough.</t> + +<t>Discussion:</t> + +<t>If present, the value MUST be positive.</t> + +<t>Informally, this acts as a stopping condition, +controlling the precision of the search. +The search stops if every goal has reached its precision.</t> + +<t>Implementations without this attribute +MUST give the Controller other ways to control the search stopping conditions.</t> + +<t>Absolute load difference and relative load difference are two popular choices, +but implementations may choose a different way to specify width.</t> + +<t>The test report MUST make it clear what specific quantity is used as Goal Width.</t> + +<t>It is RECOMMENDED to set the Goal Width (as relative difference) value +to a value no smaller than the Goal Loss Ratio. +(The reason is not obvious, see [Throughput] (#Throughput) if interested.)</t> + +</section> +<section anchor="search-goal"><name>Search Goal</name> + +<t>Definition:</t> + +<t>The Search Goal is a composite quantity consisting of several attributes, +some of them are required.</t> + +<t>Required attributes: +- Goal Final Trial Duration +- Goal Duration Sum +- Goal Loss Ratio +- Goal Exceed Ratio</t> + +<t>Optional attribute: +- Goal Width</t> + +<t>Discussion:</t> + +<t>Implementations MAY add their own attributes. +Those additional attributes may be required by the implementation +even if they are not required by MLRsearch specification. +But it is RECOMMENDED for those implementations +to support missing values by computing reasonable defaults.</t> + +<t>The meaning of listed attributes is formally given only by their indirect effect +on the search results.</t> + +<t>Informally, later sections provide additional intuitions and examples +of the Search Goal attribute values.</t> + +<t>An example of additional attributes required by some implementations +is Goal Initial Trial Duration, together with another attribute +that controls possible intermediate Trial Duration values. +The reasonable default in this case is using the Goal Final Trial Duration +and no intermediate values.</t> + +</section> +<section anchor="controller-input"><name>Controller Input</name> + +<t>Definition:</t> + +<t>Controller Input is a composite quantity +required as an input for the Controller. +The only REQUIRED attribute is a list of Search Goal instances.</t> + +<t>Discussion:</t> + +<t>MLRsearch implementations MAY use additional attributes. +Those additional attributes may be required by the implementation +even if they are not required by MLRsearch specification.</t> + +<t>Formally, the Manager does not apply any Controller configuration +apart from one Controller Input instance.</t> + +<t>For example, Traffic Profile is configured on the Measurer by the Manager +(without explicit assistance of the Controller).</t> + +<t>The order of Search Goal instances in a list SHOULD NOT +have a big impact on Controller Output (see section [Controller Output] (#Controller-Output) , +but MLRsearch implementations MAY base their behavior on the order +of Search Goal instances in a list.</t> + +<t>An example of an optional attribute (outside the list of Search Goals) +required by some implementations is Max Load. +While this is a frequently used configuration parameter, +already governed by <xref target="RFC2544"></xref> (section 20. Maximum frame rate) +and <xref target="RFC2285"></xref> (3.5.3 Maximum offered load (MOL)), +some implementations may detect or discover it instead.</t> + + + +<t>In MLRsearch specification, the [Relevant Upper Bound] (#Relevant-Upper-Bound) +is added as a required attribute precisely because it makes the search result +independent of Max Load value.</t> + + +</section> +</section> +<section anchor="search-goal-examples"><name>Search Goal Examples</name> + +<section anchor="rfc2544-goal"><name>RFC2544 Goal</name> + +<t>The following set of values makes the search result unconditionally compliant +with <xref target="RFC2544"></xref> (section 24 Trial duration)</t> + +<t><list style="symbols"> + <t>Goal Final Trial Duration = 60 seconds</t> + <t>Goal Duration Sum = 60 seconds</t> + <t>Goal Loss Ratio = 0%</t> + <t>Goal Exceed Ratio = 0%</t> +</list></t> + +<t>The latter two attributes are enough to make the search goal +conditionally compliant, adding the first attribute +makes it unconditionally compliant.</t> + +<t>The second attribute (Goal Duration Sum) only prevents MLRsearch +from repeating zero-loss full-length trials.</t> + +<t>Non-zero exceed ratio could prolong the search and allow loss inversion +between lower-load lossy short trial and higher-load full-length zero-loss trial. +From <xref target="RFC2544"></xref> alone, it is not clear whether that higher load +could be considered as compliant throughput.</t> + +</section> +<section anchor="tst009-goal"><name>TST009 Goal</name> + +<t>One of the alternatives to RFC2544 is described in +<xref target="TST009"></xref> (section 12.3.3 Binary search with loss verification). +The idea there is to repeat lossy trials, hoping for zero loss on second try, +so the results are closer to the noiseless end of performance sprectum, +and more repeatable and comparable.</t> + +<t>Only the variant with "z = infinity" is achievable with MLRsearch.</t> + + +<t>For example, for "r = 2" variant, the following search goal should be used:</t> + +<t><list style="symbols"> + <t>Goal Final Trial Duration = 60 seconds</t> + <t>Goal Duration Sum = 120 seconds</t> + <t>Goal Loss Ratio = 0%</t> + <t>Goal Exceed Ratio = 50%</t> +</list></t> + +<t>If the first 60s trial has zero loss, it is enough for MLRsearch to stop +measuring at that load, as even a second lossy trial +would still fit within the exceed ratio.</t> + +<t>But if the first trial is lossy, MLRsearch needs to perform also +the second trial to classify that load. +As Goal Duration Sum is twice as long as Goal Final Trial Duration, +third full-length trial is never needed.</t> + +</section> +</section> +<section anchor="result-terms"><name>Result Terms</name> + +<t>Before defining the output of the Controller, +it is useful to define what the Goal Result is.</t> + +<t>The Goal Result is a composite quantity.</t> + +<t>Following subsections define its attribute first, before describing the Goal Result quantity.</t> + +<t>There is a correspondence between Search Goals and Goal Results. +Most of the following subsections refer to a given Search Goal, +when defining attributes of the Goal Result. +Conversely, at the end of the search, each Search Goal +has its corresponding Goal Result.</t> + +<t>Conceptually, the search can be seen as a process of load classification, +where the Controller attempts to classify some loads as an Upper Bound +or a Lower Bound with respect to some Search Goal.</t> + +<t>Before defining real attributes of the goal result, +it is useful to define bounds in general.</t> + +<section anchor="relevant-upper-bound"><name>Relevant Upper Bound</name> + +<t>Definition:</t> + +<t>The Relevant Upper Bound is the smallest trial load value that is classified +at the end of the search as an upper bound +(see [Appendix A: Load Classification] (#Appendix-A:-Load-Classification)) +for the given Search Goal.</t> + +<t>Discussion:</t> + +<t>One search goal can have many different load classified as an upper bound. +At the end of the search, one of those loads will be the smallest, +becoming the relevant upper bound for that goal.</t> + +<t>In more detail, the set of all trial outputs (both short and full-length, +enough of them according to Goal Duration Sum) +performed at that smallest load failed to uphold all the requirements +of the given Search Goal, mainly the Goal Loss Ratio +in combination with the Goal Exceed Ratio.</t> + + +<t>If Max Load does not cause enough lossy trials, +the Relevant Upper Bound does not exist. +Conversely, if Relevant Upper Bound exists, +it is not affected by Max Load value.</t> + + + +</section> +<section anchor="relevant-lower-bound"><name>Relevant Lower Bound</name> + +<t>Definition:</t> + +<t>The Relevant Lower Bound is the largest trial load value +among those smaller than the Relevant Upper Bound, +that got classified at the end of the search as a lower bound (see +[Appendix A: Load Classification] (#Appendix-A:-Load-Classification)) +for the given Search Goal.</t> + +<t>Discussion:</t> + +<t>Only among loads smaller that the relevant upper bound, +the largest load becomes the relevant lower bound. +With loss inversion, stricter upper bound matters.</t> + +<t>In more detail, the set of all trial outputs (both short and full-length, +enough of them according to Goal Duration Sum) +performed at that largest load managed to uphold all the requirements +of the given Search Goal, mainly the Goal Loss Ratio +in combination with the Goal Exceed Ratio.</t> + +<t>Is no load had enough low-loss trials, the relevant lower bound +MAY not exist.</t> + + +<t>Strictly speaking, if the Relevant Upper Bound does not exist, +the Relevant Lower Bound also does not exist. +In that case, Max Load is classified as a lower bound, +but it is not clear whether a higher lower bound +would be found if the search used a higher Max Load value.</t> + +<t>For a regular Goal Result, the distance between the Relevant Lower Bound +and the Relevant Upper Bound MUST NOT be larger than the Goal Width, +if the implementation offers width as a goal attribute.</t> + + +<t>Searching for anther search goal may cause a loss inversion phenomenon, +where a lower load is classified as an upper bound, +but also a higher load is classified as a lower bound for the same search goal. +The definition of the Relevant Lower Bound ignores such high lower bounds.</t> + + +</section> +<section anchor="conditional-throughput"><name>Conditional Throughput</name> + +<t>Definition:</t> + +<t>The Conditional Throughput (see section [Appendix B: Conditional Throughput] (#Appendix-B:-Conditional-Throughput)) +as evaluated at the Relevant Lower Bound of the given Search Goal +at the end of the search.</t> + +<t>Discussion:</t> + +<t>Informally, this is a typical trial forwarding rate, expected to be seen +at the Relevant Lower Bound of the given Search Goal.</t> + +<t>But frequently it is only a conservative estimate thereof, +as MLRsearch implementations tend to stop gathering more data +as soon as they confirm the value cannot get worse than this estimate +within the Goal Duration Sum.</t> + +<t>This value is RECOMMENDED to be used when evaluating repeatability +and comparability if different MLRsearch implementations.</t> + + +</section> +<section anchor="goal-result"><name>Goal Result</name> + +<t>Definition:</t> + +<t>The Goal Result is a composite quantity consisting of several attributes. +Relevant Upper Bound and Relevant Lower Bound are REQUIRED attributes, +Conditional Throughput is a RECOMMENDED attribute.</t> + +<t>Discussion:</t> + +<t>Depending on SUT behavior, it is possible that one or both relevant bounds +do not exist. The goal result instance where the required attribute values exist +is informally called a Regular Goal Result instance, +so we can say some goals reached Irregular Goal Results.</t> + + +<t>A typical Irregular Goal Result is when all trials at the Max Load +have zero loss, as the Relevant Upper Bound does not exist in that case.</t> + +<t>It is RECOMMENDED that the test report will display such results appropriately, +although MLRsearch specification does not prescibe how.</t> + + +<t>Anything else regarging Irregular Goal Results, +including their role in stopping conditions of the search +is outside the scope of this document.</t> + +</section> +<section anchor="search-result"><name>Search Result</name> + +<t>Definition:</t> + +<t>The Search Result is a single composite object +that maps each Search Goal instance to a corresponding Goal Result instance.</t> + +<t>Discussion:</t> + +<t>Alternatively, the Search Result can be implemented as an ordered list +of the Goal Result instances, matching the order of Search Goal instances.</t> + + +<t>The Search Result (as a mapping) +MUST map from all the Search Goal instances present in the Controller Input.</t> + + + +</section> +<section anchor="controller-output"><name>Controller Output</name> + +<t>Definition:</t> + +<t>The Controller Output is a composite quantity returned from the Controller +to the Manager at the end of the search. +The Search Result instance is its only REQUIRED attribute.</t> + +<t>Discussion:</t> + +<t>MLRsearch implementation MAY return additional data in the Controller Output.</t> + + +</section> +</section> +<section anchor="mlrsearch-architecture"><name>MLRsearch Architecture</name> + + +<t>MLRsearch architecture consists of three main system components: +the Manager, the Controller, and the Measurer.</t> + +<t>The architecture also implies the presence of other components, +such as the SUT and the Tester (as a sub-component of the Measurer).</t> + +<t>Protocols of communication between components are generally left unspecified. +For example, when MLRsearch specification mentions "Controller calls Measurer", +it is possible that the Controller notifies the Manager +to call the Measurer indirectly instead. This way the Measurer implementations +can be fully independent from the Controller implementations, +e.g. programmed in different programming languages.</t> + +<section anchor="measurer"><name>Measurer</name> + +<t>Definition:</t> + +<t>The Measurer is an abstract system component +that when called with a [Trial Input] (#Trial-Input) instance, +performs one [Trial] (#Trial), +and returns a [Trial Output] (#Trial-Output) instance.</t> + +<t>Discussion:</t> + +<t>This definition assumes the Measurer is already initialized. +In practice, there may be additional steps before the search, +e.g. when the Manager configures the traffic profile +(either on the Measurer or on its tester sub-component directly) +and performs a warmup (if the tester requires one).</t> + +<t>It is the responsibility of the Measurer implementation to uphold +any requirements and assumptions present in MLRsearch specification, +e.g. trial forwarding ratio not being larger than one.</t> + +<t>Implementers have some freedom. +For example <xref target="RFC2544"></xref> (section 10. Verifying received frames) +gives some suggestions (but not requirements) related to +duplicated or reordered frames. +Implementations are RECOMMENDED to document their behavior +related to such freedoms in as detailed a way as possible.</t> + +<t>It is RECOMMENDED to benchmark the test equipment first, +e.g. connect sender and receiver directly (without any SUT in the path), +find a load value that guarantees the offered load is not too far +from the intended load, and use that value as the Max Load value. +When testing the real SUT, it is RECOMMENDED to turn any big difference +between the intended load and the offered load into increased Trial Loss Ratio.</t> + +<t>Neither of the two recommendations are made into requirements, +because it is not easy to tell when the difference is big enough, +in a way thay would be dis-entangled from other Measurer freedoms.</t> + +</section> +<section anchor="controller"><name>Controller</name> + +<t>Definition:</t> + +<t>The Controller is an abstract system component +that when called with a Controller Input instance +repeatedly computes Trial Input instance for the Measurer, +obtains corresponding Trial Output instances, +and eventually returns a Controller Output instance.</t> + +<t>Discussion:</t> + +<t>Informally, the Controller has big freedom in selection of Trial Inputs, +and the implementations want to achieve the Search Goals +in the shortest expected time.</t> + +<t>The Controller's role in optimizing the overall search time +distinguishes MLRsearch algorithms from simpler search procedures.</t> + +<t>Informally, each implementation can have different stopping conditions. +Goal Width is only one example. +In practice, implementation details do not matter, +as long as Goal Results are regular.</t> + +</section> +<section anchor="manager"><name>Manager</name> + +<t>Definition:</t> + +<t>The Manager is an abstract system component that is reponsible for +configuring other components, calling the Controller component once, +and for creating the test report following the reporting format as +defined in <xref target="RFC2544"></xref> (section 26. Benchmarking tests).</t> + +<t>Discussion:</t> + +<t>The Manager initializes the SUT, the Measurer (and the Tester if independent) +with their intended configurations before calling the Controller.</t> + +<t>The Manager does not need to be able to tweak any Search Goal attributes, +but it MUST report all applied attribute values even if not tweaked.</t> + + +<t>In principle, there should be a "user" (human or CI) +that "starts" or "calls" the Manager and receives the report. +The Manager MAY be able to be called more than once whis way.</t> + + +</section> +</section> +<section anchor="implementation-compliance"><name>Implementation Compliance</name> + +<t>Any networking measurement setup where there can be logically delineated system components +and there are components satisfying requirements for the Measurer, +the Controller and the Manager, is considered to be compliant with MLRsearch design.</t> + +<t>These components can be seen as abstractions present in any testing procedure. +For example, there can be a single component acting both +as the Manager and the Controller, but as long as values of required attributes +of Search Goals and Goal Results are visible in the test report, +the Controller Input instance and output instance are implied.</t> + +<t>For example, any setup for conditionally (or unconditionally) +compliant <xref target="RFC2544"></xref> throughput testing +can be understood as a MLRsearch architecture, +assuming there is enough data to reconstruct the Relevant Upper Bound.</t> + +<t>See [RFC2544 Goal] (#RFC2544-Goal) subsection for equivalent Search Goal.</t> + +<t>Any test procedure that can be understood as (one call to the Manager of) +MLRsearch architecture is said to be compliant with MLRsearch specification.</t> + +</section> +</section> +<section anchor="additional-considerations"><name>Additional Considerations</name> + +<t>This section focuses on additional considerations, intuitions and motivations +pertaining to MLRsearch methodology.</t> + + +<section anchor="mlrsearch-versions"><name>MLRsearch Versions</name> + +<t>The MLRsearch algorithm has been developed in a code-first approach, +a Python library has been created, debugged, used in production +and published in PyPI before the first descriptions +(even informal) were published.</t> + +<t>But the code (and hence the description) was evolving over time. +Multiple versions of the library were used over past several years, +and later code was usually not compatible with earlier descriptions.</t> + +<t>The code in (some version of) MLRsearch library fully determines +the search process (for a given set of configuration parameters), +leaving no space for deviations.</t> + + + +<t>This historic meaning of MLRsearch, as a family +of search algorithm implementations, +leaves plenty of space for future improvements, at the cost +of poor comparability of results of search algoritm implementations.</t> + + +<t>There are two competing needs. +There is the need for standardization in areas critical to comparability. +There is also the need to allow flexibility for implementations +to innovate and improve in other areas. +This document defines MLRsearch as a new specification +in a manner that aims to fairly balance both needs.</t> + +</section> +<section anchor="stopping-conditions"><name>Stopping Conditions</name> + +<t><xref target="RFC2544"></xref> prescribes that after performing one trial at a specific offered load, +the next offered load should be larger or smaller, based on frame loss.</t> + +<t>The usual implementation uses binary search. +Here a lossy trial becomes +a new upper bound, a lossless trial becomes a new lower bound. +The span of values between the tightest lower bound +and the tightest upper bound (including both values) forms an interval of possible results, +and after each trial the width of that interval halves.</t> + +<t>Usually the binary search implementation tracks only the two tightest bounds, +simply calling them bounds. +But the old values still remain valid bounds, +just not as tight as the new ones.</t> + +<t>After some number of trials, the tightest lower bound becomes the throughput. +<xref target="RFC2544"></xref> does not specify when, if ever, should the search stop.</t> + +<t>MLRsearch introduces a concept of [Goal Width] (#Goal-Width).</t> + +<t>The search stops +when the distance between the tightest upper bound and the tightest lower bound +is smaller than a user-configured value, called Goal Width from now on. +In other words, the interval width at the end of the search +has to be no larger than the Goal Width.</t> + +<t>This Goal Width value therefore determines the precision of the result. +Due to the fact that MLRsearch specification requires a particular +structure of the result (see [Trial Result] (#Trial-Result) section), +the result itself does contain enough information to determine its +precision, thus it is not required to report the Goal Width value.</t> + +<t>This allows MLRsearch implementations to use stopping conditions +different from Goal Width.</t> + +</section> +<section anchor="load-classification"><name>Load Classification</name> + +<t>MLRsearch keeps the basic logic of binary search (tracking tightest bounds, +measuring at the middle), perhaps with minor technical differences.</t> + +<t>MLRsearch algorithm chooses an intended load (as opposed to the offered load), +the interval between bounds does not need to be split +exactly into two equal halves, +and the final reported structure specifies both bounds.</t> + +<t>The biggest difference is that to classify a load +as an upper or lower bound, MLRsearch may need more than one trial +(depending on configuration options) to be performed at the same intended load.</t> + +<t>In consequence, even if a load already does have few trial results, +it still may be classified as undecided, neither a lower bound nor an upper bound.</t> + +<t>An explanation of the classification logic is given in the next section [Logic of Load Classification] (#Logic-of-Load-Classification), +as it heavily relies on other subsections of this section.</t> + +<t>For repeatability and comparability reasons, it is important that +given a set of trial results, all implementations of MLRsearch +classify the load equivalently.</t> + +</section> +<section anchor="loss-ratios"><name>Loss Ratios</name> + +<t>Another difference between MLRsearch and <xref target="RFC2544"></xref> binary search is in the goals of the search. +<xref target="RFC2544"></xref> has a single goal, +based on classifying full-length trials as either lossless or lossy.</t> + +<t>MLRsearch, as the name suggests, can search for multiple goals, +differing in their loss ratios. +The precise definition of the Goal Loss Ratio will be given later. +The <xref target="RFC2544"></xref> throughput goal then simply becomes a zero Goal Loss Ratio. +Different goals also may have different Goal Widths.</t> + +<t>A set of trial results for one specific intended load value +can classify the load as an upper bound for some goals, but a lower bound +for some other goals, and undecided for the rest of the goals.</t> + +<t>Therefore, the load classification depends not only on trial results, +but also on the goal. +The overall search procedure becomes more complicated, when +compared to binary search with a single goal, +but most of the complications do not affect the final result, +except for one phenomenon, loss inversion.</t> + +</section> +<section anchor="loss-inversion"><name>Loss Inversion</name> + +<t>In <xref target="RFC2544"></xref> throughput search using bisection, any load with a lossy trial +becomes a hard upper bound, meaning every subsequent trial has a smaller +intended load.</t> + +<t>But in MLRsearch, a load that is classified as an upper bound for one goal +may still be a lower bound for another goal, and due to the other goal +MLRsearch will probably perform trials at even higher loads. +What to do when all such higher load trials happen to have zero loss? +Does it mean the earlier upper bound was not real? +Does it mean the later lossless trials are not considered a lower bound? +Surely we do not want to have an upper bound at a load smaller than a lower bound.</t> + +<t>MLRsearch is conservative in these situations. +The upper bound is considered real, and the lossless trials at higher loads +are considered to be a coincidence, at least when computing the final result.</t> + +<t>This is formalized using new notions, the [Relevant Upper Bound] (#Relevant-Upper-Bound) and +the [Relevant Lower Bound] (#Relevant-Lower-Bound). +Load classification is still based just on the set of trial results +at a given intended load (trials at other loads are ignored), +making it possible to have a lower load classified as an upper bound, +and a higher load classified as a lower bound (for the same goal). +The Relevant Upper Bound (for a goal) is the smallest load classified +as an upper bound. +But the Relevant Lower Bound is not simply +the largest among lower bounds. +It is the largest load among loads +that are lower bounds while also being smaller than the Relevant Upper Bound.</t> + +<t>With these definitions, the Relevant Lower Bound is always smaller +than the Relevant Upper Bound (if both exist), and the two relevant bounds +are used analogously as the two tightest bounds in the binary search. +When they are less than the Goal Width apart, +the relevant bounds are used in the output.</t> + +<t>One consequence is that every trial result can have an impact on the search result. +That means if your SUT (or your traffic generator) needs a warmup, +be sure to warm it up before starting the search.</t> + +</section> +<section anchor="exceed-ratio"><name>Exceed Ratio</name> + +<t>The idea of performing multiple trials at the same load comes from +a model where some trial results (those with high loss) are affected +by infrequent effects, causing poor repeatability of <xref target="RFC2544"></xref> throughput results. +See the discussion about noiseful and noiseless ends +of the SUT performance spectrum in section [DUT in SUT] (#DUT-in-SUT). +Stable results are closer to the noiseless end of the SUT performance spectrum, +so MLRsearch may need to allow some frequency of high-loss trials +to ignore the rare but big effects near the noiseful end.</t> + +<t>MLRsearch can do such trial result filtering, but it needs +a configuration option to tell it how frequent can the infrequent big loss be. +This option is called the exceed ratio. +It tells MLRsearch what ratio of trials +(more exactly what ratio of trial seconds) can have a [Trial Loss Ratio] (#Trial-Loss-Ratio) +larger than the Goal Loss Ratio and still be classified as a lower bound. +Zero exceed ratio means all trials have to have a Trial Loss Ratio +equal to or smaller than the Goal Loss Ratio.</t> + +<t>For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5, +as it simplifies some later concepts by relating them to the concept of median.</t> + +</section> +<section anchor="duration-sum"><name>Duration Sum</name> + +<t>When more than one trial is intended to classify a load, +MLRsearch also needs something that controls the number of trials needed. +Therefore, each goal also has an attribute called duration sum.</t> + +<t>The meaning of a [Goal Duration Sum] (#Goal-Duration-Sum) is that +when a load has (full-length) trials +whose trial durations when summed up give a value at least as big +as the Goal Duration Sum value, +the load is guaranteed to be classified either as an upper bound +or a lower bound for that goal.</t> + +<t>Due to the fact that the duration sum has a big impact +on the overall search duration, and <xref target="RFC2544"></xref> prescribes +wait intervals around trial traffic, +the MLRsearch algorithm is allowed to sum durations that are different +from the actual trial traffic durations.</t> + +<t>In the MLRsearch specification, the different duration values are called +[Trial Effective Duration] (#Trial-Effective-Duration).</t> + +</section> +<section anchor="short-trials"><name>Short Trials</name> + +<t>MLRsearch requires each goal to specify its final trial duration. +Full-length trial is a shorter name for a trial whose intended trial duration +is equal to (or longer than) the goal final trial duration.</t> + +<t>Section 24 of <xref target="RFC2544"></xref> already anticipates possible time savings +when short trials (shorter than full-length trials) are used. +Full-length trials are the opposite of short trials, +so they may also be called long trials.</t> + +<t>Any MLRsearch implementation may include its own configuration options +which control when and how MLRsearch chooses to use short trial durations.</t> + +<t>For explainability reasons, when exceed ratio of 0.5 is used, +it is recommended for the Goal Duration Sum to be an odd multiple +of the full trial durations, so Conditional Throughput becomes identical to +a median of a particular set of trial forwarding rates.</t> + +<t>The presence of short trial results complicates the load classification logic.</t> + +<t>Full details are given later in section [Logic of Load Classification] (#Logic-of-Load-Classification). +In a nutshell, results from short trials +may cause a load to be classified as an upper bound. +This may cause loss inversion, and thus lower the Relevant Lower Bound, +below what would classification say when considering full-length trials only.</t> + + + +</section> +<section anchor="throughput"><name>Throughput</name> + + +<t>Due to the fact that testing equipment takes the intended load as an input parameter +for a trial measurement, any load search algorithm needs to deal +with intended load values internally.</t> + +<t>But in the presence of goals with a non-zero loss ratio, the intended load +usually does not match the user's intuition of what a throughput is. +The forwarding rate (as defined in <xref target="RFC2285"></xref> section 3.6.1) is better, +but it is not obvious how to generalize it +for loads with multiple trial results and a non-zero +[Goal Loss Ratio] (#Goal-Loss-Ratio).</t> + +<t>The best example is also the main motivation: hard limit performance. +Even if the medium allows higher performance, +the SUT interfaces may have their additional own limitations, +e.g. a specific fps limit on the NIC (a very common occurance).</t> + +<t>Ideally, those should be known and used when computing Max Load. +But if Max Load is higher that what interface can receive or transmit, +there will be a "hard limit" observed in trial results. +Imagine the hard limit is at 100 Mfps, Max Load is higher, +and the goal loss ratio is 0.5%. If DUT has no additional losses, +0.5% loss ratio will be achieved at 100.5025 Mfps (the relevant lower bound). +But it is not intuitive to report SUT performance as a value that is +larger than known hard limit. +We need a generalization of RFC2544 throughput, +different from just the relevant lower bound.</t> + +<t>MLRsearch defines one such generalization, called the Conditional Throughput. +It is the trial forwarding rate from one of the trials +performed at the load in question. +Determining which trial exactly is defined in +[MLRsearch Specification] (#MLRsearch-Specification), +and in [Appendix B: Conditional Throughput] (#Appendix-B:-Conditional-Throughput).</t> + +<t>In the hard limit example, 100.5 Mfps load will still have +only 100.0 Mfps forwarding rate, nicely confirming the known limitation.</t> + +<t>Conditional Throughput is partially related to load classification. +If a load is classified as a lower bound for a goal, +the Conditional Throughput can be calculated from trial results, +and guaranteed to show an loss ratio +no larger than the Goal Loss Ratio.</t> + + + + +<t>Note that when comparing the best (all zero loss) and worst case (all loss +just below Goal Loss Ratio), the same Relevant Lower Bound value +may result in the Conditional Throughput differing up to the Goal Loss Ratio.</t> + +<t>Therefore it is rarely needed to set the Goal Width (if expressed +as the relative difference of loads) below the Goal Loss Ratio. +In other words, setting the Goal Width below the Goal Loss Ratio +may cause the Conditional Throughput for a larger loss ratio to become smaller +than a Conditional Throughput for a goal with a smaller Goal Loss Ratio, +which is counter-intuitive, considering they come from the same search. +Therefore it is RECOMMENDED to set the Goal Width to a value no smaller +than the Goal Loss Ratio.</t> + +<t>Overall, this Conditional Throughput does behave well for comparability purposes.</t> + +</section> +<section anchor="search-time"><name>Search Time</name> + +<t>MLRsearch was primarily developed to reduce the time +required to determine a throughput, either the <xref target="RFC2544"></xref> compliant one, +or some generalization thereof. +The art of achieving short search times +is mainly in the smart selection of intended loads (and intended durations) +for the next trial to perform.</t> + +<t>While there is an indirect impact of the load selection on the reported values, +in practice such impact tends to be small, +even for SUTs with quite a broad performance spectrum.</t> + +<t>A typical example of two approaches to load selection leading to different +Relevant Lower Bounds is when the interval is split in a very uneven way. +Any implementation choosing loads very close to the current Relevant Lower Bound +is quite likely to eventually stumble upon a trial result +with poor performance (due to SUT noise). +For an implementation choosing loads very close +to the current Relevant Upper Bound, this is unlikely, +as it examines more loads that can see a performance +close to the noiseless end of the SUT performance spectrum.</t> + +<t>However, as even splits optimize search duration at give precision, +MLRsearch implementations that prioritize minimizing search time +are unlikely to suffer from any such bias.</t> + +<t>Therefore, this document remains quite vague on load selection +and other optimization details, and configuration attributes related to them. +Assuming users prefer libraries that achieve short overall search time, +the definition of the Relevant Lower Bound +should be strict enough to ensure result repeatability +and comparability between different implementations, +while not restricting future implementations much.</t> + + +</section> +<section anchor="rfc2544-compliance"><name><xref target="RFC2544"></xref> Compliance</name> + +<t>Some Search Goal instances lead to results compliant with RFC2544. +See [RFC2544 Goal] (#RFC2544-Goal) for more details +regarding both conditional and unconditional compliance.</t> + +<t>The presence of other Search Goals does not affect the compliance +of this Goal Result. +The Relevant Lower Bound and the Conditional Throughput are in this case +equal to each other, and the value is the <xref target="RFC2544"></xref> throughput.</t> + +</section> +</section> +<section anchor="logic-of-load-classification"><name>Logic of Load Classification</name> + +<section anchor="introductory-remarks"><name>Introductory Remarks</name> + +<t>This chapter continues with explanations, +but this time more precise definitions are needed +for readers to follow the explanations.</t> + +<t>Descriptions in this section are wordy and implementers should read +[MLRsearch Specification] (#MLRsearch-Specification) section +and Appendices for more concise definitions.</t> + +<t>The two areas of focus here are load classification +and the Conditional Throughput.</t> + +<t>To start with [Performance Spectrum] (#Performance-Spectrum) +subsection contains definitions needed to gain insight +into what Conditional Throughput means. +Remaining subsections discuss load classification.</t> + +<t>For load classification, it is useful to define <strong>good trials</strong> and <strong>bad trials</strong>:</t> + +<t><list style="symbols"> + <t><strong>Bad trial</strong>: Trial is called bad (according to a goal) +if its [Trial Loss Ratio] (#Trial-Loss-Ratio) +is larger than the [Goal Loss Ratio] (#Goal-Loss-Ratio).</t> + <t><strong>Good trial</strong>: Trial that is not bad is called good.</t> +</list></t> + +</section> +<section anchor="performance-spectrum"><name>Performance Spectrum</name> +<t>### Description</t> + +<t>There are several equivalent ways to explain the Conditional Throughput +computation. One of the ways relies on performance +spectrum.</t> + +<t>Take an intended load value, a trial duration value, and a finite set +of trial results, with all trials measured at that load value and duration value.</t> + +<t>The performance spectrum is the function that maps +any non-negative real number into a sum of trial durations among all trials +in the set, that has that number, as their trial forwarding rate, +e.g. map to zero if no trial has that particular forwarding rate.</t> + +<t>A related function, defined if there is at least one trial in the set, +is the performance spectrum divided by the sum of the durations +of all trials in the set.</t> + +<t>That function is called the performance probability function, as it satisfies +all the requirements for probability mass function +of a discrete probability distribution, +the one-dimensional random variable being the trial forwarding rate.</t> + +<t>These functions are related to the SUT performance spectrum, +as sampled by the trials in the set.</t> + + +<t>Take a set of all full-length trials performed at the Relevant Lower Bound, +sorted by decreasing trial forwarding rate. +The sum of the durations of those trials +may be less than the Goal Duration Sum, or not. +If it is less, add an imaginary trial result with zero trial forwarding rate, +such that the new sum of durations is equal to the Goal Duration Sum. +This is the set of trials to use.</t> + +<t>If the quantile touches two trials,</t> + + +<t>the larger trial forwarding rate (from the trial result sorted earlier) is used.</t> + + +<t>The resulting quantity is the Conditional Throughput of the goal in question.</t> + + +<t>A set of examples follows.</t> + +<section anchor="first-example"><name>First Example</name> + +<t><list style="symbols"> + <t>[Goal Exceed Ratio] (#Goal-Exceed-Ratio) = 0 and [Goal Duration Sum] (#Goal-Duration-Sum) has been reached.</t> + <t>Conditional Throughput is the smallest trial forwarding rate among the trials.</t> +</list></t> + +</section> +<section anchor="second-example"><name>Second Example</name> + +<t><list style="symbols"> + <t>Goal Exceed Ratio = 0 and Goal Duration Sum has not been reached yet.</t> + <t>Due to the missing duration sum, the worst case may still happen, so the Conditional Throughput is zero.</t> + <t>This is not reported to the user, as this load cannot become the Relevant Lower Bound yet.</t> +</list></t> + +</section> +<section anchor="third-example"><name>Third Example</name> + +<t><list style="symbols"> + <t>Goal Exceed Ratio = 50% and Goal Duration Sum is two seconds.</t> + <t>One trial is present with the duration of one second and zero loss.</t> + <t>The imaginary trial is added with the duration of one second and zero trial forwarding rate.</t> + <t>The median would touch both trials, so the Conditional Throughput is the trial forwarding rate of the one non-imaginary trial.</t> + <t>As that had zero loss, the value is equal to the offered load.</t> +</list></t> + + +</section> +<section anchor="summary"><name>Summary</name> + +<t>While the Conditional Throughput is a generalization of the trial forwarding rate, +its definition is not an obvious one.</t> + +<t>Other than the trial forwarding rate, the other source of intuition +is the quantile in general, and the median the recommended case.</t> + + +</section> +</section> +<section anchor="trials-with-single-duration"><name>Trials with Single Duration</name> + + +<t>When goal attributes are chosen in such a way that every trial has the same +intended duration, the load classification is simpler.</t> + +<t>The following description follows the motivation +of Goal Loss Ratio, Goal Exceed Ratio, and Goal Duration Sum.</t> + +<t>If the sum of the durations of all trials (at the given load) +is less than the Goal Duration Sum, imagine two scenarios:</t> + +<t><list style="symbols"> + <t><strong>best case scenario</strong>: all subsequent trials having zero loss, and</t> + <t><strong>worst case scenario</strong>: all subsequent trials having 100% loss.</t> +</list></t> + +<t>Here we assume there are as many subsequent trials as needed +to make the sum of all trials equal to the Goal Duration Sum.</t> + +<t>The exceed ratio is defined using sums of durations +(and number of trials does not matter), so it does not matter whether +the "subsequent trials" can consist of an integer number of full-length trials.</t> + +<t>In any of the two scenarios, best case and worst case, we can compute the load exceed ratio, +as the duration sum of good trials divided by the duration sum of all trials, +in both cases including the assumed trials.</t> + +<t>Even if, in the best case scenario, the load exceed ratio is larger +than the Goal Exceed Ratio, the load is an upper bound.</t> + +<t>MKP2 Even if, in the worst case scenario, the load exceed ratio is not larger +than the Goal Exceed Ratio, the load is a lower bound.</t> + + +<t>More specifically:</t> + +<t><list style="symbols"> + <t>Take all trials measured at a given load.</t> + <t>The sum of the durations of all bad full-length trials is called the bad sum.</t> + <t>The sum of the durations of all good full-length trials is called the good sum.</t> + <t>The result of adding the bad sum plus the good sum is called the measured sum.</t> + <t>The larger of the measured sum and the Goal Duration Sum is called the whole sum.</t> + <t>The whole sum minus the measured sum is called the missing sum.</t> + <t>The optimistic exceed ratio is the bad sum divided by the whole sum.</t> + <t>The pessimistic exceed ratio is the bad sum plus the missing sum, that divided by the whole sum.</t> + <t>If the optimistic exceed ratio is larger than the Goal Exceed Ratio, the load is classified as an upper bound.</t> + <t>If the pessimistic exceed ratio is not larger than the Goal Exceed Ratio, the load is classified as a lower bound.</t> + <t>Else, the load is classified as undecided.</t> +</list></t> + +<t>The definition of pessimistic exceed ratio is compatible with the logic in +the Conditional Throughput computation, so in this single trial duration case, +a load is a lower bound if and only if the Conditional Throughput +loss ratio is not larger than the Goal Loss Ratio.</t> + + +<t>If it is larger, the load is either an upper bound or undecided.</t> + +</section> +<section anchor="trials-with-short-duration"><name>Trials with Short Duration</name> + +<section anchor="scenarios"><name>Scenarios</name> + +<t>Trials with intended duration smaller than the goal final trial duration +are called short trials. +The motivation for load classification logic in the presence of short trials +is based around a counter-factual case: What would the trial result be +if a short trial has been measured as a full-length trial instead?</t> + +<t>There are three main scenarios where human intuition guides +the intended behavior of load classification.</t> + +<section anchor="false-good-scenario"><name>False Good Scenario</name> + +<t>The user had their reason for not configuring a shorter goal +final trial duration. +Perhaps SUT has buffers that may get full at longer +trial durations. +Perhaps SUT shows periodic decreases in performance +the user does not want to be treated as noise.</t> + +<t>In any case, many good short trials may become bad full-length trials +in the counter-factual case.</t> + +<t>In extreme cases, there are plenty of good short trials and no bad short trials.</t> + +<t>In this scenario, we want the load classification NOT to classify the load +as a lower bound, despite the abundance of good short trials.</t> + + +<t>Effectively, we want the good short trials to be ignored, so they +do not contribute to comparisons with the Goal Duration Sum.</t> + +</section> +<section anchor="true-bad-scenario"><name>True Bad Scenario</name> + +<t>When there is a frame loss in a short trial, +the counter-factual full-length trial is expected to lose at least as many +frames.</t> + +<t>In practice, bad short trials are rarely turning into +good full-length trials.</t> + +<t>In extreme cases, there are no good short trials.</t> + +<t>In this scenario, we want the load classification +to classify the load as an upper bound just based on the abundance +of short bad trials.</t> + +<t>Effectively, we want the bad short trials +to contribute to comparisons with the Goal Duration Sum, +so the load can be classified sooner.</t> + +</section> +<section anchor="balanced-scenario"><name>Balanced Scenario</name> + +<t>Some SUTs are quite indifferent to trial duration. +Performance probability function constructed from short trial results +is likely to be similar to the performance probability function constructed +from full-length trial results (perhaps with larger dispersion, +but without a big impact on the median quantiles overall).</t> + + +<t>For a moderate Goal Exceed Ratio value, this may mean there are both +good short trials and bad short trials.</t> + +<t>This scenario is there just to invalidate a simple heuristic +of always ignoring good short trials and never ignoring bad short trials, +as that simple heuristic would be too biased.</t> + +<t>Yes, the short bad trials +are likely to turn into full-length bad trials in the counter-factual case, +but there is no information on what would the good short trials turn into.</t> + +<t>The only way to decide safely is to do more trials at full length, +the same as in False Good Scenario.</t> + +</section> +</section> +<section anchor="classification-logic"><name>Classification Logic</name> + +<t>MLRsearch picks a particular logic for load classification +in the presence of short trials, but it is still RECOMMENDED +to use configurations that imply no short trials, +so the possible inefficiencies in that logic +do not affect the result, and the result has better explainability.</t> + +<t>With that said, the logic differs from the single trial duration case +only in different definition of the bad sum. +The good sum is still the sum across all good full-length trials.</t> + +<t>Few more notions are needed for defining the new bad sum:</t> + +<t><list style="symbols"> + <t>The sum of durations of all bad full-length trials is called the bad long sum.</t> + <t>The sum of durations of all bad short trials is called the bad short sum.</t> + <t>The sum of durations of all good short trials is called the good short sum.</t> + <t>One minus the Goal Exceed Ratio is called the subceed ratio.</t> + <t>The Goal Exceed Ratio divided by the subceed ratio is called the exceed coefficient.</t> + <t>The good short sum multiplied by the exceed coefficient is called the balancing sum.</t> + <t>The bad short sum minus the balancing sum is called the excess sum.</t> + <t>If the excess sum is negative, the bad sum is equal to the bad long sum.</t> + <t>Otherwise, the bad sum is equal to the bad long sum plus the excess sum.</t> +</list></t> + +<t>Here is how the new definition of the bad sum fares in the three scenarios, +where the load is close to what would the relevant bounds be +if only full-length trials were used for the search.</t> + +<section anchor="false-good-scenario-1"><name>False Good Scenario</name> + +<t>If the duration is too short, we expect to see a higher frequency +of good short trials. +This could lead to a negative excess sum, +which has no impact, hence the load classification is given just by +full-length trials. +Thus, MLRsearch using too short trials has no detrimental effect +on result comparability in this scenario. +But also using short trials does not help with overall search duration, +probably making it worse.</t> + +</section> +<section anchor="true-bad-scenario-1"><name>True Bad Scenario</name> + +<t>Settings with a small exceed ratio +have a small exceed coefficient, so the impact of the good short sum is small, +and the bad short sum is almost wholly converted into excess sum, +thus bad short trials have almost as big an impact as full-length bad trials. +The same conclusion applies to moderate exceed ratio values +when the good short sum is small. +Thus, short trials can cause a load to get classified as an upper bound earlier, +bringing time savings (while not affecting comparability).</t> + +</section> +<section anchor="balanced-scenario-1"><name>Balanced Scenario</name> + +<t>Here excess sum is small in absolute value, as the balancing sum +is expected to be similar to the bad short sum. +Once again, full-length trials are needed for final load classification; +but usage of short trials probably means MLRsearch needed +a shorter overall search time before selecting this load for measurement, +thus bringing time savings (while not affecting comparability).</t> + +<t>Note that in presence of short trial results, +the comparibility between the load classification +and the Conditional Throughput is only partial. +The Conditional Throughput still comes from a good long trial, +but a load higher than the Relevant Lower Bound may also compute to a good value.</t> + +</section> +</section> +</section> +<section anchor="trials-with-longer-duration"><name>Trials with Longer Duration</name> + +<t>If there are trial results with an intended duration larger +than the goal trial duration, the precise definitions +in Appendix A and Appendix B treat them in exactly the same way +as trials with duration equal to the goal trial duration.</t> + +<t>But in configurations with moderate (including 0.5) or small +Goal Exceed Ratio and small Goal Loss Ratio (especially zero), +bad trials with longer than goal durations may bias the search +towards the lower load values, as the noiseful end of the spectrum +gets a larger probability of causing the loss within the longer trials.</t> + + + + +</section> +</section> +<section anchor="iana-considerations"><name>IANA Considerations</name> + +<t>No requests of IANA.</t> + +</section> +<section anchor="security-considerations"><name>Security Considerations</name> + +<t>Benchmarking activities as described in this memo are limited to +technology characterization of a DUT/SUT using controlled stimuli in a +laboratory environment, with dedicated address space and the constraints +specified in the sections above.</t> + +<t>The benchmarking network topology will be an independent test setup and +MUST NOT be connected to devices that may forward the test traffic into +a production network or misroute traffic to the test management network.</t> + +<t>Further, benchmarking is performed on a "black-box" basis, relying +solely on measurements observable external to the DUT/SUT.</t> + +<t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for +benchmarking purposes. Any implications for network security arising +from the DUT/SUT SHOULD be identical in the lab and in production +networks.</t> + +</section> +<section anchor="acknowledgements"><name>Acknowledgements</name> + +<t>Some phrases and statements in this document were created +with help of Mistral AI (mistral.ai).</t> + +<t>Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough +review and numerous useful comments and suggestions in the earlier versions of this document.</t> + +<t>Special wholehearted gratitude and thanks to the late Al Morton for his +thorough reviews filled with very specific feedback and constructive +guidelines. Thank you Al for the close collaboration over the years, +for your continuous unwavering encouragement full of empathy and +positive attitude. Al, you are dearly missed.</t> + +</section> +<section anchor="appendix-a-load-classification"><name>Appendix A: Load Classification</name> + +<t>This section specifies how to perform the load classification.</t> + +<t>Any intended load value can be classified, according to a given [Search Goal] (#Search-Goal).</t> + +<t>The algorithm uses (some subsets of) the set of all available trial results +from trials measured at a given intended load at the end of the search. +All durations are those returned by the Measurer.</t> + +<t>The block at the end of this appendix holds pseudocode +which computes two values, stored in variables named +<spanx style="verb">optimistic</spanx> and <spanx style="verb">pessimistic</spanx>.</t> + + +<t>The pseudocode happens to be a valid Python code.</t> + +<t>If values of both variables are computed to be true, the load in question +is classified as a lower bound according to the given Search Goal. +If values of both variables are false, the load is classified as an upper bound. +Otherwise, the load is classified as undecided.</t> + +<t>The pseudocode expects the following variables to hold values as follows:</t> + +<t><list style="symbols"> + <t><spanx style="verb">goal_duration_sum</spanx>: The duration sum value of the given Search Goal.</t> + <t><spanx style="verb">goal_exceed_ratio</spanx>: The exceed ratio value of the given Search Goal.</t> + <t><spanx style="verb">good_long_sum</spanx>: Sum of durations across trials with trial duration +at least equal to the goal final trial duration and with a Trial Loss Ratio +not higher than the Goal Loss Ratio.</t> + <t><spanx style="verb">bad_long_sum</spanx>: Sum of durations across trials with trial duration +at least equal to the goal final trial duration and with a Trial Loss Ratio +higher than the Goal Loss Ratio.</t> + <t><spanx style="verb">good_short_sum</spanx>: Sum of durations across trials with trial duration +shorter than the goal final trial duration and with a Trial Loss Ratio +not higher than the Goal Loss Ratio.</t> + <t><spanx style="verb">bad_short_sum</spanx>: Sum of durations across trials with trial duration +shorter than the goal final trial duration and with a Trial Loss Ratio +higher than the Goal Loss Ratio.</t> +</list></t> + +<t>The code works correctly also when there are no trial results at a given load.</t> + +<figure><sourcecode type="python"><![CDATA[ +balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio) +effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum) +effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum) +quantile_duration_sum = effective_whole_sum * goal_exceed_ratio +optimistic = effective_bad_sum <= quantile_duration_sum +pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum +]]></sourcecode></figure> + +</section> +<section anchor="appendix-b-conditional-throughput"><name>Appendix B: Conditional Throughput</name> + +<t>This section specifies how to compute Conditional Throughput, as referred to in section [Conditional Throughput] (#Conditional-Throughput).</t> + +<t>Any intended load value can be used as the basis for the following computation, +but only the Relevant Lower Bound (at the end of the search) +leads to the value called the Conditional Throughput for a given Search Goal.</t> + +<t>The algorithm uses (some subsets of) the set of all available trial results +from trials measured at a given intended load at the end of the search. +All durations are those returned by the Measurer.</t> + +<t>The block at the end of this appendix holds pseudocode +which computes a value stored as variable <spanx style="verb">conditional_throughput</spanx>.</t> + + +<t>The pseudocode happens to be a valid Python code.</t> + +<t>The pseudocode expects the following variables to hold values as follows:</t> + +<t><list style="symbols"> + <t><spanx style="verb">goal_duration_sum</spanx>: The duration sum value of the given Search Goal.</t> + <t><spanx style="verb">goal_exceed_ratio</spanx>: The exceed ratio value of the given Search Goal.</t> + <t><spanx style="verb">good_long_sum</spanx>: Sum of durations across trials with trial duration +at least equal to the goal final trial duration and with a Trial Loss Ratio +not higher than the Goal Loss Ratio.</t> + <t><spanx style="verb">bad_long_sum</spanx>: Sum of durations across trials with trial duration +at least equal to the goal final trial duration and with a Trial Loss Ratio +higher than the Goal Loss Ratio.</t> + <t><spanx style="verb">long_trials</spanx>: An iterable of all trial results from trials with trial duration +at least equal to the goal final trial duration, +sorted by increasing the Trial Loss Ratio. +A trial result is a composite with the following two attributes available: <list style="symbols"> + <t><spanx style="verb">trial.loss_ratio</spanx>: The Trial Loss Ratio as measured for this trial.</t> + <t><spanx style="verb">trial.duration</spanx>: The trial duration of this trial.</t> + </list></t> +</list></t> + +<t>The code works correctly only when there if there is at least one +trial result measured at a given load.</t> + +<figure><sourcecode type="python"><![CDATA[ +all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum) +remaining = all_long_sum * (1.0 - goal_exceed_ratio) +quantile_loss_ratio = None +for trial in long_trials: + if quantile_loss_ratio is None or remaining > 0.0: + quantile_loss_ratio = trial.loss_ratio + remaining -= trial.duration + else: + break +else: + if remaining > 0.0: + quantile_loss_ratio = 1.0 +conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) +]]></sourcecode></figure> + +</section> + + + </middle> + + <back> + + +<references title='References' anchor="sec-combined-references"> + + <references title='Normative References' anchor="sec-normative-references"> + +&RFC1242; +&RFC2285; +&RFC2544; +&RFC8219; +&RFC9004; + + + </references> + + <references title='Informative References' anchor="sec-informative-references"> + +<reference anchor="TST009" target="https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf"> + <front> + <title>TST 009</title> + <author > + <organization></organization> + </author> + <date year="n.d."/> + </front> +</reference> +<reference anchor="FDio-CSIT-MLRsearch" target="https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/"> + <front> + <title>FD.io CSIT Test Methodology - MLRsearch</title> + <author > + <organization></organization> + </author> + <date year="2023" month="October"/> + </front> +</reference> +<reference anchor="PyPI-MLRsearch" target="https://pypi.org/project/MLRsearch/1.2.1/"> + <front> + <title>MLRsearch 1.2.1, Python Package Index</title> + <author > + <organization></organization> + </author> + <date year="2023" month="October"/> + </front> +</reference> + + + </references> + +</references> + + +<?line 3102?> + + + + + </back> + +<!-- ##markdown-source: +H4sIAAAAAAAAA+S9+3MbV5Iu+Hv9FRXsmGiyB4Qo+THd2pnVypLVw9uWpZFo +a+/t7ehbBApEjQAUuqogmt7Y/30zv8w8J089QMp2xOzGnYiJlgmg6jzy5MnH +l1+en59nWVd1m/Jp/vqw6ar9psy/q9s2f1d0VZ2/L4tmsc6K6+um/ERf+e5d +K39Z1otdsaVfLZti1Z1XZbc6v97e3pxvN4185fziX7Jl0dFXnlw8+ZL+6/zx +v2RZtW+e5l1zaLsnFxd/uniSFU1ZPM3rfZvd3jzNvyl3i/W2aD5Wu5v8Qy3/ +++emPuyzj7dP88tdVza7sjt/yW/NFkX3NK92qzrLFvWSvvo0P7TnRbuoqmxf +Pc3p/36XL4od/bXMi6Yp7vLTapUXm01+V7Zned3k66Jd5+uyKbM87+rFU/6A +/tnWTdeUq/YpHrEsVwUtTkvfsM/vtvIx/2dWHLp13TzNcvzfuf5vTkOjb7ye +53+pd21X7Lq7XX1bLX4On8sKvi4WVflx8kt1Q9N6UbUL2o27tiu3bfio3BbV +5mm+/Sg//T8W/K35ot6Oj+THef623hQfe+//sSm6j3Xvo/vf+qnZ8y/cS7Nd +3WxJbD6VvBTvXr14/OTLJ/rPJ0/++JX986svv9R//vHJ4z/pP/90cUF/zXg3 +3UOu3l+RmMjKdkVzU9KGr7tu3z599Oj29nZedm01p7E+WpYb+knziP/w95v2 +0fevfjynHz+6uHj894s//Yn+l/7/i/nFl3P6w9cXj27av+tX6JNPF19cfHnx +eD/fL1fyKjkRJ/RxTp+f0B9fvazq8xfvL6/OwyEYH9airbr5ajmv6kcLOiXt +o21J0rGsN/XNHf27aA9NuS13XfuITkfx9/2m2JV/79Yk4zfr/aF7RAfo7/L8 +R8lYXr2kR+Y8gvyqbLv8dXxsfh5P5gl+FA7eF+ePL+gvb+/eXt438P3dXtZy +39T/WS66R+H7jx7Pn8wfp8MJH+b4cEavoPHs8rfF4mNxU9JRXZY/jQ0mOz8/ +z4vrtmuKRZdlV+uqzWmdDrwmOb16X7dlm5c/deWurUiw+dD9VcXmb3lcqFxf +f32X0fmsdqwpinxX3uZuwen0bzblclS3qSY7DTM5m8dlzItqi1dv6cnb6ufS +Xrc8NPzr3SxrD/s9qYl8a8/e8LMb9+yynWXFbpmXu3WxW5R5U7b0XfqffUlb +f11tqu4u5y/Q6dkXjf5lzotS0lJUpAjv6NtFS+tKx0IWhfWcW5AKoyyWS3o4 +/ZN+uFjznHc3pJn42U35j0OlIkcPLVv6B63I9R2+XH4qNgdMCAPpSLLo+Vm9 +IhW46m5JO59fFy19n7TurerjVhTC73ljixwC3PKg6/yGziAr26bNt3VT5qum +LJf1dubWlXb4U7WkHaYhV/zeYkPT362qG11Yugv4f1paX96GltU1aUR6b1jo +ds3r3jVVsaEplQ2tfLFkHdeV9L809npHu8HiLV+aZV29KfnxkJFF2XRFteNf +LmgxWFrpN/imblGb31Yd3QvVzRpPb3Ujdcv5Mbx4OiXemjC2m5oHhZ9/ou3j +r0a54GVi8d9Wy+WmzLL/++lT2nremv8nw1F5Tj/FCuZ/aYrtsr7d8e7S+pBu +62g56y1dF81H/mCGH+i3Oz5Ft3S/0UTs+tQnt/mu7vgp12X+qWqraxokTZ7H +35A00dW3lEt8jgeaNqALZfGxpteuaPVZvT8qHn355IsnX3zxpwt88ZIuUnoy +LxAJW0UrTkOgsSxnYbz0UhvY03sf/uTij3/86k9/vJCF+Ctdxn/jBXoUFyj7 +Xf720LB+gKy+X9T7Us+K/pm3MdEncjpI3BZNRfN/mBqYkYxE0R5ROV7BsLSy +flhCDOzQZO64vPzhivf9+X6/gTh8KnbVZlOMK7XrqiXVy8eAxm1PwyMyE03a +vLaktSNppcNE27lt6eI8z7+pdqwvdIxd8bHkydckfiz1fCLbcGj4oUs+Jaui +EbGijaSFp+fuugOduDuazoEVQhjanF5xBaGBPqH5Vnx25diYVsTG7IsDq/Br +WoKy3Okr6eE0WoyFRa/GBDZ9rcovee9nzWf9liS4au/SE8riv6HjjrNICpBW +jjQT6TfRACQOIupLNk8+QcrdMrPWY6XFUwp/1JnR6siRpceUP9ElRYvxc9nU +tE5kL7Wz/Jq+y3OodkuyY5s7GsmKf0tLR1+FumpFGrY8Rf76rt6d4xn8YHtx +mcvFJVpv5cdXyald0Ayb/HZtqxjUEy9PtaNz1lak9HadqF9/C5ho5HwO6AU0 +/+KaFoJ0IX0a9TFr/ESg2325qFbVQvaz3O439Z08clWbJtYLDeoFsvecP5lQ +0D29zPskkzHNzYrnPH+zl9tgczfLVV3TOf8sJT0PI5HJ8VAT5SzuTP5n1tEz ++fmyWq3KZqCmeUTPd6nMwZUo6OuLLi8Leo57HJ9KurF3C34Pj+Ny15beOFjU +RcOKGkaXiEf/GTP6S7OpaDK0TiQAdHfCK+EtoWO1wdZWW1oIO1Qyzm/1V/Zs ++qBm+wUv4V/RZ3QwyFDHCeOf/EBDeVU3dMxwgN7xWp++eneWFzTk4qdqe9jS +cq9wbHiLRCerHf+3/NSU1Bfzr+dPzniMkONiw5aSnA38p40JAk8aiZaQJBcC +vSzp2/Ru7IKX5XSTZbzvSr530zNMZwQHjGdO4ig3Mv5AEg2LIMiHSgc/6b3+ +BF+gDV1WbJ7KiVLrnFVb00LhQWkHdYvftOVGJ78uDw0NuVrgkmmLTyX0p5g+ +vFH0QjazxApY01pDU32qq2V+2O3KBW1N0VT0AlLcDUntNatcvive11u9zMhc +S04bxgkXaYNLh/R6PMvJ0ZWD3rC3tuufWTyFHNyik0dUdMrxiRz6ZUn2EZ0Q +esKBvksHqfyEKfBSJrYqlrimzdjQJQi7aHXoDqyctmzoydtoQnGMo7aezGpT +/gSVXe54i2mtgol9TXYG/5Ik+RN8Q9qFjl/X4sIpbm5Y7/m/z2EXjP+Erw3e +CZWw7MDCZ8Yo7QbPcFORKy4CFO7pWcaqnK8wkp/exYVx0I3flql5P89ewBgi +qSG9NjLQMBrozOGDYXfSVcofJU+O95C/FHBrTIyf9uH7mm5nWlCSrcSFqHbs +WZRLtRPr65Y0cHATnEU1Zyvscsl3yqqiz9/qNaOOHPkeex6pWVy9u0iUJ4vJ +odVpsKtBhnJVH2DKQ7TZV1JPxF1NFbtTLM18ydI14q0d1nRxQtMb6s2Z7He/ +IzuQXqEq+KWteGKWQ/F9R4eTtHvd0ID/Jgbqv7KZ/b+//svbJ/lff3z7t/zq +zcs3T+l59Ue5cuTWhRHRFxW6KkjSZXo05nb+r4/wsGMPJmubtxZG2UzsBtp4 +Oo8dLaqsuPrAJ9EzOYlPTgxpPhp0NEkz80o7Xw8ml16MZtLEDw97VZV85LLd +YXvNV9VKr1D+Kiwk3m5Wbdi9eqk/SM49beSa1c5uwd4tixkfcB54fGpfGNgA +VsuThJSOVLUSF5bGrEMTF3Zd2tgX8Ica0rXisbGLgEfNRVq3YiM7cxTWUPlT +uTiIDc63bdTppNTojIoc0YVzXdIAS3XXF4cGSzBq1Ts7jw4r3W+LTk9GzTce +XXcZCw2/7rxckQavMPhUiT6XQdCm8Vih/d0raH0X5Z7/97DhiEPBmnR1aNRB +i7o/u11X9BEfpE6vBjmjaiBAHnAhNHr4Wp1i9E3k7zO5yB+uPiuO6yxEAfK1 +ys6f7ETF7sKiQJyW3n6zwzVGjyi2JPHQV9gLvvP3vMzBBMrwEr6ZeUWLbsbH +qym7pubbkP4TV4vzctRvoTuubLbVrkx9nCzu37IuxQqXa5UjNTTmrt7vxbvW +OeO4D3zEWUY2mERCig07JXdkAHxiCaX7dAFLjp0WXJs8hKZYludkcWXmNY3d +L3IT09edNffv9S1bJzOxJlcHEuKvL85po2o+C2KBL+sDSUOb7jNGyNPbFmbn +trBwr2E11Pweu91pTw7XiE5XMIPEOXKjYF3K55DW/v0PV7qKMBVFVZGXcI4v +FC2sMBYndZF5/cwOXZKdwce9Jsmq2KChTT9sDpAPM0dpIaAn6SDt+XqPVtuo +ffp4/vgMXmX68gX5MvwduYl5NjYcGUN7dBAYAIemctZyMFk6i+U9eFxPzhJx +U+MN7ifUEI3rsJczVOwQ+2vgcNNnbCZgVIXForJBfG4mBh7/oSKdVkER+DHT +dsw4BSJ/w2/4Mtb5z/iltGg0xkuxrRdFm14W8ZUzfIGXWM14yAcLV823tWrp +TN5kPyfddkP3hb8uvIcNL6ZeZWzAlQ3MZ/yM17jeWzRPxk1yv1voxYIFY02Q +939JO1MfGt7blv7GorRoWJRZ8etgcCdtDiyMMJ/7L6LV+HPF9wFeYJOuMC95 +Jhy+c9LqrL3K3aeqqXesdzPS6Gw5y0Tt3sJksDP8Ibl9pKBn4dMt7ndSV0sO +pyG8kJU/0ZAqXNxstDW4eRdlVG/9IWe0xfKKsPLxK7W6TXxty3Il3odahuEl +qh7Y1L9h32GekRMJB5+vl1lGmnGnIa+wvaQaSyjnOvgn+Yu3P9CEG41QV2Rs +miDzFY1P8C7SMByQYqnq6ADkLW0u3Hoa5Us6Z6Q7OdCFsC5t4opOHDwI1kzF +QQSCpk3TwRKyLwULlK9+1uUV/YiuOk5E6Jr/vg2KwBkgczkDzg6emW+mz+Nl +jTqF9qkpadHUWCG5ZzHhKFY5E3uiWkn43f2cjwt987Cj0S/JQuAbnI/MLYua +D7RKuIIDWxVdtDusFU0Ej+eF4WPI9wwfoPKndXVN53+1OSy6g6yyt6yqjjyT +1QyXmGQa2owH0pQbOIcI4eLoQEmyQuDAAQfG2ESg+wnqWkN+sGzVeOKhcwhD +I6OQI9ZfJCtdSfeUO/dYXlxDrGNarHTGO3Rdiurjm5yvvbI5J3uSzAP6hK5T +MshncF/J1IDZI64Le3zwI2mhMxpSka/v9ryA5KrzABZ0OMmYZ+uF9QlHIVUv +yAxlJbEnJV/QOxH195yPsPeqK9P2IlJ4G8fhyYKqNmR10NqEJ5IJJ3cwyVe+ +J9OJBHNn3q+FB5biGByqlmyIqid18muyW/Lf44m/t5g7/zIomCxY02HR/ZYH +UVD3JGhnNZ7JwDywdyGroF+Qm+C5Woaw6ze2XP7hXv/T1lWI8rCpiqchhOW+ +LlkVlwmSd+qpEO1SRLXxnm8seAljD4MabltaKHNjF5pS10uBd1rEQ2Qp/JD9 +T12P6xLKpKNtpUtInqSbx2aDKAtbFBktnzVO9SCrQkaqaBz+F0coN0ty/DOc +eR4EBEQvYIiOPDc+Kf72mm/Ym011wypN7sQYZs4qjkTcsn0JpW/T6u8HnY7w +WbXrh9aCWsN+zvDrYPaKvWqDTddrc6ARVfydVjeoaw50O77ZsS20tAWyD5CH +4eU7Lg76YLl5VeEt7dLRTG4hP4UC6SnoPWeUyXaZJdPMFhv4fXXcMv7x2CjX +xZ6ci1aMloZO5eZOtkcCOzCf+6OlqYlMwjmTH/HKsKfCdjw2PO50A59RfkE3 +GudlR8fEO+FXD7ZUUG0ZuUW7m/5uPieFP/JkrPLEo9klO+w21UcedYj96Ftm +IhtwvMw7w2UqeY3krDKOJ4/PqRfkD8fgtwg0+TKSgNXlyH7YYWxm+aopdItb +MtF7v+eA+QImeKYmS/Cc6YVe5++WrZ+o3y+b9IyvJnLj6jZecLZiteSyP7Bj +i1c6K+mlqZ9rj1KiBeAQZbvGDQQJ5wBVtc1YJ9v9EQ3heMHAZOP/9KiQ6NFl +nXrGGlrgODGHgMzIqbzEFzhgTamWBN/lG8SmNZqa4YTTYf1U0L3EP0/vmv6w +2IjwF8UsExgAe6JAO7W/z50pCjUeHr0hu4wMzAOCI89ZJcu+35abzTmUcnPA +hR4U+kuxolWAyCjJDP7gN5DutW5T7srFx2iQscUw4zQ4qwmNiwVbLsN2TCgb +WbLpMzIuOiWkVTOk3vF3rrjoLv970ulNtWBjVEENbF3tFnczmDka2OpBIloa +vIbLELllW5dNwRCSDUGlGdZvdJYZrEADsagWDpfYYJeRJ9YhBhHCZC2hikBV +AaTHrpf8TeL8PqVoWBqy8z+Vcb93sswSHM5oWhzy3QaX9vN2JpOjGkMqMZLY +v/Q4r+dDa6YDhon+YnPDMd81OdM0B759Oo3FZ0n6asQlloADjZunCld+ZjpC +cqoZI6C6ksPaNLZFfbOrfnbBuNZis85zCEgY/o4ltjnGmZ3SbH7iJ9G6nB2T +hs5FazckrRuNUJO0kePCMYzUYuXAzrsBXumFzwGNx8wONzfsMyDNwj8fri5d +WPQoaJv/PLB7sSuzI9l62Bt0yPnp16WP3zFCwFIjGz0kdjfPs7d13cuc8PLt +kHfB1u/5C0lWCxdEXKdE3XclRwn41NbXQBEZymeALpCQrnnXUHUc/SyXtoPB +r7FXsQuC9QhYqRhgG40rJ9iu068vcon7mVfCoTK6CySbruIUDDqJ1XMs78yS +2r1DYPkxRQvF9D85OGIXsaAEj+ionVV17fix9Q5Dg+DEglR6SJ3EUDDpIM7s +J96DZcA20RoKAVML7gMXFJ4J2az1TTIv+R0Ow8BES7NM5uVxrgzxwYLxU3nB +Dl+IvyRaKxtxkcJdEjByATpCj8Xfbkn1w1xgh4DzgpnGZ+Hm8tAOfPOJyoP8 +JtuW5B8zAe14+VeVa5a1wkYk2+ewlBGxtyhpWcsMJ6uy5B9b/i5rjwXsJW7U +8al4/6Nu2JalbJtkQo6gYHZJ8vL5YlE3Bvg5dheYOzOWNiVXkDE5Za7xu9FL +Jjo0Th7o9pKQkr1+5BLDw9flZi/RF41FTqw/TSlJDu8YPNTpnX8P7FX3Qm6x +tvd8vWfUA66A8WOVhmkuS0aTWA7FG6S4GmUIO0mZS3peUXRw/e84nRZlI7Xw +ZxoAR86ePYWMgz0KkTgt5zdzMhJpfjdiMLJvvuSwB4ecMekzTQepYHJkQ+0N +zj9InB4JPPpJyfdxmpYThVB1LrHXT6OY6MekHv2FLGQSn6Xcdg4Qhu37vt6d +/w+GbjFmUPQw49rTYP6/uJ+dWc7D71rRCgKSD6MBbATh1GmaYcfgKJU/y3VI +PFyyDYzYa8hridhdidTLCd/hemyLO33P+4oPRCmh6jKkbAD5g+Jnv9TQ2QWf +RLrXsO7q57kcHJ0lem6OvKGATgvFeugFZxAsaABaz65e1Bs5JoArHToBq4pb +fJCLgT79uCOFzg8o2ALdhIXhIUmOpzCjN85XoiQCDKG5B6Mb8cORq5rTbXuk +sNhrnWUJFE/h2uoKMHbXJlU1BvcRWyV7XdORKCJchd8H7NtrUjbNLpm3otfi +FauqiJdBLI3oeWKJkEiNosWnIx7+W/brgpPMEOL8Sq4DWr9iyXvTsvLia0iv +lEKCT4g3BRMkQtdzS72bKbJYO/sA3n1ftzo3gWFNpUCKFKijQhPni0G+V/hj +o9e7Ww58H4IsPidiTcH7D7gujc3npqpMe7gnyQYmMa51gH0jSadhB3XKiqYL +AQKsilz/pgjZsEP0HwsJk2ZzJyi8ClDqsSim3Mf0l3WxbyUsZwuLjXJwY6zd +RsUewEEHRYbdiKUDuDov9jDrt0W8G338wPy43eib+KjxAZPt7kynshZVce+9 +nl78rrwhfYz56dtoSNWS75Ra6qYsrr2NpyFWQvDRSY+XoOFFHSN93vOrZJk5 +T36+4lDCkiNuo9eexsCSr+YWH28wbprAKwE8bHFhB3dJ8vAhAXU3WrIhRrsV +bJAnoUaQWMuGlq0ZPrDEUjQkCJ9YPwpsj5YhMVYVcyFLLTpv5xa2vxKcd7sf +tCpmBRsYKbB9xBrKgjWkZlAASSqa1TsD7h0Sddf8zsxZL8ADLElLFvuA7cZO +84ql2w44BvkNS8Yili7OogFjqc1I5XcsMj/L5G7UzS8/wfgUWzuMHVrb6epo +sOpMOc4NASKFB/+WzEMDDxkYP8HdyxbLQZbdDb5cAMrhhZJmZhNH7eGwDIOz +9Y1YJjoT/LC+/gTYG/uvXe2hbL1qEv4uVmONoADExx25cUkR5CcPKpqbZN5c +eqSt6Ix36udlH4AM0vXv+bSh2soyfOoiAGfFN0fmopgKJ5lpTBL58aZqPwJS +TwM4cF4KyIQ4mgWtYJbUDLiQDZtN+w2723Gjdnzvx8e1IyDiVtY9KhQ6XZxt +va7VcZSolljN7KVOwpAlF8d5u02xkExnBU/wp05cYUCG5VOJ/csgn3w50zwj +oDmpjyqGhMHaZhmHfQ5bUj13Id5Og7wz1Z2MzrkWsnsSOIjA+/gWGaBCgoYj +/Hr+hY2RJ9geFgrRwuk+x4FWrJUMV4CES670GRmxwBHZ0Oz9PBNhMKdTTHqB +GCYzszPkfkoS8K0oJANolFJ2YIGOLt7nOJin+CedjRjFSRflLKxbLima+MX+ +RTpPX1TEioNf9CKzbYavQWCSzioHw47EIpFVV/W/YIWs+oNBEtXuUGbuelFz +yMcTEYHunbt59jL44O24Ez5mmBpqISRoNdfAxhJ8xSI/7CraYitNoxtoiXt1 +fBDZpWRPTG8EV70XIpAyz38cCkFdyR2BpHcER7aZm4M3TESsTiPyJpQKpU60 +x0yezRQ1KXavxOUcUJ1fsCh9qponLBBziXn7hORg3qyjSIdVG00P+Omy7rzh +Msr8sOdIP6z1umk4XqcvwaqW2bTyincBI+7Lnb8RXPnAe18+oIBu0xIR0D1R +bhCBU7kAaRfrHacOsqWTLIcHscJTDieuS2xEAeu8WhwAnODocQw9CahUnZqp +McxHoNt/rmvJyuGtF39kR/GOZYpuo29ef/izKO5nCaqbIdf5j2+f0oXKQWHy +5N75QCuLzjvTgqJiH72XeDcKV/lB/Lubptiv22f5aUzgJQFbKeJCIP/QAnoL +lShANi3e9x9KVAeJxAUtkZgwqCEIOM0lNoBOh0Ik49vOxqHgZBG8oWv0U0U/ +yaa2Nm5+YWhJK+A2AJ6Dr2WFQPzJ0guwPE2ThERvtaNjKStJ4+Z/A12Sbjpu +qaJainKDQjm2+5mEGQ0uIHhoqWVcVmw58msZjbUrNvUNG15+0IDPiimHcH6y +U2b94eAxvBPL7Ita9EFqqrwWU8ih+hBDI5kTv19NLDOToLFfkPpualwNE7+S +yiOLVwPxzAu4jKB6fs5rmt/NkVc35blh8REgau4gRAL/4c/r5UGNTGyHhPPE +hnB/4OzFplpULNWAdLWp+XmqTqab1iU2/SyTcgC/krCW1QzNT91P3ohwnI2d +7ONFGVI28Uos+RtWAkDkCVZSt9NNZ8Zoja1CP/jy4qnx6dTL7kglRVhwErXB +nMVu7/3xY1nuBbRnYQITmEyuAXjaA6S5FamZlKGSRww3LzxhGEEKq9bQM9E0 +mMPa4KOKrB+KgTWuaBOS+gY5U8OpZadAwbGx597elN2h2bVnoYI1BKAVXmLl +R34kk4qH7RbVPOTi417opWIk4Ky6mO98DtyNLN08v2ymHgEQIJdMcryA7XAY +3RwJQlgqWnZN/xYYe5H58cnzA3hCyg43DAOOtxbQJgEuLlaF5AMWnL7SXxmE +W4oAJKLGZXqC/NdJscOodXvXvCyaE4cny/u1IzNib16tHgQyam/g+GMc4lMk +oQ1Zos5VWQjeFQk/tflOC5+dgU8e0pdnFkaQ9df4Q9VKOdPcFoxXs5VQqMRX +/Net8pGdcZa8aiVYJPWQJZCPcHet0USc5Kb3lDPxgl87X/U/xI5k93NSEIFu +LRymwDu7/wgPEESdGqGzSVMpZMIM+OHsHjKYBZGg+Y+Q06Dlevftf/xw+e7b +l2MKDBdEh4zCRsKZeFIvmiOhrjhguy2X1VboVhD7k2m2wygPKQkZj4Jz6Q7Q +zGHRukrtUPPNETkk13e0M0AKMbfCDfM58SIOx+FqUgu5wVrGeJupPwPAEAii +qoPnCG9g+EWn8jiOanBOxZvGEYRPNPyX/s6A7aKvim7kRQaXDw+KWG66ekKq +Dc453A2NMHq/nl7Ie9+VS/c990SFHusr9Q06SlKtFQfNEhkkAf/WCgyvyoar +M+kay9mFy08Siq0rYCgEpcNC9b1i30G0Reu8Y4PjpVSEnHCuvoMfQeqMfBxn +2SPzh5fwZdl7iWcq+oyXRNO3VacvInp4UnkPO+wRR15Tz2VcT/74FX9bmWni +E9xrZmbpwkXlbUUAPacdV4AUR4E0ayguIKAQ2XMcOfLSAmKx7SVhSfeUy0xv +VNracrvvVFdydBwkLn32EhQ5JG45xsSHrEEhEKaAwoN9JZVRNhvJhDh34mGu +EagqEF0QGALt0jx/rp6S+UjPY4KYXXVa6MXHnJFiliM4DdWtiAyTq842ypkM +WGlrULRX7T5yXtgqm3yVQAOcnbzjTGIdYSXmPWPvC18o+62rroiIFQwP4QSF +D9HqaB0gU99MpCfJTOM/Fg0uy1NxHHUNy+V5zbleiQuUS2REJyp5vyBnkl9c +Ps0RkkhWi4UzmKaavT3PH5Jmns/nblncDyU6c+oifI8HP3Sn5cgvv5xrnCqE +s5Tl7nw8nnlaqZvuGUDO7CcMUM6/vhB/KikbDUWjEptkeZknv/JVzfmp+bv/ +ONCiMn6JbJqiUQhs74cJhEd/e+yHU0v4Rf4qyTqWcSnuCZvmvbBpfurD0Phi +BRgIr6+O4bB79LCq2qNjTvkUgZx4GnaDhY9tjZMlh/BUT5yI8XQihBuai9oW +3dMTF9IwesG43c4QCc9/iCTeFrwQeOpMs7otU4jQd//PvPipao/tyRdBOKG1 +9xJulVizjkFNMf+NgAi17zCaQRHqwUxDEFIDRSxFzMOzixWKz3RUL7TAJP+u +FoKb0WP7Zfo92+KXDLwA80rCxhB++JgO3zfVsmrkv8G1VHBxeFhhcVnpqis2 +QsGCWCdWEVEr99DxxTq2uo/p7Zy6vwuvNVBni1iqFdeIV2Arcmn5Mb8ig+LX +r0gMLpNM2ukl/8/ZmR+Q/Oq9+9WT/I0jpclP3yQ/kvNpEfVycj++7n8zPKDH +iFMoyFpu0wn6m8f+Z40R6ZwdWdmLObnYgnSJmzUyc57xF+GrdTLz12++C+9o +SnK8HF5oVJjopT+CM0GCW4sSZqP8wM0/ugruIc+5LGZZ/ZS/OMPV1eq4RTWI +GWXwl4Am1Js+Fyh5VH4wcUdG+Me5bos+9cx+nqQc4oSezJl8pPfORKKZXdT9 +ghaTzwBkOX/PiOwztjck3tWalAtWuzVkI5AjbHxuafr0j8u3n77EkOgfX9tv +pvf6TzYp1mqt3+OBVH4lCoGNIqcJR8Six7306h3JwrFvcQU8CvyOfoee9D79 +TjoTukyutCo955Xrl6ifuhr1My1Sd09jete/5d88+ebz9mclMQurNuUU7f7L +f672X/N+CG9C2IAp28sZhy+V8UDIWRQcDM9uKZ+w8etj7jALxkN/nxeQ7A/j +VsF5eKGuz7PxFw3ec2mjPTD8GChKKRK+5LAE+Q351R25qc9AzoPcLNzRQF6i +ZCfzo0w2sKPFxQgBn8S5kLrBclvLOYdDaxAov5qcMtAsCszeU4D9mRnUWRRn +aoFzgtQ4hCS2OJXqeehaBWyoFBkARqJpu0pgkBf/MliJZLd+NKoZWoV7Izq6 +RHgcb4bqj+ARyYp6+1VDwBXH56ob5SYxo35imr8jDx88Gy/vuZ7oXCt/c/4D +sOo4w6f027MzJGjgIbWJp/lUCW1+HUPGQ9kx+NXhvD0FAtnxSAT1q09Kh4Dk +qIvczmVlXj5sZR5rzCFZmZcPWJlfQVxy7/RRYyhrd7g+jzkcqSOY4XkIbO7E +MKR/BoJHDk5MkcHBtcdgyK29swCv5kLYC4iAvrFQwQuLZYh3rJECLhJmSjhT +enaOeBOu5Abonys5Us+Xyzwuq5RJYRUQ9A9aYUqjv/6LetPfVD8XDSkmdmgr +Dqf9vhPwlpKk+Dy6OMjNasE374z/QTLxRw468D/pspNJIUgAz39T1+w2zRmM +aTgmfgGCoPys21I0nEUnUD2aPz0biwI4fXIl2AaAejmeE0NGrh4Twbk/Y2Wu +nh/RArDpkTqFca911j7JgovaMrjxPDzEl+oLP83m+SBH7wt4YjmAoK/y/FvO +TRgYAakhfgqv074qJbIaKN9ZTFeOWyIA1w0o7wECGa6L1OFJ35dQEXhisTWT +TMh0ijPdXatALThCw0htgVQjbVEL2k05xhS2fYKXg+Ac4fdAfCvRAKvDQhyf +T3qEsdvzGArJLIAbaAF6zvVZTnaPvvWk53KehBdLwvwkvhYO7BN+wrHXhuf5 +9+b5N021vImf8lPUiyhSJLmSZERbO9oFAXslOV9+xhL0fUpdFX5hoO8gl/J8 +GsZ36WxZDiAogtAJ0PpC0MKaqeYLHLBw5b0F2kaZ24prAYtlYpvw/s6l/MK8 +Fj0f6TorpHPt9N2S34rVXXMwWCFVuC7Etn+lD3RxW/o2+cvvDrs4VQOy8eKc +5R94z5DGcdIi1i5r57ZaslOviyHbaU6bPKN0z+DUHN0nrPs+lcnTTLIFiiTY +pp/L/qVztU6oiCPcA/qJbOyKWYmqIS2xJCo2G13LLQCClqUCuVQbmUy3uRTd +0ZX17tsXb16//vb7l9++HLttfpAQbBVpnI/bqpo/ZC5FzFoBz72gz8lwO6YI +El/p0imveJCVSapVWZvn33335kM/PRfhLLgnuNLJsTW50tk4WsmxAS/PiZ9Q +HpW//uH9FQiYimUZABih5sGjNbKrGHoEunoHnAZbJVVAi4OPT2tQmawg1PNn +WjORJOkvd3pfRfcoXtWzQfTcMv9ajyF+GlTPrrO0vQQhru+U8qxUOp9tQJZo ++CUbv7V6PqnUrzlfMS5cL+iAgSLJo6SrMQaoRWC4QTR31gPCSW0XpzVkFnr3 +90x8loGYlMtCRh20lwA/MS+XwFtCSBLT90pyCCSau3vfUZamRmpaCWvkKxZw +Cx+YhaA+t1TamZ5KNITPasPxC4nW4PkoLhdiu2/E5QNMu26sYCxLzJBoe0wE +QCeNEsGp40U16tZLkBVo0bfmprXRxFIJwCIYJuE6brPXz/+7cAZEIKiWguSo +WLxmTO5BcErWkGFXooKLV5SBMjKZGtzNl8oNJ7wUlrcP6g7gCGhpQcBi+/Vm +d9iatv8XOcChPYoUzTHAjYZhZPa9xFwfIADloQ1AzBCBPGi5epEguJL8sxM5 +juuO+EQRlDYQN/zRyVZPtKBr3TBN59riBUonMWSEEqXIk/CzFi4XKPbAceVd +vVa1xwgALRzjL+u+3hrYxcR/SIPMSGwmb0Lhamvh72kjehgyjyXqg51w+DmL +R0mrAxdaT4dnWj1Z5ngQ1eGlgU6nJbIjaYnZREYie0hGAiHRXxn7n/s8ukt8 +a9VWFiVADnqx1+RTDYfiVNyBoFbPlLuwQI3swMj43sfEjtsXSCTaJv2CLMsR +I2NU8IVDMoxeaMNvEEHj9VLZbpQTPxIFaTyQ0/qKvoRVHkQkwnXk9ovINsMK +mVmiQ6q6YHL0bL7EDYxCiY1hUqm91oCEScwUfhbnwmFISULadwCekwJDoZFK +08s9TS4sHKK6Yy1PUN9ZYH5BjwnRpZwJHnlu/4DSBL5/cxUsWS5oCBR8racB +HI2VFLhZb8kraDkDy/wQjRK47S0iqxHSTcnVE5UsQq19eI7K4of13bPckZ1h +7JFAJHRF0GgGWAJWqkqvqxvJ00kAGsELEddvhZj9UylnF0a0Qv92uc/Tu6IK +k1d55HtxmXq31nWa+GazEDH3KCzOjbRmK3axPUvD+snhuUwMbgGIX4P7KQpj +YQzICfU6+BlFlwhppCJCgvgwRFwwlnpM2NCwq96xnrOyVP91NEi/KstNuLAi +R4HeLTGWbuFxmvmpGFLnLbc2OrsnQP+TFMTdru+werelFsjykN9YX5U8f+y7 +mvWMH+PTIetp0VlZiTLBVTeelwaPMjzgZVhe954ncyE3KJdVEXv/6MFSdJK5 +o+KVHNppeDD4Jvmxl8JStRCC0QU3X0CZBcqdSMq8X+qdGSmKkaHOh/mh92Vk +R3cODV/sciLSXHBVk3b/HT45j5+c4xNJ0YHIln5t2om/b/8+C/AKnCwJahjD +akX2TTnANCVb/QKoSJ9JicguPBHb4OYh8Ulw8pTQpeGHEp4uAK5qe0kdXB/6 +DXZiHtKX4HtygxhCNldOqFg4UIL8lHUhrxU62SGyZoke8SXpEDGerWxEelIT +Qrf1P9tOSXyt6AR3sMpHZ5YQXACzlhT6DL9P3ssu208d2Vh7poPnLQgPiMw0 +hVVKd5C6NRIB2G4GZoNhXdgzuuC+auZq5Sd+Cx5fo9kTj1dRucGcE7JQGcO3 +u+W+rvhyQVZs8bEUXNBM0mp6WaB3joCGgmWFn6+Lzep8eaCj/RPdkftC+x2Q +HUVP2AkAt3ds6Jwy50PVbhVigyAWA41INArh9L6MPnwgEmm1AWE0TgKWn6VH +UoE4yT4OHU7zO+3YNWEZ3o92OIF1b4WBqMlhWnJFjTuadI1WbmgxTTyU5dfG +SDuxa9l4sHinmBHgyqMRbMhKUdvkXDb9ptifaK6T2ySV9PP86l35E3CSYPq3 +SgI/bw0wh11kfBWeomlI5sLeA5/PriGc7K70MdhQH6z5avcRyUg3otzGj+oV +j3qn55VEtAH2qh/SIbk8YdFX8hCawYlIuPAs0+Ijj2qt/4wUXlVaYInOm0Nl +8WRH7IFCnJcpUcF92Q2pGxoNceAT8wcHYPFeEpEjrdGpfZqNdK5z9mLPS5XS +iGIjXOM4g71kh11EPANt30fm6YkbaHsCQ7Lk3BX0QVoElUxJkedSFjf2CZfu +tGbmq09Si829iBHZ8QCSN4Ud+7xfPW62M203IM7LFrIzwIrlUvmikuCBYjml +y4fRVeFUgYVei23NXrBOQH07pV71a6jMnLAJhiCFqNe3Tb1iuqK+1Mine/l0 +SnIMg55QQbe6VEKjcczKnGWuzlULoI2usBd4qwS5dx1YF0YS4grT7FUYjDNd +54PAbSg3cPOElGoJ4NL4N4KkpFVghlmPT+Qykyb00GFtvL5Lrja3uqzwfDLS +STKoYTWQoywSs5F9TyrS+puebVkXXauLpd+uGjuDiltYFHuRMwg1QmtFNzbi +TDYrxJ3b5EsjMXE2WSKuZgXTRB2aSOeFlgZuCBkQtrJvUl+gK+6ku7cOcwuy +pQv8/t/f/PDdy1ylVbmZrQLdHeTMZ1XvR6mRlibXnE75cgSQNgKKfCiUDaVx +0kkP5QUjQ7XgrYuYzpSxTomYxb/o7vZlr7TpN8Ar/mKk4udiFEkruSkp3iW2 +x0oilNoHLSxWSIbSeGPxi9MOFrHRgLzjxEqFXT2WYepIqTae/moQZbb4DBAl +4+jlOG2qiOwLo+2MiDHHmY/ZBlfDFanbYi0azeeN6O34QPoUwiYHtQxalA8Q +x2BDnTrnnXo3SuQf+SzI9iwbNXm/4LKCwwYx4HYdp302m5Qp9ueZv6gii/Ms +Cc33XdbJMP0qASwLVVjRqwFc0eUm3ULgmCg12aUGBMl2vCnDrX0xv8B2Pp5f +zISHA4vExRBzjdS4GrdrrccPNM8hpKJ0h1pKsTooyba6X9eBPz0zCEHdMbt3 +7/dJGOC6dE9wGnwqE6FNkhPZo0vjZOqZJ+IHZCc+dXBTdsG7CFV9V+YT8w/R +EOYkBPQHG6Ih3hiwJUEU8wOR2uzUchK+Qe7Z+HEOzOEzcf7U8ANs34ouwKEU +W4mwqUAHKpsQF9gfFsdtYyCXFZnE3tOY7n8clBrDSqkkLKnt6pXMZCAGHomB +K/O6BCm/rjxHmTUnyS0AyLxmLhzzRjmANlZQN4jTvWXrhF0DRiLxmjyqe8EU +C8NxyKgXKjoa7MGPwM/FWkt7qqrvQMehutYeTNJzkHy8fQSPTLhwAmokGWhD +t8X22cO6R76xpNGqqBrQNTEL4aUkxLp176F4lq9gZIotZBXY5Zx5qkaxCS2T +DYaCqtXO3OKbfhMC0W1+Iq8/eZa/eH95Fagy2/zxBf2O+3EpeoV8atIjtCnJ +uB4K+/VQbImESzGPwhNWiky5rpd3z45GWV8bVLynEgQSddghMOLPkAStvw33 +9wjOwtg/Lt9arIcecBPb72orprbcXjMd4nQQ3GdlA8HkUOmPMUOGoiE2U8kv +ObRRLQ4OfN8leXxx8U/3/ki7QyYl4kV6MZsDUWygdoVZeiHjOlbM47OYw2Ie +7R/DmV+ubW562RGolcGqhRCgMweODX3q1h14nBPro75nr3ybYxb+osw0xnCX +XFsh5f3wHZNL2PUvRSxK0W5SwqrMnn18hzpxYW8yBSs8sFYq2aqpWilxpDYS +fK1iGZ7JZ/9OweYyJVfJ5+TIMiDpxdQbWZ2e0nh9Le+/vsbvjC/uvzEmlTE8 +tUSxCR7Y1OFSMkE9tXtmlRYhA84ytFCOI2WlcyyfvCD71qbkG3m5hN2t1J6H +X7x6Z0ElLvKXhMULlwCM1Zt4yqkyPYS79oo29JnEGiW9NtLcRA6OcPtE8/3B +gcwPSBjiHg3ROyawltaAooChWhCL1cyRUwITaSxZ3WuhvAP7w6dKMoJc5P+A +cQVOMImQ/DPdguX+n5nVRMb073QpoVPBrQE3NzHU+8yYhvInrRCu19KJBTcM +TfiLCyNDvO8yiMnco/iwMnzNI8UKqYsJ57/XCFCJlSPeJt4k8Ri6hjA9nKnq ++12MFK5KbRR02QtYOpIUxo3noUvt2MBd4zAXRYGrYtLdC/fqAQo8JZNrEvoQ +3vVWRrl5eki9LFK3iD/KPZ2F/mepdDit2elxqBbgGQMJcWhnrYzRSTj3YNS6 +skKk5Y6vD4eoNRcR+k1uSLbb7PgeYtt0n4yz5Josb+5TrYTpYFgmhcwtD9q1 +sFpwyE6wp9pcu8fFP8vof4PK4obzm14ckSvQJVpakV38XjgltXJGcA89JCMN +wOLx+SmfMtfPVaI0Byv4S+6N4YqB1EPaqRniDF0+FNkb9gFl18Ll1otPs37K +eAT0RTJFkZaPXSt9RA52UsAYTsnqFIYJLYAcvIAVGIiOQb7BCJEm4So8fjlF +HcYj9NO0Y2isD0iYDTiwNVMkGTKcHLnUZY2EiBvzjavCOUcwxbte30I3yw+7 +LdoYG9ciPkeOe3QTDWGCu/W6mqLn6C2AdityaEOk7UpDcNOrYPyP3D33qma5 +80bVsV6HEykJSY8HAqg0GZD1zdnZ9JLEdEXPIP2NM17K3PeQlJfN3OW8Lnt5 +ABYlBfR6Q+LUrpGzBK56dTSYLpq1utmxW64Sgm6ldyP5qyxRl6uCbOSKcXAk +ALOM2zntO5Fn6+w8OiVTrZrHiDFWbZM6fO9TTCKC5X55kCtpWRegYMBV8Fk6 +LQwHfH7Yy3KcZRaEvb7zHscw0xWw2J/J16A2ofYu9V+Ra9kKXgTgCFXUd598 +z0Srv/CYvXy6GHJMk75nyeo4qmlsXROmff6eu0Zveko1tugEpfuIjnXovf/m +dZgQ8aV0vdKjW0WC5jEgxJyM+Ww/gopIyyNdgSJiUA5yrNkIK912BaFcD9q4 +/1x95uaKGf9CKG9BkO5QVQzriUgqwI6h8baknQ+c+D0Xap6lxKcmFfbJLwop +5i6kqMCZQVgx/yVhxV8SHpwIDeJZR8KDJ/ddMUKjOnrFROLFcXK9Hl4izdeG +5KvXcL/ZpZEcgOOXRpfMxV0aoor62l6TpKwbQtJoLG8mPaI9aZ8ZzbgCOCJv +v58i3UPdxG9YNOSKvIOhhXrt0QKi9TiHsMu63qR1HQL/06CNkb1qLo+hO0BD +sfcRy9BNVp5KLMF3DYEIYgVeVdHJn/BAnwPmTydxYyoflnW/b8GI+9hjbAzZ +TqsEmsdiaGxoOoxYKomYirBU81SmB+4YIcF5dr4pdzfdOngZ/BJlq7NrXpqj +utU5c6PqVFHBnnDP8+9x7GkovFspOG+msToHzGmN/MBHlj37mnjFdrVIlwiO +K/jvOMZUa58am7OkdLmcTw0wa/WN/TQstJ6wQUe+gIB/NUwDaOQQZAh+XCV9 +g3hCg5svr2KckQMTfBiyRBv3iA9xgTl+adbF/J/n8p9nI+1uxsNd6vvZ4ESF +7dEjHgvHK5sNBSQuWzgfQRjfH7YPOhdJtUULdOSkgf+LT42XMmlXjy6pQoKg +hVTd8ASQfLmmvwXwo+xn7+UwVNo/XMy1gJ8JjTrMtdXoz2B90BWauwW5yObk +WZ1l4bIZjhODc9H94FQOuodAY5JNe/X+6uLiT7ma6jBb8Jdz/cuAnki4uF1P +mcxzIowMadC3xNWgQWUM10P2Twid71+R7NS3OmPBPeOoMklXyydeEEPa2oq2 +KdCEiCXtirSxIO/sEvoBHNLfsOHFy2J/P8ffz/F3WRzILcspx7F2Zn2HkzCZ +tho/B303dyDrzyfEnIvqduxI8UkBcbc2YcEK0tU2fmu4lJmXwWK8cVtyUzB6 ++86Kevt3xOB+6B2/aoUnMFO4BK8RDGYH1T1WFXvYk+8gEt84Xm9TVYaWTiaN +a0ilnbvNRQIR1CXxe4KpFid163gtZxm95dRTJuK9Y+2v1K/sM2N6UfgWrZE/ +QxgSpahdALm7zRYqfyoS/puKDB8KuYSS+g6z5+udsQEdGGX/PiGXRPgS1l1+ +aDlSGpl9sr8GANvzp1IJ8SJhjOczZ185f/5/PYUjdZ5+R0Fp4UnfPJ2415KH +fUMPc187d1ydAe2oveMhQGHoU7zDuSIBSISz49dwcgK65FVe6hW8KlaBZkEt +HzbPwobzQwZyJZ5PzOPCMxxo5cwFL/XVxtfFw5LkK/oAGY+/0F3xh5+nIr38 +f6iW3Xog+CKTh5CDjgdBBG0BGFVmZZmMY+lXY7YDrv5BcnhlZeGy8A8yEbDq +hkEvRjohwJ3ENtmtH/oB9Ps/XHlsMfcAIAWIziSiv9bAZRYLxsehj7hri94P +FMad8lZPhsmg7r7nJgmoipMg1kKLi2O6dEDDFg/Pr8E0o9Lgcovi3W1EbQw+ +1BZr+3oPrbVY10ykphT/IyFP+gJvm+9Opj2frJPlLYvNVPYIzOJV5GspugHm +9k5JOyBfURBDy8QedUGrnbvjN9HoJcw4TvZML0WuDFCJ2tX9K2ikB+nplWSi +20FQDS1ayCpLNJcnE+bgJuftUW84P1OavHjLjkAzPOHBRFikV0XSqi8d3ehZ +NiC5sZIdtFSVfyalJ+dHLLbzESfhfGAunY/dmm9GwsjnqXpJzn0/Lv78v6OK +QyC0DExPg+oQxVHIsSpLm7blRFORDr3gk5C6/81kvDY2z/TyKFZO3Q5iPix0 +1iBzW0lcXnXh9Z26bUKDy2KGMj3NZptX6qgbGUCebJ80LtFqOTHlcHHLpKsm +cPOpd5bVO69QokPo1WnPjBjJdZBoH6wNELeo1X6ImeE6xqg7AjVIjxhnfBf9 +VkCk+8taqYq45DM09L5IQdzIPQSD10rEfC4FPTSgY9tYcI1DC1x/V/bjNTaB +qBX8dgUeUpShBVaf4y6RtKtN3xrWiVVGv2VVT2/0P56sIgrrWbSCJ4lRhDT3 +f7UOvFj9/J483PoFJOrKRT6TYz1d7c0H/DB1iP9rT3j2KrH9rOAngECY6+MO +ObokxumKJrIChUbSQWU3bDwWFkyb0oSQzwDg/znVUaEVQWDjYpPQt43xIzlT +9QK+7skNFZYC7HkMXkvVUIG6/BibG8bHTvmStLjzXwef850Z/3j+RplSxAQ5 +LjkMYlENZziG0LuZ55PdP5+hIhrLfeanlnaBvT2U/fYsu09X8S6+Ln6C9zTX +zECEcqwaaTxLIgXbJxEksJptuV5vlhUbUjrLO62Iktd9BqV7qOF4IKG7mhJj +piAXEC466TTbLoBZrDqHXzrOb5NbIgFdDazOHbmEmOY8jgwBNo0nhE46rhbd +VZ9LddY8YgsfSmn9Guhsa4jNsnjJ27X7qIjteUDHcXLJUPRrmrqmOjiZqWm3 +D8+/H0vAjoG6QemsUmJwdzczzj1JvVmshjaiamt2+vKHq3MuaEYzm3ke+KaN +X4qjLDwPemzDNUdkGxf7VhG+inQKsJlrWl9Y9JHWprGSqFaDflsaL0uXwC9h +SPt0PapMzt/biKbIhy6PUAV/viPLlgHdG+alNgOb1/oluT7RVQf/pB3aRplP +wtHJDzs0iYP6oUX1uc/Izzj77atHIneGHtr2sKWVqX4Wi4GuvlRogrwIEuov +IO4NJe7QJY6Fo0Jyb0afcyvVND8/6kkZfgmB2UljxZiuuDACmR7pdycZ5zdW +hDHzrfD0pg7tBAPaPYTAkPgyJ0xKlV1XJjXL7ri4wTAmcuEgopU2SZZWnkz8 +oX/R8pVXfXhbPyBIC8LBwJmS6yzYtgMbe89Yw8FgGK99p9VL4WO5qdbMJjCs +pV7U2qbnTqalyHZZdQ37yFkaRogAykWod+CFKT64lrIUo0mRrqkWj+9b+bNQ +77Wrq5abSgvwgGmIIwMCDmDXHLZz67DFyENxeHr0ScWnuloy+bCt+qY8kY3X +olE2fXgf8SAZlnEo0/RYsypQvJRpa61GbIcNIQgtzu2XARvdf+Scyahu66Y1 +ykO1x3cSfXBLwsvEmVcOZrMEKv1K764TBtk06IBE5Tr0ejSAdZxnS7u/0Lbz +rXR0BpWmclhFRzG0AOMSvWpHp6tiUmfZlOu7zo6NNj7KUQqJh0soylKOSfhq +0FHZW0HWt15cC7vFJByB/GfoUabV6uqoTujGtG2W7wuVpX2hfBOvfg+v7FgA +Iv+3/OsL47Mci0aMfsEd7X/LL/5pLD4hH/hSj5SRQzjAQkv1vvjw+mcTc59Z +52v+PrbfeZ2yktWRpVOrXKbk7dDB3M+Mh5M9nc41Q8+EaQjKkQcSMx3DADNq +MqUZfXIKNTNHvj+QCW7u6HTIctLrFpZZKQNO5TksSYlYu+w+fi0HWr7hRzTI +yLxKi96LDdPdT3I/w74JyqJYZgvf9FCZvYrWdS+LBHJWRSVZVjkRb3bBayoi +mylis3Z0AACINLKZpmnTsnuuq/7GN37TjoM80U/cz8OyJJrVYEYcNbZaATsD +BZyk25g6yvQC9g5Pq437lb52NzPiVo/kQ+i9MWwI3wMl1ApfHYN7gK10vmp4 +zwAOs/t2UyaX7QaF7TtFzDNtSmgNd/IznbRqh4jF3YnUCDGdHZ6RNlC/r632 +jM4MxmqQFE4SWCGdNO8dGtqhO4N4Adwy4RZhBWbhB6RLjHd6TzRBb9jRjwA4 +hSfakVYZkVJ0V5Hy8yzAaWjWQP9ZfqRe4UHfkWt62qCDJPtMDXuV0FU/OimI +XLALQ6BzvZWhBiuBSSIhgX4XKCTgtqU0RBYxL1F2xk+e5dW8BH0wzM+gJYUH +S67OqzWfha6dcJOSMAWL3UlDW/vkxPZbTHV/gwRN2aOwf/qrVf7jJ79Q53/F +Sl/hHaKav7ZKJmR2wlEyLaNXAKzuIB/oblzvs4gjMZC+VDDTg6RQyk6jO7qZ +FKK1DNylIXTmzfGIvP6l04BAsx9ryIprRjyOSPqkOtAVm6lZF++RYJFqmvDO +59yft+NYl+6WKzYcOO0I2IUm0SxHMGnwRThAILa9YBMlz2roxG8E7Scsu3rI +0jrBGCQydHWkdlfw4q3R6Li8rtIUD/44XtTAIh6E1yWN9QUJ7ZGZddc29NCp +tz8E9/gr0+mFA5EmRYA+rISD7Z5EU3ldt2FJVqNjRe9WmOZDXAdqVHdxmT2Z +1Ko/7jlHlvliR/Jal1YviWgLzISG0Ke11qBma3sw2eTJ/GiuVTjEEKuKsdZz +tbwWUubM/QraVpTgoFm9Vt3206ja0bhNxL2tFdbUagTchRIyIDg8ZqUPkumD +COdDqUXB6nBNof/kBp6UXfUMqsBrr9bIWNhjJG849jVDn0uOM+gOX9agd5XH +DUxss67YAY+/lq73SH/+FtCQs8yyEGNIpCSZwAaZv1ZYXHCfAY4U09KJqISU +hxs+qbxJiU6oskRgPMOjLegsg29vRz5Aot1b1NUrxF2TphUOY2+SH5ovyxYZ +ivoUzr1YOqwKnGqdZXothTTvYlE3imwZqvKzrAcv5NS7yYUY4TQcMWIOexgs +RhDqW5dbcm+oWADMVuuvnxsWxPa1kc3hYIXv+dt5zPx7Vy7W5eIjkwwJSxnt +17PxwNjT/FI7EVzXyE2A3VHCane8UFFWFbI+YedcuhhfyPhIlFBX/QjuzZ/A +8GPg6lOFStf66G/w1VBChFwT0raasLo39vi6XFZ0d0eTWWKX8AzYxsVBOQwD +lFyWiKNkJvQH3qfdqGNolQjFiHcHWKtQWMtiGb9p0MOKJSzayWMjoZgfUJ0J +FACybQ8An05nWXUQdAYW2uwivUhyVDXvhSAXhHiMP40l+IOYKc8RPOw5B8fQ +RiWQoEtMnnyKQ9O1aUBSETkuaNkP2AzpHYyknSNVHZ2HHYP2gi8UwOrBT7RX +9Z0i3zVFN0G4E6Eeha9/IT1KOrYThKNmtan2bYxkKrt0sl0+dg/A1LhZnOJA +k8U15D5vjxSIaVk1tGUA8UtI0F0AbWjUKQHQhSYF5X38VB6+mEMiJ6KXT3kb +rovFR1GsTO5zNp/QKi84G6cV1YqC3alFi4fjMVgftGWzRUyLxaR/6VqiSeFB +qQgKxApUZiO8mVYa58wJjVFxI61NBb62qftb56Z+T3KYXeg7ubrSWf5vEife +lV735adwcnD8z5yvxF6IOjTpFIfByd+NIoaPmTjeSFMTByHyEQsnK7a1ZVOG +WK8x9TvL9LbuEuvhmFWUyhUZRb8NXvazjCLGImCqYq24qXaTSlbuLVs6hZHy +Ce/pZTe9efYhhK1C0G8Ww91efCSn2/5/y+ZJZiuUEP/1Ns8lX/MypDX9fzAx +PLhdyv5GNwUdo5yRMWIS0GgghofWM45JCh88EE39UbjBWtGMHNHaLe7OJlKQ +MWVN1q+OnOzusTLgpG0NR4xhTbiuasnxCQyYWBKZ+TPTXHda/LiLbUsYfcod +BSo710caZL+HlCaM8hpaeYDtdqS4AenAvqWXtt0KVlvibA20hwJvp/oaxsB2 +3Ptbi6qtRB8m2klwtPazgen4Cj5vU94A/Ov8c6sGUNiQb54xqqytEnd0HYH8 +5fqd63K8cgpQULJ3VyMILgGltAIrlvVKi0bHxP2qOZRa4CfFXmz3kTIWptXN +XSrU248WHi7LrdB/6HG4aQp2HeTmPRnXFl5TnMxclFOCMyGTx4YWjprUhStd +/5SkYvsstk8L2jN6xFCAP1L0lHG+p/NBOnwXQyMmYptxAdyldwI6PaKHtc+j +3CO4AUUI9IAb6bzfy7NeTYqRUmC0Yp0jsO1eMUp8Fnp81CGpfr2pyRqrUvv0 +t7d/k1vx/xfmb2rc/S9q/34P37VajrizbLG2IybrQNjHbNgJRrqhFTtRy5ui +JH+7IqmzDIkIml/RxdUZPX1TZs5kSHDYk7NXjSPsbXd7odAc4/iZ9WljOOyb +/ZJRaq7EASnlGpU6YSH1bz5JUQgzdWzROpv1Y70C0fo01tQwNXy08xtQhPH4 +xaQlZxtNZGsRlI6Bxgt3XkS0tK0VgzwYmlLaDcgekw4mc0mggSlrpGeBZbGH +SDGCN2E4le2WgLQLB2QDcBSbCzFkOrkC92VlocScXWnvOEh4Y8qEfL5c9vtG +ecaQNn/eRRnA7reGSxyH6ZJs+CTDNDGJ+9bIEX1AoujeGpx5NmoL8Q6M25DN +KJnWLJvQFxiXFwJvECWncsDsHznf5ITEPnN8TyH03QiqrEemlmnDILFwgT50 +qjhyS8WEzAjuUtFEeEaG6vJQuhLKet8NLdLwdCAaboX2j4laHNDQSvIumxGT +dpw3NXQ/pJkyEGDFTixAYcxaxI8E95A8zQpljrfmHK1WULNQw6L9ogXF7itT +87ANq6Rpwjj8ooPgQCuL4kn2coihTrAWjWHdVtztNOalulB7FSr16Mxyu4vW +gIPV6HpL6T6QkMMywOajOPDaDi70cwpPSjz7zlHXTJzq5+GeGd196QpT9tkC +ZfHFK5J6Bpf81/D0A3xDwd+pszdeqRgavLi1QEqJnKz9ptDG6sGo2bOr0lRo +wsigf7WbJhDScTgMO19UJGLr+nYCl8zg9fuby9I6cQ3qOWeF5vl3DBjna0aS +lAH2ySB7C+xN7MvuroMjU25aFvwbcgD5P8fPKDmBAccugtfUqMcaq3pNjRHW +Jb5KY5Jw39VgTt4Ayeeia0EM6XsW1df/ybV0WptNNn8/DZ6Q7RXTyXBfB9Rr +tDRon50OTFPl4bYODh3KX7iSg5XsMLEfK2E4hCVMkYK4OFoGdD84S45Mo52W +QvnEumi2A5l7nPQRqFqBE3JuarTgayJTN9ytU5jrtCN7cKyqwtlLHZbF98bL +gqwX+YANNzSI+pymzZje27rtmKpqSefq7aVj8+LmUtt6yUeAbaQHlqV8r4Tw +5XLQJSBZz/cpOA6IssLSVZzrGrtMjhhMg7Ktcc9mSI00ajwZPbDsSbrURuce +OmxNOh8jB9UOXCX4k4kSxofWJyLzaCyzUSbRhGooIoF/7r9q06YfHJWtBNmt +nRPAmVy+4xh31DHmRjKrTXEzcexIJOKqPedYFUeSD005mo+m9UJvJ2NxJOPx +pTe4AFdESwHtaAeOxhbp+/vYgbm6xcBQwEI9jXTrrhZG2dmrT5VWZWlTGbDI +aRCFizo6xuO1emk4CUcEkCWurVcdersVbtbSTKq9a7sSmGvRI2I/LMu2utlF +rm08ykFGAxm1/LrtmoM8EwEaa+stfT7fF1ilk7j2L/H0E98x1lMsGL0qfv0j ++ZMfax2Qqfypoisx7pKwE5k/LbgD7fcF+AMnj4/VPivLw7HGIfEZybKql6VX +PRumoP/TpYr79zRzKmPWO5mzsdaMvJXJq6SmRwvb+MuyiVI063pSqrzAXFML +kZ2qwEUpTZXkEiJD+TxSFfYaYHK0zXqxYYK8GoedmXQWc48vhXUemUA35Yor +CEJrsB5vHKzdKXtRHYs2P/HFy/TYNgzvxAAoqXvYU3okcfzy1qts1uALu2hD +mbKjjrQaUTmk4A9Jvtkr8lcrR7pv+Uq8kduj/+NZBk1H1jRi+cIGFP0k+zvL +6abY3Rxo/FZ0H/jzhzddHCqsrdAFui+X2tSdd0J9W+EhMPJZWBWRfRb/eeY8 +XU1ftnDK5Tfh29qQTa6nNj4zVlXLQ62gesrIvEo1nbbxbHtbwqV+UnhcCdVC +9TNL3GVEvMzU8rPuBPG+pDUh63jQtVO3JiDd7cIPpe5ps0tru3daVjiM/TJ4 +qf2G7pYjmB4+Ez4pgQ4Li7bD28M+P037ZQc+aPr1WfDoJKTBNrzx6Q262vZU +YMgVZWzY+lyRlNDwYu87ZdgIxuekTsaKjQVSq1pjCyLJMckmvGDBz+dU2joQ +7bGrv6y3ieYYbfF3MZfyhDuJKUqfca1GOMtuUBEj1H2HmxvjQT41ik0/7TN3 +9WXmPjKrAa+5uS3aWmvABzNSihdINruEByBzFyxUtU5VSv9bRSIg2MT6p4hq +bopo6JrugrUFLnocyIoEl+0h+d0xYlj6jyvaCyvWBCnMA0sDiwU6cok474tu +TSebTuPSElQOqUv6iXtkl2M85ZoxZoTYqmiyoBtDG0AtS/AtJ7SW17R3mh3+ +4Jt9iuQXaEk0G8HCsdFuPVGYECLSLmU+eZwMJtyZ6Tx2XaytXQ54D7lUzTSA +ntjbegBkRNO3YlnK07z8AbJr5d66ZmhOwTMo6dIK+iht0sWTElwGxyhUbGgR +7/KQgF9W7TnLKscJ1LURuyEoBxPCAa3LcW/ql94yk4QjmaQFyuXGyIdIojyz +dnCk+u1FZll9za19+9h+f/e46AKULSoTpSVUvK1GnMWJC6rPwud+yXUGvDG6 +rogTkV+xsESzm1Ib20730zu32n9MitLKfnyAWa/l2mL3DOc+pKzIX5r3t+v3 +bQhacYpia/X8g7Yz/OtsKUmEAycvfQ6q2NxwTGW9NYZkjDqkflETseRLsk+U +Wwzt8ACRj4bPKImc41CzxFkNSnBcDL3bvvcOo5jUFIFgv5BXS+qG3rmsrfqk +Zm6p8ThibalhcM8xyK2YgaOrOzFZaV0ysyeie+nseJwY258xsvK8hiWmOfKc +1VJQiD6WG0txRFWiVZDgN7Zo7pP1m9H1aqK/nuff2BVjLPvSmTm11tx6BEss +uCGz1Bg57Xkl4KML9vNZZqA0UISpZk6YZ4LZNr5M83RAYy0LUePJuvW2LD7K +bTfGC9YG5JN085VFRQ+evVCJDDNISu2EW4+fXk6yzqQdifKAcIlgHbsCyGy9 +p1dey1knOOG6iSd5wopvG+8L63XIU8UGfKzoxquM77sp3ciK/ISx+if56fqw +RWg3f3F5Jjr/RMArJ/zHE7hvJ2nsLNoerZPLebJr2tnXNuq6tFtkK/Y6jEhk +9cRZuzcKGmt0d3fcz1RC/0X7cbQLiRsJuw7GQpLY9QJ66Tv9hui3ckeLHXHx +hAS4GJdy3tXnwKeIUekaQvXvWCg8WqFNsUiRJ8KkjomOHtzHuWelnQyYpdZs +/kIr0Ok65iwJHRqyZOTsu1Yy0rg8plRjw7NNfaMkx8uSzibu85FFUhXQCNmn +iya0NIzWLHrnlwyv/N5ShWiKhVuEn8yK61WGQn19WuCt0SPRHG0yoH7hnyr6 +vnOEvndqloZrcEiaHxcqzd1AqReKeiWRyYp2cGb68SMojHiTxRZNw/x22yMd +G5ZwYh+skeOwP/ZguXsWGXhRes2cisDJtOxzyfFqiQgJMMPTTZxyn7yUguIs +ixsXhT1SJNjKW1CGsV7k+nCZFSJe43E8tgPI0dVTJekhBTkjmN6JBb+T2Odk +1tVo8L1qBQOU/Pf5n8Em30OU8A7RhvGup2ih5ypHUYgshzsys1M2hCSolaYn +6tXZVPCSA9BFde+B6PMO/s5DCF7oudJQWNrEZUW+b4sYhQ+3LJKfzPo0ndu6 +qz7p42hl2ZjXLLvrmlCSc7qsScOMansO6DOx1JLWaJD1Oh6h9yF6LrKbiE9W +Wve7ZHHQAHryDTwgLchPDVIFcAr3z7rYg90lMrXh99s7R+ADHEIsGy66h8XJ +j3fDdr/6UeC5rZpMQ0NfvJkS1dmfyk29F1uRE2jL8lw5ZBgXUHDsrMjf3tEm +7XL0ZWIeCPs1TFRGsC7La47G0L8OSoOuK1op5+j+YKBJ+uzt3dtLH6STF0pR +u0SoMqn1McDOmXQxDA9RAB7/lkcsxuca/rOmQexJZ+ihWH6qN59glHNkRFyp +19aaQsHMIcdvs8QrMR38as9IHQNf3dGCqpsnrEgYBr/q0Ir7KS2Mt3vauUA9 +YsUDfqZq1+L3tDSnCG0ZvppOvNs+G5hEp7mgotlyZ6XMmQ9WwH4qXQIEuqj1 +LxNEj+3ZLNuUBdaHWar3hTrjJBvVNCjvZXDwGKDDzy9/AoZgA/I9D0+neV38 +ceTobj9+oXB4iVhqDASgD52sJMs8k50hg8LX8To5Gx+kqDR/9bKqz1+8v7w6 +d2c2iBODHJ6G9iNs1irqpOqOHbEhHoEWmk8FPX8nvdN6nUgF0p26Kra1pCy6 +urkbZGn7bezeS5RT7q0tlxqDDq9cnoPpndZc6kFD5o29/EaMjhnezRw014IP +p1WWZsFmriMNsx05CXjUQPRmuY8lxg2b58+lXzPseIcoEgtYNUkMX7ElbSTP +QmwjLBLxhTgPYTyVKOFXL+dVLTcnTrQdTd5HOZGKSbH4cqqnyetlbk1h+8NQ +/qGN9Bx6FpbWisly5KaqQUpVCrjEDQjqXhbc8D75Zfd7WruPrlXuNd1raf+n +y2+vXo2lP9WPieZzEPQNxKwNchbKlvS25+KFDf2/Mbk5/m8IeHKFTCFbSHhF +IumFjv87bIisrPQARWefwZUyyIaxTmGgC9tDSF5EzbI6iNWyZSnQYKkdyEUt +OKJ97dC9IQFiqLXBCAYDmKjv/1hyGQKgAAvGw4hXovL9KEznUfq0R/2cZtsO +CnvcodXK16LaRrpMRx4FfLdgwgvlOZXOh3Y+Ag2lkojrARLNqPRQ8lguTttw +Kp/XVgwRMXhyYZC9FWGyjJqjlqoC9oiu/50wKBkDjYaByPbmZ+k7WtMa7nR8 +e/W+YoKqOTNUwV3Y71afGHf+ce5XoQoetEI9I6zgRCQLNmvJGEQk+lEWIUWb +MThZN4qdUFiEurExXwKdF/kpse60cbzm6PfqQoyhxIOuhVzppW0xpRlip0nj +gbmIEA/Qj7jEP5VWFsOzBWhgRa6Ooco4uZBuWxleVHFsDBl1KAmlc2Sopl3I +/sw0nVBX7je162dc7Xb1p1A8wvInaoAjIeLKG0+8V1SB7mFqtfVLWoBEX4LJ +IWFFZ7NjZmiZrZ7RLDbo0JCB77YaJxykalIlmSLkpAtLRwk/GvWa88htxMsJ +ZYi6Jbp7l5yp/DnQLMj60051UhlS9ylbw5MQpguP62olHVxtyp8sAwvO6GG7 +Bd0CcZhVqSEmL6eXRzDX5LelEK0ppjPOW92M1PmAWb4tdjursKbjBHlaFRV7 +EdfFRuomGUSviwMaTou6B1A/uQPRzVbxvba+sKIRNbwlAH6jleVP423kE2gS +P9gxgirJq0XLT9PDvDVSJT4DkzqY5YUrnPHPagPDdB4Ce7hxhWcznGf/bpWG +kRtAi8kzWUJfZKhfNDLT+F1d7qTYCqirfbFzXKQ+p9jhmm3TimiL4oQPk9K3 +CDLGDslD0R9uq40ROq4V2oAN0eAvRqCLZ8vWINliTTRLrVG1TlThIeuCrCJe +zx/UDYERknBB9lEDZCt+1ICkpTjDTKyxPTJCdz4yvw2FkuaIhQZlraowMldZ +F0oJnD0JahiKp5XXWHyUt4IJbDlaggnDEYpNjH1h+tg2JHQCnmczynwwxEP/ +INL+M+u3FBwW501x6mqegDa1Q2wpVxZoxWKTTeSzQnMv/Jc1PfAVm5nL+45U +PY/K0UDIvARWbco3UYBy59w1clCrX6PtLvmGhN+u5rVH0k1pkOtmqWsdREur +oicgsuBhE3t0Vx8pvbbiMjcEAx3QoVaGM3NsDRuXNs5SLGr28hBay6+Khebk +pqI8AWTjO+dlEQCZPFsqJP/qC8QiyEl7tpm1dOZbzDImqNysRNK41wsfAI0+ +BudToDphlvybLEySF/3QOsBACPsK4Y5Z9f31s3XFnXW0wFDY10eSsllM2UIq +kk2jG2WEW8QfjY8lQ6+gb4qW3RbOGfC6purnFAoHWqSvZXqFzGRLVcvlpjyb +8b205rIHGGa0aOxdGTWRA0+0yVmNrol0EgvqNqJCGDpJ61C3sr59gIjubTgC +dkq1NGUsB9nuyVDIyMZU9CHSkTVHhoN2juiAFbijZFc5oRKk0YCWrVwaoSb9 +CtpcAgIpZqRT9HaP9yrzRfd1wubg6TxDd1ifh9P7Pztd+qq+NIgkhZHt2dG2 +ssmaC0ULSmS5eJadFsuuGlWX+ipYXljmKw7q+obCwIvKLaMYwJQvwNXW7xTB +k1II7KR3bcICJt1ZQqtWUwm9Am0RawaMVxKhzIMNFEqqvzPRn6Djwefn9WqU +igdAhgr9PT5V0vy8ksC75ibT9s5JvZokZO7pUqCtpALlbExWsxBlMq/Cgobp +uiNN3lcpPk6QOb5XrbuPWZHNnWmS0MSWV13m5eTZzpk7y9ZLBhd5z6BpbRck +wt4r3Yg/WxeuuEpYEYIxasMGjmKkV3GbqxwFW7KWf995nRMK+XZFBCgC+RE6 +ibMLEQJuyqogU4dTvlOMROwxoP2/tIfICL1Fn47Y2CNlIxGelkeMJtlQ3gE3 +WI28aB2PNbOlazfcEprQYLeJT2EP9hOvD9h0o/IkHdF3LtiV6mfh1eLVG4rV +gE4kMldoSwikUhNDqcdtod8TRmvVGCF0F5udlNZ3RHxFNlJmcRw9/WAEiOjd +KLCmvu4KmJA6iq32QUtRWzFbaJsC/SzpvYVkXtiWzOR86yU0ZH7vCz2Hph21 +b3ieBH9qxwOZXFPCKcs1O/su7JyjgOmxw7izfhk4+1n5j8phYBGCq1SpRpOk +MtZZZ+KpraOorsnpT70+C2JKK1UoTZA1OPLtwqzmrH9DgQl7l0Q/ZRBDEtsJ +OeSlQdMGFE13eiSHPDbWJxA7A0lcRrM2fuTMGhzvwEllgJRYZ4zb1LHptAyv +DaVdoTA58N4Y6Y4+geysfQkLNa1Rfpa9rKWTBK+sOAGazvJzvy3MbC02Iz+R +ZFnqj7ehWZ5vm+BX6ln2ng4BM3CVJp4GpJTIXbr+RRcId1KvKOVVcTZym3J2 +iBZmK7nqDhZNRoTCvSVFovB8Y/HPYIJJn4g2K5pyCGNhh5KhWVqwyTw+Jecc +BW0b+nb2T6RZ/qExJ5dL6Dlip5rLZnDbd+vP7mJP88nSnzkeieRn+Hvo7Pzd +iGKsLC4gd67GY/WqHl4NGXbRjKzEaI+LWndhTQWYAlInttu3IF1j2Ys1RSYt +nqLqOD8Vwi/JKTnGS3WaEFPxsdUOG6Nl/ZahBZCkz57de1c2GFuMu0xRVSLQ +gSs9oV405kbPdRUrThLKQkfyKBhAXmL/S46zb7ScTcpBHkR7SRL7QdGgiUGj +Mjo1oQIsAUFnH30FamzgO4E04SweTYHwp1wjhSX6yfQn+74+tJs7s+RG4mFm +bvaikh80qiPNPzWbNwiA5OjUaUGDZBx5GIclZqzMl4nIncsUHD653Py5iQBs +aY2nrTJdTMu0xhXIBJgYjn2vu/rQoDiEgVr4DyuGkirArm7OlCvR6pi4vCEH +bxidLP4T2gztDdiBxHoKbhSLIG3bzOcDHWhiNxjAEs1ETgk02mJrZhdufqAl +Cy5uLzcKXIR9l1qZp9b/qgtUby35rEiMKd11ds3eujE6afNiWO6iSZGNTB0r +DvuNWTKhxzEDyDTKp6hqLQ1GGxwmMJJuvK4nTuxoTFsx1iRN6g7Uz3wpxTz0 +XdbF3Iyx2nH3Q1I776Vrzme04jn2VhDhjIQLQpLESrwgn1gaXmdPKYo8CdSz +WNc8IjZEUewiq03PLJo4OF6gsndTs3QvtcwqEftVxQQWyP8puBvCmhWjAYtQ +gVNJK56w74vCiofCn3iAmMd1qUkcfQY6vSGeCmMo6aVy2eH5PhCH+nopnwvh +7OwUBr2Fi0a+Ym1nztzBtshk9MpidJL/do6/nWWjUVjnKEovOTVNj1xs8+x/ +DEjZRXE4hhsMLV6x/RFmEgHrapcLmhxZAJbuNwXp2H7YAkrfFYRJABkAzB7V +9MX8Kwun4CqUmJq05lDYFkL4aI0u1OaW3tBj4mL8aIaqXk3SoF40/0joTEIT +Xehf2AvOzZJgJZ0w0a88vG4dytlDy3AcjF4+JHS5cZ4pMkVCXMoPXYvtEGsa +VGitIR73LBg2fi80peEnGjIb9sdzNIXTy0iyGoVRCzMELcZRzkzgb6GJu6Ql +n9Ilce8Etlz3sProSVoyaHawFGAZhnqCn195prVKMRQxBmRslHKLCw5ajdT9 +YGG/rcZo4gGK3i2pepixY3Vm7aJTH99+M+vFuGKCNrstqpjiY10u7MWSC7Qm +ut0E0tNyAlafunXLHoy6ELWJtZw04kNgUjRbIPxUorjpO0ca6cZo0DLp6aoX +EgQxU232LS4B3nnb1ajVwmdB9M40yw0akyuRLXecQronHoYAS7hD5ba4UKkg +zrNXY/2ktP8bUxCw+SF2u3wq4hzPePI4zssFtXeKeGFoPXsWIj8TIyHrIXSw +TCyNAOXZddWi2hdcTxl9HGZVaQEd0TSj65ZBR9ImAj01jHOeBRN0ZClk0yDC +e+OlWiXPtw6Ad7AQ1C0wfSNtHa0HJCPiJ8k8+NeSN5c2WMwIN5p3yJSIVlSk +BjcYCpy017P0j6W+XP8QL9DHrhzhuvS3C02dbhdtr7Q0AgvfsNa8waGmUn+f +prFcBkvXLD/elf7ouMfxFJurBcEEWiMYFzaJcVuJKo8Zz9TV7hGkWobJs5H4 +xTJjMsYf28kwKNIkvKg8GyvYBKlIjEontuyvypkgdV3Q7di1azK7ZjG+jHpW +J6JZymZdjFwMI742zL74y35LAvEtD9akfMqJZYeJjWWYeQJA7a0ak0xquEeC +QxPpCA4sj+EbL9WeE+yf8eI6VDIDWenkGqYkZrtAPymkWeQUpNjGYwdjWJ5/ +QCMdHLSDnE1X46mgO88vMDQig5+HOY+RTqVoyx9aK4J8/PhP+cfyjuELIOd3 +kI+y1HMXIckCUeZmT01ZAwFyaeFFJs/tnEW007Ms2b/rktZXwbZIvzGYMyDr ++FWC8Sy062GlLHlQCbI+rIuk6qcITypzLcpLU2ozUamKTsHCmi3bK6FVIqVO +O6PL725rBoCj5TOPb6O1fLCJbaRbWoUINFfWKIDNqymmtqHsBWa6/C9lqdgC +qX8QmWmBTAj4bq0cuu5T5D+nIzEbkx29m9LyClfmFK/IlIRByaVQu0vX47lc +jwrLuF+6rpId7pFfkZ5sy+EK3rt63PrXMXdPL+aMY97wta9LaRtddZtykub4 +A4ffCzajk9vC3iTOMSCxVqhk5gSbq0zgMj7ecZNXKyMjN0kX+nb32DcUe8H3 +Vag7ybwh5cpQXc5nYM2GvqPLklubAqc7zBiK19WgyjBmc7re1SbpS00rhdZj +MfE6G04kM/FzUtAthGOez/Xv21gCJzhkRLE7z6U8187nydULJEq/av/JH7/6 +W7ggv5h/PX8MZ0sEod++I/Bf0gVD66PcXdXPLIRYaWstyAiaJLgWg0S7pVuJ +7K89rzy4fy7GYJgUoarQogsHpQUCMJYBPpVE3abacmA+Bprm2beKARGli6Zy +CmXS6Lv7tjg7wmRDK0ECWbYxAy35c1ekyNYj3pjQdDlI62rf6pDURfv+8gVt +iOC8+QzwZi4WpIfo5WBoYukTjpC69QX0H3f8MuW9WfYzN8Z6M7dOu75Ti05T ++VUM08lzQ+RHS+tZkZEztmtptFiGpgz5/iI/iat7QgLBCS0NJvt9Zq6j4kb5 +lv2GVAizPr64yF/TksxGhhehS/Bc4lnRYMs/AfTPYck18oB+G/jLDH7ir/lf +huELJcpSxzD/6uLJVxgIh2/HmxGd6UqGQ6CHTyJRipbrxzThlictSZNAmWxh +XJV59kEVcBEPVQAIWXlwPOGzPowOia6pGSTBTcOEAxDB91/6vpkPN447Az6V +M2riK1VQbDSvVvEAuKX8SKFCap69VMCi9JWpQgA2YN28Asv+Gqf13scFWImE +j86Tj5RlrvpNe1LEQIUT9FCxDjkTIVOMAQdmYEKzLskA4OAvyZEYdpTYVYty +E5owWLZDZCjqHGlBPEGtD+dMGYsCj9iISzXnTqHFgxvkFAr2mBYWqz4nqWLf +sAscuSlahfckjaO1a5Anu0OcTWF+k5juSDmW90YkmQsu/Jmj9uPaEQ7XWTlL +avt8QAbyyCSFMA7B2aAdzjd1/dGqm9RiGKYTffoRydpgI/T7r8XHnM188Mxh +taohdie+Mn7PaO6nBnK/ufqdTE3OMWv0G7qCxUVlh1arU5ZaOXqpFYzsIojT +WHUhu+jGOs9Pv7n6YHQ08p3+fLRXp3QeSArN5PVW0sT3LENMxKwSzY3fgoL3 +DDyhYSejYj2J+yfapim1wiHsNTwqqaQTtgGHbp5pulny1Em5SoBio0mCLTue +k7TN4P+XjUXRSv6qX6CokQCFHuzA87Wg3yoFQGp1tdBBLcDNee8SqCyECgde +fs29YBg+LkVV5I/rwaDT/z8dr8bf3Xr8W7Be/46J/gFPOn1M6uw8OPt/58n8 +Hcf47H+OydfLN99/m//4FvW5f3ma//t2S1MPrW9mk9oF9TjgTrCYMrejcQgE +SQ8OwxtShK02EuffuW9BgetLIYeNgrKZamo5yA1baR88aXZ8/aE7Daa7Ip1I +SrhS3BEWxGZZl9rpjgHI14cbTVSY31MBM8mqs2aGXf7RCxh7ofhsLDJm5jH8 +0ab6VPnuXCHZUSl6UB0eWoQTn104sa7I+OWuDunY1risUpNvqimHZwTHtYdY +Cef0HR24UnexGxywxzLdt1anWggsnVPktw0iw2xqkdDwgXj9F9JCTKhsLONV +i3C4HP0zLX11uubp2efGHGSxWEmfKIVHoLlqT4Kve1tq6voTCP0+9e4SNRSk +OHmm91E0KfWikJuJpTvf13sOqsbtC+iPw25TfSzRRZmLsbkN97Ls6fpjet4z +RwGquao6yFzMdaaY7v2h4TKGNpS1NpFeY/RGk+qsFj0Z+8RC/hbyrRxCgcJM +kT9H7t3A6qetkx4YaOEp3TTFfs03SOuysKsDMKV0JCVyKhGWGIMLPwQGO7RA +ZQQcKN6lTXvVBUoEiwjSezjgEC/xsZG6c2JgO0cz4KN1+UD+HhpAG+uXMtIE +ldPDRtHFR7TR2gxJVJlPy91hbBsT2MqjtPx1fGzf11acHlzYIrTIg7N/yjom +BEyAFFT1jq6A+Jg/kYI/iXv3zCa1laC2R0VU0N983kLXp2MSF1H0h70JztAI +DYlytWIYiKJ6VjVoOaiwYjBZuHAtAS1wgU8JeSstNsIsZzrj0SH0q+zohQEn +5V46+QSXwjiyGuIDqFHu3O3QWjIF0hXHnwN337Dkit7oDWumyTjgYg98O58H +3TlLEhqd9K3bltEycK0854Mt6pH/jmwQmt6IamQGmz5CcCgFbyQJrybWlEDV +KP9FVOmW0ULTalez0aIBrphs1SO2mfK5qeigVSiiN8YlxCi4mFQ8cf6Vr/aL +FYI+gjgz4EKXVHTEWDg597MsVEKkEYtOehDOtSWBdMVG3AXgTe0JEhhj2wxp +L/SdNmLabYHvOObbJEKqLZ3D30IKM3YYR62U4goCoyJjQu1K0XL8XSC7DEDG +VYxNuCHs9DxqGZ1ZRlWkkJVwij6lK7Wv1LWegVkGzDyPD02pIee0D1zQz+15 +i+UoKG7uO4G5q4CNtZSxozdgsvCs21gEXoxpwDa0ErNINMoQwUmxEf4OjVIe +dpgCKDM5td5n5eUMeBU6tktgc1O3Iai/ODSIWI3aIvQ+WY1o0zie5bY7bBl6 +cNgzvjGxOyVAD+ikX8BTLXHgyByQfmdCqSiI1QeNO5sat8MAx8agZo0ZFow3 +C5E2uLDy7MDHx6W/hR9vlqzUZ+EmSUT+nRYSucVCaWSxda3xNZd9HBC7AMA/ +xZLgyaZBOmrticUP4/icskB75uehScpip22qdtoQ7roqBjVOnrLCLEaRhU/F +zQEWfSrcCBfJ9aYztLIoQABmWo7osRyudaGLgLG3S7JsXI6SNd2DfEbZeqrA +XaFk2qK/RpivJQr2QMs7RvNb9HC3+m2W+h1cI7VG7ut3alWMMR48oEgSK1rq +ZeRtkvA3dqRkr5lgazTpH5wg1mDfvP7w57l0UeYVtt6rdJFpT+jU/hvSj70v +luzVT44aDlD507q6rqwTQkvz5LQip6FWZlHGxwajMcVPpwx1ySEYPqPwLSc1 +CtBjaaabEeav9PdkB7ThYPXIs3bRLel1eNd+4IUlMcF63cdmDZ94tT60/eMS +4gy6W0KQVofm2Ulue2SQjJVhRtCh0XGvYI2McBCrGnkKLj9rpGS12ynT5fDB +ciJBq5Mk6z0S6lpxEA6H1GNmm4pP9Gi1Wo1TcrsuDePlcutwH29OHa/v9qx9 +2qoVwjvxpAVEnZPGkpCkdegKvXkMP9dfyPyDxivomdFDDHXXDDwrpRAt2fxn +n4kFiHacZ2p+zxbcePdANiHk2R6SFaAQ+rj5QzhsUZCsvBusojM0zoyMNS62 +qJWy/i+LMN4R8JjcAwlDcTgVrr40PiOzhfWtldMSqn5E9ogLVDjuK/jDAYwJ +VChGFwuDQrvr1LD2VDIZl7NOw9SEeVsZYpjckUbNG25cuhaGYqmpdhxLElZQ +h8KSrD5GDCgnNmZY+q0lk/BZYVczhkNhRcJ8i1n4JzNu2dGOhoUxfAE/kB3S +OyOxik189Ebkd/yivJ69A3ekJu1YiIPksbLuTVClCbY0GLxoycFBnAdesJHg +anZcIvihtRQmKWTorTPa3qvRxlNxfz+3v59lju5ZyV3aZFNiEIGTLnxcuXAs +QyYAgc8JQUUxBXfx3ipDsud40Oqh8ZQg7OaRTwyYp43SI+/eH/6AS0UC7X/4 +Azb7D3+4LuKfnmbZOf3pG/sT/UWrOWLByzW4U3zbZC1kJB2JNoDtQ2tUcn5q +P3X4QOQJD/PPYTZxnFakjS5Vmi2VcSPDhGM6tvHoDeIOiSehM+JfR/GNakRW +JgKNPCJ32SKmBeb5m5h9xyMiwYd3OpwLccVIswF3jRG59qwTT/Ba5JBNFNhm +Qy4PiedE9KVisDShUnS+J5RUpvtXmL4fLVJrNZIuNIh5aFyM1mScR93RJYPo +GTo9aVmLpMxQpBBGG6sVJG8Uxxt65ZTdTN4ADiqwo+J5xsZRNeNgCEUCccNe +ei+Cmejx4SgC+mZL7wkIAZjTYrOdRRzEyoUzLLPjqoPi+DNdsdHV5E6iwYwq +w/K4ghOUDbqdjE/GLtG7w1akVWv+fcIpoLZ5mItWTxXcwKFigj1ttjjo4+B/ +vi2Q2JNnYGxQYw23YfXfA1cxG95wc/mxtDrnS7r5dq0cooYEr+baHpoYG2iS +vp3Et4ReD/Zya//j/cojZY5Fq3nZsNpjKzpkqQdRlcTakf8aab77Cw6LJNR/ +uwPDz/v1h4af8qsPzlQn4d/oOGGmv/JIafZk/FhNjf+3PWwQgM8/cDbyhxw6 +TRf+6oM3vSC/4jhqFvTeI/mQvtTG3BirHHhV6L3MeYR6o7rZg1g35LnIjn7I +o0/+cW7PPAGiayMh89uGzh9nLNm+E1fnZM8wFvqf+Iu5PPqcruOP1Z6rdp5C +Vzy+uDiPQ431crxUi1L+rL8EnoB/zszTM+t5yyIg3jwbDjdsyX/EPf6PmY4l +HXbgwioSHbMXO+Ei/1f657/mj/m3u/l8ru9ed92+ffroEf0tTGBeNzeP+L8e +/YeBTBLZPB8U6gZ0OPSZDSuIzVObdHFTPksp+nvbsv34BLuCE/Lj26f5c8n7 +Bsy+IpDQsrpEy3da1Xn+ouDQwm1pqCL0anSGXdAc6N/hR4/VQZbTbYyss+3e +s2S0z1tl/jF673FdcCvAX24kovGK/sarxrpfrUQO2k7h1YUCOK6mvWfD5QY0 +atgVycsC2YMQXsL2vGBIdGg5Eer9enGd/wXXYILmGra9fZV1/Ui51wCdO15d +1kr+65pzjOhpWlnF5dBMupq49eQPoUZb6uWuR+lMfEnjTCtsAFIV15N/MVN0 +RV4BcV70yUpwdmA/TNgaQvRg9dVgx5ZBxwH7ItvRsc0DP5JeF658XkpCGSQs +6xD2t6sPkr27rUNx69Due8EdBPpR9DRmiQDi0vNgnRx5yYlot5c1c7AobVb4 +UGI0FQPbItAjPEySbHo4Jtr/IXuqzvYoPPw0ZOSTfVLJUsKvM6t6HTOF36xn +vlN8oitTq/i3Hsy9AKMP6zvDtj7j5ooVimcBpODVbivhqNS25EL2hoeFNTbr +04prm/yrC85w3rfuV2vLGvHk5HHdnYnkhPJx3IMpEn/E/wgVNjC+LTrMTVTm ++ff1rbYXaofoUteWMkQ2UnH+Tcd+7x5dtiFriyYmbaRHM55Qn8ZT0OXYGJ5x +KUpbz1ARhQOIB4vtyc90ISA8xsJA9EOj7kR4R+AWGsXvYM7SrcL2QwSdtYov +s4QDMEV1QIi1Ac2jhAv4+Uw6kaOFhfyczLgKWL+YcFixEUrmCUtpA8J0Zuuu +uNOMQ8WhfAqx1LaeRcH963TxxES9ROztpYWBAhBXD6zXU6KMLSXG5T6wfoZ1 +kOi0daV+hcZf38qHHNCTyJ9niAqxP/mjRv/yfyODFIQZD2UpCfNqOPjPuuv8 +CHRRoS5CxjauncS7jv6ITum9pJrcnAZTCoMfMgOslT7RjzS/46DDee5qMLeV +5LE8JFiMeofBi8yTgnQGicCRM0vT5quYX2UXpmSmFVej7+ZMvAYJKotKFzsZ +NEBdUwaKzAOLRM9v7l2jry7+aWKVKrkylRqJB/zGU+5YA88AAg3LxDkpNuVk +j/jhAcIo0y4Hlgo7J0sOFTz4aRMWlzxebw7BsOL+lyybtTe4d4smPXDTuTwk +Dhj1JsIDeN5ayMdNfJZmvxJbypdJ/LK4Vz+Fy8pYZ6FLcYJVoFmc+JUQO0gS +vpHKonTd7NDNU4soQhSj4Cj9TaxuIykREnn5OlqNKUbaZwYV2TNYtJAcjH+n +5X02la2OsNlYwBYznlwcnVSFjJFrPDAT/iFYtHKlyWFjzfzMvztRJ1sBEBu9 +YMz9Tealf8enbctd+LJ766wAPx4WSE6KKzOmtL0KKeSFd6GUmcvRs+xNt/a5 +ofGHibTim219aCT9HEqxLboe7DhaAB1qzP+qlEmoLWIXWDxG21AzijH2aRxp +0ygsNgyUg+MSLCdLT22VJfG6X89/uQs1CBGioD+vtAJBegzhEZLj07hsRMAI +Gc5+v7nTFF1wGUI/r1C3oDA6bQFzXPkki1NzORndRuXg9auHE9E/UOYDy63A +ozZW6MZ8P4CfF5/urD+hoD50wgbjMPqik8FKnETLTNqblcIfIlro+9AtewYL +6BmAGeEhQAiTwHL7wUpMTiR72U9hI1TGEkjdBrWbOh9+xSwQF6rRq6QPan/F +eAWwOdKuXpqLcSBF7O52Ue/LSFgxFXigw628WXKzvReGcLtnH+rsjuzT91qK +ZI2wEpaZoHpiJdH4+MCp1wdnwXSW4Ar7wIBqcRI1H7ChrhXCxQuVDdDI0/Tt +PDSAHxrNb4ZGwr5rq5mxojiC88WZrj4qfmjazMatmhiEmIrLuEzEqUZElE2J +e5VkGnM5GqWprPKfzSfyWWlH61Yz/oDo4Ta0TzilLlzhKXs6iHp4SZwRwXPC +Y5wF+uDnPL64+CfrBIbmXty2EF64ZnxA1cqabzfkckd3BsXCMBMLX7NuGd2q +3Rcnwob3Q9Pm+0gonR7aJvGnDHj3AS+jJwihE3w2U9+u93fW0jzDTPRSf2Yn +CH+idkLo+hUFwDGT+MqxCkoQYu3uvCEU9nuWx61O63ZCxFUAC2U8Jn5VZlYC +k/AdglElQEv6Gbb+V+OuAKgvSLOiBXWLD9SGWIzNS4lCZoGEeSC1s/FRR6BJ +rzAkPZvht9WQBiyDtuuPYETgjwyB9/5zh9FjjPhcM1wHrHr2V62c2Nm/cPV6 +t8bEsH7xcn7W2HpLevRCuyRfX4MtItAnOvQTANKrn4xKQGCF2vVUAjrxGs6f +c+bIxWn4SWDatt4Y/AjcxSeX4uqcPCBe5hnLwP2T0FUpZ2MDRBAOrQR0pAFB +r0pLKtqnRKNaDXfA8ux9ABfu7URl0EDJDjDEpyPewz6IIYq9EPPBaJG3Rwoo +X9euSzLbnLjHJKEyDmoq3H1pDvmxyxaQ+WE6Js3q85dAoXv/8wTPfd8D8S33 +RA178zOWQS/qW/P95tAmv+o9LEzfPdCacq4G3wjO0GjMxT32dl1vSv/M8Acu +QTm0wyf3hqUBLPcEKRdpO7Ko+4fcT7h3rwwGsmeI9wOeExbODUURMcdeoSba +kcGO8qBMa6LjxJPhhcemFXXgL31rqg3P8283bXnsB6FbktpMaWnNsbEaJeCm +jAE1bam2O8pVExGUYkqZHyFuSw8DCVsmK8bVPVrNcZUSyhpXR1zeLCW4mlzn +4yw3Utedn1pnKK1XFLFSTPjZvV7VgJoSax7ikFEeLQXBuzCyCXQRvWXkVGmI +4LpRz+baSB/cJTGhemOSF8uRiooRa6dNeZj50kkNB4G994lIWHQ+EXsye5Vk +zH114MwN6eQnaZUzB6PxrLCSD4+OXL4ah1UHUZUzOc6Ti6pZAdMoVXcR7tqV +EsiwgGpu+DY0nU2ynNfktq6Meto5tRrKs0uNZXtwoxgC6FnSRnvNFBwg5Aue +gLbMWB+24lgoc+HNgXapDb0vpYZHohyN1bgPoei/Q0aH5s+ngi4j2z9r7gzO +Dcy0ajT0k6+UmNOKhxDQCXTb6Hw1To/9VjuCvleuuWtUNGp8mwNkfExApQwc +8w72dp/y2T+ESa2AtKjqJTOdC4IC/kiCy7YUSPTkrBXVNe9gCWAbwq6VAAvE +DxPvCv6r3NSelttRioybHAZ1HhMjeUf5U8fYQPGhZs5r3nNG9y54Z8l7haxJ +rsTkNAhnGqvXYP7dljrRicDJ92+ukhYH9r2sf78wnrPdV+pcFtf0pyIwctaD +kYxwHINM5n6F2RmJGHkU3dqMp9t1iMDO82pORq9iACJxitwYTdV+1LCgSZBV +rUvpF47todkZCpeWUZ6BdMu44gxE9lxi6Zd0uDkiUdrRyhJDd5neAqA8l+rH +0NK+atFLwa7VsQAHzugVuQM513fEE2q9i2zysTe7FJm7kQlOuy+JIzqoVYvf +GO3atJcDn4UM71GBs2L92UAgJTArRB284kavlU0Y1PccCRL5MUn7bJnPxuR9 +pB+gUKAYujKR+ixcHbEEZ35ETPork0nh5WeLgjH1h/xtjwO9resdQqAQmG+K +DQ/WC4zUJnLBaAHicD7PTNdg6YquHlPZRzF6iHOBYciYCEfyY4hyhjr2a3Bh +Vwww16jefUBA/xJpdjGU3NA2Kmk7rZYfc5Ep7ztq9TThkLT6sG3WlFJE8Wpt ++tk4GeJiXS4+GjU70yAaxmmg6x73dN3JfWt7EpKFCRDXYpt0FG8leWgsQ8zC ++YDiuDwWx2koADk8Kelk4Hn0DCY0Iogf0MALSewhEEDrmTqj4LdeknqYOXKY +jV9tI/fa/9ve1TS3kRzZO35FB+dC2CBH9o4Pq13tBE2FwowdfYRErWO9sSE1 +gCbZSxCNQAPk8DK/fSvfy8yq6g+QY+vgwxwcYwpAdXVVVlZ+vneZHnB1CcMg +xGgVryI8rV6ivEPzAMVNFcwSMaPZY4OWLWhl0UIjdyoAxvxL3Xlo/LTc9R6h +xqBYEk0DQAYYy/+t+qunKmDPxuMQb6NUqOPXiwNmhHWe6hWwbjLCekNui8bq +wIVlT1enEP4VkjPSfSiGf9GWVwreSGQ5cibpwqm9xlnzlmGaC/MeMCy1mCRv +wmWHbgq/s6kXt23OgkEzfsTInzxh3jvDWG2MlglA0URJRjrd8LQ4wK+8bgY5 +U5Ie9HUlZDt1eHZd6Z7BBJH36rPzKi+vh27Uf6CvsHOaB+duiOyLIoBlvTTv +TZaECrxNgJlG/WuC49YpgEAfYcODY5edEJXxVDBmVC62YmociJFJq2v1QIFR +RtOkBxpbyadrfEwy7vpwhgVjaO7vD/OBwqYf6xscMDsaAxFDYi09PVb/nA1F +C9PRpBYqBuH6CjX/fbufp8R1nEz/R72GqXknrtNjwls0JsY7GzafqwGi1nHY +/k97SyeWSCdymK1n8urZlwcm2badqF78R6hA7bibZXHDbolUVy5QriLsG8// +WYxFprNiIrRWRH8V6dETVlyV28oVPH39mPCbGGBgGtBTUKOOXu9ykjIOQXjN +/hl5qIyy1LlvI9PnWDjgIg+P8zpQrQhz1zIFjYIxKTa+80tOhn3Fy+iXWQV0 +6buYrK0h1SlGPW22WbiF14tkjfqFAUwe0JoPrsuAjiIOTLx6FEylyVW+PXhZ +hb8BNrJSCkzhiDMe16xSpu74JgS+R9+X5qXT8SPEbbXS9q0x1rmJM4nH3ijJ +wFUHnMVPBCxsM0DAPC+kLJDZJ8mp9jrHHNitox/q1vDZ7G7LT7p4qiugo0os +laA/4TV3qHDbNdmOg56pp5c5TY5BesGENrdsR8wo7Z4R20SQK1Z7UrvGahy3 +ZrPoN1GKJo7nNvK2JkXZRJGT77BWAdn4EA+9eg/BshMzFJKY0MMVxxF9iuaE +U/aY2E0POIDQTrnC5G5LvGDeNisHZrLMY66PJ534QN+V61yT70HeQDTzAU3U +MQYYNRw4yP8GO3ffapVotsrxLIDdNJ5jrS6JkckBkDHnPSYUGuwQK5AG0knC +cKPi+A9sSyw1BczhQcI2i9ggMNBBfxoLbTwBrVMruq+SF/BEjHyXpl7kawZW +SJMSAtL3MOpQJ0I5gMfv/IJeqNLYsAZO0ckw/MQIXkwxXFg3ebmtOo4/9dp6 +INXQrdwgs2RmHM/MeehC2ohf4bW5Z0UChiOluggdy09B8WykGu4EBT8KXmPy +Rj6pzLAYmFGkQOp4JSQDMmV1HCtvXpz+aepEvZO+NQjmYJz2LqfwcVICKtVh +07C30f9kICWyYHK20eZFwLa20j2creBQSaWvMQ0+VNuU6Ml1S8oYbdeJ9exO +gqZsIxhuGh8J3zSGb47fttZhx79j2HcwGP25dReJELkpPnz1M0moBE2vuq9J +v1JV9JqMoQrfHiDcA3wrUA8jjqXg9C6Ejhgd+Zttvaj6gGyia8JOInvRhVq8 +WlU/qwbo/9DDXUqQBq1kZp217AN6bzqAA9c2PaDAT395//mn18YQ3id0G4Cp +Awj9TNEe5WAGtdbspaSf5tx9Uy+HhuqPNHeLhl6UiJ9BJuNEDeWVht/LMlZU +DoTBdI28qsx7HQPhe0ZVc0Q9V9NQ2zHWxd1jcV9XD6z8/RTLeiToUxz9CkHx +XuVnyMaR5Zojx53RG4z8gndu8neLZgcLByKqFvbzGs1iGQ5JihSm6Zb6DtEg +UCNKkT+O5zxIwo0sovwhtdTNspGYRaV0ERdsxpFIOWHjpYQp1TeRTNUyPWTq +1g5IWwxr8SsVhO5NTQa9Oomg627UJBFaNz8W77dSI3YBjDS/+tFtQhSIkb7Q +fiXff374oXifFnLLFSV0iPP/E5cIyxzrxVVrfFecKw6u5VDaHw9RPRjZGrY4 +7QoBM+AxsA/u0YDmpf7TmZexy5mRiBruclBnoLs0S7bpqGEpfpSuGFBKPIeE +4pJDY9/yeHoKZyu/15xFnD4OqrTeOGXQB63uL17i9i8uRSTcAJBB5LTFynyv +6AewImEtnzNpcm45yyrpIBlGx1IV2nR1b/4OkQvWT/VHPOfZuv4wgtoU8eXo +elsvjzxbbyxH7Hd/emA7U37HA2idOjkc5ut1tbSUWk3GMiXKWzRl8B4Leboe +TD3WzIC/e/2RCOEQsA/hrxRNMoIturrC9ehAASfN9kTkgmRy05RNPPIfg/CO +rAiU2pva6+aAbJImiQBwI6SCXH81Bq4IudYEG4e1MO61ZUA1N/VVmBwiArIE +8tI49KcHMEry7YPH2jLPJ49IQO1dBwvDpODP82Ww494jYvYvoN6qk3CJIKEj +M4EcGj0l6BSTDdHGt5k7Xcl7tXa4jlC+IvavhXhXj0echbjMApxjWU0/OzCh +VODFumXlpVEJzmwxt4T9SdEl2ApFu1ALZ4J8WM9LmDByCUZ30VaVkybaHhbH +9DXUgls0QSm2m4Y/iCs7HeDbybLDye77/fIcPZB0lS2N4Edko90F+88ULqxQ +Jf96kOk855i/RkuERlUMR9SwKK+aYBEBDnfdZx/5URF33so7nZMoXYqlzj5c +SMIWNWnOVfiWPio+NYAf8eyOYMMRgusOHK+IER3F51BhyPc/vjn/wx9/+CMk +IohTOCMbFYUOfZJw5L3+fKnrKz9FFVi1jrAlD2VN0l51N6UmozW7Dga3WWXw +DJKRGk3QoMQNXMz3pFgoyutrMJOBRuzghIbv7P6l/VdFiREyQK7QEhFyMWDg +Hfs13b+beoldUbiGgmsbrNBfwYDdLIlPlVUT4F9pz+0qUz5Sba5LAKMvzeUJ +xsX6UTKZpyK0YPR14idACzDgwZHelo9z2KKleatAHwsKoNr2bql34DcPi5pg +1reEOxCuAIcFIYwS8lKVagXO1tNggKmX7lPBtXcMhzhsbjnGSKf19GlV/19g +Px3l5GvtkdLliMCheVBeSZuRDlcQwgoKRwyPRstkr/xwoPDGdZjJa52ukOpe +WLb5PEe4zRi/BpcVunuZ1ZWiTgZ7UYDZcbZUBr8bsoUMl6pNDqBkcIeDxt7A +nZP7Gh5TvxNWKU9I7gy/Wj9wl0k57uzltXWQrk6j91XQUPJ7tAoxUru0iCnl +tGGPVIdDQmGtVbfvtACkNLq/EYEyc5dX2Rgif9iA26radNetu17e9aEzlQyd +hE436WrGUPhACM+O+iAnJMZkg7buBl+tU9aZ5nnVM8NCXN/stJ3a7YNYijUU +MSKvVzT5OsnhCI3FatsNrvWMaQaBOkpXFkItQUw2wK4AoddKG+bCKkW4Ia1Q +t86Dov5dceGLmreSWjAeX3xjyW5n3+n0nc6Gj7UivZAVAxFr+JyseJDoPOEK +NGTM96xsdp8tAt098WJ7IbTpV9ZMa3U2avpIMUa8GNVU0tLEbSk5Fn2GXN1X +5XZndEGDuWBc/EKDe2yRNnR1SqXvuppGgEjYaJuwOvXCDLWqs4cuAixmZe2s +ipJOiaatc8AnLAk3AuHfGwmkCckwGOSYbhJFQGHclaZojNdhGvXfaxCHi/k5 +6cs7zf+eFpaQ81BSx0DodtXalFY7KMEuvRp+GVGxI9udRkpgN8uE5D3CqsnU +K5YPGXNmMEKBISSXWluBOlMGcr4dwqfLfhLGSGZ2feNwm5FPO0yU5Ts1LKR9 +3d4ko0ByuJrtLslUqhGsKz6SFbgpFfXR6BUl58IC3WrtHQmsqI0B2kGYUKpp +j3wllwd35x41MHwFK7k65us7TcqqmtoehF8vqiCAflTfDpzQxJ4E+Ka5h7Iy +jxqE9qOzAaFOB4I2bfNXQqLgUgNYV/ZNGAj+9cWLH/7X9MUaak7iL4t6u9jf +KRXEzM1XtxNeq4CgrL3a7dUwilxTMh6kJ1fu8l6gHsJTbmq9fB8Myle0FXe+ +bqMYUGqkjdi/Fz5+3cF0vVoJc6SZIr1jR+SXx03VxodI7sXq6psmI3FlVFpi +hRJeb2vfKb/Mq7XsaauRZFQS4N+RibP9RFSYYr9pxHRm3kLEhxEDpY0GjL+Z +Ygs1M0oLLA4cYb6hUz8RIjKcMki/3nIr4upy6x0vJazbySccqXCoy0USsNRT +WUnGFQA1TVGFFWkeUdmuB4yBUanaeBRMovmqXNyezJufs/Cp2jZKdLMQf5BZ +RmDGy4qCQCoVo10lhs9u+xjV6Mde8Oo8DV4NHx3DhMh04yxXdql+PR1Q0YJU +bIQuY2a6MWsk6fdwxVmNKswPpXUiCUU4W7gR2s7O7dcMTaiJP3Dtom8Hl66f +kZ318rgRpO7zDIJMHqCrIfvHyz8fSjAb600fFsYWwlUzNbfCB+R3UpZClZsd +dbKoBfSwWrYBEVdEvu1QUAymbBwRRWnb8nJCuzEVUDWZZfTkDL9z5CrQIN4x +9sxJ1tH8I6PFJrxp8iYDADOHU8Z8SuKKSQFCjlfTse8TV0tqSDWVzww2s58G +RWVB15SU2Wp8b6SXSr9vgSpYRgdmyXsLOzvkcwSJt6b1i87H4EjRg3TYm/NI +OgR0v2siuXhYi0WacOvUIrWtbJdKQ07/lrx69RgkXVCRGUpPue8Q/5Nb2UQk +Mn4brZJV1cDXuGlWy4QIEZdcZAInEqkM8BMTvOWycc9KS0K0NkPar672NIgX +KySamnX+egwNanXEOPGepR/KZRVVYoYcHGT5XZDlv4ksS4JcTXIyfkvYYL9x +vF43zpNMutuuQzrQ61f8uCStqokMuU2aVAb3PatjKzcok5ocDzGZC8qWDILj +mBDpfZNY1xdrhSuRoCLTZh+1daNzRXOjW+G0hAE4IhV2Vuf7GhTAWueQV0IO +a5aZ2yb+M5RZgI5txNEeKQp3j4qTSd6RRELOFxU3Y9eEhYLxE2MktpdXsWiB +flVydYzB4SPRcuXE924VJBCuzlQoislUjhGGH4xrxaWyKgDYtVo00Rml50eO +KCPRVYAQVF12bN7ZdJQCkjazcxSble5SMuvgldX5hagqu3TKWCtstQRcN+t2 +2oXFKi7O3p2JPCGuoBfB5F2D4AByUWEP5DugA/tUhamIUuz+4M9pulpO7T2x +28o2+jZe1XlX3TX0yeq7mvVwk121uFlLhvsxcfIirl4p5tn3IipU9wuL7C/l +2g4Wbo2I7WRVzoXTQMjIqvV9vW3WjMLRIxBnng2rDEwHWSsXlbcSeKgsHN4Y +unXeBWN0EKQT7fnIsvRr4qaFt9nwRQxOndS+nnpAqp6OijSvv/386RLtpHPM +YO0lglJEs6iSHl8NW/FgE6mVlhiaA0sx5IWJTZbM5iLWXt0GBbGLdpupBhkh +HLmgWe8I2oefSOcBr4xZ/np1CswO2oQjt7ePRH3WwTmTnkWBh2mDMiAkcFIH +GGRpDsmmI2AFQZyO7m94/CcaKQylidSKIGlxjawTfFvbFhOLDJA+THKSzd15 +qwtjC1Y9RwVoq9WaeEtHobyG94XYY3Qa827+VQJG5Vx1b7IREx0acLnF2eJ2 +3TwEob3WKDD7Cjc3W7ReI8Iba17suDjFJqrPF+y5Jtcw0gASgxaOkzCVs4vi ++I7//7Ss5Zp6KzaAWIq3cAvPguESzH2Yjqp/33949+a/ivA/rJjM3UsvzIWb +hIuqrh4Khf4KKn/vXGuqS3T2xJGuHfqusqpcQ/1pPdvjWMpxx4ERcRN+gdIr +US67/dLOp70C1zrI89mqeBsuNG2rD0NO3OXkfCUdBhWBtQJankeVroIyn5eL +WyPoZZNk0LgToAGshC/5VOht1rdASz9b+XXAVgLxI6lsoKLuNeb4GGYfLGL5 +LsCOlPwQC7Z+ELxWlMpZmRf2FX1gAvcqWCE3cC4nQVwZJwp3BVYhiO5qhqmU +IK9EgknAXIgvkZRavhwma7xMU2ym3rTdwnnJxwpkhflLTk6fkq3fTTsrulR5 +aCT4n4QYU9oq+SdZOVWdlqtrYXa+uRPhCh4SAjNAicM9NFVF7NwR5X1Zrxic +zZpmeWzHAZHy11BvLa2k1KaOs1XGL4ViKdn74Pzst+toy1oe2W6FVSOC1RlW +onW2R7TwN221D6cgeOMTI8nVwhbBr7OKz3YnHfFynIycKBgAwSVYTr5GGJSv +kOOvCRTN16H6zb9qUl2hYz3fmgCqnKKAiXyzctoiF4lzI61L5O7Efn02OqYQ +Y9COjaiSYVkeDLpdPjv8PsnrB5OGmNc0maUajRVMbGlpCuv/ofWaR1cRcpR+ +X4UAzWvdIJn44AQfOKy603qOMZtI4NM31EsAFMxfSwI/PAYBWpOoACXZGgsI +K4GQYHxFIpZBHDwquN1nKEWRZ2ByGOIoP4+wi3EMkgN5+uRkrsonQJK6WE6d +zrDnISslK0jPTMniHJU0Tiq8ihwjr4lysH00QH4V0/+LHd4v7f7uK3mmMmTG +hN54cFF8IAacvuCnOlC/2eXJgZrlFwmi6mw+dVsgtbA2LR7vZeO9gqdfBT8Y +GESkmn1LFPTocofR0DTVaUDoQzyFuc/Lf7apP2vaWHK4vf/IxC2l+TTg0rdc +7n+eaT895Ut4T0tW6basRGOEJc0WxYqzPAbRgyqc/PLLL8UGunLiXVSyFMWr +It/S4ndF73wW3xfHfzh9UZz0P5pOKoMa+YI1xpCpcBe/Dz7Rz8cvTl8AlyV5 +0kmRTSUdCnarDia/zk56GLH30FnRU0/TieFmZP8cRhx6zsBrTxJQtFf9Rxb/ +/qoYfMIkxa97VRwPPe6kyF5pOj5Y2LnMGv3zyzGwuScMUqv0HAl1ib73SD2g +LHyoX88C87R1i55f7+trtcwsv5pSzD50eCGYOxqgPx6zO6cTaed1X8fm4b3c +I2kFZt2G7p7fTOvEtC51QdWsDnvqdu3XpDb7S6zNHrSkzy/FmDm/PGUYX9gR +2IqNHhIkUpXX4VuYyKgbiXbyyDzjm3wbA/nXsyl9cxP5N5vwN5vwG9uEmDIn +FWZ9FlSWlB5YzZ4hKedpoG/5DpJ6iNSVnfqo7kvJ+T3LkTJRySvqTCJECaxs +PA8SPUhZJEyVvxRFFFaAiH2StMmEt/tsoNaZiud1V+t2nmYj2ZvpOJ0tMw1t +Pxy1FQmglCDzjfBfT7LlGAe8Tq3IsK/RIDMjraMMZkXXbkslfDph6aMs8Ksi +G+93B+xNt5Licoefv5PXwJISO3BdJFLJ3oLw9kO/DWshPyaVns3nP4pgr1pL +QjH4u1dFd9f963GcE/tWVv4tHDBx8HkQ19tJ/Lf66tdOJKzVZPgGCx+a/fAF +9oOv7MBIU9qaJydimC9uB+5o9jICZm4VzNv8Ika3Cb9xLt9oT4v3a+CCCD2h +hKfX1YDngtyWXKHLxlJBA/0sGPxjtQZmBttXvg/G1sn3ZqPizkNdh7cs1Vvv +aBjukPmXtEPG4PM6uWHLMgFK67qRQM6uuTvZb0ZnyeE+rwXU0XMy0VZod49S +PjjfhiUOpuKMKTP5r74JeTkUCE7QLadPPOpsuYzd4KtaovqeCj8PS1ULoVY4 +z5fV9o7ko9ry+Kz+wHJT7zJGrnfVA4ZqX1rGApsq9pfVgDFb4JViYrpfaDXQ +fUOwBaXDXOTDa5kFtXzY37ta09TGs4ruFsRaP745b2XUSoHifKTKKkvIOO8C +Uu9W46KlDVhXLKPahNerZW9QnM8aCHT1SUp9ZVsIs47Z07OL6bCJ9v/XgXit +I+IBAA== + +--> + +</rfc> + diff --git a/docs/ietf/process.txt b/docs/ietf/process.txt index 128c31bff1..6492861163 100644 --- a/docs/ietf/process.txt +++ b/docs/ietf/process.txt @@ -22,7 +22,7 @@ $ sudo gem install kramdown-rfc2629 $ kdrfc --version Main: -$ kdrfc draft-ietf-bmwg-mlrsearch-06.md +$ kdrfc draft-ietf-bmwg-mlrsearch-07.md If that complains, do it manually at https://author-tools.ietf.org/ |