diff options
author | 2025-02-25 14:12:27 +0100 | |
---|---|---|
committer | 2025-02-25 14:12:27 +0100 | |
commit | 573d6c09705acc9cd0423c18912cfc48a97390cc (patch) | |
tree | ae5542b70e2d463f06f61b6372d1eacb34c1a920 | |
parent | 7e053e65069df061748d625cea306075efdc75a8 (diff) |
feat(ietf): Replace MLRsearch with draft-09
+ Also update the MLR methodology page:
+ Remove old descriptions when covered by draft.
+ List facts needed by test report requirements.
+ Add a skeleton of future "Additional details" documents.
- The skeleton is quite unreadable for now.
Change-Id: I0d3e6f4419f190fcedc5cbeb7c3e3a44b0d0d975
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
-rw-r--r-- | docs/content/methodology/measurements/data_plane_throughput/mlr_search.md | 426 | ||||
-rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-09.md (renamed from docs/ietf/draft-ietf-bmwg-mlrsearch-08.md) | 1539 | ||||
-rw-r--r-- | docs/ietf/process.txt | 4 |
3 files changed, 848 insertions, 1121 deletions
diff --git a/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md b/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md index 71e4471905..3bf004e277 100644 --- a/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md +++ b/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md @@ -5,46 +5,9 @@ weight: 2 # MLR Search -## Overview - -Multiple Loss Ratio search (MLRsearch) tests use an optimized search algorithm -implemented in FD.io CSIT project. MLRsearch discovers conditional throughput -corresponding to any number of loss ratio goals, within a single search. - -Two loss ratio goals are of interest in FD.io CSIT, leading to Non-Drop Rate -(NDR, loss ratio goal is exact zero) and Partial Drop Rate -(PDR, 0.5% loss ratio goal). -Instead of a single long trial, a sequence of short (1s) trials is done. -Thus, instead of final trial duration, a duration sum (21s) is prescribed. -This allows the algorithm to make a decision sooner, -when the results are quite one-sided. -Also, only one half of the trial results is required to meet -the loss ratio requirement, making the conditional throughput more stable. -The conditional throughput in this case is in principle the median forwarding rate -among all trials at the relevant lower bound intended load. -In practice, the search stops when missing trial results cannot -disprove the load as a lower bound, so conditional throughput -is the worst forwarding rate among the measured good trials. - -MLRsearch discovers all the loads in single search, reducing required time -duration compared to separate `binary search`es[^1] for each rate. Overall -search time is reduced even further by relying on shorter trial -duration sums for intermediate targets, with only measurements for -final targets require the full duration sum. This results in the -shorter overall execution time when compared to standard NDR/PDR binary -search, while guaranteeing similar results. - - Note: The conditional throughput is *always* reported by Robot code - as a bi-directional aggregate of two (usually symmetric) - uni-directional packet rates received and reported by an - external traffic generator (TRex), unless the test specifically requires - unidirectional traffic. The underlying Python library uses - unidirectional values instead, as min and max load are given for those. - -## Search Implementation - -Detailed description of the MLRsearch algorithm is included in the IETF -draft +## Specification + +Detailed description of the MLRsearch specification is included in the IETF draft [draft-ietf-bmwg-mlrsearch](https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-mlrsearch) that is in the process of being standardized in the IETF Benchmarking Methodology Working Group (BMWG). @@ -52,61 +15,340 @@ Methodology Working Group (BMWG). MLRsearch is also available as a [PyPI (Python Package Index) library](https://pypi.org/project/MLRsearch/). -## Algorithm highlights +## Search Goals + +In CSIT we use two search goals, traditionally called NDR (non-drop rate) +and PDR (partial drop rate): + +NDR: +* Goal Initial Trial Duration = 1 second +* Goal Final Trial Duration = 1 second +* Goal Duration Sum = 21 seconds +* Goal Loss Ratio = 0.0% +* Goal Exceed Ratio = 50% +* Goal Width = 0.5% + +PDR: +* Goal Initial Trial Duration = 1 second +* Goal Final Trial Duration = 1 second +* Goal Duration Sum = 21 seconds +* Goal Loss Ratio = 0.5% +* Goal Exceed Ratio = 50% +* Goal Width = 0.5% + +## Test Report Information + +The MLRsearch specification requires (or recommends) the test report to contain +some information, here is the summary. + +CSIT uses TRex as the traffic generator, over SSH, bash and Python layers +that add roughly 0.5 second delay between trials. +That is enough to replace all the wait times listed in RFC 2544. + +The trial effective duration includes that delay, so as little as 7 trials +is sometimes enough for load classification (instead of 11). + +TRex uses multiple worker threads (typically 8) so the traffic +can be bursty on small time scales, but even the small buffers on DUT side +usually make this effect invisible. + +TRex is usually precise in the number of packets sent, +but in high-performance setups it may show "duration stretching" +where it takes considerably longer than the intended duration +to send all the traffic. To combat this behavior, CSIT applies +"start+sleep+stop" approach, which can cause some number of packets +to remain unsent. We allow 10 microseconds worth of traffic to be missing +to allow for not all TRex workers starting at the same time, +but larger values are included in trial loss ratio. + +Most our tests use symmetric bidirectional traffic profiles, +and the results are presented as aggregate values, e.g. east-west plust west-east. +Other specifics of traffic profiles are too numerous to be listed here, +CSIT code is the authoritative documentation. + +Max load is computed from known values of per-direction NIC limits, +usually large packets are limited by media bandwidth and small frames +are limited by intrinsid packets-per-second (pps) limits. +Min load is set to 9.001 kpps to ensure enough packets for latency histogram. + +If min load is classified as an upper bound for the NDR goal, +the test fails immediatelly (not searching for PDR). +Also, if the search takes too long, the test fails firhout a result. +If max load is classified as a lower bound, this situation is reported +as zero-width irregular result, usually not distinguished from regular results. + +Relevant lower bound and relevant upper bound are recorded together with +conditional throughput for both goals, but only the conditional throughput +is presented in our WebUI frontend. + +TRex uses lightweight way to count forwarded packets, +so it does not identify duplicate and reordered packets. +Some testbeds contain a link for TRex self-test, the results of such "trex" tests +are measured as separate tests, but they are not influencing real SUT tests. + +As different test require different SUT setups, those are lightly documented +on suite descriptions, but CSIT code is the authoritative documentation. + +<!-- +TODO: Are all requirements addressed in "Deviations from RFC 2544"? + +Compliant in principle, but: ++ trial wait times ++ trial time overhead to effdur ++ start+sleep+stop ++ 10us buffer ++ TRex is bursty in principle but not in practice ++ test report shows aggregate values ++ traffic profile details are in code only ++ fails on timeout or irregular NDR +- intermediate phases increase dursum ++ min load 9kbps per direction for latency ++ max load by nic known pps and bps limits ++ hitting that max load is treated as regular result of zero width ++ only conditional throughputs are processed +- (at least PDR is "less discrete") ++ relevant bounds are also stored ++ duplicate and reordered packets are not detected ++ self-test is done for repeatability reasons +- (not for max load reasons) ++ SUT config is in code ++ (only partly in suite documentation) + +TODO: Update the above when the below progresses. +--> + +## Heuristics MRR and receive rate at MRR load are used as initial guesses for the search. All previously measured trials (except the very first one which acts as a warm-up) are taken into consideration. -For every loss ratio goal, the relevant upper and lower bound -(intended loads, among loads of large enough duration sum) form an interval. -Exit condition is given by that interval reaching low enough relative width. -Small enough width is achieved by bisecting the current interval. -The bisection can be uneven, to save measurements based on information theory. -The width value is 0.5%, the same as PDR goal loss ratio, -as smaller values may report PDR conditional throughput smaller than NDR. - -Switching to higher trial duration sum generally requires additional trials -at a load from previous duration sum target. -When this refinement does not confirm previous bound classification -(e.g. a lower bound for preceding target -becomes an upper bound of the new target due to new trail results), -external search is used to find close enough bound of the lost type. -External search is a generalization of the first stage of -`exponential search`[^2]. - -A preceding target uses double of the next width goal, -because one bisection is always safe before risking external search. - -As different search targets are interested at different loads, -lower intended load are measured first, -as that approach saves more time when trial results are not very consistent. -Other heuristics are there, aimed to prevent unneccessarily narrow intervals, -and to handle corner cases around min and max load. - -## Deviations from RFC 2544 - -RFC 2544 implies long final trial duration (just one long trial is needed -for classification to lower or uper bound, so exceed ratio does not matter). -With 1s trials and 0.5 exceed ratio, NDR values reported by CSIT -are likely higher than RFC 2544 throughput (especially for less stable tests). - -CSIT does not have any explicit wait times before and after trial traffic. -(But the TRex-based measurer takes almost half a second between targets.) - -Small difference between intended load and offered load is tolerated, -mainly due to various time overheads preventing precise measurement -of the traffic duration (and TRex can sometimes suffer from duration -stretching). Large difference is reported as unsent packets -(measurement is forcibly stopped after given time), counted as -a packet loss, so search focuses on loads actually achievable by TRex. - -In some tests, negative loss count is observed (TRex sees more packets -coming back to it than TRex sent this trial). CSIT code treats that -as a packet loss (as if VPP duplicated the packets), -but TRex does not check other packets for duplication -(as many traffic profiles generate non-unique packets). - -[^1]: [binary search](https://en.wikipedia.org/wiki/Binary_search) -[^2]: [exponential search](https://en.wikipedia.org/wiki/Exponential_search) +At the start of the search, a discrete set of possible loads is pre-computed +and used to avoid rounding errors. + +The specified search goals are treated as "final targets", +but receded by "intermediate targets" of smaller duration sum +and larger goal width. This allows the algorithm to quickly converge +towards the "interesting region" where full duration sum is needed. + +Generally, smaller candidate loads are measured first. +For more tricks (uneven splits and multiple selection strategies) +see the source of the Python implementation. + +# Additional details + +There will be future documents linked from here. +For now, there is only an outline of future (ambitious) content. + +## History of early versions in CSIT? + +### MLRsearch as a class of algorithms + +(mutually incompatible) + +### Example tests benefiting from different goals? + +## Design principles + +### Independence of components + +### Implementation freedom + +#### Optional and implementation-required inputs + +#### Reasonable default values + +#### Better outputs in future + +#### "allowed if makes worse" principle + +### Follow intuition, avoid surprises + +### Usage + +(anomaly detection in trending, comparison tables with low stdev for release) + +### Max load and min load + +### Size of loss + +(does not matter, only binary low-loss vs high-loss) + +### Goals used + +### Simulator + +(PLRsearch fitting functions, exotic goals) + +(Example of time savings between RFC2544 and CSIT goal at the same accuracy?) + +### Long trials vs many trials + +### Conservativeness + +### Fail fast + +### Timeout + +## Measurer questions + +### Capabilities + +(Traffic profiles specific to TRex, TG TA and Yang) + +### Self test + +### Warm-up + +### Time overhead + +### Predicting offered count + +### Duration stretching + +(start+sleep+stop) + +### Burstiness + +### Negative loss + +### Aggregate limits + +(RX+TX, sum over ports, number of queues, CPU limits, baseline vs burst in cloud) + +### Other Oload issues + +(duplex and other interferences; DUT-DUT links with encapsulation overhead) + +## Test report + +### Definition + +### Alternative units + +#### Unidirectional vs bidirectional + +#### Bandwidth + +## Heuristics + +### FRMOL and FRFRMOL + +### Intermediate phases + +### Relative width + +### Discrete loads + +### Expansion coefficient + +### Uneven splits + +### Selector strategies + +### Candidate ordering + +## DUT behaviors + +### Periodic interrupts + +### More details on distribution of big and small loss spikes + +(performance spectrum as a probabilistic distribution over trial forwarding rate) + +(trial results as small population) + +(median and other quantiles, "touching" quantiles) + +### Exceed probability + +(load regions, common patterns seen in practice) + +### Large buffers + +### Performance decrease due to resource leaks + +### Energy mode switching can cause loss inversion? + +## Correctness + +### Balancing sum from short trials + +### Optimistic and pessimistic estimates + +### Load is eventually classified + +### Gaming is possible but slow + +### Brittle heuristics + +### Goal ordering + +### Discouraged goals + +#### Goal Width > Goal Loss Ratio + +#### Goal Duration Sum value lower than Goal Final Trial Duration + +#### Incomparable goals + +(worst case: slow race to bottom) + +### When a load can become undecided again? + +## Related test procedures + +### Latency + +### Passive Telemetry + +### Active Telemetry + +### Comparison with FRMOL + +### Comparison with PLRsearch + +## Beyond frames + +### Transactions + +### Fragmentation + +### Throttled TCP + +### Ramp-up + +### Fixed scale + +### Reset + +### Bisecting for telemetry thresholds + +### Bisecting for B2B burst size + +## Future improvements + +### Return trials at relevant bounds + +### Allow flipping the conservativeness? + +(return the larger load when Loss Inversion happens) + +### Short high-loss trials to affect Conditional Throughput? + +### Multiple runs between hard SUT resets + +### Duration sum based on misclassification probability + +(needs a prior on exceed probability distribution; and error/time balance) + +### Heavy loss should be worse than narrow loss + +### Predict goodput based on loss and latency + +## Examples? + +(take a real run and discuss heuristic decisions?) + +## Summarize how MLRsearch addressed the Identified Problems? diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-08.md b/docs/ietf/draft-ietf-bmwg-mlrsearch-09.md index ff63224a9d..58b3ebb646 100644 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-08.md +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-09.md @@ -2,8 +2,8 @@ title: Multiple Loss Ratio Search abbrev: MLRsearch -docname: draft-ietf-bmwg-mlrsearch-08 -date: 2024-10-21 +docname: draft-ietf-bmwg-mlrsearch-09 +date: 2025-02-24 ipr: trust200902 area: ops @@ -33,10 +33,11 @@ normative: RFC1242: RFC2285: RFC2544: + RFC5180: RFC8219: - RFC9004: informative: + RFC6349: TST009: target: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf title: "TST 009" @@ -48,6 +49,15 @@ informative: target: https://pypi.org/project/MLRsearch/1.2.1/ title: "MLRsearch 1.2.1, Python Package Index" date: 2023-10 + Lencze-Shima: + target: https://datatracker.ietf.org/doc/html/draft-lencse-bmwg-rfc2544-bis-00 + title: "An Upgrade to Benchmarking Methodology for Network Interconnect Devices" + Lencze-Kovacs-Shima: + target: http://dx.doi.org/10.11601/ijates.v9i2.288 + title: "Gaming with the Throughput and the Latency Benchmarking Measurement Procedures of RFC 2544" + Ott-Mathis-Semke-Mahdavi: + target: https://www.cs.cornell.edu/people/egs/cornellonly/syslunch/fall02/ott.pdf + title: "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm" --- abstract @@ -58,8 +68,7 @@ support multiple loss ratio searches, and enhance result repeatability and comparability. The primary reason for extending [RFC2544] is to address the challenges -and requirements presented by the evaluation and testing -the data planes of software-based networking systems. +of evaluating and testing the data planes of software-based networking systems. To give users more freedom, MLRsearch provides additional configuration options such as allowing multiple short trials per load instead of one large trial, @@ -86,7 +95,7 @@ The purpose of this document is to describe the Multiple Loss Ratio search (MLRsearch) methodology, optimized for determining data plane throughput in software-based networking devices and functions. -Applying vanilla [RFC2544] throughput bisection to software DUTs +Applying the vanilla [RFC2544] throughput bisection method to software DUTs results in several problems: - Binary search takes too long as most trials are done far from the @@ -96,11 +105,11 @@ results in several problems: - Software DUTs show noisy trial results, leading to a big spread of possible discovered throughput values. - Throughput requires a loss of exactly zero frames, but the industry - frequently allows for small but non-zero losses. + frequently allows for low but non-zero losses. - The definition of throughput is not clear when trial results are inconsistent. To address these problems, -the MLRsearch test methodology specification employs the following enhancements: +the MLRsearch test methodology employs the following enhancements: - Allow multiple short trials instead of one big trial per load. - Optionally, tolerate a percentage of trial results with higher loss. @@ -123,10 +132,10 @@ thus achieving high comparability without limiting future improvements. MLRsearch configuration options are flexible enough to support both conservative settings and aggressive settings. -The conservative settings lead to results +Conservative enough settings lead to results unconditionally compliant with [RFC2544], -but longer search duration and worse repeatability. -Conversely, aggressive settings lead to shorter search duration +but without much improvement on search duration and repeatability. +Conversely, aggressive settings lead to shorter search durations and better repeatability, but the results are not compliant with [RFC2544]. No part of [RFC2544] is intended to be obsoleted by this document. @@ -150,15 +159,16 @@ for time-efficiency improvements. A more generalized throughput concept could enable further enhancements while maintaining the precision of simpler methods. -The bisection method, when unconditionally compliant with [RFC2544], -is excessively slow. +The bisection method, when used in a manner unconditionally compliant +with [RFC2544], is excessively slow. + This is because a significant amount of time is spent on trials with loads that, in retrospect, are far from the final determined throughput. [RFC2544] does not specify any stopping condition for throughput search, so users already have an access to a limited trade-off between search duration and achieved precision. -However, each full 60-second trials doubles the precision, +However, each of the full 60-second trials doubles the precision, so not many trials can be removed without a substantial loss of precision. ## DUT in SUT @@ -175,9 +185,9 @@ SUT as: - The collective set of network devices as a single entity to which stimulus is offered and response measured [RFC2285] (Section 3.1.2). -[RFC2544] specifies a test setup with an external tester stimulating the -networking system, treating it either as a single DUT, or as a system -of devices, an SUT. +[RFC2544] (Section 19) specifies a test setup +with an external tester stimulating the networking system, +treating it either as a single DUT, or as a system of devices, an SUT. In the case of software networking, the SUT consists of not only the DUT as a software program processing frames, but also of @@ -191,8 +201,7 @@ experience interference from the operating system or other software operating on the same server. Some of this interference can be mitigated. -For instance, -pinning DUT program threads to specific CPU cores +For instance, pinning DUT program threads to specific CPU cores and isolating those cores can prevent context switching. Despite taking all feasible precautions, some adverse effects may still impact @@ -201,12 +210,12 @@ In this document, these effects are collectively referred to as SUT noise, even if the effects are not as unpredictable as what other engineering disciplines call noise. -DUT can also exhibit fluctuating performance itself, for reasons -not related to the rest of SUT. For example due to pauses in execution -as needed for internal stateful processing. -In many cases this -may be an expected per-design behavior, as it would be observable even -in a hypothetical scenario where all sources of SUT noise are eliminated. +DUT can also exhibit fluctuating performance itself, +for reasons not related to the rest of SUT. For example +due to pauses in execution as needed for internal stateful processing. +In many cases this may be an expected per-design behavior, +as it would be observable even in a hypothetical scenario +where all sources of SUT noise are eliminated. Such behavior affects trial results in a way similar to SUT noise. As the two phenomenons are hard to distinguish, in this document the term 'noise' is used to encompass @@ -225,14 +234,14 @@ it lowers the observed SUT performance as observed in trial results. In this model, SUT does not have a single performance value, it has a spectrum. One end of the spectrum is the idealized noiseless performance value, the other end can be called a noiseful performance. -In practice, trial result -close to the noiseful end of the spectrum happens only rarely. -The worse the performance value is, the more rarely it is seen in a trial. +In practice, trial results close to the noiseful end of the spectrum +happen only rarely. +The worse a possible performance value is, the more rarely it is seen in a trial. Therefore, the extreme noiseful end of the SUT spectrum is not observable among trial results. -Also, the extreme noiseless end of the SUT spectrum +Furthermore, the extreme noiseless end of the SUT spectrum is unlikely to be observable, this time because some small noise effects -are likely to occur multiple times during a trial. +are very likely to occur multiple times during a trial. Unless specified otherwise, this document's focus is on the potentially observable ends of the SUT performance spectrum, @@ -240,17 +249,15 @@ as opposed to the extreme ones. When focusing on the DUT, the benchmarking effort should ideally aim to eliminate only the SUT noise from SUT measurements. -However, -this is currently not feasible in practice, as there are no realistic enough -models available to distinguish SUT noise from DUT fluctuations, -based on authors' experience and available literature. - -Assuming a well-constructed SUT, the DUT is likely its -primary performance bottleneck. -In this case, we can define the DUT's -ideal noiseless performance as the noiseless end of the SUT performance spectrum, -especially for throughput. -However, other performance metrics, such as latency, +However, this is currently not feasible in practice, +as there are no realistic enough models that would be capable +to distinguish SUT noise from DUT fluctuations +(at least based on authors' experience and available literature). + +Assuming a well-constructed SUT, the DUT is likely its primary bottleneck. +In this case, we can define the DUT's ideal noiseless performance +as the noiseless end of the SUT performance spectrum. +That is true for throughput. Other performance metrics, such as latency, may require additional considerations. Note that by this definition, DUT noiseless performance @@ -262,7 +269,7 @@ by estimating the noiseless end of the SUT performance spectrum using a limited number of trial results. Any improvements to the throughput search algorithm, aimed at better -dealing with software networking SUT and DUT setup, should employ +dealing with software networking SUT and DUT setups, should employ strategies recognizing the presence of SUT noise, allowing the discovery of (proxies for) DUT noiseless performance at different levels of sensitivity to SUT noise. @@ -270,11 +277,14 @@ at different levels of sensitivity to SUT noise. ## Repeatability and Comparability [RFC2544] does not suggest to repeat throughput search. -And from just one -discovered throughput value, it cannot be determined how repeatable that value is. +And from just one discovered throughput value, +it cannot be determined how repeatable that value is. Poor repeatability then leads to poor comparability, as different benchmarking teams may obtain varying throughput values for the same SUT, exceeding the expected differences from search precision. +Repeatability is important also when the test procedure is kept the same, +but SUT is varied in small ways. For example, during development +of software-based DUTs, repeatability is needed to detect small regressions. [RFC2544] throughput requirements (60 seconds trial and no tolerance of a single frame loss) affect the throughput results @@ -283,7 +293,7 @@ The SUT behavior close to the noiseful end of its performance spectrum consists of rare occasions of significantly low performance, but the long trial duration makes those occasions not so rare on the trial level. Therefore, the binary search results tend to wander away from the noiseless end -of SUT performance spectrum, more frequently and more widely than short +of SUT performance spectrum, more frequently and more widely than shorter trials would, thus causing poor throughput repeatability. The repeatability problem can be addressed by defining a search procedure @@ -299,10 +309,9 @@ Conversely, any alteration to [RFC2544] throughput search that improves repeatability should be considered as less dependent on the SUT noise. -An alternative option is to simply run a search multiple times, and report some -statistics (e.g. average and standard deviation). -This can be used -for a subset of tests deemed more important, +An alternative option is to simply run a search multiple times, +and report some statistics (e.g. average and standard deviation). +This can be used for a subset of tests deemed more important, but it makes the search duration problem even more pronounced. ## Throughput with Non-Zero Loss @@ -318,7 +327,7 @@ Then, it says: it is useful to know the actual maximum data rate that the device can support. -However, many benchmarking teams accept a small, +However, many benchmarking teams accept a low, non-zero loss ratio as the goal for their load search. Motivations are many: @@ -339,6 +348,9 @@ Motivations are many: - If an approximation of the SUT noise impact on the Trial Loss Ratio is known, it can be set as the Goal Loss Ratio. +- For more information, see an earlier draft [Lencze-Shima] (Section 5) + and references there. + Regardless of the validity of all similar motivations, support for non-zero loss goals makes any search algorithm more user-friendly. [RFC2544] throughput is not user-friendly in this regard. @@ -358,29 +370,38 @@ for the load that satisfies a non-zero Goal Loss Ratio. But it is not that obvious how to search for multiple goals at once, hence the support for multiple Search Goals remains a problem. +There does not seem to be a consensus on which ratio value is the best. +For users, performance of higher protocol layers is important, for +example goodput of TCP connection (TCP throughput), but relationship +between goodput and loss ratio is not simple. See +[Lencze-Kovacs-Shima] for examples of various corner cases, +[RFC6349] Section 3 for loss ratios acceptable for an accurate +measurement of TCP throughput, and [Ott-Mathis-Semke-Mahdavi] for +models and calculations of TCP performance in presence of packet loss. + ## Inconsistent Trial Results While performing throughput search by executing a sequence of measurement trials, there is a risk of encountering inconsistencies between trial results. +Examples include: + +- A trial at the same load (same or different trial duration) results + in a different Trial Loss Ratio. +- A trial at a larger load (same or different trial duration) results + in a lower Trial Loss Ratio. + The plain bisection never encounters inconsistent trials. But [RFC2544] hints about the possibility of inconsistent trial results, in two places in its text. -The first place is section 24, where full trial durations are required, +The first place is Section 24, where full trial durations are required, presumably because they can be inconsistent with the results from short trial durations. -The second place is section 26.3, where two successive zero-loss trials +The second place is Section 26.3, where two successive zero-loss trials are recommended, presumably because after one zero-loss trial there can be a subsequent inconsistent non-zero-loss trial. -Examples include: - -- A trial at the same load (same or different trial duration) results - in a different Trial Loss Ratio. -- A trial at a higher load (same or different trial duration) results - in a smaller Trial Loss Ratio. - Any robust throughput search algorithm needs to decide how to continue the search in the presence of such inconsistencies. Definitions of throughput in [RFC1242] and [RFC2544] are not specific enough @@ -401,25 +422,19 @@ MLRsearch specification describes all technical definitions needed for evaluating whether a particular test procedure complies with MLRsearch specification. -{::comment} - [Good idea for 08, maybe ask BMWG first?] - - <mark>TODO VP: Separate Requirements and Recommendations/Suggestions - paragraphs? (currently requirements are in discussion subsections - - discussion should only clarify things without adding new - requirements)</mark> -{:/comment} - Some terms used in the specification are capitalized. It is just a stylistic choice for this document, reminding the reader this term is introduced, defined or explained -elsewhere in the document. +elsewhere in the document. See [Index ](#index) for list of such terms. Lowercase variants are equally valid. Each per term subsection contains a short **Definition** paragraph -containing a minimal definition and all strict REQUIREMENTS, followed -by **Discussion** paragraphs containing some important consequences and -RECOMMENDATIONS. +containing a minimal definition and all strict requirements, +followed by **Discussion** paragraphs focusing on +important consequences and recommendations. +Requirements on the way other components can use the defined quantity +are also present in the discussion paragraphs. + Other text in this section discusses document structure and non-authoritative summaries. @@ -455,22 +470,32 @@ they may have different requirements and stopping conditions. Search results are based on Load Classification. When measured enough, any chosen Load can either achieve or fail each Search Goal (separately), thus becoming -a Lower Bound or an Upper Bound for that Search Goal. +a Lower Bound or an Upper Bound for that Search Goal, respectively. + +For repeatability and comparability reasons, it is important that +all implementations of MLRsearch classify the Load equivalently, +based on all Trials measured at that Load. When the Relevant Lower Bound is close enough to Relevant Upper Bound according to Goal Width, the Regular Goal Result is found. Search stops when all Regular Goal Results are found, or when some Search Goals are proven to have only Irregular Goal Results. -{::comment} +### Behavior Correctness - TODO-P1: An implementation may add additional attributes to inputs and outputs. +MLRsearch Specification by itself does not guarantee +the Search ends in finite time, as the freedom the Controller has +for Load selection also allows for clearly deficient choices. - TODO-P1: An implementation may require some attributes not required by specification. +Although the authors believe that any MLRsearch Implementation +that aims to shorten the Search Duration (with fixed Controller Input) +will necessarily also become good at repeatability and comparability, +any attempts to prove such claims are outside of the scope of this document. - TODO-P1: An implementation may support "missing" attributes by applying "reasonable defaults". +For deeper insights, see [FDio-CSIT-MLRsearch]. -{:/comment} +The primary MLRsearch Implementation, used as the prototype +for this specification, is [PyPI-MLRsearch]. ## Quantities @@ -490,19 +515,36 @@ Some attributes are not independent from others, and they can be calculated from other attributes. Such quantites are called derived quantities. -## Existing Terms +### Current and Final Values -{::comment} +Some quantites are defined in a way that allows computing their values +in the middle of the Search. Other quantities are specified in a way +that allows their values to be computed only after the Search ends. +And some quantities are important only after the Search ended, +but their values are computable also before the Search ends. - TODO-P1: Merge into Glossary! MK - IMV this section should stay here as is. +For a quantity that is computable before the Search ends, +the adjective **current** is used to mark a value of that quantity +available before the Search ends. +When such value is relevant for the search result, the adjective **final** +is used to denote the value of that quantity at the end of the Search. -{:/comment} +If a time evolution of such a dynamic quantity is guided +by configuration quantities, those adjectives can be used +to distinguish quantities. +For example if the current value of "duration" (dynamic quantity) increases +from "initial duration" to "final duration" (configuration quantities), +all the quoted names denote separate but related quantites. +As the naming suggests, the final value od "duration" is expected +to be equal to "final duration" value. + +## Existing Terms This specification relies on the following three documents that should be consulted before attempting to make use of this document: - RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" - contains basic term definitions. + contains basic term definitions. - RFC 2285 "Benchmarking Terminology for LAN Switching Devices" adds more terms and discussions, describing some known network @@ -515,65 +557,6 @@ be consulted before attempting to make use of this document: Definitions of some central terms from above documents are copied and discussed in the following subsections. -{::comment} - [Good idea for 08, but needs more work. Ask BMWG?] - - Alternatively, quick list of all (existing and new here) terms, - with links (external or internal respectively) to definitions. - - <mark>MKP3 [VP] TODO: Even if the following list will not be in final draft, - it is useful to keep it around (maybe commented-out) while editing.</mark> - - <mark>MKP3 VP note: rough list of all RFC references: - - [RFC1242] (section 3.17 Throughput) ... definition - - [RFC2544] (section 26.1 Throughput) ... methodology - - [RFC2544] (section 24. Trial duration): - - full trial durations (implies short trials) - - Also 60s for unconditional compliance is here. - - Also "the search" (without quotes) appears there. - - Also "binary search" (with quotes) appears there. - - [RFC2544] (section 26.3 Frame loss rate): - - two successive zero-loss trials are recommended (hints about loss inversion) - - un/conditionally compliant with [RFC2544] - - [RFC2544] (section 26. Benchmarking tests:) - - all its "dot sections" have "Reporting format:" paragraphs - - (implies test report) - - [RFC2544] (section 26.1 Throughput) wants graph, frame size on X axis. - - [RFC2544] (section 23. Trial description) trial - - general description of trial - - wait times specifically, maybe also learning frames? - - Data Rate of [RFC2544] (section 14. Bidirectional traffic) - - seems equal to input frame rate [RFC2544] (23. Trial description). - - [RFC2544] (section 21. Bursty traffic) suggests non-constant loads? - - Intended Load of [RFC2285] (section 3.5.1 Intended load (Iload)) - - [RFC2285] (Section 3.5.2 Offered load (Oload)) - - Forwarding Rate as defined in [RFC2285] (section 3.6.1 Forwarding rate (FR)) - - [RFC2285] (3.5.3 Maximum offered load (MOL)) - - reordered frames [RFC2544] (section 10. Verifying received frames) - - For example, [RFC2544] (Appendix C) lists frame formats and protocol addresses, - as recommended from [RFC2544] (section 8. Frame formats) - and [RFC2544] (section 12. Protocol addresses). - - [RFC8219] (section 5.3. Traffic Setup) introduces traffic setups consisting of a mix of IPv4 and IPv6 traffic - - [RFC2544] (section 9. Frame sizes) - - [RFC1242] (section 3.5 Data link frame size) - - [RFC2285] (section 3.6.2) FRMOL - - [RFC2285] (section 3.1.1) DUT - - [RFC2285] (section 3.1.2) SUT - - [RFC2544] (section 6. Test set up) test setup with (an external) tester - - [RFC9004] B2B - - [RFC8219] (section 5.3. Traffic Setup) for an example of ip4+ip6 mixed traffic - </mark> - -{:/comment} - -{::comment} - [Important, just not enough time in 07.] - - <mark>MKP3 [VP] TODO: Verify that MLRsearch specification does not discuss - meaning of existing terms without quoting their original definition.</mark> - -{:/comment} - ### SUT Defined in [RFC2285] (Section 3.1.2) as follows. @@ -630,11 +613,12 @@ Definition: Discussion: +The traffic is sent only in phase c) and received in phases c) and d). + The definition describes some traits, and it is not clear whether all of them -are REQUIRED, or some of them are only RECOMMENDED. +are required, or some of them are only recommended. -Trials are the only stimuli the SUT is expected to experience -during the Search. +Trials are the only stimuli the SUT is expected to experience during the Search. For the purposes of the MLRsearch specification, it is ALLOWED for the test procedure to deviate from the [RFC2544] description, @@ -644,31 +628,31 @@ In some discussion paragraphs, it is useful to consider the traffic as sent and received by a tester, as implicitly defined in [RFC2544] (Section 6). -{::comment} - - TODO-P2: Assert traffic is sent only in phase c) and received in phases c) and d). - -{:/comment} - An example of deviation from [RFC2544] is using shorter wait times, -compared to those described in phases b), d) and e). +compared to those described in phases a), b), d) and e). + +The [RFC2544] document itself seems to be treating phase b) +as any type of configuration that cannot be configured only once (by Manager, +before Search starts), as some crucial SUT state could time-out during the Search. +This document RECOMMENDS to understand "learning frames" to be +any such time-sensitive per-trial configuration method, +with bridge MAC learning being only one possibe example. +[RFC2544] (Section C.2.4.1) lists another example: ARP with wait time 5 seconds. ## Trial Terms This section defines new and redefine existing terms for quantities relevant as inputs or outputs of a Trial, as used by the Measurer component. +This includes also any derived quantities related to one trial result. ### Trial Duration Definition: -Trial Duration is the intended duration of the traffic part of a Trial. +Trial Duration is the intended duration of the phase c) of a Trial. Discussion: -This quantity does not include any preparation nor waiting -described in section 23 of [RFC2544] (Section 23). - While any positive real value may be provided, some Measurer implementations MAY limit possible values, e.g. by rounding down to nearest integer in seconds. In that case, it is RECOMMENDED to give such inputs to the Controller @@ -698,11 +682,6 @@ and Intended Load of [RFC2285] (Section 3.5.1), in the sense that all three definitions specify that this value applies to one (input or output) interface. -For test report purposes, multi-interface aggregate load MAY be reported, -and is understood as the same quantity expressed using different units. -From the report it MUST be clear whether a particular Trial Load value -is per one interface, or an aggregate over all interfaces. - Similarly to Trial Duration, some Measurers may limit the possible values of trial load. Contrary to trial duration, the test report is NOT REQUIRED to document such behavior, as in practice the load differences @@ -715,43 +694,16 @@ If a particular Trial Load value is not tied to a single Trial, e.g. if there are no Trials yet or if there are multiple Trials, this document uses a shorthand **Load**. -{::comment} - [I feel this is important, to be discussed separately (not in-scope).] - - <mark>MKP2 [VP] TODO: Explain why are we not using Oload. - 1. MLRsearch implementations cannot react correctly to big differences - between Iload and Oload. - 2. The media between the tested and the DUT are thus considered to be part of SUT. - If DUT causes congestion control, it is not expected to handle Iload. - </mark> - - See further discussion in [Trial Forwarding Ratio](#trial-forwarding-ratio) - and in [Measurer ](#measurer) sections for other related issues. - - <mark>MKP2 [VP] TODO: Create a separate subsection for Oload discussion, - or clearly separate which aspects are discussed under which term.</mark> - - <mark>MKP2 [VP] TODO: New idea. Compare the tester to an ordinary router - in some datacenter. The Intended Load is not jst some abstract input. - It is the real traffic coming from routers next hop farther. - It does not matter that DUT has forwarded each frame it received, - if the tester was unable to sent all the traffic in time. - Endpoint see packet loss, they do not care about [RFC2285] - half-duplex, spanning trees, nor congestion control mechanisms. - Formally speaking, I consider even the sending interface of the sender - to be the part of SUT. - Reading [RFC2285] (section 3.5.3 Maximum offered load (MOL)) - "This will be the case when an external source lacks the resources - to transmit frames at the minimum legal inter-frame gap" - that means TRex workers are also part of SUT. If they do not have - enough CPU power to generate frames are required, those frames are lost. - </mark> - - <mark>MKP2 [VP] TODO: That new idea warants some discussion in "DUT within SUT", - as it is just another case of ther rest of SUT ruining - otherwise good DUT performance.</mark> +For test report purposes, multi-interface aggregate load MAY be reported, +and is understood as the same quantity expressed using different units. +From the report it MUST be clear whether a particular Trial Load value +is per one interface, or an aggregate over all interfaces. +This implies there is a known and constant coefficient between +single-interface and multi-interface load values. +The single-interface value is still the primary one, +as most other documents deal with single-interface quantites only. -{:/comment} +The last paragraph also applies to other terms related to Load. ### Trial Input @@ -767,24 +719,31 @@ to denote all corresponding Trial Input instances. A Trial Input instance acts as the input for one call of the Measurer component. -Contrary to other composite quantities, MLRsearch implementations +Contrary to other composite quantities, MLRsearch Implementations are NOT ALLOWED to add optional attributes here. This improves interoperability between various implementations of the Controller and the Measurer. +Please note that both attributes are **intended** quantities, +as only those can be fully controlled by the Controller. +The actual offered quantities, as realized by the Measurer, can be different +(and must be different if not multiplying into integer number of frames), +but questions around those offered quantities are generally +outside of the scope of this document. + ### Traffic Profile Definition: Traffic Profile is a composite quantity containing all attributes other than Trial Load and Trial Duration, -that are needed for unique determination of the trial to be performed. +that are needed for unique determination of the Trial to be performed. Discussion: All the attributes are assumed to be constant during the search, and the composite is configured on the Measurer by the Manager -before the search starts. +before the Search starts. This is why the traffic profile is not part of the Trial Input. As a consequence, implementations of the Manager and the Measurer @@ -816,7 +775,7 @@ Profile, if they apply to the test scenario, include: - fully meshed traffic from [RFC2285] (Section 3.3.3), -- and modifiers from [RFC2544] (Section 11). +- modifiers from [RFC2544] (Section 11). ### Trial Forwarding Ratio @@ -831,53 +790,20 @@ by the total number of frames expected to be forwarded during the trial. Discussion: For most Traffic Profiles, "expected to be forwarded" means -"intended to get transmitted from Tester towards SUT". +"intended to get transmitted from tester towards SUT". Only if this is not the case, the test report MUST describe the Traffic Profile in a way that implies how Trial Forwarding Ratio should be calculated. Trial Forwarding Ratio MAY be expressed in other units (e.g. as a percentage) in the test report. -Note that, contrary to loads, frame counts used to compute -trial forwarding ratio are aggregates over all SUT output interfaces. - -Questions around what is the correct number of frames -that should have been forwarded -is generally outside of the scope of this document. - -{::comment} - - TODO-P0: Mention iload/oload difference is also out of scope. - - TODO-P2: Mention duplicate, previous-trial and other "more than - expected" frame counts are out of scope. Recommend to count them as - loss? MK there should be a reference about the last TODO in 1242 2285 - or 2544. - -{:/comment} - -{::comment} - [Part two of iload/oload discussion.] - - See discussion in [Measurer ](#measurer) section - for more details about calibrating test equipment. - - <mark>MKP2 [VP] TODO: Define unsent frames?</mark> - - <mark>MKP2 [VP] TODO: If Oload is fairly below Iload, the unsent frames - should be counted as lost, otherwise search outputs are misleading. - But what is "fairly"? CSIT tolerates 10 microseconds worth of unsent frames.</mark> - -{:/comment} - -{::comment} - [Low priority, but maybe useful for somebody?] +Note that, contrary to Load terms, frame counts used to compute +Trial Forwarding Ratio are generally aggregates over all SUT output interfaces, +as most test procedures verify all outgoung frames. - <mark>MKP2 [VP] TODO: Mention traffic profiles with uneven frame counts? - E.g. when SUT is expected to perform IP packet fragmentation or reassembly. - </mark> - -{:/comment} +For example, in a test with symmetric bidirectional traffic, +if one direction is forwarded without losses, but the opposite direction +does not forward at all, the trial forwarding ratio would be 0.5 (50%). ### Trial Loss Ratio @@ -890,7 +816,7 @@ Discussion: 100% minus the Trial Forwarding Ratio, when expressed as a percentage. This is almost identical to Frame Loss Rate of [RFC1242] (Section 3.6). -Te only minor differences are that Trial Loss Ratio +The only minor differences are that Trial Loss Ratio does not need to be expressed as a percentage, and Trial Loss Ratio is explicitly based on aggregate frame counts. @@ -905,31 +831,18 @@ Discussion: It is important to note that while similar, this quantity is not identical to the Forwarding Rate as defined in [RFC2285] (Section 3.6.1). -The latter is specific to one output interface only, -whereas the Trial Forwarding Ratio is based -on frame counts aggregated over all SUT output interfaces. - -In consequence, for symmetric traffic profiles the Trial Forwarding Rate value -is equal to arithmetric average of [RFC2285] Forwarding Rate values -across all active interfaces. +The latter is based on frame counts on one output interface only, +so each output interface can have different forwarding rate, +whereas the Trial Forwarding Rate is based on frame counts +aggregated over all SUT output interfaces, while stil being a multiple of Load. -{::comment} - [Part 3 of iload/oload discussion.] - - <mark>MKP2 [VP] TODO: If some unsent frames were tolerated (not counted as lost), - this value is actually higher than the real fps output of the SUT. - Should we use the real FR as the basis for Conditional Throughput - (instead of this TFR)? That would require additional Trial Output attribute. - </mark> - - <mark>MKP2 [VP] TODO: What about duration stretching? - This also causes difference between Iload and Oload, - but in an invisible way.</mark> - - <mark>MKP2 [VP] TODO: Recommend start+sleep+stop? - How long wait for late frames? RFC2544 2s is too much even at 30s trial.</mark> +Consequently, for symmetric bidirectional Traffic Profiles, +the Trial Forwarding Rate value is equal to arithmetic average +of [RFC2285] Forwarding Rate values across both output interfaces. -{:/comment} +Given that Trial Forwarding Rate is a quantity based on Load, +it is ALLOWED to express this quantity using multi-interface values +in test report, e.g. as sum of per-interface forwarding rate values. ### Trial Effective Duration @@ -960,17 +873,6 @@ and use that as the Trial Effective Duration. This is also a way for the Measurer to inform the Controller about its surprising behavior, for example when rounding the Trial Duration value. -{::comment} - [Not very important, but easy and nice recommendation.] - - <mark>MKP2 [VP] TODO: Recommend for Controller to return all trials at relevant bounds, - as that may better inform users when surprisingly small amount of trials - was performed, just because the the trial effective duration values were big.</mark> - - <mark>MKP2 [VP] TODO: Repeat that this is not here to deal with duration stretching.</mark> - -{:/comment} - ### Trial Output Definition: @@ -998,31 +900,6 @@ to be reported for forwarding rate measurements, it is NOT REQUIRED in MLRsearch Specification, as search results do not depend on it. -{::comment} - - TODO-P1: MK note - i know that Offered Load can be calculated from Trial - Loss Ratio and Trial Forwarding Rate but still most/all network users - would expect to know what Trial Load was used. Also, saying that search - results do not depend on Offered Load or Trial Load is not true :) - VP note - I partially disagree and partially do not understand. - -{:/comment} - -{::comment} - - [Side tangent from iload/oload discussion. Stilll recommendation is not obvious.] - - <mark>MKP2 mk edit note: we need to more explicitly address - the relevance or irrelevance of [RFC2285] (Section 3.5.2 Offered load (Oload)). - Current text in [Trial Load](#trial-load) is ambiguous - quoted below.</mark> - - <mark>MKP2 "Questions around what is the correct number of frames that should - have been forwarded is generally outside of the scope of this document. - See discussion in [Measurer ](#measurer) section for more details about - calibrating test equipment."</mark> - -{:/comment} - ### Trial Result Definition: @@ -1032,7 +909,7 @@ consisting of the Trial Input and the Trial Output. Discussion: -When talking about multiple trials, it is common to say "trial results" +When talking about multiple trials, it is common to say "Trial Results" to denote all corresponding Trial Result instances. While implementations SHOULD NOT include additional attributes @@ -1041,121 +918,106 @@ with independent values, they MAY include derived quantities. ## Goal Terms This section defines new terms for quantities relevant (directly or indirectly) -for inputs or outputs of the Controller component. +for inputs and outputs of the Controller component. Several goal attributes are defined before introducing the main composite quantity: the Search Goal. -{::comment} - - TODO-P0: Mention definitions are not informative? - E.g. Goal Final Trial Duration and Goal Initial Trial Duration - have the same Definition text. - Note that these are already fixed for now, but other attributes need review. - -{:/comment} +Contrary to other sections, definitions in subsections of this section +are necessarily vague, as their fundamental meaning is to act as +coefficients in formulas for Controller Output, which are not defined yet. -Discussions within this section are short, informal, -and referencing future sections, with the impact on search results -discussed only after introducing complete set of auxiliary terms. +The discussions here relate the attributes to concepts mentioned in chapter +[Identified Problems](#identified-problems), but even these discussion +paragraphs are short, informal, and mostly referencing later sections, +where the impact on search results is discussed after introducing +the complete set of auxiliary terms. ### Goal Final Trial Duration -{::comment} - - TODO-P0: review updated definition, check if any informal explanation is needed. - -{:/comment} - Definition: -Minimum value for Trial Duration required for classifying the Load -as a Lower Bound. +Minimal value for Trial Duration that has to be reached. +The value MUST be positive. Discussion: -This attribute value MUST be positive. - -Informally, while MLRsearch is allowed to perform trials shorter than this value, -the results from such short trials have only limited impact on search results. +Some Trials have to be at least this long +to allow a Load to be classified as a Lower Bound. +The Controller is allowed to choose shorter durations, +results of those may be enough for classification as an Upper Bound. It is RECOMMENDED for all search goals to share the same -Goal Final Trial Duration value. -Otherwise, Trial Duration values larger than the Goal Final Trial Duration -may occur, weakening the assumptions +Goal Final Trial Duration value. Otherwise, Trial Duration values larger than +the Goal Final Trial Duration may occur, weakening the assumptions the [Load Classification Logic](#load-classification-logic) is based on. -{::comment} - - TODO-P2: Currently not covered well in Logic chapter? - - TODO-P2: Maybe change fourth goal there to show this? - -{:/comment} - ### Goal Duration Sum Definition: A threshold value for a particular sum of Trial Effective Duration values. +The value MUST be positive. Discussion: -This attribute value MUST be positive. - -Informally, this prescribes the maximum amount of trials performed +Informally, this prescribes the sufficient amount of trials performed at a specific Trial Load and Goal Final Trial Duration during the search. If the Goal Duration Sum is larger than the Goal Final Trial Duration, -multiple trials may need to be performed at the same load. +multiple trials may be needed to be performed at the same load. -See [MLRsearch Compliant with TST009](#mlrsearch-compliant-with-tst009) -for an example where possibility of multiple trials at the same load is intended. +See section [MLRsearch Compliant with TST009](#mlrsearch-compliant-with-tst009) +of this document for an example where the possibility of multiple trials +at the same load is intended. -A Goal Duration Sum value lower than the Goal Final Trial Duration -(of the same goal) could save some search time, but is NOT RECOMMENDED. +A Goal Duration Sum value shorter than the Goal Final Trial Duration +(of the same goal) could save some search time, but is NOT RECOMMENDED, +as the time savings come at the cost of decreased repeatability. -{::comment} - - TODO-P2: Currently not covered in the classification logic chapter. - -{:/comment} +In practice, the Search can spend less than Goal Duration Sum measuring +a Load value when the results are particularly one-sided, +but also the Search can spend more than Goal Duration Sum measuring a Load +when the results are balanced and include +trials shorter than Goal Final Trial Duration. ### Goal Loss Ratio Definition: A threshold value for Trial Loss Ratio values. +The value MUST be non-negative and smaller than one. Discussion: -Attribute value MUST be non-negative and smaller than one. - A trial with Trial Loss Ratio larger than this value signals the SUT may be unable to process this Trial Load well enough. See [Throughput with Non-Zero Loss](#throughput-with-non-zero-loss) -why users may want to set this value above zero. +for reasons why users may want to set this value above zero. + +Since multiple trials may be needed for one Load value, +the Load Classification is generally more complicated than mere comparison +of Trial Loss Ratio to Goal Loss Ratio. ### Goal Exceed Ratio Definition: -A threshold value for a particular ratio of sums of Trial Effective Duration -values. +A threshold value for a particular ratio of sums +of Trial Effective Duration values. +The value MUST be non-negative and smaller than one. Discussion: -Attribute value MUST be non-negative and smaller than one. - -Informally, up to this proportion of High-Loss Trials -(Trial Results with Trial Loss Ratio above Goal Loss Ratio) -is tolerated at a Lower Bound. +Informally, up to this proportion of Trial Results +with Trial Loss Ratio above Goal Loss Ratio is tolerated at a Lower Bound. +This is the full impact if every Trial was measured at Goal Final Trial Duration. +The actual full logic is more complicated, as shorter Trials are allowed. For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5 (50%), -as it simplifies some concepts by relating them to the concept of median. -Also, the value of 50% leads to smallest variation in overall Search Duration -in practice. +as in practice that value leads to +the smallest variation in overall Search Duration. See [Exceed Ratio and Multiple Trials](#exceed-ratio-and-multiple-trials) section for more details. @@ -1165,13 +1027,12 @@ section for more details. Definition: A threshold value for deciding whether two Trial Load values are close enough. +This is an OPTIONAL attribute. If present, the value MUST be positive. Discussion: -It is an optional attribute. If present, the value MUST be positive. - Informally, this acts as a stopping condition, -controlling the precision of the search. +controlling the precision of the search result. The search stops if every goal has reached its precision. Implementations without this attribute @@ -1182,30 +1043,17 @@ but implementations may choose a different way to specify width. The test report MUST make it clear what specific quantity is used as Goal Width. -{::comment} - - TODO-P2: Comment: While not needed for precision purposes - larger-than-width result (e.g. when time is up) is still an Irregular result, - so this is the way to make sure it looks irregular in report. - -{:/comment} - It is RECOMMENDED to set the Goal Width (as relative difference) value -to a value no smaller than the Goal Loss Ratio. +to a value no lower than the Goal Loss Ratio. If the reason is not obvious, see the details in [Generalized Throughput](#generalized-throughput). ### Goal Initial Trial Duration -{::comment} - - TODO-P0: review updated definition, check if any informal explanation is needed. - -{:/comment} - Definition: -Minimum value for Trial Duration required for classifying the Load as any Bound. +Minimal value for Trial Duration suggested to use for this goal. +If present, this value MUST be positive. Discussion: @@ -1213,15 +1061,16 @@ This is an example of an OPTIONAL Search Goal some implementations may support. The reasonable default value is equal to the Goal Final Trial Duration value. -If present, this value MUST be positive. - -Informally, this is the smallest Trial Duration the Controller will select +Informally, this is the shortest Trial Duration the Controller should select when focusing on the goal. -Strictly speaking, Trial Results with smaller Trial Duration values -are still accepted by the Load Classification logic. -This is just a way for the user to discourage trials with Trial Duration -values deemed as too unreliable for this SUT and this Search Goal. +Note that shorter Trial Duration values can still be used, +for example selected while focusing on a different Search Goal. +Such results MUST be still accepted by the Load Classification logic. + +Goal Initial Trial Duration is just a way for the user to discourage +trials with Trial Duration values deemed as too unreliable +for particular SUT and this Search Goal. ### Search Goal @@ -1246,14 +1095,10 @@ Implementations MAY add their own attributes. Those additional attributes may be required by the implementation even if they are not required by MLRsearch specification. But it is RECOMMENDED for those implementations -to support missing values by providing reasonable default values. +to support missing attributes by providing reasonable default values. -{::comment} - - TODO2: MK last sentence doesn't make sense. - VP: Added TODOs to Overview section. - -{:/comment} +For example, implementations with Goal Initial Trial Durations +may also require users to specify "how quickly" should Trial Durations increase. See [Compliance ](#compliance) for important Search Goal instances. @@ -1267,7 +1112,7 @@ The only REQUIRED attribute is a list of Search Goal instances. Discussion: -MLRsearch implementations MAY use additional attributes. +MLRsearch Implementations MAY use additional attributes. Those additional attributes may be required by the implementation even if they are not required by MLRsearch specification. @@ -1277,32 +1122,11 @@ apart from one Controller Input instance. For example, Traffic Profile is configured on the Measurer by the Manager, without explicit assistance of the Controller. -{::comment} - - TODO-P0: This paragraph is for implementers. - - TODO2: MK implementation hints are fine, and do not have to be preceded -with any remark of the sort you're suggesting IMV. - -{:/comment} - The order of Search Goal instances in a list SHOULD NOT have a big impact on Controller Output, -but MLRsearch implementations MAY base their behavior on the order +but MLRsearch Implementations MAY base their behavior on the order of Search Goal instances in a list. -{::comment} - [User recommendation, we should have separate section summarizing those.] - - Also, it is recommended to avoid "incomparable" goals, e.g. one with - lower loss ratio but higher exceed ratio, and other with higher loss ratio - but lower loss ratio. In worst case, this can make the search to last too long. - Implementations are RECOMMENDED to sort the goals and start with - stricter ones first, as bounds for those will not get invalidated - byt measureing for less trict goal later in the search. - -{:/comment} - #### Max Load Definition: @@ -1316,7 +1140,7 @@ Max Load is an example of an optional attribute (outside the list of Search Goal required by some implementations of MLRsearch. In theory, each search goal could have its own Max Load value, -but as all trials are possibly affecting all Search Goals, +but as all Trial Results are possibly affecting all Search Goals, it makes more sense for a single Max Load value to apply to all Search Goal instances. @@ -1326,43 +1150,13 @@ and (as maximum offered load) by [RFC2285] (Section 3.5.3), some implementations may detect or discover it (instead of requiring a user-supplied value). -{::comment} - - TODO-P0: Move this (and goal width) to RUB discussion or other explanation instead. - - TODO2: MK i think it belongs here, as input parameter. may refer to - section "Hard Performance Limit" though. - -{:/comment} - In MLRsearch specification, one reason for listing the [Relevant Upper Bound](#relevant-upper-bound) as a required attribute is that it makes the search result independent of Max Load value. -{::comment} - - TODO2: MK RUB is not an attribute, it's Result Term. Hence above - sentence does not make sense and should be removed. - VP: RUB is an attribure of Goal Result composite quantity. - -{:/comment} - -{::comment} - [Not important directly, may matter for iload/oload.] - - <mark>MKP2 [VP] TODO: 2544 and 2285 care about half-duplex media. Should we?</mark> - -{:/comment} - -{::comment} - [Maybe obvious but I think useful. RFC2544 talks about header compression in WANs.] - - <mark>MKP2 [VP] TODO: Mention that Max Load should care about all media within SUT, - including DUT-DUT links. Important when that link carries encapsulated traffic, - as bandwidth limit there implies lower max rate - (than implied by tester-SUT links).</mark> - -{:/comment} +Given that Max Load is a quantity based on Load, +it is ALLOWED to express this quantity using multi-interface values +in test report, e.g. as sum of per-interface maximal loads. #### Min Load @@ -1382,19 +1176,15 @@ Min Load is mainly useful for saving time by failing early, arriving at an Irregular Goal Result when Min Load gets classified as an Upper Bound. -For implementations, it is useful to require Min Load to be non-zero +For implementations, it is RECOMMENDED to require Min Load to be non-zero and large enough to result in at least one frame being forwarded -even at smallest allowed Trial Duration, -so Trial Loss Ratio is always well-defined, -and the implementation can use relative Goal Width -(without running into issues around zero Trial Load value). - -{::comment} - - TODO2: MK last 3 lines need to be reworded, as they don't make sense, - and i can't suggest alternative wording. +even at shortest allowed Trial Duration, +so that Trial Loss Ratio is always well-defined, +and the implementation can apply relative Goal Width safely. -{:/comment} +Given that Min Load is a quantity based on Load, +it is ALLOWED to express this quantity using multi-interface values +in test report, e.g. as sum of per-interface minimal loads. ## Auxiliary Terms @@ -1402,58 +1192,14 @@ While the terms defined in this section are not strictly needed when formulating MLRsearch requirements, they simplify the language used in discussion paragraphs and explanation chapters. -### Current and Final Quantities - -{::comment} - - TODO2: MK doesn't this content belong to "Quantities" section at the - beginning of the doc? - VP: Probably yes, should be moved. - -{:/comment} - -Some quantites are defined in a way that allows them to be computed -in the middle of the Search. Other quantities are specified in a way -that allows them to be computed only after the Search ends. -And some quantities are important only after the Search ended, -but are computable also before the Search ends. - -The adjective **current** marks a quantity that is computable -before the Search ends, but the computed value may change during the Search. -When such value is relevant for the search result, the adjective **final** -may be used to denote the value at the end of the Search. - -{::comment} - - TODO2: MK **current** and **final** adjectives seem to relate to values - of quantities, and not quantities themselves, or? - -{:/comment} - ### Trial Classification -{::comment} - - TODO2: MK do we need this explanation below. Can't we just leave this - section header and then list trial types as is? - -{:/comment} - When one Trial Result instance is compared to one Search Goal instance, several relations can be named using short adjectives. As trial results do not affect each other, this **Trial Classification** does not change during the Search. -{::comment} - - TODO-P0: Is it obvious the adjectives can be combined? - - TODO2: MK **current** and **final** adjectives seem to relate to values - of quantities, and not quantities themselves, or? - -{:/comment} - #### High-Loss Trial A trial with Trial Loss Ratio larger than a Goal Loss Ratio value @@ -1463,7 +1209,7 @@ is called a **high-loss trial**, with respect to given Search Goal #### Low-Loss Trial If a trial is not high-loss, it is called a **low-loss trial** -(or even zero-loss trial, if Goal Loss Ratio is zero). +(or zero-loss trial, if Goal Loss Ratio is zero). #### Short Trial @@ -1481,36 +1227,21 @@ Note that this includes Trial Durations larger than Goal Final Trial Duration. A trial with Trial Duration longer than the Goal Final Trial Duration is called a **long trial**. -{::comment} - - TODO-P0: If used in Logic chapter, add to Glossary and maybe move before full-length. - - TODO-P2: Maybe change fourth goal there to show this better? - - TODO-P0: If not used, delete. - -{:/comment} - ### Load Classification -{::comment} - - TODO-P0: Turn into a precise definition paragraph. - -{:/comment} - -When the set of all Trial Result instances performed so far -at one Trial Load is compared to one Search Goal instance, -two relations can be named using the concept of a bound. +When a set of all Trial Result instances, performed so far +at one Trial Load, is compared to one Search Goal instance, +their relation can be named using the concept of a bound. In general, such bounds are a current quantity, -even though cases of changing bounds is rare in practice. +even though cases of a Load changing its classification more than once +during the Search is rare in practice. #### Upper Bound Definition: -A Trial Load value is called an Upper Bound if and only if it is classified +A Load value is called an Upper Bound if and only if it is classified as such by [Appendix A: Load Classification](#appendix-a-load-classification) algorithm for the given Search Goal at the current moment of the Search. @@ -1523,78 +1254,27 @@ mainly the Goal Loss Ratio in combination with the Goal Exceed Ratio. Here "certain to fail" relates to any possible results within the time remaining till Goal Duration Sum. -{::comment} - - TODO2: MK not sure above paragraph adds any explanation value whatsover. - It verges into the domain of discussing all possible outcomes and does - nothing to clarify what upper bound is about. And as there is no clear - explanation of upper bound i added one above. - -{:/comment} - One search goal can have multiple different Trial Load values classified as its Upper Bounds. -As search progresses and more trials are measured, -any load value can become an Upper Bound. +While search progresses and more trials are measured, +any load value can become an Upper Bound in principle. -Also, a load can stop being an Upper Bound, but that +Moreover, a load can stop being an Upper Bound, but that can only happen when more than Goal Duration Sum of trials are measured (e.g. because another Search Goal needs more trials at this load). -In that case the load becomes a Lower Bound (see next subsection), +In practice, the load becomes a Lower Bound (see next subsection), and we say the previous Upper Bound got Invalidated. -{::comment} - [Medium priority, depends on how many user recommendations we have.] - - With non-zero exceed ratio values, a short high-loss trial may not be enough - to classify a load as the relevant upper bound. - Users MAY apply Goal Duration Sum value lower than Goal Final Trial Duration - to force such classification in hope to save time, - but it is RECOMMENDED not to do so, as in practice - it hurts comparability and repeatability. - -{:/comment} - -{::comment} - [Probably too technical, unless relation to repeatability is found.] - - In general, a load starts as as undecided, then maybe flips to become - an upper bound. MLRsearch stops measuring at that load for this goal, - but it may be forced to measure more for some other search goals, - in which case the load may flip to a lower bound (and back and forth). - - <mark>[VP] TODO: Confirm the load can never flip back to being undecided.</mark> - - Even though the load classification may change during the search, - the goal results are established at the end of the search. - - If the exceed ratio is zero, an upper bound can never flip; - one high-loss trial (even short) is enough to pin the classification. - -{:/comment} - #### Lower Bound Definition: -A Trial Load value is called a Lower Bound if and only if it is classified +A Load value is called a Lower Bound if and only if it is classified as such by [Appendix A: Load Classification](#appendix-a-load-classification) algorithm for the given Search Goal at the current moment of the search. Discussion: -{::comment} - - MK: - It is the minimum value in a range being searched, together with Upper - Bound, defining the interval within which MLRsearch operates for - specific Search Goal, iteratively narrowing down to arrive to Search - Result. - - VP: That is wrong in situations with Loss Inversions. - -{:/comment} - In more detail, the set of all Trial Results performed so far at the Trial Load (and any Trial Duration) is certain to uphold all the requirements of the given Search Goal, @@ -1602,31 +1282,37 @@ mainly the Goal Loss Ratio in combination with the Goal Exceed Ratio. Here "certain to uphold" relates to any possible results within the time remaining till Goal Duration Sum. -{::comment} - - TODO2: MK similar to previous section - not sure above paragraph adds - any explanation value whatsover. It verges into the domain of - discussing all possible outcomes and does nothing to clarify what upper - bound is about. And as there is no clear explanation of upper bound i - added one above. - -{:/comment} - One search goal can have multiple different Trial Load values classified as its Lower Bounds. As search progresses and more trials are measured, -any load value can become a Lower Bound. +any load value can become a Lower Bound in principle. No load can be both an Upper Bound and a Lower Bound for the same Search goal -at the same time, but it is possible for a higher load to be a Lower Bound +at the same time, but it is possible for a larger load to be a Lower Bound while a smaller load is an Upper Bound. -Also, a load can stop being a Lower Bound, but that +Moreover, a load can stop being a Lower Bound, but that can only happen when more than Goal Duration Sum of trials are measured (e.g. because another Search Goal needs more trials at this load). -In that case the load becomes an Upper Bound, +In that case, the load becomes an Upper Bound, and we say the previous Lower Bound got Invalidated. +#### Undecided + +Definition: + +A Load value is called Undecided if it is currently +neither an Upper Bound nor a Lower Bound. + +Discussion: + +A Load value that has not been measured so far is Undecided. + +It is possible for a Load to transition from an Upper Bound to Undecided +by adding Short Trials with Low-Loss results. +That is yet another reason for users to avoid using Search Goal instances +with diferent Goal Final Trial Duration values. + ## Result Terms Before defining the full structure of Controller Output, @@ -1645,45 +1331,45 @@ has its corresponding Goal Result instance. Definition: The Relevant Upper Bound is the smallest Trial Load value -classified as an Upper Bound for the given Search Goal at the end of the search. +classified as an Upper Bound for a given Search Goal at the end of the Search. Discussion: -If no measured load had enough high-loss trials, -the Relevant Upper Bound MAY be not-existent. +If no measured load had enough High-Loss Trials, +the Relevant Upper Bound MAY be non-existent. For example, when Max Load is classified as a Lower Bound. -{::comment} - - TODO-P0: Delete or move: - - TODO2: MK duplicate content explaining the same as above but with - inverse logic. - -{:/comment} - -Conversely, if Relevant Upper Bound exists, +Conversely, when Relevant Upper Bound does exist, it is not affected by Max Load value. +Given that Relevant Upper Bound is a quantity based on Load, +it is ALLOWED to express this quantity using multi-interface values +in test report, e.g. as sum of per-interface loads. + ### Relevant Lower Bound Definition: The Relevant Lower Bound is the largest Trial Load value among those smaller than the Relevant Upper Bound, that got classified -as a Lower Bound for the given Search Goal at the end of the search. +as a Lower Bound for a given Search Goal at the end of the search. Discussion: -If no load had enough low-loss trials, the relevant lower bound +If no load had enough Low-Loss Trials, the Relevant Lower Bound MAY be non-existent. Strictly speaking, if the Relevant Upper Bound does not exist, the Relevant Lower Bound also does not exist. In a typical case, Max Load is classified as a Lower Bound, -but it is not clear whether a higher value -would be found as a Lower Bound if the search was not limited -by this Max Load value. +making it impossible to increase the Load to continue the search +for an Upper Bound. +Thus, it is not clear whether a larger value would be found +for a Relevant Lower Bound if larger Loads were possible. + +Given that Relevant Lower Bound is a quantity based on Load, +it is ALLOWED to express this quantity using multi-interface values +in test report, e.g. as sum of per-interface loads. ### Conditional Throughput @@ -1695,33 +1381,29 @@ according to algorithm defined in Discussion: -The Relevant Lower Bound is defined only at the end of the search, +The Relevant Lower Bound is defined only at the end of the Search, and so is the Conditional Throughput. But the algorithm can be applied at any time on any Lower Bound load, so the final Conditional Throughput value may appear sooner -than at the end of the search. +than at the end of the Search. Informally, the Conditional Throughput should be a typical Trial Forwarding Rate, expected to be seen at the Relevant Lower Bound of the given Search Goal. But frequently it is only a conservative estimate thereof, -as MLRsearch implementations tend to stop gathering more trials +as MLRsearch Implementations tend to stop measuring more Trials as soon as they confirm the value cannot get worse than this estimate within the Goal Duration Sum. This value is RECOMMENDED to be used when evaluating repeatability -and comparability of different MLRsearch implementations. +and comparability of different MLRsearch Implementations. See [Generalized Throughput](#generalized-throughput) for more details. -{::comment} - [Low priority but useful for comparabuility.] - - <mark>[VP] TODO: Add subsection for Trial Results At Relevant Bounds - as an optional attribute of Goal Result.</mark> - -{:/comment} +Given that Conditional Throughput is a quantity based on Load, +it is ALLOWED to express this quantity using multi-interface values +in test report, e.g. as sum of per-interface foerwarding rates. ### Goal Results @@ -1749,7 +1431,7 @@ and the Relevant Upper Bound MUST NOT be larger than the Goal Width, Implementations MAY add their own attributes. -Test report MUST display Relevant Lower Value, +Test report MUST display Relevant Lower Bound. Displaying Relevant Upper Bound is NOT REQUIRED, but it is RECOMMENDED, especially if the implementation does not use Goal Width. @@ -1763,7 +1445,9 @@ Discussion: It is RECOMMENDED to report any useful quantity even if it does not satisfy all the requirements. For example if Max Load is classified -as a Lower Bound, it is fine to report it as the Relevant Lower Bound, +as a Lower Bound, it is fine to report it as an "effective" Relevant Lower Bound +(although not a real one, as that requires +Relevant Upper Bound which does not exist in this case), and compute Conditional Throughput for it. In this case, only the missing Relevant Upper Bound signals this result instance is irregular. @@ -1776,19 +1460,12 @@ they MUST be clearly marked as comming from irregular results. The implementation MAY define additional attributes. -{::comment} - [Useful.] - - <mark>MKP2 [VP] TODO: Also allways-fail. Link to bounds to avoid duplication.</mark> - -{:/comment} - #### Goal Result Definition: -Goal Result is a composite quantity. Each instance is either a Regular Goal Result -or an Irregular Goal Result. +Goal Result is a composite quantity. +Each instance is either a Regular Goal Result or an Irregular Goal Result. Discussion: @@ -1814,13 +1491,6 @@ but their status as regular or irregular may be different. For example if two goals differ only in Goal Width value, and the relevant bound values are close enough according to only one of them. -{::comment} - [Not important.] - - <mark>[VP] Postponed: API independence, modularity.</mark> - -{:/comment} - ### Controller Output Definition: @@ -1831,14 +1501,8 @@ The Search Result instance is its only REQUIRED attribute. Discussion: -MLRsearch implementation MAY return additional data in the Controller Output, -for example number of trials performed and the total Search duration. - -{::comment} - - TODO-P0: "max search time exceeded" flag? - -{:/comment} +MLRsearch Implementation MAY return additional data in the Controller Output, +for example number of trials performed and the total Search Duration. ## MLRsearch Architecture @@ -1846,14 +1510,14 @@ MLRsearch architecture consists of three main system components: the Manager, the Controller, and the Measurer. The architecture also implies the presence of other components, -such as the SUT and the Tester (as a sub-component of the Measurer). +such as the SUT and the tester (as a sub-component of the Measurer). Protocols of communication between components are generally left unspecified. For example, when MLRsearch specification mentions "Controller calls Measurer", it is possible that the Controller notifies the Manager -to call the Measurer indirectly instead. This way the Measurer implementations +to call the Measurer indirectly instead. This way the Measurer Implementations can be fully independent from the Controller implementations, -e.g. programmed in different programming languages. +e.g. developed in different programming languages. ### Measurer @@ -1869,7 +1533,7 @@ This definition assumes the Measurer is already initialized. In practice, there may be additional steps before the Search, e.g. when the Manager configures the traffic profile (either on the Measurer or on its tester sub-component directly) -and performs a warmup (if the test procedure requires one). +and performs a warm-up (if the tester or the test procedure requires one). It is the responsibility of the Measurer implementation to uphold any requirements and assumptions present in MLRsearch specification, @@ -1891,7 +1555,11 @@ between the Intended Load and the Offered Load into increased Trial Loss Ratio. Neither of the two recommendations are made into requirements, because it is not easy to tell when the difference is big enough, -in a way thay would be dis-entangled from other Measurer freedoms. +in a way that would be disentangled from other Measurer freedoms. + +For a simple example of a situation where the Offered Load cannot keep up +with the Intended Load, and the consequences on MLRsearch result, +see [Hard Performance Limit](#hard-performance-limit). ### Controller @@ -1907,9 +1575,9 @@ Discussion: Informally, the Controller has big freedom in selection of Trial Inputs, and the implementations want to achieve all the Search Goals -in the shortest expected time. +in the shortest average time. -The Controller's role in optimizing the overall search time +The Controller's role in optimizing the overall Search Duration distinguishes MLRsearch algorithms from simpler search procedures. Informally, each implementation can have different stopping conditions. @@ -1928,33 +1596,40 @@ defined in [RFC2544] (Section 26). Discussion: -The Manager initializes the SUT, the Measurer (and the Tester if independent) +The Manager initializes the SUT, the Measurer +(and the tester if independent from Measurer) with their intended configurations before calling the Controller. +Note that [RFC2544] (Section 7) already puts requirements on SUT setups: + + It is expected that all of the tests will be run without changing the + configuration or setup of the DUT in any way other than that required + to do the specific test. For example, it is not acceptable to change + the size of frame handling buffers between tests of frame handling + rates or to disable all but one transport protocol when testing the + throughput of that protocol. + +It is REQUIRED for the test report to encompass all the SUT configuration +details, perhaps by describing a "default" configuration common for most tests +and only describe configuration changes if required by a specific test. + +For example, [RFC5180] (Section 5.1.1) recommends testing jumbo frames +if SUT can forward them, even though they are outside the scope +of the 802.3 IEEE standard. In this case, it is fair +for the SUT default configuration to not support jumbo frames, +and only enable this support when testing jumbo traffic profiles, +as the handling of jumbo frames typically has different packet buffer +requirements and potentially higher processing overhead. +Ideally, non-jumbo frame sizes should also be tested on the jumbo-enabled setup. + The Manager does not need to be able to tweak any Search Goal attributes, but it MUST report all applied attribute values even if not tweaked. -In principle, there should be a "user" (human or CI) +In principle, there should be a "user" (human or automated) that "starts" or "calls" the Manager and receives the report. The Manager MAY be able to be called more than once whis way, thus triggering multiple independent Searches. -{::comment} - [Not important, unless anybody else asks.] - - <mark>MKP2 The Manager may use the Measurer or other system components - to perform other tests, e.g. back-to-back frames, - as the Controller is only replacing the search from - [RFC2544] (Section 26.1).</mark> - -{:/comment} - -{::comment} - - TODO-P2: Summarize test report requirements here? - -{:/comment} - ## Compliance This section discusses compliance relations between MLRsearch @@ -1962,8 +1637,8 @@ and other test procedures. ### Test Procedure Compliant with MLRsearch -Any networking measurement setup where there can be logically delineated -system components and there are abstract components satisfying requirements +Any networking measurement setup that could be understood as consisting of +abstract components satisfying requirements for the Measurer, the Controller and the Manager, is considered to be compliant with MLRsearch specification. @@ -1979,15 +1654,8 @@ can be understood as a MLRsearch architecture, as long as there is enough data to reconstruct the Relevant Upper Bound. See the next subsection for an equivalent Search Goal. -Any test procedure that can be understood as (one call to the Manager of) -MLRsearch architecture is said to be compliant with MLRsearch specification. - -{::comment} - - TODO-P0: Delete occurances of "MLRsearch Implementation", review - occurances of "MLRsearch implementation". - -{:/comment} +Any test procedure that can be understood as one call to the Manager of +MLRsearch architecture is said to be compliant with MLRsearch Specification. ### MLRsearch Compliant with RFC2544 @@ -2005,28 +1673,15 @@ Adding the first attribute, Goal Final Trial Duration, makes the Search Goal unconditionally compliant. The second attribute (Goal Duration Sum) only prevents MLRsearch -from repeating zero-loss full-length trials. +from repeating zero-loss Full-Length Trials. The presence of other Search Goals does not affect the compliance of this Goal Result. The Relevant Lower Bound and the Conditional Throughput are in this case equal to each other, and the value is the [RFC2544] throughput. -{::comment} - - TODO-P1: Move the rest into Load Classification Logic chapter. - -{:/comment} - Non-zero exceed ratio is not strictly disallowed, but it could -needlessly prolong the search when low-loss short trials are present. - -{::comment} - - TODO-P2: Also it would open more questions re Loss Inversion, - but no need to say that anywhere. - -{:/comment} +needlessly prolong the search when Low-Loss short trials are present. ### MLRsearch Compliant with TST009 @@ -2039,17 +1694,6 @@ thus more repeatable and comparable. Only the variant with "z = infinity" is achievable with MLRsearch. -{::comment} - [Low priority, unless a short sentence is found.] - - <mark>MKP2 MK note: Shouldn't we add a note about how MLRsearch goes about - addressing the TST009 point related to z, that is "z is threshold of - Lord(r) to override Loss Verification when the count of lost frames is - very high and unnecessary verification trials."? i.e. by have Goal Loss - Ratio. Thoughts?</mark> - -{:/comment} - For example, for "max(r) = 2" variant, the following Search Goal instance should be used to get compatible Search Result: @@ -2100,19 +1744,19 @@ assuming initial bounds are confirmed. MLRsearch specification requires listing both Relevant Bounds for each Search Goal, and the difference between the bounds implies -whether the result precision achieved. -Therefore it is not necessary to report the specific stopping condition used. +whether the result precision is achieved. +Therefore, it is not necessary to report the specific stopping condition used. -MLRsearch implementations may use Goal Width +MLRsearch Implementations may use Goal Width to allow direct control of result precision, -and indirect control of the search duration. +and indirect control of the Search Duration. -Other MLRsearch implementations may use different stopping conditions; -for example based on the search duration, trading off precision control +Other MLRsearch Implementations may use different stopping conditions; +for example based on the Search Duration, trading off precision control for duration control. Due to various possible time optimizations, there is no longer a strict -correspondence between the overall search duration and Goal Width values. +correspondence between the Search Duration and Goal Width values. In practice, noisy SUT performance increases both average search time and its variance. @@ -2122,7 +1766,7 @@ The most obvious difference between MLRsearch and [RFC2544] binary search is in the goals of the search. [RFC2544] has a single goal, based on classifying a single full-length trial as either zero-loss or non-zero-loss. -MLRsearch supports searching for multiple goals at once, +MLRsearch supports searching for multiple Search Goals at once, usually differing in their Goal Loss Ratio values. ### Single Goal and Hard Bounds @@ -2131,12 +1775,12 @@ Each bound in [RFC2544] simple binary search is "hard", in the sense that all further Trial Load values are smaller than any current upper bound and larger than any current lower bound. -This is also possible for MLRsearch implementations, +This is also possible for MLRsearch Implementations, when the search is started with only one Search Goal instance. ### Multiple Goals and Loss Inversion -MLRsearch supports multiple goals, making the search procedure +MLRsearch supports multiple Search Goals, making the search procedure more complicated compared to binary search with single goal, but most of the complications do not affect the final results much. Except for one phenomenon: Loss Inversion. @@ -2147,21 +1791,15 @@ But for larger amounts, a Load that is classified as an Upper Bound for one Search Goal may still be a Lower Bound for another Search Goal. And, due to this other goal, MLRsearch will probably perform subsequent Trials -at Trial Loads even higher than the original value. - -{::comment} - - TODO-P2: Unify load adjectives: higher/lower xor larger/smaller. => higher/lower. - -{:/comment} +at Trial Loads even larger than the original value. This introduces questions any many-goals search algorithm has to address. -What to do when all such higher load trials happen to have zero loss? +What to do when all such larger load trials happen to have zero loss? Does it mean the earlier upper bound was not real? -Does it mean the later low-loss trials are not considered a lower bound? +Does it mean the later Low-Loss trials are not considered a lower bound? -The situation where a smaller load is classified as an Upper Bound, -while a larger load is classified as a Lower Bound (for the same search goal), +The situation where a smaller Load is classified as an Upper Bound, +while a larger Load is classified as a Lower Bound (for the same search goal), is called Loss Inversion. Conversely, only single-goal search algorithms can have hard bounds @@ -2176,9 +1814,10 @@ is considered to be a fluke, at least when computing the final result. This is formalized using definitions of [Relevant Upper Bound](#relevant-upper-bound) and [Relevant Lower Bound](#relevant-lower-bound). -The Relevant Upper Bound (for specific goal) is the smallest load classified + +The Relevant Upper Bound (for specific goal) is the smallest Load classified as an Upper Bound. But the Relevant Lower Bound is not simply -the largest among Lower Bounds. It is the largest load among loads +the largest among Lower Bounds. It is the largest Load among Loads that are Lower Bounds while also being smaller than the Relevant Upper Bound. With these definitions, the Relevant Lower Bound is always smaller @@ -2193,14 +1832,15 @@ every Trial Result can have an impact on any current Relevant Bound larger than that Trial Load, namely by becoming a new Upper Bound. -This also applies when that trial happens -before that bound could have become current. +This also applies when that Load is measured +before another Load gets enough measurements to become a current Relevant Bound. -This means if your SUT (or your Traffic Generator) needs a warmup, -be sure to warm it up before starting the Search. +This also implies that if the SUT tested (or the Traffic Generator used) +needs a warm-up, it should be warmed up before starting the Search, +otherwise the first few measurements could become unjustly limiting. -Also, for MLRsearch implementation, it means it is better to measure -at smaller loads first, so bounds found earlier are less likely +For MLRsearch Implementations, it means it is better to measure +at smaller Loads first, so bounds found earlier are less likely to get invalidated later. ## Exceed Ratio and Multiple Trials @@ -2215,7 +1855,7 @@ so MLRsearch may need to allow some frequency of high-loss trials to ignore the rare but big effects near the noiseful end. For MLRsearch to perform such Trial Result filtering, it needs -a configuration option to tell how frequent can the "infrequent" big loss be. +a configuration option to tell how frequent the "infrequent" big loss can be. This option is called the [Goal Exceed Ratio](#goal-exceed-ratio). It tells MLRsearch what ratio of trials (more specifically, what ratio of Trial Effective Duration seconds) @@ -2223,89 +1863,58 @@ can have a [Trial Loss Ratio](#trial-loss-ratio) larger than the [Goal Loss Ratio](#goal-loss-ratio) and still be classified as a [Lower Bound](#lower-bound). -Zero exceed ratio means all trials must have a Trial Loss Ratio -equal to or smaller than the Goal Loss Ratio. +Zero exceed ratio means all Trials must have a Trial Loss Ratio +equal to or lower than the Goal Loss Ratio. -When more than one trial is intended to classify a Load, +When more than one Trial is intended to classify a Load, MLRsearch also needs something that controls the number of trials needed. Therefore, each goal also has an attribute called Goal Duration Sum. The meaning of a [Goal Duration Sum](#goal-duration-sum) is that -when a load has (full-length) trials +when a Load has (Full-Length) Trials whose Trial Effective Durations when summed up give a value at least as big as the Goal Duration Sum value, -the load is guaranteed to be classified either as an Upper Bound +the Load is guaranteed to be classified either as an Upper Bound or a Lower Bound for that Search Goal instance. -{::comment} - - TODO-P2: Move some discussion on Trial Effective Duration from spec chapter - to around here? Probably no time to dwell on this, delete the todo. - TODO2: my pref is to keep it in spec section - -{:/comment} - ## Short Trials and Duration Selection -MLRsearch requires each goal to specify its Goal Final Trial Duration. +MLRsearch requires each Searcg Goal to specify its Goal Final Trial Duration. Section 24 of [RFC2544] already anticipates possible time savings when Short Trials are used. -Any MLRsearch implementation MAY include its own configuration options +Any MLRsearch Implementation may include its own configuration options which control when and how MLRsearch chooses to use short trial durations. -While MLRsearch implementations are free to use any logic to select -Trial Input values, comparability between MLRsearch implementations +While MLRsearch Implementations are free to use any logic to select +Trial Input values, comparability between MLRsearch Implementations is only assured when the Load Classification logic handles any possible set of Trial Results in the same way. -The presence of short trial results complicates -the load classification logic, see details in +The presence of Short Trial Results complicates +the Load Classification logic, see details in [Load Classification Logic](#load-classification-logic) chapter. While the Load Classification algorithm is designed to avoid any unneeded Trials, -for explainability reasons it is RECOMMENDED for users to use +for explainability reasons it is recommended for users to use such Controller Input instances that lead to all Trial Duration values selected by Controller to be the same, e.g. by setting any Goal Initial Trial Duration to be a single value also used in all Goal Final Trial Duration attributes. -{::comment} - - TODO-P0: last statement is confusing. it implies GITD = GFTD, which doesn't make sense to me. - - TODO-P0: below to be removed once Load Classification Logic is done. - -{:/comment} - -In a nutshell, results from short trials -may cause a load to be classified as an upper bound. -This may cause loss inversion, and thus lower the Relevant Lower Bound, -below what would classification say when considering full-length trials only. - -{::comment} - [Important. Keeping compatibility slows search considerably.] - - <mark>Alas, such configurations are usually not compliant with [RFC2544] requirements, - or not time-saving enough.</mark> - - <mark>mk edit note: This statement does not make sense to me. Suggest to remove it.</mark> - -{:/comment} - ## Generalized Throughput Due to the fact that testing equipment takes the Intended Load -as an input parameter for a trial measurement, +as an input parameter for a Trial measurement, any load search algorithm needs to deal with Intended Load values internally. -But in the presence of goals with a non-zero [Goal Loss Ratio](#goal-loss-ratio), -the Intended Load usually does not match +But in the presence of Search Goals with a non-zero +[Goal Loss Ratio](#goal-loss-ratio), the Load usually does not match the user's intuition of what a throughput is. -The forwarding rate (as defined in [RFC2285] section 3.6.1) is better, +The forwarding rate as defined in [RFC2285] (Section 3.6.1) is better, but it is not obvious how to generalize it -for loads with multiple trials and a non-zero goal loss ratio. +for Loads with multiple Trials and a non-zero Goal Loss Ratio. The best example is also the main motivation: hard performance limit. @@ -2313,15 +1922,15 @@ The best example is also the main motivation: hard performance limit. Even if bandwidth of the medium allows higher performance, the SUT interfaces may have their additional own limitations, -e.g. a specific frames-per-second limit on the NIC (a common occurance). +e.g. a specific frames-per-second limit on the NIC (a common occurence). Ideally, those should be known and provided as [Max Load](#max-load). -But if Max Load is set higher than what the interface can receive or transmit, -there will be a "hard limit" observed in trial results. +But if Max Load is set larger than what the interface can receive or transmit, +there will be a "hard limit" behavior observed in Trial Results. Imagine the hard limit is at hundred million frames per second (100 Mfps), -Max Load is higher, and the goal loss ratio is 0.5%. -If DUT has no additional losses, 0.5% loss ratio will be achieved +Max Load is larger, and the Goal Loss Ratio is 0.5%. +If DUT has no additional losses, 0.5% Trial Loss Ratio will be achieved at Relevant Lower Bound of 100.5025 Mfps. But it is not intuitive to report SUT performance as a value that is larger than the known hard limit. @@ -2330,18 +1939,18 @@ different from just the Relevant Lower Bound. MLRsearch defines one such generalization, the [Conditional Throughput](#conditional-throughput). -It is the Trial Forwarding Rate from one of the full-length trials +It is the Trial Forwarding Rate from one of the Full-Length Trials performed at the Relevant Lower Bound. The algorithm to determine which trial exactly is in [Appendix B: Conditional Throughput](#appendix-b-conditional-throughput). -In the hard limit example, 100.5025 Mfps load will still have +In the hard limit example, 100.5025 Mfps Load will still have only 100.0 Mfps forwarding rate, nicely confirming the known limitation. ### Performance Variability With non-zero Goal Loss Ratio, and without hard performance limits, -low-loss trials at the same Load may achieve different Trial Forwarding Rate +Low-Loss trials at the same Load may achieve different Trial Forwarding Rate values just due to DUT performance variability. By comparing the best case (all Relevant Lower Bound trials have zero loss) @@ -2349,71 +1958,17 @@ and the worst case (all Trial Loss Ratios at Relevant Lower Bound are equal to the Goal Loss Ratio), we find the possible Conditional Throughput values may have up to the Goal Loss Ratio relative difference. -Therefore, it is rarely needed to set the Goal Width (if expressed -as the relative difference of loads) below the Goal Loss Ratio. -In other words, setting the Goal Width below the Goal Loss Ratio -may cause the Conditional Throughput for a larger loss ratio to become smaller -than a Conditional Throughput for a goal with a smaller Goal Loss Ratio, -which is counter-intuitive, considering they come from the same search. -Therefore it is RECOMMENDED to set the Goal Width to a value no smaller -than the Goal Loss Ratio. +Setting the Goal Width below the Goal Loss Ratio +may cause the Conditional Throughput for a larger Goal Loss Ratio to become smaller +than a Conditional Throughput for a goal with a lower Goal Loss Ratio, +which is counter-intuitive, considering they come from the same Search. +Therefore it is RECOMMENDED to set the Goal Width to a value no lower +than the Goal Loss Ratio of the higher-loss Search Goal. Despite this variability, in practice Conditional Throughput behaves better -than Relevant Lower Bound for comparability purposes. - -{::comment} - - TODO-P0: Move the rest into the last chapter. - -{:/comment} - -Conditional Throughput is partially related to load classification. -If a load is classified as a Relevant Lower Bound for a goal, -the Conditional Throughput comes from a trial result, -that is guaranteed to have Trial Loss Ratio no larger than the Goal Loss Ratio. - -{::comment} - [Important only for "design principles" chapter we may never have.] - - <mark>In the future, other intuitive values may become popular, - but they are unlikely to supersede the definition of the Relevant Lower Bound - as the most fitting value for comparability purposes, - therefore the Relevant Lower Bound remains a required attribute - of the Goal Result structure, while the Conditional Throughput is only optional.</mark> - - <mark>mk edit note: This paragraph adds to the confusion. I would remove - this paragraph, as with the new text above it doesn't seem to add any - value.</mark> - - <mark>[VP] TODO: This is an example of MLRsearch design principles.</mark> - -{:/comment} - -{::comment} - [Useful.] - - <mark>[VP] TODO: Mention somewhere that trending is a specific case - of repeatability/comparability.</mark> - -{:/comment} - -{::comment} - [Important for BMWG. Configurability is bad for comparability.] - - <mark>MKP2 Sadly, different implementations may exhibit their sweet spot of</mark> - <mark>the best repeatability for a given search duration</mark> - <mark>at different goals attribute values, especially concerning</mark> - <mark>any optional goal attributes such as the initial trial duration.</mark> - <mark>Thus, this document does not comment much on which configurations</mark> - <mark>are good for comparability between different implementations.</mark> - <mark>For comparability between different SUTs using the same implementation,</mark> - <mark>refer to configurations recommended by that particular implementation.</mark> - - <mark>MKP2 mk edit note: Isn't this going off on a tangent, hypothesising and - second guessing about different possible implementations. What is the - value of this content to this document? Suggest to remove it.</mark> - -{:/comment} +than Relevant Lower Bound for comparability purposes, +especially if deterministic Load selection is likely to produce +exactly the same Relevant Lower Bound value across multiple runs. # MLRsearch Logic and Example @@ -2424,17 +1979,11 @@ reflecting formal pseudocode representation present in and [Appendix B: Conditional Throughput](#appendix-b-conditional-throughput). This is followed by example search. -{::comment} - - TODO-P1: Move this paragraph to a better place. - TODO-P1: This is an answer to the questions of "why are algorithms this strict"? - TODO-P1: Pose that question somewhere, pose this answer there or in another place. - -{:/comment} - -For repeatability and comparability reasons, it is important that -all implementations of MLRsearch classify the load equivalently, -based on all trials measured at the given load. +The logic as described here is equivalent but not identical to the pseudocode +on appendices. The pseudocode is designed to be short and frequently +combines multiple operation into one expression. +The logic as described here lists each operation separately +and uses more intuitive names for te intermediate values. ## Load Classification Logic @@ -2463,7 +2012,7 @@ Note: For explanation clarity variables are taged as (I)nput, - Effective high-loss sum (T) is the full-length high-loss sum plus the positive excess sum. - Effective full sum (T) is the effective high-loss sum - plus the full-length low-loss sum. + plus the full-length low-loss sum. - Effective whole sum (T) is the larger of the effective full sum and the Goal Duration Sum. - Missing sum (T) is the effective whole sum minus the effective full sum. @@ -2504,19 +2053,19 @@ Note: For explanation clarity variables are taged as (I)nput, Loss Ratio: - If remaining sum is not larger than zero, exit the loop. - Set current loss ratio to this trial's Trial Loss Ratio (I). - - Decrease the remaining sum by this trial's Trial Effective - Duration (I). + - Decrease the remaining sum by this trial's Trial Effective Duration (I). - Current forwarding ratio (T) is One minus the current loss ratio. - Conditional Throughput (T) is the current forwarding ratio multiplied by the Load value. -{::comment} - TODO-P0: Move somewhere else? MK: I think it's okay to leave it here. -{:/comment} - -By definition, Conditional Throughput logic results in a value -that represents Trial Loss Ratio at most equal to Goal Loss Ratio. +This shows that Conditional Throughput is partially related to Load Classification. +If a Load is classified as a Relevant Lower Bound for a Search Goal instance, +the Conditional Throughput comes from a Trial Result +that is guaranteed to have Trial Loss Ratio no larger than the Goal Loss Ratio. +The converse is not true if Goal Width is smaller than the Goal Loss Ratio, +as in that case it is possible for the Conditional Throughput +to be larger than the Relevant Upper Bound. ## SUT Behaviors @@ -2560,16 +2109,6 @@ can be summarized into a single number: Exceed Probability. It is the probability (according to the expert) that the measured Trial Loss Ratio will be higher than the Goal Loss Ratio. -{::comment} - - TODO-P2: Do we need to say small EP means low load? - - TODO-P3: Mention how ER relates to EP here? - - TODO-P2: Tie to Relevant Lower Bound and Conditional Throughput somewhere. - -{:/comment} - ### Trial Duration Dependence When comparing Exceed Probability values for the same Trial Load value @@ -2578,16 +2117,16 @@ there are several patterns that commonly occur in practice. #### Strong Increase -Exceed Probability is very small at short durations but very high at full-length. +Exceed Probability is very low at short durations but very high at full-length. This SUT behavior is undesirable, and may hint at faulty SUT, e.g. SUT leaks resources and is unable to sustain the desired performance. But this behavior is also seen when SUT uses large amount of buffers. -This is the main reasons users may want to set high Goal Final Trial Duration. +This is the main reasons users may want to set large Goal Final Trial Duration. #### Mild Increase -Short trials have smaller exceed probability, but the difference is not as high. +Short trials have lower exceed probability, but the difference is not as high. This behavior is quite common if the noise contains infrequent but large loss spikes, as the more performant parts of a full-length trial are unable to compensate for all the frame loss from a less performant part. @@ -2606,22 +2145,6 @@ This can be possible only for non-zero Goal Loss Ratio, for example if SUT needs to "warm up" to best performance within each trial. Not sommonly seen in practice. -{::comment} - - TODO-P2: Define loss spikes? Mention loss spikes when discussing noise? - -{:/comment} - -{::comment} - - ### Loss Spikes - - #### Frequent Small Loss Spikes - - #### Rare Big Loss Spikes - -{:/comment} - ## Example Search The following example Search is related to @@ -2644,7 +2167,7 @@ In all points in time, only one Search Goal instance is marked as "in focus". That explains Trial Duration of the new Trials, but is otherwise unrelated to the logic applied. -MLRsearch implementations are not required to "focus" on one goal at time, +MLRsearch Implementations are not required to "focus" on one goal at time, but this example is useful to show a load can be classified also for goals not "in focus". @@ -2702,12 +2225,6 @@ the other two cover less frequent cases. ### Example Trial Results -{::comment} - - TODO-P1: Merge this with Point computations so all trial data is localized. - -{:/comment} - The following six sets of trial results are selected for the example Search. The sets are defined as points in time, describing which Trial Results were added since the previous point. @@ -2897,8 +2414,8 @@ Optimistic exceed ratio | 100% | 0% | 50% | 75% Pessimistic exceed ratio | 100% | 100% | 50% | 100% Classification Result | Upper Bound | Undecided | Lower Bound | Upper Bound -The one missing trial for "1s final" was low-loss, -half of trial results are low-loss which exactly matches 50% exceed ratio. +The one missing trial for "1s final" was Low-Loss, +half of trial results are Low-Loss which exactly matches 50% exceed ratio. This shows time savings are not guaranteed. #### Point 5 @@ -2933,7 +2450,7 @@ to compensate for losses seen in 1-second results. But Load Classification logic does not have that knowledge hardcoded, so optimistic exceed ratio is still only 50%. -But the 0.1% Trial Loss Ratio is smaller than "20% exceed" Goal Loss Ratio, +But the 0.1% Trial Loss Ratio is lower than "20% exceed" Goal Loss Ratio, so this unexpected Full-Length Low-Loss trial changed the classification result of this Load to Lower Bound. @@ -2963,7 +2480,7 @@ Classification Result | Upper Bound | Lower Bound | Lower Bound | Lower Bo This is the Low-Loss Trial the "TST009" goal was waiting for. This Load is now classified for all goals, the search may end. -Or, more realistically, it can focus on higher load only, +Or, more realistically, it can focus on larger load only, as the three goals will want an Upper Bound (unless this Load is Max Load). ### Conditional Throughput Computations @@ -3071,17 +2588,6 @@ One has Trial Loss Ratio of 0%, the other of 0.1%. Due to stricter Goal Exceed Ratio, this Conditional Throughput is smaller than Conditional Throughput of the other two goals. -{::comment} - - TODO-P2: Example of long trial being too strict? - - TODO-P2: Unless a set of Search Goals is recommended, comparability is not there. - - TODO-P2: Spell out how MLRsearch addressed the Problems. - -{:/comment} - - # IANA Considerations No requests of IANA. @@ -3107,73 +2613,68 @@ networks. # Acknowledgements -Some phrases and statements in this document were created -with help of Mistral AI (mistral.ai). - -Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough -review and numerous useful comments and suggestions in the earlier versions of this document. - Special wholehearted gratitude and thanks to the late Al Morton for his thorough reviews filled with very specific feedback and constructive -guidelines. Thank you Al for the close collaboration over the years, -for your continuous unwavering encouragement full of empathy and -positive attitude. Al, you are dearly missed. +guidelines. Thank You Al for the close collaboration over the years, Your Mentorship, +Your continuous unwavering encouragement full of empathy and energizing +positive attitude. Al, You are dearly missed. + +Thanks to Gabor Lencse, Giuseppe Fioccola and BMWG contributors for good +discussions and thorough reviews, guiding and helping us to improve the +clarity and formality of this document. + +Many thanks to Alec Hothan of the OPNFV NFVbench project for a thorough +review and numerous useful comments and suggestions in the earlier +versions of this document. # Appendix A: Load Classification -This section specifies how to perform the load classification. +This section specifies how to perform the Load Classification. Any Trial Load value can be classified, -according to a given [Search Goal](#search-goal). +according to a given [Search Goal](#search-goal) instance. -The algorithm uses (some subsets of) the set of all available trial results -from trials measured at a given intended load at the end of the search. -All durations are those returned by the Measurer. +The algorithm uses (some subsets of) the set of all available Trial Results +from Trials measured at a given Load at the end of the Search. The block at the end of this appendix holds pseudocode which computes two values, stored in variables named `optimistic_is_lower` and `pessimistic_is_lower`. -{::comment} - [We have other section re optimistic. Not going to talk about variable naming here.] - - <mark>MKP2 mk edit note: Need to add the description of what - the `optimistic` and `pessimistic` variables represent. - Or a reference to where this is described - e.g. in [Single Trial Duration](#single-trial-duration) section.</mark> - -{:/comment} - The pseudocode happens to be valid Python code. -If values of both variables are computed to be true, the load in question -is classified as a lower bound according to the given Search Goal. -If values of both variables are false, the load is classified as an upper bound. -Otherwise, the load is classified as undecided. +If values of both variables are computed to be true, the Load in question +is classified as a Lower Bound according to the given Search Goal instance. +If values of both variables are false, the Load is classified as an Upper Bound. +Otherwise, the load is classified as Undecided. The pseudocode expects the following variables to hold the following values: -- `goal_duration_sum`: The duration sum value of the given Search Goal. +- `goal_duration_sum`: The Goal Duration Sum value of the given Search Goal. -- `goal_exceed_ratio`: The exceed ratio value of the given Search Goal. +- `goal_exceed_ratio`: The Goal Exceed Ratio value of the given Search Goal. -- `full_length_low_loss_sum`: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial Loss Ratio - not higher than the Goal Loss Ratio. +- `full_length_low_loss_sum`: Sum of Trial Effective Durations across Trials + with Trial Duration at least equal to the Goal Final Trial Dduration + and with Trial Loss Ratio not higher than the Goal Loss Ratio + (across Full-Length Low-Loss Trials). -- `full_length_high_loss_sum`: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial Loss Ratio - higher than the Goal Loss Ratio. +- `full_length_high_loss_sum`: Sum of Trial Effective Durations across Trials + with Trial Duration at least equal to the Goal Final Trial Duration + and with Trial Loss Ratio higher than the Goal Loss Ratio + (across Full-Length High-Loss Trials). -- `short_low_loss_sum`: Sum of durations across trials with trial duration - shorter than the goal final trial duration and with a Trial Loss Ratio - not higher than the Goal Loss Ratio. +- `short_low_loss_sum`: Sum of Trial Effective Durations across Trials + with Trial Duration shorter than the Goal Final Trial Duration + and with Trial Loss Ratio not higher than the Goal Loss Ratio + (across Short Low-Loss Trials). -- `short_high_loss_sum`: Sum of durations across trials with trial duration - shorter than the goal final trial duration and with a Trial Loss Ratio - higher than the Goal Loss Ratio. +- `short_high_loss_sum`: Sum of Trial Effective Durations across Trials + with Trial Duration shorter than the Goal Final Trial Duration + and with Trial Loss Ratio higher than the Goal Loss Ratio + (across Short High-Loss Trials). -The code works correctly also when there are no trial results at a given load. +The code works correctly also when there are no Trial Results at a given Load. ~~~ python exceed_coefficient = goal_exceed_ratio / (1.0 - goal_exceed_ratio) @@ -3190,56 +2691,48 @@ optimistic_is_lower = effective_high_loss_sum <= quantile_duration_sum # Appendix B: Conditional Throughput -This section specifies how to compute Conditional Throughput, as referred to in section [Conditional Throughput](#conditional-throughput). +This section specifies how to compute Conditional Throughput, +as referred to in section [Conditional Throughput](#conditional-throughput). -Any intended load value can be used as the basis for the following computation, -but only the Relevant Lower Bound (at the end of the search) +Any Load value can be used as the basis for the following computation, +but only the Relevant Lower Bound (at the end of the Search) leads to the value called the Conditional Throughput for a given Search Goal. -The algorithm uses (some subsets of) the set of all available trial results -from trials measured at a given intended load at the end of the search. -All durations are those returned by the Measurer. +The algorithm uses (some subsets of) the set of all available Trial Results +from Trials measured at a given Load at the end of the Search. The block at the end of this appendix holds pseudocode which computes a value stored as variable `conditional_throughput`. -{::comment} - [CT is CT. But text could make more obvious.] - - <mark>MKP2 mk edit note: Need to add the description of what does - the `conditional_throughput` variable represent. - Or a reference to where this is described - e.g. in [Conditional Throughput](#conditional-throughput) section.</mark> - -{:/comment} - The pseudocode happens to be valid Python code. The pseudocode expects the following variables to hold the following values: -- `goal_duration_sum`: The duration sum value of the given Search Goal. +- `goal_duration_sum`: The Goal Duration Sum value of the given Search Goal. -- `goal_exceed_ratio`: The exceed ratio value of the given Search Goal. +- `goal_exceed_ratio`: The Goal Exceed Ratio value of the given Search Goal. -- `full_length_low_loss_sum`: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial Loss Ratio - not higher than the Goal Loss Ratio. +- `full_length_low_loss_sum`: Sum of Trial Effective Durations across Trials + with Trial Duration at least equal to the Goal Final Trial Dduration + and with Trial Loss Ratio not higher than the Goal Loss Ratio + (across Full-Length Low-Loss Trials). -- `full_length_high_loss_sum`: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial Loss Ratio - higher than the Goal Loss Ratio. +- `full_length_high_loss_sum`: Sum of Trial Effective Durations across Trials + with Trial Duration at least equal to the Goal Final Trial Duration + and with Trial Loss Ratio higher than the Goal Loss Ratio + (across Full-Length High-Loss Trials). -- `full_length_trials`: An iterable of all trial results from trials with trial duration - at least equal to the goal final trial duration, - sorted by increasing the Trial Loss Ratio. - A trial result is a composite with the following two attributes available: +- `full_length_trials`: An iterable of all Trial Results from Trials + with Trial Duration at least equal to the Goal Final Trial Duration + (all Full-Length Trials), sorted by increasing Trial Loss Ratio. + One item `trial` is a composite with the following two attributes available: - - `trial.loss_ratio`: The Trial Loss Ratio as measured for this trial. + - `trial.loss_ratio`: The Trial Loss Ratio as measured for this Trial. - - `trial.duration`: The trial duration of this trial. + - `trial.effective_duration`: The Trial Effective Duration of this Trial. The code works correctly only when there if there is at least one -trial result measured at a given load. +Trial Tesult measured at the given Load. ~~~ python full_length_sum = full_length_low_loss_sum + full_length_high_loss_sum @@ -3249,7 +2742,7 @@ quantile_loss_ratio = None for trial in full_length_trials: if quantile_loss_ratio is None or remaining > 0.0: quantile_loss_ratio = trial.loss_ratio - remaining -= trial.duration + remaining -= trial.effective_duration else: break else: @@ -3294,7 +2787,7 @@ conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) - Measurer: introduced in [Overview ](#overview), defined in [Meaurer ](#measurer). - Min Load: defined in [Min Load](#min-load). - MLRsearch Specification: introduced in [Purpose and Scope](#purpose-and-scope) -and in [Overview ](#overview), defined in [Test Procedure Compliant with MLRsearch](#test-procedure-compliant-with-mlrsearch). + and in [Overview ](#overview), defined in [Test Procedure Compliant with MLRsearch](#test-procedure-compliant-with-mlrsearch). - MLRsearch Implementation: defined in [Test Procedure Compliant with MLRsearch](#test-procedure-compliant-with-mlrsearch). - Offered Load: defined in [RFC2285] (Section 3.5.2). - Regular Goal Result: defined in [Regular Goal Result](#regular-goal-result). @@ -3318,34 +2811,26 @@ and in [Overview ](#overview), defined in [Test Procedure Compliant with MLRsear - Trial Load: defined in [Trial Load](#trial-load). - Trial Output: defined in [Trial Output](#trial-output). - Trial Result: defined in [Trial Result](#trial-result). +- Undecided: defined in [Undecided ](#undecided). - Upper Bound: defined in [Upper Bound](#upper-bound). -{::comment} - - - Test Procedure: defined in [RFC2544] (Section 26), TODO-P3: That lists several procedures in subsection, - but does not define what "a test procedure" is. - - Test Report: defined in [RFC2544] (Section 26), TODO-P3: Lists reporting formats without actually defining what the report is. - - Tester: defined in [RFC2544] (Section 6), TODO-P3: Not used enough to be in Glossary. - -{:/comment} - --- back {::comment} [Final checklist.] <mark>[VP] Final Checks. Only mark as done when there are no active todos above.</mark> - + <mark>[VP] Rename chapter/sub-/section to better match their content.</mark> - + <mark>MKP3 [VP] TODO: Recheck the definition dependencies go bottom-up.</mark> - + <mark>[VP] TODO: Unify external reference style (brackets, spaces, section numbers and names).</mark> - + <mark>MKP2 [VP] TODO: Capitalization of New Terms: useful when editing and reviewing, but I still vote to remove capitalization before final submit, because all other RFCs I see only capitalize due to being section title.</mark> - + <mark>[VP] TODO: If time permits, keep improving formal style (e.g. using AI).</mark> {:/comment} diff --git a/docs/ietf/process.txt b/docs/ietf/process.txt index f1545e96e8..d9945cedb6 100644 --- a/docs/ietf/process.txt +++ b/docs/ietf/process.txt @@ -1,4 +1,4 @@ -# Copyright (c) 2024 Cisco and/or its affiliates. +# Copyright (c) 2025 Cisco and/or its affiliates. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at: @@ -25,7 +25,7 @@ $ sudo gem install kramdown-rfc $ kdrfc --version Main: -$ kdrfc draft-ietf-bmwg-mlrsearch-08.md +$ kdrfc draft-ietf-bmwg-mlrsearch-09.md If that complains, do it manually at https://author-tools.ietf.org/ |