diff options
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt | 2800 | ||||
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml | 3136 | ||||
| -rw-r--r-- | docs/ietf/draft-ietf-bmwg-mlrsearch-08.md (renamed from docs/ietf/draft-ietf-bmwg-mlrsearch-07.md) | 4 |
3 files changed, 2 insertions, 5938 deletions
diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt deleted file mode 100644 index c5c94410a8..0000000000 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.txt +++ /dev/null @@ -1,2800 +0,0 @@ - - - - -Benchmarking Working Group M. Konstantynowicz -Internet-Draft V. Polak -Intended status: Informational Cisco Systems -Expires: 18 January 2025 18 July 2024 - - - Multiple Loss Ratio Search - draft-ietf-bmwg-mlrsearch-07 - -Abstract - - This document proposes extensions to [RFC2544] throughput search by - defining a new methodology called Multiple Loss Ratio search - (MLRsearch). MLRsearch aims to minimize search duration, support - multiple loss ratio searches, and enhance result repeatability and - comparability. - - The primary reason for extending [RFC2544] is to address the - challenges and requirements presented by the evaluation and testing - of software-based networking systems' data planes. - - To give users more freedom, MLRsearch provides additional - configuration options such as allowing multiple short trials per load - instead of one large trial, tolerating a certain percentage of trial - results with higher loss, and supporting the search for multiple - goals with varying loss ratios. - -Status of This Memo - - This Internet-Draft is submitted in full conformance with the - provisions of BCP 78 and BCP 79. - - Internet-Drafts are working documents of the Internet Engineering - Task Force (IETF). Note that other groups may also distribute - working documents as Internet-Drafts. The list of current Internet- - Drafts is at https://datatracker.ietf.org/drafts/current/. - - Internet-Drafts are draft documents valid for a maximum of six months - and may be updated, replaced, or obsoleted by other documents at any - time. It is inappropriate to use Internet-Drafts as reference - material or to cite them other than as "work in progress." - - This Internet-Draft will expire on 18 January 2025. - -Copyright Notice - - Copyright (c) 2024 IETF Trust and the persons identified as the - document authors. All rights reserved. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 1] - -Internet-Draft MLRsearch July 2024 - - - This document is subject to BCP 78 and the IETF Trust's Legal - Provisions Relating to IETF Documents (https://trustee.ietf.org/ - license-info) in effect on the date of publication of this document. - Please review these documents carefully, as they describe your rights - and restrictions with respect to this document. Code Components - extracted from this document must include Revised BSD License text as - described in Section 4.e of the Trust Legal Provisions and are - provided without warranty as described in the Revised BSD License. - -Table of Contents - - 1. Purpose and Scope . . . . . . . . . . . . . . . . . . . . . . 4 - 2. Identified Problems . . . . . . . . . . . . . . . . . . . . . 5 - 2.1. Long Search Duration . . . . . . . . . . . . . . . . . . 5 - 2.2. DUT in SUT . . . . . . . . . . . . . . . . . . . . . . . 6 - 2.3. Repeatability and Comparability . . . . . . . . . . . . . 8 - 2.4. Throughput with Non-Zero Loss . . . . . . . . . . . . . . 8 - 2.5. Inconsistent Trial Results . . . . . . . . . . . . . . . 9 - 3. MLRsearch Specification . . . . . . . . . . . . . . . . . . . 10 - 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 10 - 3.2. Measurement Quantities . . . . . . . . . . . . . . . . . 11 - 3.3. Existing Terms . . . . . . . . . . . . . . . . . . . . . 12 - 3.3.1. SUT . . . . . . . . . . . . . . . . . . . . . . . . . 12 - 3.3.2. DUT . . . . . . . . . . . . . . . . . . . . . . . . . 12 - 3.3.3. Trial . . . . . . . . . . . . . . . . . . . . . . . . 12 - 3.4. Trial Terms . . . . . . . . . . . . . . . . . . . . . . . 13 - 3.4.1. Trial Duration . . . . . . . . . . . . . . . . . . . 14 - 3.4.2. Trial Load . . . . . . . . . . . . . . . . . . . . . 14 - 3.4.3. Trial Input . . . . . . . . . . . . . . . . . . . . . 15 - 3.4.4. Traffic Profile . . . . . . . . . . . . . . . . . . . 15 - 3.4.5. Trial Forwarding Ratio . . . . . . . . . . . . . . . 16 - 3.4.6. Trial Loss Ratio . . . . . . . . . . . . . . . . . . 16 - 3.4.7. Trial Forwarding Rate . . . . . . . . . . . . . . . . 17 - 3.4.8. Trial Effective Duration . . . . . . . . . . . . . . 17 - 3.4.9. Trial Output . . . . . . . . . . . . . . . . . . . . 18 - 3.4.10. Trial Result . . . . . . . . . . . . . . . . . . . . 18 - 3.5. Goal Terms . . . . . . . . . . . . . . . . . . . . . . . 19 - 3.5.1. Goal Final Trial Duration . . . . . . . . . . . . . . 19 - 3.5.2. Goal Duration Sum . . . . . . . . . . . . . . . . . . 19 - 3.5.3. Goal Loss Ratio . . . . . . . . . . . . . . . . . . . 20 - 3.5.4. Goal Exceed Ratio . . . . . . . . . . . . . . . . . . 20 - 3.5.5. Goal Width . . . . . . . . . . . . . . . . . . . . . 21 - 3.5.6. Search Goal . . . . . . . . . . . . . . . . . . . . . 21 - 3.5.7. Controller Input . . . . . . . . . . . . . . . . . . 22 - 3.6. Search Goal Examples . . . . . . . . . . . . . . . . . . 23 - 3.6.1. RFC2544 Goal . . . . . . . . . . . . . . . . . . . . 23 - 3.6.2. TST009 Goal . . . . . . . . . . . . . . . . . . . . . 24 - 3.7. Result Terms . . . . . . . . . . . . . . . . . . . . . . 24 - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 2] - -Internet-Draft MLRsearch July 2024 - - - 3.7.1. Relevant Upper Bound . . . . . . . . . . . . . . . . 25 - 3.7.2. Relevant Lower Bound . . . . . . . . . . . . . . . . 25 - 3.7.3. Conditional Throughput . . . . . . . . . . . . . . . 26 - 3.7.4. Goal Result . . . . . . . . . . . . . . . . . . . . . 26 - 3.7.5. Search Result . . . . . . . . . . . . . . . . . . . . 27 - 3.7.6. Controller Output . . . . . . . . . . . . . . . . . . 27 - 3.8. MLRsearch Architecture . . . . . . . . . . . . . . . . . 28 - 3.8.1. Measurer . . . . . . . . . . . . . . . . . . . . . . 28 - 3.8.2. Controller . . . . . . . . . . . . . . . . . . . . . 29 - 3.8.3. Manager . . . . . . . . . . . . . . . . . . . . . . . 29 - 3.9. Implementation Compliance . . . . . . . . . . . . . . . . 30 - 4. Additional Considerations . . . . . . . . . . . . . . . . . . 30 - 4.1. MLRsearch Versions . . . . . . . . . . . . . . . . . . . 31 - 4.2. Stopping Conditions . . . . . . . . . . . . . . . . . . . 31 - 4.3. Load Classification . . . . . . . . . . . . . . . . . . . 32 - 4.4. Loss Ratios . . . . . . . . . . . . . . . . . . . . . . . 32 - 4.5. Loss Inversion . . . . . . . . . . . . . . . . . . . . . 33 - 4.6. Exceed Ratio . . . . . . . . . . . . . . . . . . . . . . 34 - 4.7. Duration Sum . . . . . . . . . . . . . . . . . . . . . . 34 - 4.8. Short Trials . . . . . . . . . . . . . . . . . . . . . . 35 - 4.9. Throughput . . . . . . . . . . . . . . . . . . . . . . . 35 - 4.10. Search Time . . . . . . . . . . . . . . . . . . . . . . . 37 - 4.11. RFC2544 Compliance . . . . . . . . . . . . . . . . . . . 38 - 5. Logic of Load Classification . . . . . . . . . . . . . . . . 38 - 5.1. Introductory Remarks . . . . . . . . . . . . . . . . . . 38 - 5.2. Performance Spectrum . . . . . . . . . . . . . . . . . . 38 - 5.2.1. First Example . . . . . . . . . . . . . . . . . . . . 39 - 5.2.2. Second Example . . . . . . . . . . . . . . . . . . . 40 - 5.2.3. Third Example . . . . . . . . . . . . . . . . . . . . 40 - 5.2.4. Summary . . . . . . . . . . . . . . . . . . . . . . . 40 - 5.3. Trials with Single Duration . . . . . . . . . . . . . . . 40 - 5.4. Trials with Short Duration . . . . . . . . . . . . . . . 42 - 5.4.1. Scenarios . . . . . . . . . . . . . . . . . . . . . . 42 - 5.4.2. Classification Logic . . . . . . . . . . . . . . . . 43 - 5.5. Trials with Longer Duration . . . . . . . . . . . . . . . 45 - 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 45 - 7. Security Considerations . . . . . . . . . . . . . . . . . . . 45 - 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 46 - 9. Appendix A: Load Classification . . . . . . . . . . . . . . . 46 - 10. Appendix B: Conditional Throughput . . . . . . . . . . . . . 47 - 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 49 - 11.1. Normative References . . . . . . . . . . . . . . . . . . 49 - 11.2. Informative References . . . . . . . . . . . . . . . . . 49 - Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 49 - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 3] - -Internet-Draft MLRsearch July 2024 - - -1. Purpose and Scope - - The purpose of this document is to describe Multiple Loss Ratio - search (MLRsearch), a data plane throughput search methodology - optimized for software networking DUTs. - - Applying vanilla [RFC2544] throughput bisection to software DUTs - results in several problems: - - * Binary search takes too long as most trials are done far from the - eventually found throughput. - - * The required final trial duration and pauses between trials - prolong the overall search duration. - - * Software DUTs show noisy trial results, leading to a big spread of - possible discovered throughput values. - - * Throughput requires a loss of exactly zero frames, but the - industry frequently allows for small but non-zero losses. - - * The definition of throughput is not clear when trial results are - inconsistent. - - To address the problems mentioned above, the MLRsearch test - methodology specification employs the following enhancements: - - * Allow multiple short trials instead of one big trial per load. - - - Optionally, tolerate a percentage of trial results with higher - loss. - - * Allow searching for multiple Search Goals, with differing loss - ratios. - - - Any trial result can affect each Search Goal in principle. - - * Insert multiple coarse targets for each Search Goal, earlier ones - need to spend less time on trials. - - - Earlier targets also aim for lesser precision. - - - Use Forwarding Rate (FR) at maximum offered load [RFC2285] - (section 3.6.2) to initialize the initial targets. - - * Take care when dealing with inconsistent trial results. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 4] - -Internet-Draft MLRsearch July 2024 - - - - Reported throughput is smaller than the smallest load with high - loss. - - - Smaller load candidates are measured first. - - * Apply several load selection heuristics to save even more time by - trying hard to avoid unnecessarily narrow bounds. - - Some of these enhancements are formalized as MLRsearch specification, - the remaining enhancements are treated as implementation details, - thus achieving high comparability without limiting future - improvements. - - MLRsearch configuration options are flexible enough to support both - conservative settings and aggressive settings. The conservative - settings lead to results unconditionally compliant with [RFC2544], - but longer search duration and worse repeatability. Conversely, - aggressive settings lead to shorter search duration and better - repeatability, but the results are not compliant with [RFC2544]. - - No part of [RFC2544] is intended to be obsoleted by this document. - -2. Identified Problems - - This chapter describes the problems affecting usability of various - performance testing methodologies, mainly a binary search for - [RFC2544] unconditionally compliant throughput. - -2.1. Long Search Duration - - The emergence of software DUTs, with frequent software updates and a - number of different frame processing modes and configurations, has - increased both the number of performance tests required to verify the - DUT update and the frequency of running those tests. This makes the - overall test execution time even more important than before. - - The current [RFC2544] throughput definition restricts the potential - for time-efficiency improvements. A more generalized throughput - concept could enable further enhancements while maintaining the - precision of simpler methods. - - The bisection method, when unconditionally compliant with [RFC2544], - is excessively slow. This is because a significant amount of time is - spent on trials with loads that, in retrospect, are far from the - final determined throughput. - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 5] - -Internet-Draft MLRsearch July 2024 - - - [RFC2544] does not specify any stopping condition for throughput - search, so users already have an access to a limited trade-off - between search duration and achieved precision. However, each full - 60-second trials doubles the precision, so not many trials can be - removed without a substantial loss of precision. - -2.2. DUT in SUT - - [RFC2285] defines: - DUT as - The network forwarding device to which - stimulus is offered and response measured [RFC2285] (section 3.1.1). - - SUT as - The collective set of network devices to which stimulus is - offered as a single entity and response measured [RFC2285] (section - 3.1.2). - - [RFC2544] specifies a test setup with an external tester stimulating - the networking system, treating it either as a single DUT, or as a - system of devices, an SUT. - - In the case of software networking, the SUT consists of not only the - DUT as a software program processing frames, but also of server - hardware and operating system functions, with that server hardware - resources shared across all programs including the operating system. - - Given that the SUT is a shared multi-tenant environment encompassing - the DUT and other components, the DUT might inadvertently experience - interference from the operating system or other software operating on - the same server. - - Some of this interference can be mitigated. For instance, pinning - DUT program threads to specific CPU cores and isolating those cores - can prevent context switching. - - Despite taking all feasible precautions, some adverse effects may - still impact the DUT's network performance. In this document, these - effects are collectively referred to as SUT noise, even if the - effects are not as unpredictable as what other engineering - disciplines call noise. - - DUT can also exhibit fluctuating performance itself, for reasons not - related to the rest of SUT. For example due to pauses in execution - as needed for internal stateful processing. In many cases this may - be an expected per-design behavior, as it would be observable even in - a hypothetical scenario where all sources of SUT noise are - eliminated. Such behavior affects trial results in a way similar to - SUT noise. As the two phenomenons are hard to distinguish, in this - document the term 'noise' is used to encompass both the internal - performance fluctuations of the DUT and the genuine noise of the SUT. - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 6] - -Internet-Draft MLRsearch July 2024 - - - A simple model of SUT performance consists of an idealized noiseless - performance, and additional noise effects. For a specific SUT, the - noiseless performance is assumed to be constant, with all observed - performance variations being attributed to noise. The impact of the - noise can vary in time, sometimes wildly, even within a single trial. - The noise can sometimes be negligible, but frequently it lowers the - observed SUT performance as observed in trial results. - - In this model, SUT does not have a single performance value, it has a - spectrum. One end of the spectrum is the idealized noiseless - performance value, the other end can be called a noiseful - performance. In practice, trial result close to the noiseful end of - the spectrum happens only rarely. The worse the performance value - is, the more rarely it is seen in a trial. Therefore, the extreme - noiseful end of the SUT spectrum is not observable among trial - results. Also, the extreme noiseless end of the SUT spectrum is - unlikely to be observable, this time because some small noise effects - are likely to occur multiple times during a trial. - - Unless specified otherwise, this document's focus is on the - potentially observable ends of the SUT performance spectrum, as - opposed to the extreme ones. - - When focusing on the DUT, the benchmarking effort should ideally aim - to eliminate only the SUT noise from SUT measurements. However, this - is currently not feasible in practice, as there are no realistic - enough models available to distinguish SUT noise from DUT - fluctuations, based on authors' experience and available literature. - - Assuming a well-constructed SUT, the DUT is likely its primary - performance bottleneck. In this case, we can define the DUT's ideal - noiseless performance as the noiseless end of the SUT performance - spectrum, especially for throughput. However, other performance - metrics, such as latency, may require additional considerations. - - Note that by this definition, DUT noiseless performance also - minimizes the impact of DUT fluctuations, as much as realistically - possible for a given trial duration. - - MLRsearch methodology aims to solve the DUT in SUT problem by - estimating the noiseless end of the SUT performance spectrum using a - limited number of trial results. - - Any improvements to the throughput search algorithm, aimed at better - dealing with software networking SUT and DUT setup, should employ - strategies recognizing the presence of SUT noise, allowing the - discovery of (proxies for) DUT noiseless performance at different - levels of sensitivity to SUT noise. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 7] - -Internet-Draft MLRsearch July 2024 - - -2.3. Repeatability and Comparability - - [RFC2544] does not suggest to repeat throughput search. And from - just one discovered throughput value, it cannot be determined how - repeatable that value is. Poor repeatability then leads to poor - comparability, as different benchmarking teams may obtain varying - throughput values for the same SUT, exceeding the expected - differences from search precision. - - [RFC2544] throughput requirements (60 seconds trial and no tolerance - of a single frame loss) affect the throughput results in the - following way. The SUT behavior close to the noiseful end of its - performance spectrum consists of rare occasions of significantly low - performance, but the long trial duration makes those occasions not so - rare on the trial level. Therefore, the binary search results tend - to wander away from the noiseless end of SUT performance spectrum, - more frequently and more widely than short trials would, thus causing - poor throughput repeatability. - - The repeatability problem can be addressed by defining a search - procedure that identifies a consistent level of performance, even if - it does not meet the strict definition of throughput in [RFC2544]. - - According to the SUT performance spectrum model, better repeatability - will be at the noiseless end of the spectrum. Therefore, solutions - to the DUT in SUT problem will help also with the repeatability - problem. - - Conversely, any alteration to [RFC2544] throughput search that - improves repeatability should be considered as less dependent on the - SUT noise. - - An alternative option is to simply run a search multiple times, and - report some statistics (e.g. average and standard deviation). This - can be used for a subset of tests deemed more important, but it makes - the search duration problem even more pronounced. - -2.4. Throughput with Non-Zero Loss - - [RFC1242] (section 3.17 Throughput) defines throughput as: The - maximum rate at which none of the offered frames are dropped by the - device. - - Then, it says: Since even the loss of one frame in a data stream can - cause significant delays while waiting for the higher level protocols - to time out, it is useful to know the actual maximum data rate that - the device can support. - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 8] - -Internet-Draft MLRsearch July 2024 - - - However, many benchmarking teams accept a small, non-zero loss ratio - as the goal for their load search. - - Motivations are many: - - * Modern protocols tolerate frame loss better, compared to the time - when [RFC1242] and [RFC2544] were specified. - - * Trials nowadays send way more frames within the same duration, - increasing the chance of a small SUT performance fluctuation being - enough to cause frame loss. - - * Small bursts of frame loss caused by noise have otherwise smaller - impact on the average frame loss ratio observed in the trial, as - during other parts of the same trial the SUT may work more closely - to its noiseless performance, thus perhaps lowering the Trial Loss - Ratio below the Goal Loss Ratio value. - - * If an approximation of the SUT noise impact on the Trial Loss - Ratio is known, it can be set as the Goal Loss Ratio. - - Regardless of the validity of all similar motivations, support for - non-zero loss goals makes any search algorithm more user-friendly. - [RFC2544] throughput is not user-friendly in this regard. - - Furthermore, allowing users to specify multiple loss ratio values, - and enabling a single search to find all relevant bounds, - significantly enhances the usefulness of the search algorithm. - - Searching for multiple Search Goals also helps to describe the SUT - performance spectrum better than the result of a single Search Goal. - For example, the repeated wide gap between zero and non-zero loss - loads indicates the noise has a large impact on the observed - performance, which is not evident from a single goal load search - procedure result. - - It is easy to modify the vanilla bisection to find a lower bound for - the intended load that satisfies a non-zero Goal Loss Ratio. But it - is not that obvious how to search for multiple goals at once, hence - the support for multiple Search Goals remains a problem. - -2.5. Inconsistent Trial Results - - While performing throughput search by executing a sequence of - measurement trials, there is a risk of encountering inconsistencies - between trial results. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 9] - -Internet-Draft MLRsearch July 2024 - - - The plain bisection never encounters inconsistent trials. But - [RFC2544] hints about the possibility of inconsistent trial results, - in two places in its text. The first place is section 24, where full - trial durations are required, presumably because they can be - inconsistent with the results from short trial durations. The second - place is section 26.3, where two successive zero-loss trials are - recommended, presumably because after one zero-loss trial there can - be a subsequent inconsistent non-zero-loss trial. - - Examples include: - - * A trial at the same load (same or different trial duration) - results in a different Trial Loss Ratio. - - * A trial at a higher load (same or different trial duration) - results in a smaller Trial Loss Ratio. - - Any robust throughput search algorithm needs to decide how to - continue the search in the presence of such inconsistencies. - Definitions of throughput in [RFC1242] and [RFC2544] are not specific - enough to imply a unique way of handling such inconsistencies. - - Ideally, there will be a definition of a new quantity which both - generalizes throughput for non-zero-loss (and other possible - repeatability enhancements), while being precise enough to force a - specific way to resolve trial result inconsistencies. But until such - a definition is agreed upon, the correct way to handle inconsistent - trial results remains an open problem. - -3. MLRsearch Specification - - This section describes MLRsearch specification including all - technical definitions needed for evaluating whether a particular test - procedure complies with MLRsearch specification. - -3.1. Overview - - MLRsearch specification describes a set of abstract system - components, acting as functions with specified inputs and outputs. - - A test procedure is said to comply with MLRsearch specification if it - can be conceptually divided into analogous components, each - satisfying requirements for the corresponding MLRsearch component. - - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 10] - -Internet-Draft MLRsearch July 2024 - - - The Measurer component is tasked to perform trials, the Controller - component is tasked to select trial loads and durations, the Manager - component is tasked to pre-configure everything and to produce the - test report. The test report explicitly states Search Goals (as the - Controller Inputs) and corresponding Goal Results (Controller - Outputs). - - The Manager calls the Controller once, the Controller keeps calling - the Measurer until all stopping conditions are met. - - The part where Controller calls the Measurer is called the search. - Any activity done by the Manager before it calls the Controller (or - after Controller returns) is not considered to be part of the search. - - MLRsearch specification prescribes regular search results and - recommends their stopping conditions. Irregular search results are - also allowed, they may have different requirements and stopping - conditions. - - Search results are based on load classification. When measured - enough, any chosen load either achieves of fails each search goal, - thus becoming a lower or an upper bound for that goal. When the - relevant bounds are at loads that are close enough (according to goal - precision), the regular result is found. Search stops when all - regular results are found (or if some goals are proven to have only - irregular results). - -3.2. Measurement Quantities - - MLRsearch specification uses a number of measurement quantities. - - In general, MLRsearch specification does not require particular units - to be used, but it is REQUIRED for the test report to state all the - units. For example, ratio quantities can be dimensionless numbers - between zero and one, but may be expressed as percentages instead. - - For convenience, a group of quantities can be treated as a composite - quantity, One constituent of a composite quantity is called an - attribute, and a group of attribute values is called an instance of - that composite quantity. - - Some attributes are not independent from others, and they can be - calculated from other attributes. Such quantites are called derived - quantities. - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 11] - -Internet-Draft MLRsearch July 2024 - - -3.3. Existing Terms - - RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" - contains basic definitions, and RFC 2544 "Benchmarking Methodology - for Network Interconnect Devices" contains discussions of a number of - terms and additional methodology requirements. RFC 2285 adds more - terms and discussions, describing some known situations in more - precise way. - - All three documents should be consulted before attempting to make use - of this document. - - Definitions of some central terms are copied and discussed in - subsections. - -3.3.1. SUT - - Defined in [RFC2285] (section 3.1.2 System Under Test (SUT)) as - follows. - - Definition: - - The collective set of network devices to which stimulus is offered as - a single entity and response measured. - - Discussion: - - An SUT consisting of a single network device is also allowed. - -3.3.2. DUT - - Defined in [RFC2285] (section 3.1.1 Device Under Test (DUT)) as - follows. - - Definition: - - The network forwarding device to which stimulus is offered and - response measured. - - Discussion: - - DUT, as a sub-component of SUT, is only indirectly mentioned in - MLRsearch specification, but is of key relevance for its motivation. - -3.3.3. Trial - - A trial is the part of the test described in [RFC2544] (section 23. - Trial description). - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 12] - -Internet-Draft MLRsearch July 2024 - - - Definition: - - A particular test consists of multiple trials. Each trial returns - one piece of information, for example the loss rate at a particular - input frame rate. Each trial consists of a number of phases: - - a) If the DUT is a router, send the routing update to the "input" - port and pause two seconds to be sure that the routing has settled. - - b) Send the "learning frames" to the "output" port and wait 2 seconds - to be sure that the learning has settled. Bridge learning frames are - frames with source addresses that are the same as the destination - addresses used by the test frames. Learning frames for other - protocols are used to prime the address resolution tables in the DUT. - The formats of the learning frame that should be used are shown in - the Test Frame Formats document. - - c) Run the test trial. - - d) Wait for two seconds for any residual frames to be received. - - e) Wait for at least five seconds for the DUT to restabilize. - - Discussion: - - The definition describes some traits, it is not clear whether all of - them are REQUIRED, or some of them are only RECOMMENDED. - - For the purposes of the MLRsearch specification, it is ALLOWED for - the test procedure to deviate from the [RFC2544] description, but any - such deviation MUST be made explicit in the test report. - - Trials are the only stimuli the SUT is expected to experience during - the search. - - In some discussion paragraphs, it is useful to consider the traffic - as sent and received by a tester, as implicitly defined in [RFC2544] - (section 6. Test set up). - - An example of deviation from [RFC2544] is using shorter wait times. - -3.4. Trial Terms - - This section defines new and redefine existing terms for quantities - relevant as inputs or outputs of trial, as used by the Measurer - component. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 13] - -Internet-Draft MLRsearch July 2024 - - -3.4.1. Trial Duration - - Definition: - - Trial duration is the intended duration of the traffic for a trial. - - Discussion: - - In general, this quantity does not include any preparation nor - waiting described in section 23 of [RFC2544] (section 23. Trial - description). - - While any positive real value may be provided, some Measurer - implementations MAY limit possible values, e.g. by rounding down to - neared integer in seconds. In that case, it is RECOMMENDED to give - such inputs to the Controller so the Controller only proposes the - accepted values. Alternatively, the test report MUST present the - rounded values as Search Goal attributes. - -3.4.2. Trial Load - - Definition: - - The trial load is the intended load for a trial - - Discussion: - - For test report purposes, it is assumed that this is a constant load - by default. This MAY be only an average load, e.g. when the traffic - is intended to be busty, e.g. as suggested in [RFC2544] (section 21. - Bursty traffic), but the test report MUST explicitly mention how non- - constant the traffic is. - - Trial load is the quantity defined as Constant Load of [RFC1242] - (section 3.4 Constant Load), Data Rate of [RFC2544] (section 14. - Bidirectional traffic) and Intended Load of [RFC2285] (section 3.5.1 - Intended load (Iload)). All three definitions specify that this - value applies to one (input or output) interface. - - For test report purposes, multi-interface aggregate load MAY be - reported, this is understood as the same quantity expressed using - different units. From the report it MUST be clear whether a - particular trial load value is per one interface, or an aggregate - over all interfaces. - - Similarly to trial duration, some Measurers may limit the possible - values of trial load. Contrary to trial duration, the test report is - NOT REQUIRED to document such behavior. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 14] - -Internet-Draft MLRsearch July 2024 - - - It is ALLOWED to combine trial load and trial duration in a way that - would not be possible to achieve using any integer number of data - frames. - -3.4.3. Trial Input - - Definition: - - Trial Input is a composite quantity, consisting of two attributes: - trial duration and trial load. - - Discussion: - - When talking about multiple trials, it is common to say "Trial - Inputs" to denote all corresponding Trial Input instances. - - A Trial Input instance acts as the input for one call of the Measurer - component. - - Contrary to other composite quantities, MLRsearch implementations are - NOT ALLOWED to add optional attributes here. This improves - interoperability between various implementations of the Controller - and the Measurer. - -3.4.4. Traffic Profile - - Definition: - - Traffic profile is a composite quantity containing attributes other - than trial load and trial duration, needed for unique determination - of the trial to be performed. - - Discussion: - - All its attributes are assumed to be constant during the search, and - the composite is configured on the Measurer by the Manager before the - search starts. This is why the traffic profile is not part of the - Trial Input. - - As a consequence, implementations of the Manager and the Measurer - must be aware of their common set of capabilities, so that the - traffic profile uniquely defines the traffic during the search. The - important fact is that none of those capabilities have to be known by - the Controller implementations. - - The traffic profile SHOULD contain some specific quantities, for - example [RFC2544] (section 9. Frame sizes) governs data link frame - size as defined in [RFC1242] (section 3.5 Data link frame size). - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 15] - -Internet-Draft MLRsearch July 2024 - - - Several more specific quantities may be RECOMMENDED, depending on - media type. For example, [RFC2544] (Appendix C) lists frame formats - and protocol addresses, as recommended from [RFC2544] (section 8. - Frame formats) and [RFC2544] (section 12. Protocol addresses). - - Depending on SUT configuration, e.g. when testing specific protocols, - additional attributes MUST be included in the traffic profile and in - the test report. - - Example: [RFC8219] (section 5.3. Traffic Setup) introduces traffic - setups consisting of a mix of IPv4 and IPv6 traffic - the implied - traffic profile therefore must include an attribute for their - percentage. - - Other traffic properties that need to be somehow specified in Traffic - Profile include: [RFC2544] (section 14. Bidirectional traffic), - [RFC2285] (section 3.3.3 Fully meshed traffic), and [RFC2544] - (section 11. Modifiers). - -3.4.5. Trial Forwarding Ratio - - Definition: - - The trial forwarding ratio is a dimensionless floating point value. - It MUST range between 0.0 and 1.0, both inclusive. It is calculated - by dividing the number of frames successfully forwarded by the SUT by - the total number of frames expected to be forwarded during the trial - - Discussion: - - For most traffic profiles, "expected to be forwarded" means "intended - to get transmitted from Tester towards SUT". - - Trial forwarding ratio MAY be expressed in other units (e.g. as a - percentage) in the test report. - - Note that, contrary to loads, frame counts used to compute trial - forwarding ratio are aggregates over all SUT output interfaces. - - Questions around what is the correct number of frames that should - have been forwarded is generally outside of the scope of this - document. - -3.4.6. Trial Loss Ratio - - Definition: - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 16] - -Internet-Draft MLRsearch July 2024 - - - The Trial Loss Ratio is equal to one minus the trial forwarding - ratio. - - Discussion: - - 100% minus the trial forwarding ratio, when expressed as a - percentage. - - This is almost identical to Frame Loss Rate of [RFC1242] (section 3.6 - Frame Loss Rate), the only minor difference is that Trial Loss Ratio - does not need to be expressed as a percentage. - -3.4.7. Trial Forwarding Rate - - Definition: - - The trial forwarding rate is a derived quantity, calculated by - multiplying the trial load by the trial forwarding ratio. - - Discussion: - - It is important to note that while similar, this quantity is not - identical to the Forwarding Rate as defined in [RFC2285] (section - 3.6.1 Forwarding rate (FR)). The latter is specific to one output - interface only, whereas the trial forwarding ratio is based on frame - counts aggregated over all SUT output interfaces. - -3.4.8. Trial Effective Duration - - Definition: - - Trial effective duration is a time quantity related to the trial, by - default equal to the trial duration. - - Discussion: - - This is an optional feature. If the Measurer does not return any - trial effective duration value, the Controller MUST use the trial - duration value instead. - - Trial effective duration may be any time quantity chosen by the - Measurer to be used for time-based decisions in the Controller. - - The test report MUST explain how the Measurer computes the returned - trial effective duration values, if they are not always equal to the - trial duration. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 17] - -Internet-Draft MLRsearch July 2024 - - - This feature can be beneficial for users who wish to manage the - overall search duration, rather than solely the traffic portion of - it. Simply measure the duration of the whole trial (waits including) - and use that as the trial effective duration. - - Also, this is a way for the Measurer to inform the Controller about - its surprising behavior, for example when rounding the trial duration - value. - -3.4.9. Trial Output - - Definition: - - Trial Output is a composite quantity. The REQUIRED attributes are - Trial Loss Ratio, trial effective duration and trial forwarding rate. - - Discussion: - - When talking about multiple trials, it is common to say "Trial - Outputs" to denote all corresponding Trial Output instances. - - Implementations may provide additional (optional) attributes. The - Controller implementations MUST ignore values of any optional - attribute they are not familiar with, except when passing Trial - Output instance to the Manager. - - Example of an optional attribute: The aggregate number of frames - expected to be forwarded during the trial, especially if it is not - just (a rounded-up value) implied by trial load and trial duration. - - While [RFC2285] (Section 3.5.2 Offered load (Oload)) requires the - offered load value to be reported for forwarding rate measurements, - it is NOT REQUIRED in MLRsearch specification. - -3.4.10. Trial Result - - Definition: - - Trial result is a composite quantity, consisting of the Trial Input - and the Trial Output. - - Discussion: - - When talking about multiple trials, it is common to say "trial - results" to denote all corresponding trial result instances. - - While implementations SHOULD NOT include additional attributes with - independent values, they MAY include derived quantities. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 18] - -Internet-Draft MLRsearch July 2024 - - -3.5. Goal Terms - - This section defines new and redefine existing terms for quantities - indirectly relevant for inputs or outputs of the Controller - component. - - Several goal attributes are defined before introducing the main - component quantity: the Search Goal. - -3.5.1. Goal Final Trial Duration - - Definition: - - A threshold value for trial durations. - - Discussion: - - This attribute value MUST be positive. - - A trial with Trial Duration at least as long as the Goal Final Trial - Duration is called a full-length trial (with respect to the given - Search Goal). - - A trial that is not full-length is called a short trial. - - Informally, while MLRsearch is allowed to perform short trials, the - results from such short trials have only limited impact on search - results. - - One trial may be full-length for some Search Goals, but not for - others. - - The full relation of this goal to Controller Output is defined later - in this document in subsections of [Goal Result] (#Goal-Result). For - example, the Conditional Throughput for this goal is computed only - from full-length trial results. - -3.5.2. Goal Duration Sum - - Definition: - - A threshold value for a particular sum of trial effective durations. - - Discussion: - - This attribute value MUST be positive. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 19] - -Internet-Draft MLRsearch July 2024 - - - Informally, even when looking only at full-length trials, MLRsearch - may spend up to this time measuring the same load value. - - If the Goal Duration Sum is larger than the Goal Final Trial - Duration, multiple full-length trials may need to be performed at the - same load. - - See [TST009 Example] (#TST009-Example) for an example where - possibility of multiple full-length trials at the same load is - intended. - - A Goal Duration Sum value lower than the Goal Final Trial Duration - (of the same goal) could save some search time, but is NOT - RECOMMENDED. See [Relevant Upper Bound] (#Relevant-Upper-Bound) for - partial explanation. - -3.5.3. Goal Loss Ratio - - Definition: - - A threshold value for Trial Loss Ratios. - - Discussion: - - Attribute value MUST be non-negative and smaller than one. - - A trial with Trial Loss Ratio larger than a Goal Loss Ratio value is - called a lossy trial, with respect to given Search Goal. - - Informally, if a load causes too many lossy trials, the Relevant - Lower Bound for this goal will be smaller than that load. - - If a trial is not lossy, it is called a low-loss trial, or - (specifically for zero Goal Loss Ratio value) zero-loss trial. - -3.5.4. Goal Exceed Ratio - - Definition: - - A threshold value for a particular ratio of sums of Trial Effective - Durations. - - Discussion: - - Attribute value MUST be non-negative and smaller than one. - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 20] - -Internet-Draft MLRsearch July 2024 - - - See later sections for details on which sums. Specifically, the - direct usage is only in [Appendix A: Load Classification] (#Appendix- - A:-Load-Classification) and [Appendix B: Conditional Throughput] - (#Appendix-B:-Conditional-Throughput). The impact of that usage is - discussed in subsections leading to [Goal Result] (#Goal-Result). - - Informally, the impact of lossy trials is controlled by this value. - Effectively, Goal Exceed Ratio is a percentage of full-length trials - that may be lossy without the load being classified as the [Relevant - Upper Bound] (#Relevant-Upper-Bound). - -3.5.5. Goal Width - - Definition: - - A value used as a threshold for deciding whether two trial load - values are close enough. - - Discussion: - - If present, the value MUST be positive. - - Informally, this acts as a stopping condition, controlling the - precision of the search. The search stops if every goal has reached - its precision. - - Implementations without this attribute MUST give the Controller other - ways to control the search stopping conditions. - - Absolute load difference and relative load difference are two popular - choices, but implementations may choose a different way to specify - width. - - The test report MUST make it clear what specific quantity is used as - Goal Width. - - It is RECOMMENDED to set the Goal Width (as relative difference) - value to a value no smaller than the Goal Loss Ratio. (The reason is - not obvious, see [Throughput] (#Throughput) if interested.) - -3.5.6. Search Goal - - Definition: - - The Search Goal is a composite quantity consisting of several - attributes, some of them are required. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 21] - -Internet-Draft MLRsearch July 2024 - - - Required attributes: - Goal Final Trial Duration - Goal Duration Sum - - Goal Loss Ratio - Goal Exceed Ratio - - Optional attribute: - Goal Width - - Discussion: - - Implementations MAY add their own attributes. Those additional - attributes may be required by the implementation even if they are not - required by MLRsearch specification. But it is RECOMMENDED for those - implementations to support missing values by computing reasonable - defaults. - - The meaning of listed attributes is formally given only by their - indirect effect on the search results. - - Informally, later sections provide additional intuitions and examples - of the Search Goal attribute values. - - An example of additional attributes required by some implementations - is Goal Initial Trial Duration, together with another attribute that - controls possible intermediate Trial Duration values. The reasonable - default in this case is using the Goal Final Trial Duration and no - intermediate values. - -3.5.7. Controller Input - - Definition: - - Controller Input is a composite quantity required as an input for the - Controller. The only REQUIRED attribute is a list of Search Goal - instances. - - Discussion: - - MLRsearch implementations MAY use additional attributes. Those - additional attributes may be required by the implementation even if - they are not required by MLRsearch specification. - - Formally, the Manager does not apply any Controller configuration - apart from one Controller Input instance. - - For example, Traffic Profile is configured on the Measurer by the - Manager (without explicit assistance of the Controller). - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 22] - -Internet-Draft MLRsearch July 2024 - - - The order of Search Goal instances in a list SHOULD NOT have a big - impact on Controller Output (see section [Controller Output] - (#Controller-Output) , but MLRsearch implementations MAY base their - behavior on the order of Search Goal instances in a list. - - An example of an optional attribute (outside the list of Search - Goals) required by some implementations is Max Load. While this is a - frequently used configuration parameter, already governed by - [RFC2544] (section 20. Maximum frame rate) and [RFC2285] (3.5.3 - Maximum offered load (MOL)), some implementations may detect or - discover it instead. - - In MLRsearch specification, the [Relevant Upper Bound] (#Relevant- - Upper-Bound) is added as a required attribute precisely because it - makes the search result independent of Max Load value. - -3.6. Search Goal Examples - -3.6.1. RFC2544 Goal - - The following set of values makes the search result unconditionally - compliant with [RFC2544] (section 24 Trial duration) - - * Goal Final Trial Duration = 60 seconds - - * Goal Duration Sum = 60 seconds - - * Goal Loss Ratio = 0% - - * Goal Exceed Ratio = 0% - - The latter two attributes are enough to make the search goal - conditionally compliant, adding the first attribute makes it - unconditionally compliant. - - The second attribute (Goal Duration Sum) only prevents MLRsearch from - repeating zero-loss full-length trials. - - Non-zero exceed ratio could prolong the search and allow loss - inversion between lower-load lossy short trial and higher-load full- - length zero-loss trial. From [RFC2544] alone, it is not clear - whether that higher load could be considered as compliant throughput. - - - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 23] - -Internet-Draft MLRsearch July 2024 - - -3.6.2. TST009 Goal - - One of the alternatives to RFC2544 is described in [TST009] (section - 12.3.3 Binary search with loss verification). The idea there is to - repeat lossy trials, hoping for zero loss on second try, so the - results are closer to the noiseless end of performance sprectum, and - more repeatable and comparable. - - Only the variant with "z = infinity" is achievable with MLRsearch. - - For example, for "r = 2" variant, the following search goal should be - used: - - * Goal Final Trial Duration = 60 seconds - - * Goal Duration Sum = 120 seconds - - * Goal Loss Ratio = 0% - - * Goal Exceed Ratio = 50% - - If the first 60s trial has zero loss, it is enough for MLRsearch to - stop measuring at that load, as even a second lossy trial would still - fit within the exceed ratio. - - But if the first trial is lossy, MLRsearch needs to perform also the - second trial to classify that load. As Goal Duration Sum is twice as - long as Goal Final Trial Duration, third full-length trial is never - needed. - -3.7. Result Terms - - Before defining the output of the Controller, it is useful to define - what the Goal Result is. - - The Goal Result is a composite quantity. - - Following subsections define its attribute first, before describing - the Goal Result quantity. - - There is a correspondence between Search Goals and Goal Results. - Most of the following subsections refer to a given Search Goal, when - defining attributes of the Goal Result. Conversely, at the end of - the search, each Search Goal has its corresponding Goal Result. - - Conceptually, the search can be seen as a process of load - classification, where the Controller attempts to classify some loads - as an Upper Bound or a Lower Bound with respect to some Search Goal. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 24] - -Internet-Draft MLRsearch July 2024 - - - Before defining real attributes of the goal result, it is useful to - define bounds in general. - -3.7.1. Relevant Upper Bound - - Definition: - - The Relevant Upper Bound is the smallest trial load value that is - classified at the end of the search as an upper bound (see - [Appendix A: Load Classification] (#Appendix-A:-Load-Classification)) - for the given Search Goal. - - Discussion: - - One search goal can have many different load classified as an upper - bound. At the end of the search, one of those loads will be the - smallest, becoming the relevant upper bound for that goal. - - In more detail, the set of all trial outputs (both short and full- - length, enough of them according to Goal Duration Sum) performed at - that smallest load failed to uphold all the requirements of the given - Search Goal, mainly the Goal Loss Ratio in combination with the Goal - Exceed Ratio. - - If Max Load does not cause enough lossy trials, the Relevant Upper - Bound does not exist. Conversely, if Relevant Upper Bound exists, it - is not affected by Max Load value. - -3.7.2. Relevant Lower Bound - - Definition: - - The Relevant Lower Bound is the largest trial load value among those - smaller than the Relevant Upper Bound, that got classified at the end - of the search as a lower bound (see [Appendix A: Load Classification] - (#Appendix-A:-Load-Classification)) for the given Search Goal. - - Discussion: - - Only among loads smaller that the relevant upper bound, the largest - load becomes the relevant lower bound. With loss inversion, stricter - upper bound matters. - - In more detail, the set of all trial outputs (both short and full- - length, enough of them according to Goal Duration Sum) performed at - that largest load managed to uphold all the requirements of the given - Search Goal, mainly the Goal Loss Ratio in combination with the Goal - Exceed Ratio. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 25] - -Internet-Draft MLRsearch July 2024 - - - Is no load had enough low-loss trials, the relevant lower bound MAY - not exist. - - Strictly speaking, if the Relevant Upper Bound does not exist, the - Relevant Lower Bound also does not exist. In that case, Max Load is - classified as a lower bound, but it is not clear whether a higher - lower bound would be found if the search used a higher Max Load - value. - - For a regular Goal Result, the distance between the Relevant Lower - Bound and the Relevant Upper Bound MUST NOT be larger than the Goal - Width, if the implementation offers width as a goal attribute. - - Searching for anther search goal may cause a loss inversion - phenomenon, where a lower load is classified as an upper bound, but - also a higher load is classified as a lower bound for the same search - goal. The definition of the Relevant Lower Bound ignores such high - lower bounds. - -3.7.3. Conditional Throughput - - Definition: - - The Conditional Throughput (see section [Appendix B: Conditional - Throughput] (#Appendix-B:-Conditional-Throughput)) as evaluated at - the Relevant Lower Bound of the given Search Goal at the end of the - search. - - Discussion: - - Informally, this is a typical trial forwarding rate, expected to be - seen at the Relevant Lower Bound of the given Search Goal. - - But frequently it is only a conservative estimate thereof, as - MLRsearch implementations tend to stop gathering more data as soon as - they confirm the value cannot get worse than this estimate within the - Goal Duration Sum. - - This value is RECOMMENDED to be used when evaluating repeatability - and comparability if different MLRsearch implementations. - -3.7.4. Goal Result - - Definition: - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 26] - -Internet-Draft MLRsearch July 2024 - - - The Goal Result is a composite quantity consisting of several - attributes. Relevant Upper Bound and Relevant Lower Bound are - REQUIRED attributes, Conditional Throughput is a RECOMMENDED - attribute. - - Discussion: - - Depending on SUT behavior, it is possible that one or both relevant - bounds do not exist. The goal result instance where the required - attribute values exist is informally called a Regular Goal Result - instance, so we can say some goals reached Irregular Goal Results. - - A typical Irregular Goal Result is when all trials at the Max Load - have zero loss, as the Relevant Upper Bound does not exist in that - case. - - It is RECOMMENDED that the test report will display such results - appropriately, although MLRsearch specification does not prescibe - how. - - Anything else regarging Irregular Goal Results, including their role - in stopping conditions of the search is outside the scope of this - document. - -3.7.5. Search Result - - Definition: - - The Search Result is a single composite object that maps each Search - Goal instance to a corresponding Goal Result instance. - - Discussion: - - Alternatively, the Search Result can be implemented as an ordered - list of the Goal Result instances, matching the order of Search Goal - instances. - - The Search Result (as a mapping) MUST map from all the Search Goal - instances present in the Controller Input. - -3.7.6. Controller Output - - Definition: - - The Controller Output is a composite quantity returned from the - Controller to the Manager at the end of the search. The Search - Result instance is its only REQUIRED attribute. - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 27] - -Internet-Draft MLRsearch July 2024 - - - Discussion: - - MLRsearch implementation MAY return additional data in the Controller - Output. - -3.8. MLRsearch Architecture - - MLRsearch architecture consists of three main system components: the - Manager, the Controller, and the Measurer. - - The architecture also implies the presence of other components, such - as the SUT and the Tester (as a sub-component of the Measurer). - - Protocols of communication between components are generally left - unspecified. For example, when MLRsearch specification mentions - "Controller calls Measurer", it is possible that the Controller - notifies the Manager to call the Measurer indirectly instead. This - way the Measurer implementations can be fully independent from the - Controller implementations, e.g. programmed in different programming - languages. - -3.8.1. Measurer - - Definition: - - The Measurer is an abstract system component that when called with a - [Trial Input] (#Trial-Input) instance, performs one [Trial] (#Trial), - and returns a [Trial Output] (#Trial-Output) instance. - - Discussion: - - This definition assumes the Measurer is already initialized. In - practice, there may be additional steps before the search, e.g. when - the Manager configures the traffic profile (either on the Measurer or - on its tester sub-component directly) and performs a warmup (if the - tester requires one). - - It is the responsibility of the Measurer implementation to uphold any - requirements and assumptions present in MLRsearch specification, e.g. - trial forwarding ratio not being larger than one. - - Implementers have some freedom. For example [RFC2544] (section 10. - Verifying received frames) gives some suggestions (but not - requirements) related to duplicated or reordered frames. - Implementations are RECOMMENDED to document their behavior related to - such freedoms in as detailed a way as possible. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 28] - -Internet-Draft MLRsearch July 2024 - - - It is RECOMMENDED to benchmark the test equipment first, e.g. connect - sender and receiver directly (without any SUT in the path), find a - load value that guarantees the offered load is not too far from the - intended load, and use that value as the Max Load value. When - testing the real SUT, it is RECOMMENDED to turn any big difference - between the intended load and the offered load into increased Trial - Loss Ratio. - - Neither of the two recommendations are made into requirements, - because it is not easy to tell when the difference is big enough, in - a way thay would be dis-entangled from other Measurer freedoms. - -3.8.2. Controller - - Definition: - - The Controller is an abstract system component that when called with - a Controller Input instance repeatedly computes Trial Input instance - for the Measurer, obtains corresponding Trial Output instances, and - eventually returns a Controller Output instance. - - Discussion: - - Informally, the Controller has big freedom in selection of Trial - Inputs, and the implementations want to achieve the Search Goals in - the shortest expected time. - - The Controller's role in optimizing the overall search time - distinguishes MLRsearch algorithms from simpler search procedures. - - Informally, each implementation can have different stopping - conditions. Goal Width is only one example. In practice, - implementation details do not matter, as long as Goal Results are - regular. - -3.8.3. Manager - - Definition: - - The Manager is an abstract system component that is reponsible for - configuring other components, calling the Controller component once, - and for creating the test report following the reporting format as - defined in [RFC2544] (section 26. Benchmarking tests). - - Discussion: - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 29] - -Internet-Draft MLRsearch July 2024 - - - The Manager initializes the SUT, the Measurer (and the Tester if - independent) with their intended configurations before calling the - Controller. - - The Manager does not need to be able to tweak any Search Goal - attributes, but it MUST report all applied attribute values even if - not tweaked. - - In principle, there should be a "user" (human or CI) that "starts" or - "calls" the Manager and receives the report. The Manager MAY be able - to be called more than once whis way. - -3.9. Implementation Compliance - - Any networking measurement setup where there can be logically - delineated system components and there are components satisfying - requirements for the Measurer, the Controller and the Manager, is - considered to be compliant with MLRsearch design. - - These components can be seen as abstractions present in any testing - procedure. For example, there can be a single component acting both - as the Manager and the Controller, but as long as values of required - attributes of Search Goals and Goal Results are visible in the test - report, the Controller Input instance and output instance are - implied. - - For example, any setup for conditionally (or unconditionally) - compliant [RFC2544] throughput testing can be understood as a - MLRsearch architecture, assuming there is enough data to reconstruct - the Relevant Upper Bound. - - See [RFC2544 Goal] (#RFC2544-Goal) subsection for equivalent Search - Goal. - - Any test procedure that can be understood as (one call to the Manager - of) MLRsearch architecture is said to be compliant with MLRsearch - specification. - -4. Additional Considerations - - This section focuses on additional considerations, intuitions and - motivations pertaining to MLRsearch methodology. - - - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 30] - -Internet-Draft MLRsearch July 2024 - - -4.1. MLRsearch Versions - - The MLRsearch algorithm has been developed in a code-first approach, - a Python library has been created, debugged, used in production and - published in PyPI before the first descriptions (even informal) were - published. - - But the code (and hence the description) was evolving over time. - Multiple versions of the library were used over past several years, - and later code was usually not compatible with earlier descriptions. - - The code in (some version of) MLRsearch library fully determines the - search process (for a given set of configuration parameters), leaving - no space for deviations. - - This historic meaning of MLRsearch, as a family of search algorithm - implementations, leaves plenty of space for future improvements, at - the cost of poor comparability of results of search algoritm - implementations. - - There are two competing needs. There is the need for standardization - in areas critical to comparability. There is also the need to allow - flexibility for implementations to innovate and improve in other - areas. This document defines MLRsearch as a new specification in a - manner that aims to fairly balance both needs. - -4.2. Stopping Conditions - - [RFC2544] prescribes that after performing one trial at a specific - offered load, the next offered load should be larger or smaller, - based on frame loss. - - The usual implementation uses binary search. Here a lossy trial - becomes a new upper bound, a lossless trial becomes a new lower - bound. The span of values between the tightest lower bound and the - tightest upper bound (including both values) forms an interval of - possible results, and after each trial the width of that interval - halves. - - Usually the binary search implementation tracks only the two tightest - bounds, simply calling them bounds. But the old values still remain - valid bounds, just not as tight as the new ones. - - After some number of trials, the tightest lower bound becomes the - throughput. [RFC2544] does not specify when, if ever, should the - search stop. - - MLRsearch introduces a concept of [Goal Width] (#Goal-Width). - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 31] - -Internet-Draft MLRsearch July 2024 - - - The search stops when the distance between the tightest upper bound - and the tightest lower bound is smaller than a user-configured value, - called Goal Width from now on. In other words, the interval width at - the end of the search has to be no larger than the Goal Width. - - This Goal Width value therefore determines the precision of the - result. Due to the fact that MLRsearch specification requires a - particular structure of the result (see [Trial Result] (#Trial- - Result) section), the result itself does contain enough information - to determine its precision, thus it is not required to report the - Goal Width value. - - This allows MLRsearch implementations to use stopping conditions - different from Goal Width. - -4.3. Load Classification - - MLRsearch keeps the basic logic of binary search (tracking tightest - bounds, measuring at the middle), perhaps with minor technical - differences. - - MLRsearch algorithm chooses an intended load (as opposed to the - offered load), the interval between bounds does not need to be split - exactly into two equal halves, and the final reported structure - specifies both bounds. - - The biggest difference is that to classify a load as an upper or - lower bound, MLRsearch may need more than one trial (depending on - configuration options) to be performed at the same intended load. - - In consequence, even if a load already does have few trial results, - it still may be classified as undecided, neither a lower bound nor an - upper bound. - - An explanation of the classification logic is given in the next - section [Logic of Load Classification] (#Logic-of-Load- - Classification), as it heavily relies on other subsections of this - section. - - For repeatability and comparability reasons, it is important that - given a set of trial results, all implementations of MLRsearch - classify the load equivalently. - -4.4. Loss Ratios - - Another difference between MLRsearch and [RFC2544] binary search is - in the goals of the search. [RFC2544] has a single goal, based on - classifying full-length trials as either lossless or lossy. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 32] - -Internet-Draft MLRsearch July 2024 - - - MLRsearch, as the name suggests, can search for multiple goals, - differing in their loss ratios. The precise definition of the Goal - Loss Ratio will be given later. The [RFC2544] throughput goal then - simply becomes a zero Goal Loss Ratio. Different goals also may have - different Goal Widths. - - A set of trial results for one specific intended load value can - classify the load as an upper bound for some goals, but a lower bound - for some other goals, and undecided for the rest of the goals. - - Therefore, the load classification depends not only on trial results, - but also on the goal. The overall search procedure becomes more - complicated, when compared to binary search with a single goal, but - most of the complications do not affect the final result, except for - one phenomenon, loss inversion. - -4.5. Loss Inversion - - In [RFC2544] throughput search using bisection, any load with a lossy - trial becomes a hard upper bound, meaning every subsequent trial has - a smaller intended load. - - But in MLRsearch, a load that is classified as an upper bound for one - goal may still be a lower bound for another goal, and due to the - other goal MLRsearch will probably perform trials at even higher - loads. What to do when all such higher load trials happen to have - zero loss? Does it mean the earlier upper bound was not real? Does - it mean the later lossless trials are not considered a lower bound? - Surely we do not want to have an upper bound at a load smaller than a - lower bound. - - MLRsearch is conservative in these situations. The upper bound is - considered real, and the lossless trials at higher loads are - considered to be a coincidence, at least when computing the final - result. - - This is formalized using new notions, the [Relevant Upper Bound] - (#Relevant-Upper-Bound) and the [Relevant Lower Bound] (#Relevant- - Lower-Bound). Load classification is still based just on the set of - trial results at a given intended load (trials at other loads are - ignored), making it possible to have a lower load classified as an - upper bound, and a higher load classified as a lower bound (for the - same goal). The Relevant Upper Bound (for a goal) is the smallest - load classified as an upper bound. But the Relevant Lower Bound is - not simply the largest among lower bounds. It is the largest load - among loads that are lower bounds while also being smaller than the - Relevant Upper Bound. - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 33] - -Internet-Draft MLRsearch July 2024 - - - With these definitions, the Relevant Lower Bound is always smaller - than the Relevant Upper Bound (if both exist), and the two relevant - bounds are used analogously as the two tightest bounds in the binary - search. When they are less than the Goal Width apart, the relevant - bounds are used in the output. - - One consequence is that every trial result can have an impact on the - search result. That means if your SUT (or your traffic generator) - needs a warmup, be sure to warm it up before starting the search. - -4.6. Exceed Ratio - - The idea of performing multiple trials at the same load comes from a - model where some trial results (those with high loss) are affected by - infrequent effects, causing poor repeatability of [RFC2544] - throughput results. See the discussion about noiseful and noiseless - ends of the SUT performance spectrum in section [DUT in SUT] (#DUT- - in-SUT). Stable results are closer to the noiseless end of the SUT - performance spectrum, so MLRsearch may need to allow some frequency - of high-loss trials to ignore the rare but big effects near the - noiseful end. - - MLRsearch can do such trial result filtering, but it needs a - configuration option to tell it how frequent can the infrequent big - loss be. This option is called the exceed ratio. It tells MLRsearch - what ratio of trials (more exactly what ratio of trial seconds) can - have a [Trial Loss Ratio] (#Trial-Loss-Ratio) larger than the Goal - Loss Ratio and still be classified as a lower bound. Zero exceed - ratio means all trials have to have a Trial Loss Ratio equal to or - smaller than the Goal Loss Ratio. - - For explainability reasons, the RECOMMENDED value for exceed ratio is - 0.5, as it simplifies some later concepts by relating them to the - concept of median. - -4.7. Duration Sum - - When more than one trial is intended to classify a load, MLRsearch - also needs something that controls the number of trials needed. - Therefore, each goal also has an attribute called duration sum. - - The meaning of a [Goal Duration Sum] (#Goal-Duration-Sum) is that - when a load has (full-length) trials whose trial durations when - summed up give a value at least as big as the Goal Duration Sum - value, the load is guaranteed to be classified either as an upper - bound or a lower bound for that goal. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 34] - -Internet-Draft MLRsearch July 2024 - - - Due to the fact that the duration sum has a big impact on the overall - search duration, and [RFC2544] prescribes wait intervals around trial - traffic, the MLRsearch algorithm is allowed to sum durations that are - different from the actual trial traffic durations. - - In the MLRsearch specification, the different duration values are - called [Trial Effective Duration] (#Trial-Effective-Duration). - -4.8. Short Trials - - MLRsearch requires each goal to specify its final trial duration. - Full-length trial is a shorter name for a trial whose intended trial - duration is equal to (or longer than) the goal final trial duration. - - Section 24 of [RFC2544] already anticipates possible time savings - when short trials (shorter than full-length trials) are used. Full- - length trials are the opposite of short trials, so they may also be - called long trials. - - Any MLRsearch implementation may include its own configuration - options which control when and how MLRsearch chooses to use short - trial durations. - - For explainability reasons, when exceed ratio of 0.5 is used, it is - recommended for the Goal Duration Sum to be an odd multiple of the - full trial durations, so Conditional Throughput becomes identical to - a median of a particular set of trial forwarding rates. - - The presence of short trial results complicates the load - classification logic. - - Full details are given later in section [Logic of Load - Classification] (#Logic-of-Load-Classification). In a nutshell, - results from short trials may cause a load to be classified as an - upper bound. This may cause loss inversion, and thus lower the - Relevant Lower Bound, below what would classification say when - considering full-length trials only. - -4.9. Throughput - - Due to the fact that testing equipment takes the intended load as an - input parameter for a trial measurement, any load search algorithm - needs to deal with intended load values internally. - - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 35] - -Internet-Draft MLRsearch July 2024 - - - But in the presence of goals with a non-zero loss ratio, the intended - load usually does not match the user's intuition of what a throughput - is. The forwarding rate (as defined in [RFC2285] section 3.6.1) is - better, but it is not obvious how to generalize it for loads with - multiple trial results and a non-zero [Goal Loss Ratio] (#Goal-Loss- - Ratio). - - The best example is also the main motivation: hard limit performance. - Even if the medium allows higher performance, the SUT interfaces may - have their additional own limitations, e.g. a specific fps limit on - the NIC (a very common occurance). - - Ideally, those should be known and used when computing Max Load. But - if Max Load is higher that what interface can receive or transmit, - there will be a "hard limit" observed in trial results. Imagine the - hard limit is at 100 Mfps, Max Load is higher, and the goal loss - ratio is 0.5%. If DUT has no additional losses, 0.5% loss ratio will - be achieved at 100.5025 Mfps (the relevant lower bound). But it is - not intuitive to report SUT performance as a value that is larger - than known hard limit. We need a generalization of RFC2544 - throughput, different from just the relevant lower bound. - - MLRsearch defines one such generalization, called the Conditional - Throughput. It is the trial forwarding rate from one of the trials - performed at the load in question. Determining which trial exactly - is defined in [MLRsearch Specification] (#MLRsearch-Specification), - and in [Appendix B: Conditional Throughput] (#Appendix-B:- - Conditional-Throughput). - - In the hard limit example, 100.5 Mfps load will still have only 100.0 - Mfps forwarding rate, nicely confirming the known limitation. - - Conditional Throughput is partially related to load classification. - If a load is classified as a lower bound for a goal, the Conditional - Throughput can be calculated from trial results, and guaranteed to - show an loss ratio no larger than the Goal Loss Ratio. - - Note that when comparing the best (all zero loss) and worst case (all - loss just below Goal Loss Ratio), the same Relevant Lower Bound value - may result in the Conditional Throughput differing up to the Goal - Loss Ratio. - - - - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 36] - -Internet-Draft MLRsearch July 2024 - - - Therefore it is rarely needed to set the Goal Width (if expressed as - the relative difference of loads) below the Goal Loss Ratio. In - other words, setting the Goal Width below the Goal Loss Ratio may - cause the Conditional Throughput for a larger loss ratio to become - smaller than a Conditional Throughput for a goal with a smaller Goal - Loss Ratio, which is counter-intuitive, considering they come from - the same search. Therefore it is RECOMMENDED to set the Goal Width - to a value no smaller than the Goal Loss Ratio. - - Overall, this Conditional Throughput does behave well for - comparability purposes. - -4.10. Search Time - - MLRsearch was primarily developed to reduce the time required to - determine a throughput, either the [RFC2544] compliant one, or some - generalization thereof. The art of achieving short search times is - mainly in the smart selection of intended loads (and intended - durations) for the next trial to perform. - - While there is an indirect impact of the load selection on the - reported values, in practice such impact tends to be small, even for - SUTs with quite a broad performance spectrum. - - A typical example of two approaches to load selection leading to - different Relevant Lower Bounds is when the interval is split in a - very uneven way. Any implementation choosing loads very close to the - current Relevant Lower Bound is quite likely to eventually stumble - upon a trial result with poor performance (due to SUT noise). For an - implementation choosing loads very close to the current Relevant - Upper Bound, this is unlikely, as it examines more loads that can see - a performance close to the noiseless end of the SUT performance - spectrum. - - However, as even splits optimize search duration at give precision, - MLRsearch implementations that prioritize minimizing search time are - unlikely to suffer from any such bias. - - Therefore, this document remains quite vague on load selection and - other optimization details, and configuration attributes related to - them. Assuming users prefer libraries that achieve short overall - search time, the definition of the Relevant Lower Bound should be - strict enough to ensure result repeatability and comparability - between different implementations, while not restricting future - implementations much. - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 37] - -Internet-Draft MLRsearch July 2024 - - -4.11. [RFC2544] Compliance - - Some Search Goal instances lead to results compliant with RFC2544. - See [RFC2544 Goal] (#RFC2544-Goal) for more details regarding both - conditional and unconditional compliance. - - The presence of other Search Goals does not affect the compliance of - this Goal Result. The Relevant Lower Bound and the Conditional - Throughput are in this case equal to each other, and the value is the - [RFC2544] throughput. - -5. Logic of Load Classification - -5.1. Introductory Remarks - - This chapter continues with explanations, but this time more precise - definitions are needed for readers to follow the explanations. - - Descriptions in this section are wordy and implementers should read - [MLRsearch Specification] (#MLRsearch-Specification) section and - Appendices for more concise definitions. - - The two areas of focus here are load classification and the - Conditional Throughput. - - To start with [Performance Spectrum] (#Performance-Spectrum) - subsection contains definitions needed to gain insight into what - Conditional Throughput means. Remaining subsections discuss load - classification. - - For load classification, it is useful to define *good trials* and - *bad trials*: - - * *Bad trial*: Trial is called bad (according to a goal) if its - [Trial Loss Ratio] (#Trial-Loss-Ratio) is larger than the [Goal - Loss Ratio] (#Goal-Loss-Ratio). - - * *Good trial*: Trial that is not bad is called good. - -5.2. Performance Spectrum - - ### Description - - There are several equivalent ways to explain the Conditional - Throughput computation. One of the ways relies on performance - spectrum. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 38] - -Internet-Draft MLRsearch July 2024 - - - Take an intended load value, a trial duration value, and a finite set - of trial results, with all trials measured at that load value and - duration value. - - The performance spectrum is the function that maps any non-negative - real number into a sum of trial durations among all trials in the - set, that has that number, as their trial forwarding rate, e.g. map - to zero if no trial has that particular forwarding rate. - - A related function, defined if there is at least one trial in the - set, is the performance spectrum divided by the sum of the durations - of all trials in the set. - - That function is called the performance probability function, as it - satisfies all the requirements for probability mass function of a - discrete probability distribution, the one-dimensional random - variable being the trial forwarding rate. - - These functions are related to the SUT performance spectrum, as - sampled by the trials in the set. - - Take a set of all full-length trials performed at the Relevant Lower - Bound, sorted by decreasing trial forwarding rate. The sum of the - durations of those trials may be less than the Goal Duration Sum, or - not. If it is less, add an imaginary trial result with zero trial - forwarding rate, such that the new sum of durations is equal to the - Goal Duration Sum. This is the set of trials to use. - - If the quantile touches two trials, - - the larger trial forwarding rate (from the trial result sorted - earlier) is used. - - The resulting quantity is the Conditional Throughput of the goal in - question. - - A set of examples follows. - -5.2.1. First Example - - * [Goal Exceed Ratio] (#Goal-Exceed-Ratio) = 0 and [Goal Duration - Sum] (#Goal-Duration-Sum) has been reached. - - * Conditional Throughput is the smallest trial forwarding rate among - the trials. - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 39] - -Internet-Draft MLRsearch July 2024 - - -5.2.2. Second Example - - * Goal Exceed Ratio = 0 and Goal Duration Sum has not been reached - yet. - - * Due to the missing duration sum, the worst case may still happen, - so the Conditional Throughput is zero. - - * This is not reported to the user, as this load cannot become the - Relevant Lower Bound yet. - -5.2.3. Third Example - - * Goal Exceed Ratio = 50% and Goal Duration Sum is two seconds. - - * One trial is present with the duration of one second and zero - loss. - - * The imaginary trial is added with the duration of one second and - zero trial forwarding rate. - - * The median would touch both trials, so the Conditional Throughput - is the trial forwarding rate of the one non-imaginary trial. - - * As that had zero loss, the value is equal to the offered load. - -5.2.4. Summary - - While the Conditional Throughput is a generalization of the trial - forwarding rate, its definition is not an obvious one. - - Other than the trial forwarding rate, the other source of intuition - is the quantile in general, and the median the recommended case. - -5.3. Trials with Single Duration - - When goal attributes are chosen in such a way that every trial has - the same intended duration, the load classification is simpler. - - The following description follows the motivation of Goal Loss Ratio, - Goal Exceed Ratio, and Goal Duration Sum. - - If the sum of the durations of all trials (at the given load) is less - than the Goal Duration Sum, imagine two scenarios: - - * *best case scenario*: all subsequent trials having zero loss, and - - * *worst case scenario*: all subsequent trials having 100% loss. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 40] - -Internet-Draft MLRsearch July 2024 - - - Here we assume there are as many subsequent trials as needed to make - the sum of all trials equal to the Goal Duration Sum. - - The exceed ratio is defined using sums of durations (and number of - trials does not matter), so it does not matter whether the - "subsequent trials" can consist of an integer number of full-length - trials. - - In any of the two scenarios, best case and worst case, we can compute - the load exceed ratio, as the duration sum of good trials divided by - the duration sum of all trials, in both cases including the assumed - trials. - - Even if, in the best case scenario, the load exceed ratio is larger - than the Goal Exceed Ratio, the load is an upper bound. - - MKP2 Even if, in the worst case scenario, the load exceed ratio is - not larger than the Goal Exceed Ratio, the load is a lower bound. - - More specifically: - - * Take all trials measured at a given load. - - * The sum of the durations of all bad full-length trials is called - the bad sum. - - * The sum of the durations of all good full-length trials is called - the good sum. - - * The result of adding the bad sum plus the good sum is called the - measured sum. - - * The larger of the measured sum and the Goal Duration Sum is called - the whole sum. - - * The whole sum minus the measured sum is called the missing sum. - - * The optimistic exceed ratio is the bad sum divided by the whole - sum. - - * The pessimistic exceed ratio is the bad sum plus the missing sum, - that divided by the whole sum. - - * If the optimistic exceed ratio is larger than the Goal Exceed - Ratio, the load is classified as an upper bound. - - * If the pessimistic exceed ratio is not larger than the Goal Exceed - Ratio, the load is classified as a lower bound. - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 41] - -Internet-Draft MLRsearch July 2024 - - - * Else, the load is classified as undecided. - - The definition of pessimistic exceed ratio is compatible with the - logic in the Conditional Throughput computation, so in this single - trial duration case, a load is a lower bound if and only if the - Conditional Throughput loss ratio is not larger than the Goal Loss - Ratio. - - If it is larger, the load is either an upper bound or undecided. - -5.4. Trials with Short Duration - -5.4.1. Scenarios - - Trials with intended duration smaller than the goal final trial - duration are called short trials. The motivation for load - classification logic in the presence of short trials is based around - a counter-factual case: What would the trial result be if a short - trial has been measured as a full-length trial instead? - - There are three main scenarios where human intuition guides the - intended behavior of load classification. - -5.4.1.1. False Good Scenario - - The user had their reason for not configuring a shorter goal final - trial duration. Perhaps SUT has buffers that may get full at longer - trial durations. Perhaps SUT shows periodic decreases in performance - the user does not want to be treated as noise. - - In any case, many good short trials may become bad full-length trials - in the counter-factual case. - - In extreme cases, there are plenty of good short trials and no bad - short trials. - - In this scenario, we want the load classification NOT to classify the - load as a lower bound, despite the abundance of good short trials. - - Effectively, we want the good short trials to be ignored, so they do - not contribute to comparisons with the Goal Duration Sum. - -5.4.1.2. True Bad Scenario - - When there is a frame loss in a short trial, the counter-factual - full-length trial is expected to lose at least as many frames. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 42] - -Internet-Draft MLRsearch July 2024 - - - In practice, bad short trials are rarely turning into good full- - length trials. - - In extreme cases, there are no good short trials. - - In this scenario, we want the load classification to classify the - load as an upper bound just based on the abundance of short bad - trials. - - Effectively, we want the bad short trials to contribute to - comparisons with the Goal Duration Sum, so the load can be classified - sooner. - -5.4.1.3. Balanced Scenario - - Some SUTs are quite indifferent to trial duration. Performance - probability function constructed from short trial results is likely - to be similar to the performance probability function constructed - from full-length trial results (perhaps with larger dispersion, but - without a big impact on the median quantiles overall). - - For a moderate Goal Exceed Ratio value, this may mean there are both - good short trials and bad short trials. - - This scenario is there just to invalidate a simple heuristic of - always ignoring good short trials and never ignoring bad short - trials, as that simple heuristic would be too biased. - - Yes, the short bad trials are likely to turn into full-length bad - trials in the counter-factual case, but there is no information on - what would the good short trials turn into. - - The only way to decide safely is to do more trials at full length, - the same as in False Good Scenario. - -5.4.2. Classification Logic - - MLRsearch picks a particular logic for load classification in the - presence of short trials, but it is still RECOMMENDED to use - configurations that imply no short trials, so the possible - inefficiencies in that logic do not affect the result, and the result - has better explainability. - - With that said, the logic differs from the single trial duration case - only in different definition of the bad sum. The good sum is still - the sum across all good full-length trials. - - Few more notions are needed for defining the new bad sum: - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 43] - -Internet-Draft MLRsearch July 2024 - - - * The sum of durations of all bad full-length trials is called the - bad long sum. - - * The sum of durations of all bad short trials is called the bad - short sum. - - * The sum of durations of all good short trials is called the good - short sum. - - * One minus the Goal Exceed Ratio is called the subceed ratio. - - * The Goal Exceed Ratio divided by the subceed ratio is called the - exceed coefficient. - - * The good short sum multiplied by the exceed coefficient is called - the balancing sum. - - * The bad short sum minus the balancing sum is called the excess - sum. - - * If the excess sum is negative, the bad sum is equal to the bad - long sum. - - * Otherwise, the bad sum is equal to the bad long sum plus the - excess sum. - - Here is how the new definition of the bad sum fares in the three - scenarios, where the load is close to what would the relevant bounds - be if only full-length trials were used for the search. - -5.4.2.1. False Good Scenario - - If the duration is too short, we expect to see a higher frequency of - good short trials. This could lead to a negative excess sum, which - has no impact, hence the load classification is given just by full- - length trials. Thus, MLRsearch using too short trials has no - detrimental effect on result comparability in this scenario. But - also using short trials does not help with overall search duration, - probably making it worse. - -5.4.2.2. True Bad Scenario - - Settings with a small exceed ratio have a small exceed coefficient, - so the impact of the good short sum is small, and the bad short sum - is almost wholly converted into excess sum, thus bad short trials - have almost as big an impact as full-length bad trials. The same - conclusion applies to moderate exceed ratio values when the good - short sum is small. Thus, short trials can cause a load to get - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 44] - -Internet-Draft MLRsearch July 2024 - - - classified as an upper bound earlier, bringing time savings (while - not affecting comparability). - -5.4.2.3. Balanced Scenario - - Here excess sum is small in absolute value, as the balancing sum is - expected to be similar to the bad short sum. Once again, full-length - trials are needed for final load classification; but usage of short - trials probably means MLRsearch needed a shorter overall search time - before selecting this load for measurement, thus bringing time - savings (while not affecting comparability). - - Note that in presence of short trial results, the comparibility - between the load classification and the Conditional Throughput is - only partial. The Conditional Throughput still comes from a good - long trial, but a load higher than the Relevant Lower Bound may also - compute to a good value. - -5.5. Trials with Longer Duration - - If there are trial results with an intended duration larger than the - goal trial duration, the precise definitions in Appendix A and - Appendix B treat them in exactly the same way as trials with duration - equal to the goal trial duration. - - But in configurations with moderate (including 0.5) or small Goal - Exceed Ratio and small Goal Loss Ratio (especially zero), bad trials - with longer than goal durations may bias the search towards the lower - load values, as the noiseful end of the spectrum gets a larger - probability of causing the loss within the longer trials. - -6. IANA Considerations - - No requests of IANA. - -7. Security Considerations - - Benchmarking activities as described in this memo are limited to - technology characterization of a DUT/SUT using controlled stimuli in - a laboratory environment, with dedicated address space and the - constraints specified in the sections above. - - The benchmarking network topology will be an independent test setup - and MUST NOT be connected to devices that may forward the test - traffic into a production network or misroute traffic to the test - management network. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 45] - -Internet-Draft MLRsearch July 2024 - - - Further, benchmarking is performed on a "black-box" basis, relying - solely on measurements observable external to the DUT/SUT. - - Special capabilities SHOULD NOT exist in the DUT/SUT specifically for - benchmarking purposes. Any implications for network security arising - from the DUT/SUT SHOULD be identical in the lab and in production - networks. - -8. Acknowledgements - - Some phrases and statements in this document were created with help - of Mistral AI (mistral.ai). - - Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough - review and numerous useful comments and suggestions in the earlier - versions of this document. - - Special wholehearted gratitude and thanks to the late Al Morton for - his thorough reviews filled with very specific feedback and - constructive guidelines. Thank you Al for the close collaboration - over the years, for your continuous unwavering encouragement full of - empathy and positive attitude. Al, you are dearly missed. - -9. Appendix A: Load Classification - - This section specifies how to perform the load classification. - - Any intended load value can be classified, according to a given - [Search Goal] (#Search-Goal). - - The algorithm uses (some subsets of) the set of all available trial - results from trials measured at a given intended load at the end of - the search. All durations are those returned by the Measurer. - - The block at the end of this appendix holds pseudocode which computes - two values, stored in variables named optimistic and pessimistic. - - The pseudocode happens to be a valid Python code. - - If values of both variables are computed to be true, the load in - question is classified as a lower bound according to the given Search - Goal. If values of both variables are false, the load is classified - as an upper bound. Otherwise, the load is classified as undecided. - - The pseudocode expects the following variables to hold values as - follows: - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 46] - -Internet-Draft MLRsearch July 2024 - - - * goal_duration_sum: The duration sum value of the given Search - Goal. - - * goal_exceed_ratio: The exceed ratio value of the given Search - Goal. - - * good_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial - Loss Ratio not higher than the Goal Loss Ratio. - - * bad_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial - Loss Ratio higher than the Goal Loss Ratio. - - * good_short_sum: Sum of durations across trials with trial duration - shorter than the goal final trial duration and with a Trial Loss - Ratio not higher than the Goal Loss Ratio. - - * bad_short_sum: Sum of durations across trials with trial duration - shorter than the goal final trial duration and with a Trial Loss - Ratio higher than the Goal Loss Ratio. - - The code works correctly also when there are no trial results at a - given load. - - balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio) - effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum) - effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum) - quantile_duration_sum = effective_whole_sum * goal_exceed_ratio - optimistic = effective_bad_sum <= quantile_duration_sum - pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum - -10. Appendix B: Conditional Throughput - - This section specifies how to compute Conditional Throughput, as - referred to in section [Conditional Throughput] (#Conditional- - Throughput). - - Any intended load value can be used as the basis for the following - computation, but only the Relevant Lower Bound (at the end of the - search) leads to the value called the Conditional Throughput for a - given Search Goal. - - The algorithm uses (some subsets of) the set of all available trial - results from trials measured at a given intended load at the end of - the search. All durations are those returned by the Measurer. - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 47] - -Internet-Draft MLRsearch July 2024 - - - The block at the end of this appendix holds pseudocode which computes - a value stored as variable conditional_throughput. - - The pseudocode happens to be a valid Python code. - - The pseudocode expects the following variables to hold values as - follows: - - * goal_duration_sum: The duration sum value of the given Search - Goal. - - * goal_exceed_ratio: The exceed ratio value of the given Search - Goal. - - * good_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial - Loss Ratio not higher than the Goal Loss Ratio. - - * bad_long_sum: Sum of durations across trials with trial duration - at least equal to the goal final trial duration and with a Trial - Loss Ratio higher than the Goal Loss Ratio. - - * long_trials: An iterable of all trial results from trials with - trial duration at least equal to the goal final trial duration, - sorted by increasing the Trial Loss Ratio. A trial result is a - composite with the following two attributes available: - - - trial.loss_ratio: The Trial Loss Ratio as measured for this - trial. - - - trial.duration: The trial duration of this trial. - - The code works correctly only when there if there is at least one - trial result measured at a given load. - - all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum) - remaining = all_long_sum * (1.0 - goal_exceed_ratio) - quantile_loss_ratio = None - for trial in long_trials: - if quantile_loss_ratio is None or remaining > 0.0: - quantile_loss_ratio = trial.loss_ratio - remaining -= trial.duration - else: - break - else: - if remaining > 0.0: - quantile_loss_ratio = 1.0 - conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 48] - -Internet-Draft MLRsearch July 2024 - - -11. References - -11.1. Normative References - - [RFC1242] Bradner, S., "Benchmarking Terminology for Network - Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, - July 1991, <https://www.rfc-editor.org/info/rfc1242>. - - [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN - Switching Devices", RFC 2285, DOI 10.17487/RFC2285, - February 1998, <https://www.rfc-editor.org/info/rfc2285>. - - [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for - Network Interconnect Devices", RFC 2544, - DOI 10.17487/RFC2544, March 1999, - <https://www.rfc-editor.org/info/rfc2544>. - - [RFC8219] Georgescu, M., Pislaru, L., and G. Lencse, "Benchmarking - Methodology for IPv6 Transition Technologies", RFC 8219, - DOI 10.17487/RFC8219, August 2017, - <https://www.rfc-editor.org/info/rfc8219>. - - [RFC9004] Morton, A., "Updates for the Back-to-Back Frame Benchmark - in RFC 2544", RFC 9004, DOI 10.17487/RFC9004, May 2021, - <https://www.rfc-editor.org/info/rfc9004>. - -11.2. Informative References - - [FDio-CSIT-MLRsearch] - "FD.io CSIT Test Methodology - MLRsearch", October 2023, - <https://csit.fd.io/cdocs/methodology/measurements/ - data_plane_throughput/mlr_search/>. - - [PyPI-MLRsearch] - "MLRsearch 1.2.1, Python Package Index", October 2023, - <https://pypi.org/project/MLRsearch/1.2.1/>. - - [TST009] "TST 009", n.d., <https://www.etsi.org/deliver/etsi_gs/ - NFV-TST/001_099/009/03.04.01_60/gs_NFV- - TST009v030401p.pdf>. - -Authors' Addresses - - Maciek Konstantynowicz - Cisco Systems - Email: mkonstan@cisco.com - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 49] - -Internet-Draft MLRsearch July 2024 - - - Vratko Polak - Cisco Systems - Email: vrpolak@cisco.com - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -Konstantynowicz & Polak Expires 18 January 2025 [Page 50] diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml b/docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml deleted file mode 100644 index c3aede3d3b..0000000000 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.xml +++ /dev/null @@ -1,3136 +0,0 @@ -<?xml version="1.0" encoding="us-ascii"?> - <?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?> - <!-- generated by https://github.com/cabo/kramdown-rfc version 1.7.18 (Ruby 3.1.2) --> - - -<!DOCTYPE rfc [ - <!ENTITY nbsp " "> - <!ENTITY zwsp "​"> - <!ENTITY nbhy "‑"> - <!ENTITY wj "⁠"> - -<!ENTITY RFC1242 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.1242.xml"> -<!ENTITY RFC2285 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2285.xml"> -<!ENTITY RFC2544 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.2544.xml"> -<!ENTITY RFC8219 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8219.xml"> -<!ENTITY RFC9004 SYSTEM "https://bib.ietf.org/public/rfc/bibxml/reference.RFC.9004.xml"> -]> - - -<rfc ipr="trust200902" docName="draft-ietf-bmwg-mlrsearch-07" category="info" tocInclude="true" sortRefs="true" symRefs="true"> - <front> - <title abbrev="MLRsearch">Multiple Loss Ratio Search</title> - - <author initials="M." surname="Konstantynowicz" fullname="Maciek Konstantynowicz"> - <organization>Cisco Systems</organization> - <address> - <email>mkonstan@cisco.com</email> - </address> - </author> - <author initials="V." surname="Polak" fullname="Vratko Polak"> - <organization>Cisco Systems</organization> - <address> - <email>vrpolak@cisco.com</email> - </address> - </author> - - <date year="2024" month="July" day="18"/> - - <area>ops</area> - <workgroup>Benchmarking Working Group</workgroup> - <keyword>Internet-Draft</keyword> - - <abstract> - - -<?line 52?> - -<t>This document proposes extensions to <xref target="RFC2544"></xref> throughput search by -defining a new methodology called Multiple Loss Ratio search -(MLRsearch). MLRsearch aims to minimize search duration, -support multiple loss ratio searches, -and enhance result repeatability and comparability.</t> - -<t>The primary reason for extending <xref target="RFC2544"></xref> is to address the challenges -and requirements presented by the evaluation and testing -of software-based networking systems' data planes.</t> - -<t>To give users more freedom, MLRsearch provides additional configuration options -such as allowing multiple short trials per load instead of one large trial, -tolerating a certain percentage of trial results with higher loss, -and supporting the search for multiple goals with varying loss ratios.</t> - - - - </abstract> - - - - </front> - - <middle> - - -<?line 69?> - - -<section anchor="purpose-and-scope"><name>Purpose and Scope</name> - -<t>The purpose of this document is to describe Multiple Loss Ratio search -(MLRsearch), a data plane throughput search methodology optimized for software -networking DUTs.</t> - -<t>Applying vanilla <xref target="RFC2544"></xref> throughput bisection to software DUTs -results in several problems:</t> - -<t><list style="symbols"> - <t>Binary search takes too long as most trials are done far from the -eventually found throughput.</t> - <t>The required final trial duration and pauses between trials -prolong the overall search duration.</t> - <t>Software DUTs show noisy trial results, -leading to a big spread of possible discovered throughput values.</t> - <t>Throughput requires a loss of exactly zero frames, but the industry -frequently allows for small but non-zero losses.</t> - <t>The definition of throughput is not clear when trial results are inconsistent.</t> -</list></t> - -<t>To address the problems mentioned above, -the MLRsearch test methodology specification employs the following enhancements:</t> - -<t><list style="symbols"> - <t>Allow multiple short trials instead of one big trial per load. - <list style="symbols"> - <t>Optionally, tolerate a percentage of trial results with higher loss.</t> - </list></t> - <t>Allow searching for multiple Search Goals, with differing loss ratios. - <list style="symbols"> - <t>Any trial result can affect each Search Goal in principle.</t> - </list></t> - <t>Insert multiple coarse targets for each Search Goal, earlier ones need -to spend less time on trials. - <list style="symbols"> - <t>Earlier targets also aim for lesser precision.</t> - <t>Use Forwarding Rate (FR) at maximum offered load -<xref target="RFC2285"></xref> (section 3.6.2) to initialize the initial targets.</t> - </list></t> - <t>Take care when dealing with inconsistent trial results. - <list style="symbols"> - <t>Reported throughput is smaller than the smallest load with high loss.</t> - <t>Smaller load candidates are measured first.</t> - </list></t> - <t>Apply several load selection heuristics to save even more time -by trying hard to avoid unnecessarily narrow bounds.</t> -</list></t> - -<t>Some of these enhancements are formalized as MLRsearch specification, -the remaining enhancements are treated as implementation details, -thus achieving high comparability without limiting future improvements.</t> - -<t>MLRsearch configuration options are flexible enough to -support both conservative settings and aggressive settings. -The conservative settings lead to results -unconditionally compliant with <xref target="RFC2544"></xref>, -but longer search duration and worse repeatability. -Conversely, aggressive settings lead to shorter search duration -and better repeatability, but the results are not compliant with <xref target="RFC2544"></xref>.</t> - -<t>No part of <xref target="RFC2544"></xref> is intended to be obsoleted by this document.</t> - -</section> -<section anchor="identified-problems"><name>Identified Problems</name> - -<t>This chapter describes the problems affecting usability -of various performance testing methodologies, -mainly a binary search for <xref target="RFC2544"></xref> unconditionally compliant throughput.</t> - -<section anchor="long-search-duration"><name>Long Search Duration</name> - - -<t>The emergence of software DUTs, with frequent software updates and a -number of different frame processing modes and configurations, -has increased both the number of performance tests -required to verify the DUT update and the frequency of running those tests. -This makes the overall test execution time even more important than before.</t> - -<t>The current <xref target="RFC2544"></xref> throughput definition restricts the potential -for time-efficiency improvements. -A more generalized throughput concept could enable further enhancements -while maintaining the precision of simpler methods.</t> - -<t>The bisection method, when unconditionally compliant with <xref target="RFC2544"></xref>, -is excessively slow. -This is because a significant amount of time is spent on trials -with loads that, in retrospect, are far from the final determined throughput.</t> - -<t><xref target="RFC2544"></xref> does not specify any stopping condition for throughput search, -so users already have an access to a limited trade-off -between search duration and achieved precision. -However, each full 60-second trials doubles the precision, -so not many trials can be removed without a substantial loss of precision.</t> - -</section> -<section anchor="dut-in-sut"><name>DUT in SUT</name> - -<t><xref target="RFC2285"></xref> defines: -- DUT as - - The network forwarding device to which stimulus is offered and - response measured <xref target="RFC2285"></xref> (section 3.1.1). -- SUT as - - The collective set of network devices to which stimulus is offered - as a single entity and response measured <xref target="RFC2285"></xref> (section 3.1.2).</t> - -<t><xref target="RFC2544"></xref> specifies a test setup with an external tester stimulating the -networking system, treating it either as a single DUT, or as a system -of devices, an SUT.</t> - -<t>In the case of software networking, the SUT consists of not only the DUT -as a software program processing frames, but also of -server hardware and operating system functions, -with that server hardware resources shared across all programs including -the operating system.</t> - -<t>Given that the SUT is a shared multi-tenant environment -encompassing the DUT and other components, the DUT might inadvertently -experience interference from the operating system -or other software operating on the same server.</t> - -<t>Some of this interference can be mitigated. -For instance, -pinning DUT program threads to specific CPU cores -and isolating those cores can prevent context switching.</t> - -<t>Despite taking all feasible precautions, some adverse effects may still impact -the DUT's network performance. -In this document, these effects are collectively -referred to as SUT noise, even if the effects are not as unpredictable -as what other engineering disciplines call noise.</t> - -<t>DUT can also exhibit fluctuating performance itself, for reasons -not related to the rest of SUT. For example due to pauses in execution -as needed for internal stateful processing. -In many cases this -may be an expected per-design behavior, as it would be observable even -in a hypothetical scenario where all sources of SUT noise are eliminated. -Such behavior affects trial results in a way similar to SUT noise. -As the two phenomenons are hard to distinguish, -in this document the term 'noise' is used to encompass -both the internal performance fluctuations of the DUT -and the genuine noise of the SUT.</t> - -<t>A simple model of SUT performance consists of an idealized noiseless performance, -and additional noise effects. -For a specific SUT, the noiseless performance is assumed to be constant, -with all observed performance variations being attributed to noise. -The impact of the noise can vary in time, sometimes wildly, -even within a single trial. -The noise can sometimes be negligible, but frequently -it lowers the observed SUT performance as observed in trial results.</t> - -<t>In this model, SUT does not have a single performance value, it has a spectrum. -One end of the spectrum is the idealized noiseless performance value, -the other end can be called a noiseful performance. -In practice, trial result -close to the noiseful end of the spectrum happens only rarely. -The worse the performance value is, the more rarely it is seen in a trial. -Therefore, the extreme noiseful end of the SUT spectrum is not observable -among trial results. -Also, the extreme noiseless end of the SUT spectrum -is unlikely to be observable, this time because some small noise effects -are likely to occur multiple times during a trial.</t> - -<t>Unless specified otherwise, this document's focus is -on the potentially observable ends of the SUT performance spectrum, -as opposed to the extreme ones.</t> - -<t>When focusing on the DUT, the benchmarking effort should ideally aim -to eliminate only the SUT noise from SUT measurements. -However, -this is currently not feasible in practice, as there are no realistic enough -models available to distinguish SUT noise from DUT fluctuations, -based on authors' experience and available literature.</t> - -<t>Assuming a well-constructed SUT, the DUT is likely its -primary performance bottleneck. -In this case, we can define the DUT's -ideal noiseless performance as the noiseless end of the SUT performance spectrum, -especially for throughput. -However, other performance metrics, such as latency, -may require additional considerations.</t> - -<t>Note that by this definition, DUT noiseless performance -also minimizes the impact of DUT fluctuations, as much as realistically possible -for a given trial duration.</t> - -<t>MLRsearch methodology aims to solve the DUT in SUT problem -by estimating the noiseless end of the SUT performance spectrum -using a limited number of trial results.</t> - -<t>Any improvements to the throughput search algorithm, aimed at better -dealing with software networking SUT and DUT setup, should employ -strategies recognizing the presence of SUT noise, allowing the discovery of -(proxies for) DUT noiseless performance -at different levels of sensitivity to SUT noise.</t> - -</section> -<section anchor="repeatability-and-comparability"><name>Repeatability and Comparability</name> - -<t><xref target="RFC2544"></xref> does not suggest to repeat throughput search. -And from just one -discovered throughput value, it cannot be determined how repeatable that value is. -Poor repeatability then leads to poor comparability, -as different benchmarking teams may obtain varying throughput values -for the same SUT, exceeding the expected differences from search precision.</t> - -<t><xref target="RFC2544"></xref> throughput requirements (60 seconds trial and -no tolerance of a single frame loss) affect the throughput results -in the following way. -The SUT behavior close to the noiseful end of its performance spectrum -consists of rare occasions of significantly low performance, -but the long trial duration makes those occasions not so rare on the trial level. -Therefore, the binary search results tend to wander away from the noiseless end -of SUT performance spectrum, more frequently and more widely than short -trials would, thus causing poor throughput repeatability.</t> - -<t>The repeatability problem can be addressed by defining a search procedure -that identifies a consistent level of performance, -even if it does not meet the strict definition of throughput in <xref target="RFC2544"></xref>.</t> - -<t>According to the SUT performance spectrum model, better repeatability -will be at the noiseless end of the spectrum. -Therefore, solutions to the DUT in SUT problem -will help also with the repeatability problem.</t> - -<t>Conversely, any alteration to <xref target="RFC2544"></xref> throughput search -that improves repeatability should be considered -as less dependent on the SUT noise.</t> - -<t>An alternative option is to simply run a search multiple times, and report some -statistics (e.g. average and standard deviation). -This can be used -for a subset of tests deemed more important, -but it makes the search duration problem even more pronounced.</t> - -</section> -<section anchor="throughput-with-non-zero-loss"><name>Throughput with Non-Zero Loss</name> - -<t><xref target="RFC1242"></xref> (section 3.17 Throughput) defines throughput as: - The maximum rate at which none of the offered frames - are dropped by the device.</t> - -<t>Then, it says: - Since even the loss of one frame in a - data stream can cause significant delays while - waiting for the higher level protocols to time out, - it is useful to know the actual maximum data - rate that the device can support.</t> - -<t>However, many benchmarking teams accept a small, -non-zero loss ratio as the goal for their load search.</t> - -<t>Motivations are many:</t> - -<t><list style="symbols"> - <t>Modern protocols tolerate frame loss better, -compared to the time when <xref target="RFC1242"></xref> and <xref target="RFC2544"></xref> were specified.</t> - <t>Trials nowadays send way more frames within the same duration, -increasing the chance of a small SUT performance fluctuation -being enough to cause frame loss.</t> - <t>Small bursts of frame loss caused by noise have otherwise smaller impact -on the average frame loss ratio observed in the trial, -as during other parts of the same trial the SUT may work more closely -to its noiseless performance, thus perhaps lowering the Trial Loss Ratio -below the Goal Loss Ratio value.</t> - <t>If an approximation of the SUT noise impact on the Trial Loss Ratio is known, -it can be set as the Goal Loss Ratio.</t> -</list></t> - -<t>Regardless of the validity of all similar motivations, -support for non-zero loss goals makes any search algorithm more user-friendly. -<xref target="RFC2544"></xref> throughput is not user-friendly in this regard.</t> - -<t>Furthermore, allowing users to specify multiple loss ratio values, -and enabling a single search to find all relevant bounds, -significantly enhances the usefulness of the search algorithm.</t> - -<t>Searching for multiple Search Goals also helps to describe the SUT performance -spectrum better than the result of a single Search Goal. -For example, the repeated wide gap between zero and non-zero loss loads -indicates the noise has a large impact on the observed performance, -which is not evident from a single goal load search procedure result.</t> - -<t>It is easy to modify the vanilla bisection to find a lower bound -for the intended load that satisfies a non-zero Goal Loss Ratio. -But it is not that obvious how to search for multiple goals at once, -hence the support for multiple Search Goals remains a problem.</t> - -</section> -<section anchor="inconsistent-trial-results"><name>Inconsistent Trial Results</name> - -<t>While performing throughput search by executing a sequence of -measurement trials, there is a risk of encountering inconsistencies -between trial results.</t> - -<t>The plain bisection never encounters inconsistent trials. -But <xref target="RFC2544"></xref> hints about the possibility of inconsistent trial results, -in two places in its text. -The first place is section 24, where full trial durations are required, -presumably because they can be inconsistent with the results -from short trial durations. -The second place is section 26.3, where two successive zero-loss trials -are recommended, presumably because after one zero-loss trial -there can be a subsequent inconsistent non-zero-loss trial.</t> - -<t>Examples include:</t> - -<t><list style="symbols"> - <t>A trial at the same load (same or different trial duration) results -in a different Trial Loss Ratio.</t> - <t>A trial at a higher load (same or different trial duration) results -in a smaller Trial Loss Ratio.</t> -</list></t> - -<t>Any robust throughput search algorithm needs to decide how to continue -the search in the presence of such inconsistencies. -Definitions of throughput in <xref target="RFC1242"></xref> and <xref target="RFC2544"></xref> are not specific enough -to imply a unique way of handling such inconsistencies.</t> - -<t>Ideally, there will be a definition of a new quantity which both generalizes -throughput for non-zero-loss (and other possible repeatability enhancements), -while being precise enough to force a specific way to resolve trial result -inconsistencies. -But until such a definition is agreed upon, the correct way to handle -inconsistent trial results remains an open problem.</t> - -</section> -</section> -<section anchor="mlrsearch-specification"><name>MLRsearch Specification</name> - -<t>This section describes MLRsearch specification including all technical -definitions needed for evaluating whether a particular test procedure -complies with MLRsearch specification.</t> - - -<section anchor="overview"><name>Overview</name> - -<t>MLRsearch specification describes a set of abstract system components, -acting as functions with specified inputs and outputs.</t> - -<t>A test procedure is said to comply with MLRsearch specification -if it can be conceptually divided into analogous components, -each satisfying requirements for the corresponding MLRsearch component.</t> - -<t>The Measurer component is tasked to perform trials, -the Controller component is tasked to select trial loads and durations, -the Manager component is tasked to pre-configure everything -and to produce the test report. -The test report explicitly states Search Goals (as the Controller Inputs) -and corresponding Goal Results (Controller Outputs).</t> - - -<t>The Manager calls the Controller once, -the Controller keeps calling the Measurer -until all stopping conditions are met.</t> - -<t>The part where Controller calls the Measurer is called the search. -Any activity done by the Manager before it calls the Controller -(or after Controller returns) is not considered to be part of the search.</t> - -<t>MLRsearch specification prescribes regular search results and recommends -their stopping conditions. Irregular search results are also allowed, -they may have different requirements and stopping conditions.</t> - -<t>Search results are based on load classification. -When measured enough, any chosen load either achieves of fails each search goal, -thus becoming a lower or an upper bound for that goal. -When the relevant bounds are at loads that are close enough -(according to goal precision), the regular result is found. -Search stops when all regular results are found -(or if some goals are proven to have only irregular results).</t> - -</section> -<section anchor="measurement-quantities"><name>Measurement Quantities</name> - -<t>MLRsearch specification uses a number of measurement quantities.</t> - -<t>In general, MLRsearch specification does not require particular units to be used, -but it is REQUIRED for the test report to state all the units. -For example, ratio quantities can be dimensionless numbers between zero and one, -but may be expressed as percentages instead.</t> - -<t>For convenience, a group of quantities can be treated as a composite quantity, -One constituent of a composite quantity is called an attribute, -and a group of attribute values is called an instance of that composite quantity.</t> - -<t>Some attributes are not independent from others, -and they can be calculated from other attributes. -Such quantites are called derived quantities.</t> - -</section> -<section anchor="existing-terms"><name>Existing Terms</name> - -<t>RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" -contains basic definitions, and -RFC 2544 "Benchmarking Methodology for Network Interconnect Devices" -contains discussions of a number of terms and additional methodology requirements. -RFC 2285 adds more terms and discussions, describing some known situations -in more precise way.</t> - -<t>All three documents should be consulted -before attempting to make use of this document.</t> - -<t>Definitions of some central terms are copied and discussed in subsections.</t> - - - - - -<section anchor="sut"><name>SUT</name> - -<t>Defined in <xref target="RFC2285"></xref> (section 3.1.2 System Under Test (SUT)) as follows.</t> - -<t>Definition:</t> - -<t>The collective set of network devices to which stimulus is offered -as a single entity and response measured.</t> - -<t>Discussion:</t> - -<t>An SUT consisting of a single network device is also allowed.</t> - -</section> -<section anchor="dut"><name>DUT</name> - -<t>Defined in <xref target="RFC2285"></xref> (section 3.1.1 Device Under Test (DUT)) as follows.</t> - -<t>Definition:</t> - -<t>The network forwarding device to which stimulus is offered and -response measured.</t> - -<t>Discussion:</t> - -<t>DUT, as a sub-component of SUT, is only indirectly mentioned -in MLRsearch specification, but is of key relevance for its motivation.</t> - - -</section> -<section anchor="trial"><name>Trial</name> - -<t>A trial is the part of the test described in <xref target="RFC2544"></xref> (section 23. Trial description).</t> - -<t>Definition:</t> - -<t>A particular test consists of multiple trials. Each trial returns - one piece of information, for example the loss rate at a particular - input frame rate. Each trial consists of a number of phases:</t> - -<t>a) If the DUT is a router, send the routing update to the "input" - port and pause two seconds to be sure that the routing has settled.</t> - -<t>b) Send the "learning frames" to the "output" port and wait 2 - seconds to be sure that the learning has settled. Bridge learning - frames are frames with source addresses that are the same as the - destination addresses used by the test frames. Learning frames for - other protocols are used to prime the address resolution tables in - the DUT. The formats of the learning frame that should be used are - shown in the Test Frame Formats document.</t> - -<t>c) Run the test trial.</t> - -<t>d) Wait for two seconds for any residual frames to be received.</t> - -<t>e) Wait for at least five seconds for the DUT to restabilize.</t> - -<t>Discussion:</t> - -<t>The definition describes some traits, it is not clear whether all of them -are REQUIRED, or some of them are only RECOMMENDED.</t> - - -<t>For the purposes of the MLRsearch specification, -it is ALLOWED for the test procedure to deviate from the <xref target="RFC2544"></xref> description, -but any such deviation MUST be made explicit in the test report.</t> - -<t>Trials are the only stimuli the SUT is expected to experience -during the search.</t> - -<t>In some discussion paragraphs, it is useful to consider the traffic -as sent and received by a tester, as implicitly defined -in <xref target="RFC2544"></xref> (section 6. Test set up).</t> - -<t>An example of deviation from <xref target="RFC2544"></xref> is using shorter wait times.</t> - -</section> -</section> -<section anchor="trial-terms"><name>Trial Terms</name> - -<t>This section defines new and redefine existing terms for quantities -relevant as inputs or outputs of trial, as used by the Measurer component.</t> - -<section anchor="trial-duration"><name>Trial Duration</name> - -<t>Definition:</t> - -<t>Trial duration is the intended duration of the traffic for a trial.</t> - -<t>Discussion:</t> - -<t>In general, this quantity does not include any preparation nor waiting -described in section 23 of <xref target="RFC2544"></xref> (section 23. Trial description).</t> - -<t>While any positive real value may be provided, some Measurer implementations -MAY limit possible values, e.g. by rounding down to neared integer in seconds. -In that case, it is RECOMMENDED to give such inputs to the Controller -so the Controller only proposes the accepted values. -Alternatively, the test report MUST present the rounded values -as Search Goal attributes.</t> - -</section> -<section anchor="trial-load"><name>Trial Load</name> - -<t>Definition:</t> - -<t>The trial load is the intended load for a trial</t> - -<t>Discussion:</t> - -<t>For test report purposes, it is assumed that this is a constant load by default. -This MAY be only an average load, e.g. when the traffic is intended to be busty, -e.g. as suggested in <xref target="RFC2544"></xref> (section 21. Bursty traffic), -but the test report MUST explicitly mention how non-constant the traffic is.</t> - -<t>Trial load is the quantity defined as Constant Load of <xref target="RFC1242"></xref> -(section 3.4 Constant Load), Data Rate of <xref target="RFC2544"></xref> -(section 14. Bidirectional traffic) -and Intended Load of <xref target="RFC2285"></xref> (section 3.5.1 Intended load (Iload)). -All three definitions specify -that this value applies to one (input or output) interface.</t> - - -<t>For test report purposes, multi-interface aggregate load MAY be reported, -this is understood as the same quantity expressed using different units. -From the report it MUST be clear whether a particular trial load value -is per one interface, or an aggregate over all interfaces.</t> - -<t>Similarly to trial duration, some Measurers may limit the possible values -of trial load. Contrary to trial duration, the test report is NOT REQUIRED -to document such behavior.</t> - - -<t>It is ALLOWED to combine trial load and trial duration in a way -that would not be possible to achieve using any integer number of data frames.</t> - - -</section> -<section anchor="trial-input"><name>Trial Input</name> - -<t>Definition:</t> - -<t>Trial Input is a composite quantity, consisting of two attributes: -trial duration and trial load.</t> - -<t>Discussion:</t> - -<t>When talking about multiple trials, it is common to say "Trial Inputs" -to denote all corresponding Trial Input instances.</t> - -<t>A Trial Input instance acts as the input for one call of the Measurer component.</t> - -<t>Contrary to other composite quantities, MLRsearch implementations -are NOT ALLOWED to add optional attributes here. -This improves interoperability between various implementations of -the Controller and the Measurer.</t> - -</section> -<section anchor="traffic-profile"><name>Traffic Profile</name> - -<t>Definition:</t> - -<t>Traffic profile is a composite quantity -containing attributes other than trial load and trial duration, -needed for unique determination of the trial to be performed.</t> - -<t>Discussion:</t> - -<t>All its attributes are assumed to be constant during the search, -and the composite is configured on the Measurer by the Manager -before the search starts. -This is why the traffic profile is not part of the Trial Input.</t> - -<t>As a consequence, implementations of the Manager and the Measurer -must be aware of their common set of capabilities, so that the traffic profile -uniquely defines the traffic during the search. -The important fact is that none of those capabilities -have to be known by the Controller implementations.</t> - -<t>The traffic profile SHOULD contain some specific quantities, -for example <xref target="RFC2544"></xref> (section 9. Frame sizes) governs -data link frame size as defined in <xref target="RFC1242"></xref> (section 3.5 Data link frame size).</t> - -<t>Several more specific quantities may be RECOMMENDED, depending on media type. -For example, <xref target="RFC2544"></xref> (Appendix C) lists frame formats and protocol addresses, -as recommended from <xref target="RFC2544"></xref> (section 8. Frame formats) -and <xref target="RFC2544"></xref> (section 12. Protocol addresses).</t> - -<t>Depending on SUT configuration, e.g. when testing specific protocols, -additional attributes MUST be included in the traffic profile -and in the test report.</t> - -<t>Example: <xref target="RFC8219"></xref> (section 5.3. Traffic Setup) introduces traffic setups -consisting of a mix of IPv4 and IPv6 traffic - the implied traffic profile -therefore must include an attribute for their percentage.</t> - -<t>Other traffic properties that need to be somehow specified -in Traffic Profile include: -<xref target="RFC2544"></xref> (section 14. Bidirectional traffic), -<xref target="RFC2285"></xref> (section 3.3.3 Fully meshed traffic), -and <xref target="RFC2544"></xref> (section 11. Modifiers).</t> - -</section> -<section anchor="trial-forwarding-ratio"><name>Trial Forwarding Ratio</name> - -<t>Definition:</t> - -<t>The trial forwarding ratio is a dimensionless floating point value. -It MUST range between 0.0 and 1.0, both inclusive. -It is calculated by dividing the number of frames -successfully forwarded by the SUT -by the total number of frames expected to be forwarded during the trial</t> - -<t>Discussion:</t> - -<t>For most traffic profiles, "expected to be forwarded" means -"intended to get transmitted from Tester towards SUT".</t> - -<t>Trial forwarding ratio MAY be expressed in other units -(e.g. as a percentage) in the test report.</t> - -<t>Note that, contrary to loads, frame counts used to compute -trial forwarding ratio are aggregates over all SUT output interfaces.</t> - -<t>Questions around what is the correct number of frames -that should have been forwarded -is generally outside of the scope of this document.</t> - - - -</section> -<section anchor="trial-loss-ratio"><name>Trial Loss Ratio</name> - -<t>Definition:</t> - -<t>The Trial Loss Ratio is equal to one minus the trial forwarding ratio.</t> - -<t>Discussion:</t> - -<t>100% minus the trial forwarding ratio, when expressed as a percentage.</t> - -<t>This is almost identical to Frame Loss Rate of <xref target="RFC1242"></xref> -(section 3.6 Frame Loss Rate), -the only minor difference is that Trial Loss Ratio -does not need to be expressed as a percentage.</t> - -</section> -<section anchor="trial-forwarding-rate"><name>Trial Forwarding Rate</name> - -<t>Definition:</t> - -<t>The trial forwarding rate is a derived quantity, calculated by -multiplying the trial load by the trial forwarding ratio.</t> - -<t>Discussion:</t> - -<t>It is important to note that while similar, this quantity is not identical -to the Forwarding Rate as defined in <xref target="RFC2285"></xref> -(section 3.6.1 Forwarding rate (FR)). -The latter is specific to one output interface only, -whereas the trial forwarding ratio is based -on frame counts aggregated over all SUT output interfaces.</t> - - -</section> -<section anchor="trial-effective-duration"><name>Trial Effective Duration</name> - -<t>Definition:</t> - -<t>Trial effective duration is a time quantity related to the trial, -by default equal to the trial duration.</t> - -<t>Discussion:</t> - -<t>This is an optional feature. -If the Measurer does not return any trial effective duration value, -the Controller MUST use the trial duration value instead.</t> - -<t>Trial effective duration may be any time quantity chosen by the Measurer -to be used for time-based decisions in the Controller.</t> - -<t>The test report MUST explain how the Measurer computes the returned -trial effective duration values, if they are not always -equal to the trial duration.</t> - -<t>This feature can be beneficial for users -who wish to manage the overall search duration, -rather than solely the traffic portion of it. -Simply measure the duration of the whole trial (waits including) -and use that as the trial effective duration.</t> - -<t>Also, this is a way for the Measurer to inform the Controller about -its surprising behavior, for example when rounding the trial duration value.</t> - - -</section> -<section anchor="trial-output"><name>Trial Output</name> - -<t>Definition:</t> - -<t>Trial Output is a composite quantity. The REQUIRED attributes are -Trial Loss Ratio, trial effective duration and trial forwarding rate.</t> - -<t>Discussion:</t> - -<t>When talking about multiple trials, it is common to say "Trial Outputs" -to denote all corresponding Trial Output instances.</t> - -<t>Implementations may provide additional (optional) attributes. -The Controller implementations MUST ignore values of any optional attribute -they are not familiar with, -except when passing Trial Output instance to the Manager.</t> - -<t>Example of an optional attribute: -The aggregate number of frames expected to be forwarded during the trial, -especially if it is not just (a rounded-up value) -implied by trial load and trial duration.</t> - -<t>While <xref target="RFC2285"></xref> (Section 3.5.2 Offered load (Oload)) -requires the offered load value to be reported for forwarding rate measurements, -it is NOT REQUIRED in MLRsearch specification.</t> - - -</section> -<section anchor="trial-result"><name>Trial Result</name> - -<t>Definition:</t> - -<t>Trial result is a composite quantity, -consisting of the Trial Input and the Trial Output.</t> - -<t>Discussion:</t> - -<t>When talking about multiple trials, it is common to say "trial results" -to denote all corresponding trial result instances.</t> - -<t>While implementations SHOULD NOT include additional attributes -with independent values, they MAY include derived quantities.</t> - -</section> -</section> -<section anchor="goal-terms"><name>Goal Terms</name> - -<t>This section defines new and redefine existing terms for quantities -indirectly relevant for inputs or outputs of the Controller component.</t> - -<t>Several goal attributes are defined before introducing -the main component quantity: the Search Goal.</t> - -<section anchor="goal-final-trial-duration"><name>Goal Final Trial Duration</name> - -<t>Definition:</t> - -<t>A threshold value for trial durations.</t> - -<t>Discussion:</t> - -<t>This attribute value MUST be positive.</t> - -<t>A trial with Trial Duration at least as long as the Goal Final Trial Duration -is called a full-length trial (with respect to the given Search Goal).</t> - -<t>A trial that is not full-length is called a short trial.</t> - -<t>Informally, while MLRsearch is allowed to perform short trials, -the results from such short trials have only limited impact on search results.</t> - -<t>One trial may be full-length for some Search Goals, but not for others.</t> - -<t>The full relation of this goal to Controller Output is defined later in -this document in subsections of [Goal Result] (#Goal-Result). -For example, the Conditional Throughput for this goal is computed only from -full-length trial results.</t> - -</section> -<section anchor="goal-duration-sum"><name>Goal Duration Sum</name> - -<t>Definition:</t> - -<t>A threshold value for a particular sum of trial effective durations.</t> - -<t>Discussion:</t> - -<t>This attribute value MUST be positive.</t> - -<t>Informally, even when looking only at full-length trials, -MLRsearch may spend up to this time measuring the same load value.</t> - -<t>If the Goal Duration Sum is larger than the Goal Final Trial Duration, -multiple full-length trials may need to be performed at the same load.</t> - -<t>See [TST009 Example] (#TST009-Example) for an example where possibility -of multiple full-length trials at the same load is intended.</t> - -<t>A Goal Duration Sum value lower than the Goal Final Trial Duration -(of the same goal) could save some search time, but is NOT RECOMMENDED. -See [Relevant Upper Bound] (#Relevant-Upper-Bound) for partial explanation.</t> - -</section> -<section anchor="goal-loss-ratio"><name>Goal Loss Ratio</name> - -<t>Definition:</t> - -<t>A threshold value for Trial Loss Ratios.</t> - -<t>Discussion:</t> - -<t>Attribute value MUST be non-negative and smaller than one.</t> - -<t>A trial with Trial Loss Ratio larger than a Goal Loss Ratio value -is called a lossy trial, with respect to given Search Goal.</t> - -<t>Informally, if a load causes too many lossy trials, -the Relevant Lower Bound for this goal will be smaller than that load.</t> - -<t>If a trial is not lossy, it is called a low-loss trial, -or (specifically for zero Goal Loss Ratio value) zero-loss trial.</t> - -</section> -<section anchor="goal-exceed-ratio"><name>Goal Exceed Ratio</name> - -<t>Definition:</t> - -<t>A threshold value for a particular ratio of sums of Trial Effective Durations.</t> - -<t>Discussion:</t> - -<t>Attribute value MUST be non-negative and smaller than one.</t> - -<t>See later sections for details on which sums. -Specifically, the direct usage is only in -[Appendix A: Load Classification] (#Appendix-A:-Load-Classification) -and [Appendix B: Conditional Throughput] (#Appendix-B:-Conditional-Throughput). -The impact of that usage is discussed in subsections leading to -[Goal Result] (#Goal-Result).</t> - -<t>Informally, the impact of lossy trials is controlled by this value. -Effectively, Goal Exceed Ratio is a percentage of full-length trials -that may be lossy without the load being classified -as the [Relevant Upper Bound] (#Relevant-Upper-Bound).</t> - -</section> -<section anchor="goal-width"><name>Goal Width</name> - -<t>Definition:</t> - -<t>A value used as a threshold for deciding -whether two trial load values are close enough.</t> - -<t>Discussion:</t> - -<t>If present, the value MUST be positive.</t> - -<t>Informally, this acts as a stopping condition, -controlling the precision of the search. -The search stops if every goal has reached its precision.</t> - -<t>Implementations without this attribute -MUST give the Controller other ways to control the search stopping conditions.</t> - -<t>Absolute load difference and relative load difference are two popular choices, -but implementations may choose a different way to specify width.</t> - -<t>The test report MUST make it clear what specific quantity is used as Goal Width.</t> - -<t>It is RECOMMENDED to set the Goal Width (as relative difference) value -to a value no smaller than the Goal Loss Ratio. -(The reason is not obvious, see [Throughput] (#Throughput) if interested.)</t> - -</section> -<section anchor="search-goal"><name>Search Goal</name> - -<t>Definition:</t> - -<t>The Search Goal is a composite quantity consisting of several attributes, -some of them are required.</t> - -<t>Required attributes: -- Goal Final Trial Duration -- Goal Duration Sum -- Goal Loss Ratio -- Goal Exceed Ratio</t> - -<t>Optional attribute: -- Goal Width</t> - -<t>Discussion:</t> - -<t>Implementations MAY add their own attributes. -Those additional attributes may be required by the implementation -even if they are not required by MLRsearch specification. -But it is RECOMMENDED for those implementations -to support missing values by computing reasonable defaults.</t> - -<t>The meaning of listed attributes is formally given only by their indirect effect -on the search results.</t> - -<t>Informally, later sections provide additional intuitions and examples -of the Search Goal attribute values.</t> - -<t>An example of additional attributes required by some implementations -is Goal Initial Trial Duration, together with another attribute -that controls possible intermediate Trial Duration values. -The reasonable default in this case is using the Goal Final Trial Duration -and no intermediate values.</t> - -</section> -<section anchor="controller-input"><name>Controller Input</name> - -<t>Definition:</t> - -<t>Controller Input is a composite quantity -required as an input for the Controller. -The only REQUIRED attribute is a list of Search Goal instances.</t> - -<t>Discussion:</t> - -<t>MLRsearch implementations MAY use additional attributes. -Those additional attributes may be required by the implementation -even if they are not required by MLRsearch specification.</t> - -<t>Formally, the Manager does not apply any Controller configuration -apart from one Controller Input instance.</t> - -<t>For example, Traffic Profile is configured on the Measurer by the Manager -(without explicit assistance of the Controller).</t> - -<t>The order of Search Goal instances in a list SHOULD NOT -have a big impact on Controller Output (see section [Controller Output] (#Controller-Output) , -but MLRsearch implementations MAY base their behavior on the order -of Search Goal instances in a list.</t> - -<t>An example of an optional attribute (outside the list of Search Goals) -required by some implementations is Max Load. -While this is a frequently used configuration parameter, -already governed by <xref target="RFC2544"></xref> (section 20. Maximum frame rate) -and <xref target="RFC2285"></xref> (3.5.3 Maximum offered load (MOL)), -some implementations may detect or discover it instead.</t> - - - -<t>In MLRsearch specification, the [Relevant Upper Bound] (#Relevant-Upper-Bound) -is added as a required attribute precisely because it makes the search result -independent of Max Load value.</t> - - -</section> -</section> -<section anchor="search-goal-examples"><name>Search Goal Examples</name> - -<section anchor="rfc2544-goal"><name>RFC2544 Goal</name> - -<t>The following set of values makes the search result unconditionally compliant -with <xref target="RFC2544"></xref> (section 24 Trial duration)</t> - -<t><list style="symbols"> - <t>Goal Final Trial Duration = 60 seconds</t> - <t>Goal Duration Sum = 60 seconds</t> - <t>Goal Loss Ratio = 0%</t> - <t>Goal Exceed Ratio = 0%</t> -</list></t> - -<t>The latter two attributes are enough to make the search goal -conditionally compliant, adding the first attribute -makes it unconditionally compliant.</t> - -<t>The second attribute (Goal Duration Sum) only prevents MLRsearch -from repeating zero-loss full-length trials.</t> - -<t>Non-zero exceed ratio could prolong the search and allow loss inversion -between lower-load lossy short trial and higher-load full-length zero-loss trial. -From <xref target="RFC2544"></xref> alone, it is not clear whether that higher load -could be considered as compliant throughput.</t> - -</section> -<section anchor="tst009-goal"><name>TST009 Goal</name> - -<t>One of the alternatives to RFC2544 is described in -<xref target="TST009"></xref> (section 12.3.3 Binary search with loss verification). -The idea there is to repeat lossy trials, hoping for zero loss on second try, -so the results are closer to the noiseless end of performance sprectum, -and more repeatable and comparable.</t> - -<t>Only the variant with "z = infinity" is achievable with MLRsearch.</t> - - -<t>For example, for "r = 2" variant, the following search goal should be used:</t> - -<t><list style="symbols"> - <t>Goal Final Trial Duration = 60 seconds</t> - <t>Goal Duration Sum = 120 seconds</t> - <t>Goal Loss Ratio = 0%</t> - <t>Goal Exceed Ratio = 50%</t> -</list></t> - -<t>If the first 60s trial has zero loss, it is enough for MLRsearch to stop -measuring at that load, as even a second lossy trial -would still fit within the exceed ratio.</t> - -<t>But if the first trial is lossy, MLRsearch needs to perform also -the second trial to classify that load. -As Goal Duration Sum is twice as long as Goal Final Trial Duration, -third full-length trial is never needed.</t> - -</section> -</section> -<section anchor="result-terms"><name>Result Terms</name> - -<t>Before defining the output of the Controller, -it is useful to define what the Goal Result is.</t> - -<t>The Goal Result is a composite quantity.</t> - -<t>Following subsections define its attribute first, before describing the Goal Result quantity.</t> - -<t>There is a correspondence between Search Goals and Goal Results. -Most of the following subsections refer to a given Search Goal, -when defining attributes of the Goal Result. -Conversely, at the end of the search, each Search Goal -has its corresponding Goal Result.</t> - -<t>Conceptually, the search can be seen as a process of load classification, -where the Controller attempts to classify some loads as an Upper Bound -or a Lower Bound with respect to some Search Goal.</t> - -<t>Before defining real attributes of the goal result, -it is useful to define bounds in general.</t> - -<section anchor="relevant-upper-bound"><name>Relevant Upper Bound</name> - -<t>Definition:</t> - -<t>The Relevant Upper Bound is the smallest trial load value that is classified -at the end of the search as an upper bound -(see [Appendix A: Load Classification] (#Appendix-A:-Load-Classification)) -for the given Search Goal.</t> - -<t>Discussion:</t> - -<t>One search goal can have many different load classified as an upper bound. -At the end of the search, one of those loads will be the smallest, -becoming the relevant upper bound for that goal.</t> - -<t>In more detail, the set of all trial outputs (both short and full-length, -enough of them according to Goal Duration Sum) -performed at that smallest load failed to uphold all the requirements -of the given Search Goal, mainly the Goal Loss Ratio -in combination with the Goal Exceed Ratio.</t> - - -<t>If Max Load does not cause enough lossy trials, -the Relevant Upper Bound does not exist. -Conversely, if Relevant Upper Bound exists, -it is not affected by Max Load value.</t> - - - -</section> -<section anchor="relevant-lower-bound"><name>Relevant Lower Bound</name> - -<t>Definition:</t> - -<t>The Relevant Lower Bound is the largest trial load value -among those smaller than the Relevant Upper Bound, -that got classified at the end of the search as a lower bound (see -[Appendix A: Load Classification] (#Appendix-A:-Load-Classification)) -for the given Search Goal.</t> - -<t>Discussion:</t> - -<t>Only among loads smaller that the relevant upper bound, -the largest load becomes the relevant lower bound. -With loss inversion, stricter upper bound matters.</t> - -<t>In more detail, the set of all trial outputs (both short and full-length, -enough of them according to Goal Duration Sum) -performed at that largest load managed to uphold all the requirements -of the given Search Goal, mainly the Goal Loss Ratio -in combination with the Goal Exceed Ratio.</t> - -<t>Is no load had enough low-loss trials, the relevant lower bound -MAY not exist.</t> - - -<t>Strictly speaking, if the Relevant Upper Bound does not exist, -the Relevant Lower Bound also does not exist. -In that case, Max Load is classified as a lower bound, -but it is not clear whether a higher lower bound -would be found if the search used a higher Max Load value.</t> - -<t>For a regular Goal Result, the distance between the Relevant Lower Bound -and the Relevant Upper Bound MUST NOT be larger than the Goal Width, -if the implementation offers width as a goal attribute.</t> - - -<t>Searching for anther search goal may cause a loss inversion phenomenon, -where a lower load is classified as an upper bound, -but also a higher load is classified as a lower bound for the same search goal. -The definition of the Relevant Lower Bound ignores such high lower bounds.</t> - - -</section> -<section anchor="conditional-throughput"><name>Conditional Throughput</name> - -<t>Definition:</t> - -<t>The Conditional Throughput (see section [Appendix B: Conditional Throughput] (#Appendix-B:-Conditional-Throughput)) -as evaluated at the Relevant Lower Bound of the given Search Goal -at the end of the search.</t> - -<t>Discussion:</t> - -<t>Informally, this is a typical trial forwarding rate, expected to be seen -at the Relevant Lower Bound of the given Search Goal.</t> - -<t>But frequently it is only a conservative estimate thereof, -as MLRsearch implementations tend to stop gathering more data -as soon as they confirm the value cannot get worse than this estimate -within the Goal Duration Sum.</t> - -<t>This value is RECOMMENDED to be used when evaluating repeatability -and comparability if different MLRsearch implementations.</t> - - -</section> -<section anchor="goal-result"><name>Goal Result</name> - -<t>Definition:</t> - -<t>The Goal Result is a composite quantity consisting of several attributes. -Relevant Upper Bound and Relevant Lower Bound are REQUIRED attributes, -Conditional Throughput is a RECOMMENDED attribute.</t> - -<t>Discussion:</t> - -<t>Depending on SUT behavior, it is possible that one or both relevant bounds -do not exist. The goal result instance where the required attribute values exist -is informally called a Regular Goal Result instance, -so we can say some goals reached Irregular Goal Results.</t> - - -<t>A typical Irregular Goal Result is when all trials at the Max Load -have zero loss, as the Relevant Upper Bound does not exist in that case.</t> - -<t>It is RECOMMENDED that the test report will display such results appropriately, -although MLRsearch specification does not prescibe how.</t> - - -<t>Anything else regarging Irregular Goal Results, -including their role in stopping conditions of the search -is outside the scope of this document.</t> - -</section> -<section anchor="search-result"><name>Search Result</name> - -<t>Definition:</t> - -<t>The Search Result is a single composite object -that maps each Search Goal instance to a corresponding Goal Result instance.</t> - -<t>Discussion:</t> - -<t>Alternatively, the Search Result can be implemented as an ordered list -of the Goal Result instances, matching the order of Search Goal instances.</t> - - -<t>The Search Result (as a mapping) -MUST map from all the Search Goal instances present in the Controller Input.</t> - - - -</section> -<section anchor="controller-output"><name>Controller Output</name> - -<t>Definition:</t> - -<t>The Controller Output is a composite quantity returned from the Controller -to the Manager at the end of the search. -The Search Result instance is its only REQUIRED attribute.</t> - -<t>Discussion:</t> - -<t>MLRsearch implementation MAY return additional data in the Controller Output.</t> - - -</section> -</section> -<section anchor="mlrsearch-architecture"><name>MLRsearch Architecture</name> - - -<t>MLRsearch architecture consists of three main system components: -the Manager, the Controller, and the Measurer.</t> - -<t>The architecture also implies the presence of other components, -such as the SUT and the Tester (as a sub-component of the Measurer).</t> - -<t>Protocols of communication between components are generally left unspecified. -For example, when MLRsearch specification mentions "Controller calls Measurer", -it is possible that the Controller notifies the Manager -to call the Measurer indirectly instead. This way the Measurer implementations -can be fully independent from the Controller implementations, -e.g. programmed in different programming languages.</t> - -<section anchor="measurer"><name>Measurer</name> - -<t>Definition:</t> - -<t>The Measurer is an abstract system component -that when called with a [Trial Input] (#Trial-Input) instance, -performs one [Trial] (#Trial), -and returns a [Trial Output] (#Trial-Output) instance.</t> - -<t>Discussion:</t> - -<t>This definition assumes the Measurer is already initialized. -In practice, there may be additional steps before the search, -e.g. when the Manager configures the traffic profile -(either on the Measurer or on its tester sub-component directly) -and performs a warmup (if the tester requires one).</t> - -<t>It is the responsibility of the Measurer implementation to uphold -any requirements and assumptions present in MLRsearch specification, -e.g. trial forwarding ratio not being larger than one.</t> - -<t>Implementers have some freedom. -For example <xref target="RFC2544"></xref> (section 10. Verifying received frames) -gives some suggestions (but not requirements) related to -duplicated or reordered frames. -Implementations are RECOMMENDED to document their behavior -related to such freedoms in as detailed a way as possible.</t> - -<t>It is RECOMMENDED to benchmark the test equipment first, -e.g. connect sender and receiver directly (without any SUT in the path), -find a load value that guarantees the offered load is not too far -from the intended load, and use that value as the Max Load value. -When testing the real SUT, it is RECOMMENDED to turn any big difference -between the intended load and the offered load into increased Trial Loss Ratio.</t> - -<t>Neither of the two recommendations are made into requirements, -because it is not easy to tell when the difference is big enough, -in a way thay would be dis-entangled from other Measurer freedoms.</t> - -</section> -<section anchor="controller"><name>Controller</name> - -<t>Definition:</t> - -<t>The Controller is an abstract system component -that when called with a Controller Input instance -repeatedly computes Trial Input instance for the Measurer, -obtains corresponding Trial Output instances, -and eventually returns a Controller Output instance.</t> - -<t>Discussion:</t> - -<t>Informally, the Controller has big freedom in selection of Trial Inputs, -and the implementations want to achieve the Search Goals -in the shortest expected time.</t> - -<t>The Controller's role in optimizing the overall search time -distinguishes MLRsearch algorithms from simpler search procedures.</t> - -<t>Informally, each implementation can have different stopping conditions. -Goal Width is only one example. -In practice, implementation details do not matter, -as long as Goal Results are regular.</t> - -</section> -<section anchor="manager"><name>Manager</name> - -<t>Definition:</t> - -<t>The Manager is an abstract system component that is reponsible for -configuring other components, calling the Controller component once, -and for creating the test report following the reporting format as -defined in <xref target="RFC2544"></xref> (section 26. Benchmarking tests).</t> - -<t>Discussion:</t> - -<t>The Manager initializes the SUT, the Measurer (and the Tester if independent) -with their intended configurations before calling the Controller.</t> - -<t>The Manager does not need to be able to tweak any Search Goal attributes, -but it MUST report all applied attribute values even if not tweaked.</t> - - -<t>In principle, there should be a "user" (human or CI) -that "starts" or "calls" the Manager and receives the report. -The Manager MAY be able to be called more than once whis way.</t> - - -</section> -</section> -<section anchor="implementation-compliance"><name>Implementation Compliance</name> - -<t>Any networking measurement setup where there can be logically delineated system components -and there are components satisfying requirements for the Measurer, -the Controller and the Manager, is considered to be compliant with MLRsearch design.</t> - -<t>These components can be seen as abstractions present in any testing procedure. -For example, there can be a single component acting both -as the Manager and the Controller, but as long as values of required attributes -of Search Goals and Goal Results are visible in the test report, -the Controller Input instance and output instance are implied.</t> - -<t>For example, any setup for conditionally (or unconditionally) -compliant <xref target="RFC2544"></xref> throughput testing -can be understood as a MLRsearch architecture, -assuming there is enough data to reconstruct the Relevant Upper Bound.</t> - -<t>See [RFC2544 Goal] (#RFC2544-Goal) subsection for equivalent Search Goal.</t> - -<t>Any test procedure that can be understood as (one call to the Manager of) -MLRsearch architecture is said to be compliant with MLRsearch specification.</t> - -</section> -</section> -<section anchor="additional-considerations"><name>Additional Considerations</name> - -<t>This section focuses on additional considerations, intuitions and motivations -pertaining to MLRsearch methodology.</t> - - -<section anchor="mlrsearch-versions"><name>MLRsearch Versions</name> - -<t>The MLRsearch algorithm has been developed in a code-first approach, -a Python library has been created, debugged, used in production -and published in PyPI before the first descriptions -(even informal) were published.</t> - -<t>But the code (and hence the description) was evolving over time. -Multiple versions of the library were used over past several years, -and later code was usually not compatible with earlier descriptions.</t> - -<t>The code in (some version of) MLRsearch library fully determines -the search process (for a given set of configuration parameters), -leaving no space for deviations.</t> - - - -<t>This historic meaning of MLRsearch, as a family -of search algorithm implementations, -leaves plenty of space for future improvements, at the cost -of poor comparability of results of search algoritm implementations.</t> - - -<t>There are two competing needs. -There is the need for standardization in areas critical to comparability. -There is also the need to allow flexibility for implementations -to innovate and improve in other areas. -This document defines MLRsearch as a new specification -in a manner that aims to fairly balance both needs.</t> - -</section> -<section anchor="stopping-conditions"><name>Stopping Conditions</name> - -<t><xref target="RFC2544"></xref> prescribes that after performing one trial at a specific offered load, -the next offered load should be larger or smaller, based on frame loss.</t> - -<t>The usual implementation uses binary search. -Here a lossy trial becomes -a new upper bound, a lossless trial becomes a new lower bound. -The span of values between the tightest lower bound -and the tightest upper bound (including both values) forms an interval of possible results, -and after each trial the width of that interval halves.</t> - -<t>Usually the binary search implementation tracks only the two tightest bounds, -simply calling them bounds. -But the old values still remain valid bounds, -just not as tight as the new ones.</t> - -<t>After some number of trials, the tightest lower bound becomes the throughput. -<xref target="RFC2544"></xref> does not specify when, if ever, should the search stop.</t> - -<t>MLRsearch introduces a concept of [Goal Width] (#Goal-Width).</t> - -<t>The search stops -when the distance between the tightest upper bound and the tightest lower bound -is smaller than a user-configured value, called Goal Width from now on. -In other words, the interval width at the end of the search -has to be no larger than the Goal Width.</t> - -<t>This Goal Width value therefore determines the precision of the result. -Due to the fact that MLRsearch specification requires a particular -structure of the result (see [Trial Result] (#Trial-Result) section), -the result itself does contain enough information to determine its -precision, thus it is not required to report the Goal Width value.</t> - -<t>This allows MLRsearch implementations to use stopping conditions -different from Goal Width.</t> - -</section> -<section anchor="load-classification"><name>Load Classification</name> - -<t>MLRsearch keeps the basic logic of binary search (tracking tightest bounds, -measuring at the middle), perhaps with minor technical differences.</t> - -<t>MLRsearch algorithm chooses an intended load (as opposed to the offered load), -the interval between bounds does not need to be split -exactly into two equal halves, -and the final reported structure specifies both bounds.</t> - -<t>The biggest difference is that to classify a load -as an upper or lower bound, MLRsearch may need more than one trial -(depending on configuration options) to be performed at the same intended load.</t> - -<t>In consequence, even if a load already does have few trial results, -it still may be classified as undecided, neither a lower bound nor an upper bound.</t> - -<t>An explanation of the classification logic is given in the next section [Logic of Load Classification] (#Logic-of-Load-Classification), -as it heavily relies on other subsections of this section.</t> - -<t>For repeatability and comparability reasons, it is important that -given a set of trial results, all implementations of MLRsearch -classify the load equivalently.</t> - -</section> -<section anchor="loss-ratios"><name>Loss Ratios</name> - -<t>Another difference between MLRsearch and <xref target="RFC2544"></xref> binary search is in the goals of the search. -<xref target="RFC2544"></xref> has a single goal, -based on classifying full-length trials as either lossless or lossy.</t> - -<t>MLRsearch, as the name suggests, can search for multiple goals, -differing in their loss ratios. -The precise definition of the Goal Loss Ratio will be given later. -The <xref target="RFC2544"></xref> throughput goal then simply becomes a zero Goal Loss Ratio. -Different goals also may have different Goal Widths.</t> - -<t>A set of trial results for one specific intended load value -can classify the load as an upper bound for some goals, but a lower bound -for some other goals, and undecided for the rest of the goals.</t> - -<t>Therefore, the load classification depends not only on trial results, -but also on the goal. -The overall search procedure becomes more complicated, when -compared to binary search with a single goal, -but most of the complications do not affect the final result, -except for one phenomenon, loss inversion.</t> - -</section> -<section anchor="loss-inversion"><name>Loss Inversion</name> - -<t>In <xref target="RFC2544"></xref> throughput search using bisection, any load with a lossy trial -becomes a hard upper bound, meaning every subsequent trial has a smaller -intended load.</t> - -<t>But in MLRsearch, a load that is classified as an upper bound for one goal -may still be a lower bound for another goal, and due to the other goal -MLRsearch will probably perform trials at even higher loads. -What to do when all such higher load trials happen to have zero loss? -Does it mean the earlier upper bound was not real? -Does it mean the later lossless trials are not considered a lower bound? -Surely we do not want to have an upper bound at a load smaller than a lower bound.</t> - -<t>MLRsearch is conservative in these situations. -The upper bound is considered real, and the lossless trials at higher loads -are considered to be a coincidence, at least when computing the final result.</t> - -<t>This is formalized using new notions, the [Relevant Upper Bound] (#Relevant-Upper-Bound) and -the [Relevant Lower Bound] (#Relevant-Lower-Bound). -Load classification is still based just on the set of trial results -at a given intended load (trials at other loads are ignored), -making it possible to have a lower load classified as an upper bound, -and a higher load classified as a lower bound (for the same goal). -The Relevant Upper Bound (for a goal) is the smallest load classified -as an upper bound. -But the Relevant Lower Bound is not simply -the largest among lower bounds. -It is the largest load among loads -that are lower bounds while also being smaller than the Relevant Upper Bound.</t> - -<t>With these definitions, the Relevant Lower Bound is always smaller -than the Relevant Upper Bound (if both exist), and the two relevant bounds -are used analogously as the two tightest bounds in the binary search. -When they are less than the Goal Width apart, -the relevant bounds are used in the output.</t> - -<t>One consequence is that every trial result can have an impact on the search result. -That means if your SUT (or your traffic generator) needs a warmup, -be sure to warm it up before starting the search.</t> - -</section> -<section anchor="exceed-ratio"><name>Exceed Ratio</name> - -<t>The idea of performing multiple trials at the same load comes from -a model where some trial results (those with high loss) are affected -by infrequent effects, causing poor repeatability of <xref target="RFC2544"></xref> throughput results. -See the discussion about noiseful and noiseless ends -of the SUT performance spectrum in section [DUT in SUT] (#DUT-in-SUT). -Stable results are closer to the noiseless end of the SUT performance spectrum, -so MLRsearch may need to allow some frequency of high-loss trials -to ignore the rare but big effects near the noiseful end.</t> - -<t>MLRsearch can do such trial result filtering, but it needs -a configuration option to tell it how frequent can the infrequent big loss be. -This option is called the exceed ratio. -It tells MLRsearch what ratio of trials -(more exactly what ratio of trial seconds) can have a [Trial Loss Ratio] (#Trial-Loss-Ratio) -larger than the Goal Loss Ratio and still be classified as a lower bound. -Zero exceed ratio means all trials have to have a Trial Loss Ratio -equal to or smaller than the Goal Loss Ratio.</t> - -<t>For explainability reasons, the RECOMMENDED value for exceed ratio is 0.5, -as it simplifies some later concepts by relating them to the concept of median.</t> - -</section> -<section anchor="duration-sum"><name>Duration Sum</name> - -<t>When more than one trial is intended to classify a load, -MLRsearch also needs something that controls the number of trials needed. -Therefore, each goal also has an attribute called duration sum.</t> - -<t>The meaning of a [Goal Duration Sum] (#Goal-Duration-Sum) is that -when a load has (full-length) trials -whose trial durations when summed up give a value at least as big -as the Goal Duration Sum value, -the load is guaranteed to be classified either as an upper bound -or a lower bound for that goal.</t> - -<t>Due to the fact that the duration sum has a big impact -on the overall search duration, and <xref target="RFC2544"></xref> prescribes -wait intervals around trial traffic, -the MLRsearch algorithm is allowed to sum durations that are different -from the actual trial traffic durations.</t> - -<t>In the MLRsearch specification, the different duration values are called -[Trial Effective Duration] (#Trial-Effective-Duration).</t> - -</section> -<section anchor="short-trials"><name>Short Trials</name> - -<t>MLRsearch requires each goal to specify its final trial duration. -Full-length trial is a shorter name for a trial whose intended trial duration -is equal to (or longer than) the goal final trial duration.</t> - -<t>Section 24 of <xref target="RFC2544"></xref> already anticipates possible time savings -when short trials (shorter than full-length trials) are used. -Full-length trials are the opposite of short trials, -so they may also be called long trials.</t> - -<t>Any MLRsearch implementation may include its own configuration options -which control when and how MLRsearch chooses to use short trial durations.</t> - -<t>For explainability reasons, when exceed ratio of 0.5 is used, -it is recommended for the Goal Duration Sum to be an odd multiple -of the full trial durations, so Conditional Throughput becomes identical to -a median of a particular set of trial forwarding rates.</t> - -<t>The presence of short trial results complicates the load classification logic.</t> - -<t>Full details are given later in section [Logic of Load Classification] (#Logic-of-Load-Classification). -In a nutshell, results from short trials -may cause a load to be classified as an upper bound. -This may cause loss inversion, and thus lower the Relevant Lower Bound, -below what would classification say when considering full-length trials only.</t> - - - -</section> -<section anchor="throughput"><name>Throughput</name> - - -<t>Due to the fact that testing equipment takes the intended load as an input parameter -for a trial measurement, any load search algorithm needs to deal -with intended load values internally.</t> - -<t>But in the presence of goals with a non-zero loss ratio, the intended load -usually does not match the user's intuition of what a throughput is. -The forwarding rate (as defined in <xref target="RFC2285"></xref> section 3.6.1) is better, -but it is not obvious how to generalize it -for loads with multiple trial results and a non-zero -[Goal Loss Ratio] (#Goal-Loss-Ratio).</t> - -<t>The best example is also the main motivation: hard limit performance. -Even if the medium allows higher performance, -the SUT interfaces may have their additional own limitations, -e.g. a specific fps limit on the NIC (a very common occurance).</t> - -<t>Ideally, those should be known and used when computing Max Load. -But if Max Load is higher that what interface can receive or transmit, -there will be a "hard limit" observed in trial results. -Imagine the hard limit is at 100 Mfps, Max Load is higher, -and the goal loss ratio is 0.5%. If DUT has no additional losses, -0.5% loss ratio will be achieved at 100.5025 Mfps (the relevant lower bound). -But it is not intuitive to report SUT performance as a value that is -larger than known hard limit. -We need a generalization of RFC2544 throughput, -different from just the relevant lower bound.</t> - -<t>MLRsearch defines one such generalization, called the Conditional Throughput. -It is the trial forwarding rate from one of the trials -performed at the load in question. -Determining which trial exactly is defined in -[MLRsearch Specification] (#MLRsearch-Specification), -and in [Appendix B: Conditional Throughput] (#Appendix-B:-Conditional-Throughput).</t> - -<t>In the hard limit example, 100.5 Mfps load will still have -only 100.0 Mfps forwarding rate, nicely confirming the known limitation.</t> - -<t>Conditional Throughput is partially related to load classification. -If a load is classified as a lower bound for a goal, -the Conditional Throughput can be calculated from trial results, -and guaranteed to show an loss ratio -no larger than the Goal Loss Ratio.</t> - - - - -<t>Note that when comparing the best (all zero loss) and worst case (all loss -just below Goal Loss Ratio), the same Relevant Lower Bound value -may result in the Conditional Throughput differing up to the Goal Loss Ratio.</t> - -<t>Therefore it is rarely needed to set the Goal Width (if expressed -as the relative difference of loads) below the Goal Loss Ratio. -In other words, setting the Goal Width below the Goal Loss Ratio -may cause the Conditional Throughput for a larger loss ratio to become smaller -than a Conditional Throughput for a goal with a smaller Goal Loss Ratio, -which is counter-intuitive, considering they come from the same search. -Therefore it is RECOMMENDED to set the Goal Width to a value no smaller -than the Goal Loss Ratio.</t> - -<t>Overall, this Conditional Throughput does behave well for comparability purposes.</t> - -</section> -<section anchor="search-time"><name>Search Time</name> - -<t>MLRsearch was primarily developed to reduce the time -required to determine a throughput, either the <xref target="RFC2544"></xref> compliant one, -or some generalization thereof. -The art of achieving short search times -is mainly in the smart selection of intended loads (and intended durations) -for the next trial to perform.</t> - -<t>While there is an indirect impact of the load selection on the reported values, -in practice such impact tends to be small, -even for SUTs with quite a broad performance spectrum.</t> - -<t>A typical example of two approaches to load selection leading to different -Relevant Lower Bounds is when the interval is split in a very uneven way. -Any implementation choosing loads very close to the current Relevant Lower Bound -is quite likely to eventually stumble upon a trial result -with poor performance (due to SUT noise). -For an implementation choosing loads very close -to the current Relevant Upper Bound, this is unlikely, -as it examines more loads that can see a performance -close to the noiseless end of the SUT performance spectrum.</t> - -<t>However, as even splits optimize search duration at give precision, -MLRsearch implementations that prioritize minimizing search time -are unlikely to suffer from any such bias.</t> - -<t>Therefore, this document remains quite vague on load selection -and other optimization details, and configuration attributes related to them. -Assuming users prefer libraries that achieve short overall search time, -the definition of the Relevant Lower Bound -should be strict enough to ensure result repeatability -and comparability between different implementations, -while not restricting future implementations much.</t> - - -</section> -<section anchor="rfc2544-compliance"><name><xref target="RFC2544"></xref> Compliance</name> - -<t>Some Search Goal instances lead to results compliant with RFC2544. -See [RFC2544 Goal] (#RFC2544-Goal) for more details -regarding both conditional and unconditional compliance.</t> - -<t>The presence of other Search Goals does not affect the compliance -of this Goal Result. -The Relevant Lower Bound and the Conditional Throughput are in this case -equal to each other, and the value is the <xref target="RFC2544"></xref> throughput.</t> - -</section> -</section> -<section anchor="logic-of-load-classification"><name>Logic of Load Classification</name> - -<section anchor="introductory-remarks"><name>Introductory Remarks</name> - -<t>This chapter continues with explanations, -but this time more precise definitions are needed -for readers to follow the explanations.</t> - -<t>Descriptions in this section are wordy and implementers should read -[MLRsearch Specification] (#MLRsearch-Specification) section -and Appendices for more concise definitions.</t> - -<t>The two areas of focus here are load classification -and the Conditional Throughput.</t> - -<t>To start with [Performance Spectrum] (#Performance-Spectrum) -subsection contains definitions needed to gain insight -into what Conditional Throughput means. -Remaining subsections discuss load classification.</t> - -<t>For load classification, it is useful to define <strong>good trials</strong> and <strong>bad trials</strong>:</t> - -<t><list style="symbols"> - <t><strong>Bad trial</strong>: Trial is called bad (according to a goal) -if its [Trial Loss Ratio] (#Trial-Loss-Ratio) -is larger than the [Goal Loss Ratio] (#Goal-Loss-Ratio).</t> - <t><strong>Good trial</strong>: Trial that is not bad is called good.</t> -</list></t> - -</section> -<section anchor="performance-spectrum"><name>Performance Spectrum</name> -<t>### Description</t> - -<t>There are several equivalent ways to explain the Conditional Throughput -computation. One of the ways relies on performance -spectrum.</t> - -<t>Take an intended load value, a trial duration value, and a finite set -of trial results, with all trials measured at that load value and duration value.</t> - -<t>The performance spectrum is the function that maps -any non-negative real number into a sum of trial durations among all trials -in the set, that has that number, as their trial forwarding rate, -e.g. map to zero if no trial has that particular forwarding rate.</t> - -<t>A related function, defined if there is at least one trial in the set, -is the performance spectrum divided by the sum of the durations -of all trials in the set.</t> - -<t>That function is called the performance probability function, as it satisfies -all the requirements for probability mass function -of a discrete probability distribution, -the one-dimensional random variable being the trial forwarding rate.</t> - -<t>These functions are related to the SUT performance spectrum, -as sampled by the trials in the set.</t> - - -<t>Take a set of all full-length trials performed at the Relevant Lower Bound, -sorted by decreasing trial forwarding rate. -The sum of the durations of those trials -may be less than the Goal Duration Sum, or not. -If it is less, add an imaginary trial result with zero trial forwarding rate, -such that the new sum of durations is equal to the Goal Duration Sum. -This is the set of trials to use.</t> - -<t>If the quantile touches two trials,</t> - - -<t>the larger trial forwarding rate (from the trial result sorted earlier) is used.</t> - - -<t>The resulting quantity is the Conditional Throughput of the goal in question.</t> - - -<t>A set of examples follows.</t> - -<section anchor="first-example"><name>First Example</name> - -<t><list style="symbols"> - <t>[Goal Exceed Ratio] (#Goal-Exceed-Ratio) = 0 and [Goal Duration Sum] (#Goal-Duration-Sum) has been reached.</t> - <t>Conditional Throughput is the smallest trial forwarding rate among the trials.</t> -</list></t> - -</section> -<section anchor="second-example"><name>Second Example</name> - -<t><list style="symbols"> - <t>Goal Exceed Ratio = 0 and Goal Duration Sum has not been reached yet.</t> - <t>Due to the missing duration sum, the worst case may still happen, so the Conditional Throughput is zero.</t> - <t>This is not reported to the user, as this load cannot become the Relevant Lower Bound yet.</t> -</list></t> - -</section> -<section anchor="third-example"><name>Third Example</name> - -<t><list style="symbols"> - <t>Goal Exceed Ratio = 50% and Goal Duration Sum is two seconds.</t> - <t>One trial is present with the duration of one second and zero loss.</t> - <t>The imaginary trial is added with the duration of one second and zero trial forwarding rate.</t> - <t>The median would touch both trials, so the Conditional Throughput is the trial forwarding rate of the one non-imaginary trial.</t> - <t>As that had zero loss, the value is equal to the offered load.</t> -</list></t> - - -</section> -<section anchor="summary"><name>Summary</name> - -<t>While the Conditional Throughput is a generalization of the trial forwarding rate, -its definition is not an obvious one.</t> - -<t>Other than the trial forwarding rate, the other source of intuition -is the quantile in general, and the median the recommended case.</t> - - -</section> -</section> -<section anchor="trials-with-single-duration"><name>Trials with Single Duration</name> - - -<t>When goal attributes are chosen in such a way that every trial has the same -intended duration, the load classification is simpler.</t> - -<t>The following description follows the motivation -of Goal Loss Ratio, Goal Exceed Ratio, and Goal Duration Sum.</t> - -<t>If the sum of the durations of all trials (at the given load) -is less than the Goal Duration Sum, imagine two scenarios:</t> - -<t><list style="symbols"> - <t><strong>best case scenario</strong>: all subsequent trials having zero loss, and</t> - <t><strong>worst case scenario</strong>: all subsequent trials having 100% loss.</t> -</list></t> - -<t>Here we assume there are as many subsequent trials as needed -to make the sum of all trials equal to the Goal Duration Sum.</t> - -<t>The exceed ratio is defined using sums of durations -(and number of trials does not matter), so it does not matter whether -the "subsequent trials" can consist of an integer number of full-length trials.</t> - -<t>In any of the two scenarios, best case and worst case, we can compute the load exceed ratio, -as the duration sum of good trials divided by the duration sum of all trials, -in both cases including the assumed trials.</t> - -<t>Even if, in the best case scenario, the load exceed ratio is larger -than the Goal Exceed Ratio, the load is an upper bound.</t> - -<t>MKP2 Even if, in the worst case scenario, the load exceed ratio is not larger -than the Goal Exceed Ratio, the load is a lower bound.</t> - - -<t>More specifically:</t> - -<t><list style="symbols"> - <t>Take all trials measured at a given load.</t> - <t>The sum of the durations of all bad full-length trials is called the bad sum.</t> - <t>The sum of the durations of all good full-length trials is called the good sum.</t> - <t>The result of adding the bad sum plus the good sum is called the measured sum.</t> - <t>The larger of the measured sum and the Goal Duration Sum is called the whole sum.</t> - <t>The whole sum minus the measured sum is called the missing sum.</t> - <t>The optimistic exceed ratio is the bad sum divided by the whole sum.</t> - <t>The pessimistic exceed ratio is the bad sum plus the missing sum, that divided by the whole sum.</t> - <t>If the optimistic exceed ratio is larger than the Goal Exceed Ratio, the load is classified as an upper bound.</t> - <t>If the pessimistic exceed ratio is not larger than the Goal Exceed Ratio, the load is classified as a lower bound.</t> - <t>Else, the load is classified as undecided.</t> -</list></t> - -<t>The definition of pessimistic exceed ratio is compatible with the logic in -the Conditional Throughput computation, so in this single trial duration case, -a load is a lower bound if and only if the Conditional Throughput -loss ratio is not larger than the Goal Loss Ratio.</t> - - -<t>If it is larger, the load is either an upper bound or undecided.</t> - -</section> -<section anchor="trials-with-short-duration"><name>Trials with Short Duration</name> - -<section anchor="scenarios"><name>Scenarios</name> - -<t>Trials with intended duration smaller than the goal final trial duration -are called short trials. -The motivation for load classification logic in the presence of short trials -is based around a counter-factual case: What would the trial result be -if a short trial has been measured as a full-length trial instead?</t> - -<t>There are three main scenarios where human intuition guides -the intended behavior of load classification.</t> - -<section anchor="false-good-scenario"><name>False Good Scenario</name> - -<t>The user had their reason for not configuring a shorter goal -final trial duration. -Perhaps SUT has buffers that may get full at longer -trial durations. -Perhaps SUT shows periodic decreases in performance -the user does not want to be treated as noise.</t> - -<t>In any case, many good short trials may become bad full-length trials -in the counter-factual case.</t> - -<t>In extreme cases, there are plenty of good short trials and no bad short trials.</t> - -<t>In this scenario, we want the load classification NOT to classify the load -as a lower bound, despite the abundance of good short trials.</t> - - -<t>Effectively, we want the good short trials to be ignored, so they -do not contribute to comparisons with the Goal Duration Sum.</t> - -</section> -<section anchor="true-bad-scenario"><name>True Bad Scenario</name> - -<t>When there is a frame loss in a short trial, -the counter-factual full-length trial is expected to lose at least as many -frames.</t> - -<t>In practice, bad short trials are rarely turning into -good full-length trials.</t> - -<t>In extreme cases, there are no good short trials.</t> - -<t>In this scenario, we want the load classification -to classify the load as an upper bound just based on the abundance -of short bad trials.</t> - -<t>Effectively, we want the bad short trials -to contribute to comparisons with the Goal Duration Sum, -so the load can be classified sooner.</t> - -</section> -<section anchor="balanced-scenario"><name>Balanced Scenario</name> - -<t>Some SUTs are quite indifferent to trial duration. -Performance probability function constructed from short trial results -is likely to be similar to the performance probability function constructed -from full-length trial results (perhaps with larger dispersion, -but without a big impact on the median quantiles overall).</t> - - -<t>For a moderate Goal Exceed Ratio value, this may mean there are both -good short trials and bad short trials.</t> - -<t>This scenario is there just to invalidate a simple heuristic -of always ignoring good short trials and never ignoring bad short trials, -as that simple heuristic would be too biased.</t> - -<t>Yes, the short bad trials -are likely to turn into full-length bad trials in the counter-factual case, -but there is no information on what would the good short trials turn into.</t> - -<t>The only way to decide safely is to do more trials at full length, -the same as in False Good Scenario.</t> - -</section> -</section> -<section anchor="classification-logic"><name>Classification Logic</name> - -<t>MLRsearch picks a particular logic for load classification -in the presence of short trials, but it is still RECOMMENDED -to use configurations that imply no short trials, -so the possible inefficiencies in that logic -do not affect the result, and the result has better explainability.</t> - -<t>With that said, the logic differs from the single trial duration case -only in different definition of the bad sum. -The good sum is still the sum across all good full-length trials.</t> - -<t>Few more notions are needed for defining the new bad sum:</t> - -<t><list style="symbols"> - <t>The sum of durations of all bad full-length trials is called the bad long sum.</t> - <t>The sum of durations of all bad short trials is called the bad short sum.</t> - <t>The sum of durations of all good short trials is called the good short sum.</t> - <t>One minus the Goal Exceed Ratio is called the subceed ratio.</t> - <t>The Goal Exceed Ratio divided by the subceed ratio is called the exceed coefficient.</t> - <t>The good short sum multiplied by the exceed coefficient is called the balancing sum.</t> - <t>The bad short sum minus the balancing sum is called the excess sum.</t> - <t>If the excess sum is negative, the bad sum is equal to the bad long sum.</t> - <t>Otherwise, the bad sum is equal to the bad long sum plus the excess sum.</t> -</list></t> - -<t>Here is how the new definition of the bad sum fares in the three scenarios, -where the load is close to what would the relevant bounds be -if only full-length trials were used for the search.</t> - -<section anchor="false-good-scenario-1"><name>False Good Scenario</name> - -<t>If the duration is too short, we expect to see a higher frequency -of good short trials. -This could lead to a negative excess sum, -which has no impact, hence the load classification is given just by -full-length trials. -Thus, MLRsearch using too short trials has no detrimental effect -on result comparability in this scenario. -But also using short trials does not help with overall search duration, -probably making it worse.</t> - -</section> -<section anchor="true-bad-scenario-1"><name>True Bad Scenario</name> - -<t>Settings with a small exceed ratio -have a small exceed coefficient, so the impact of the good short sum is small, -and the bad short sum is almost wholly converted into excess sum, -thus bad short trials have almost as big an impact as full-length bad trials. -The same conclusion applies to moderate exceed ratio values -when the good short sum is small. -Thus, short trials can cause a load to get classified as an upper bound earlier, -bringing time savings (while not affecting comparability).</t> - -</section> -<section anchor="balanced-scenario-1"><name>Balanced Scenario</name> - -<t>Here excess sum is small in absolute value, as the balancing sum -is expected to be similar to the bad short sum. -Once again, full-length trials are needed for final load classification; -but usage of short trials probably means MLRsearch needed -a shorter overall search time before selecting this load for measurement, -thus bringing time savings (while not affecting comparability).</t> - -<t>Note that in presence of short trial results, -the comparibility between the load classification -and the Conditional Throughput is only partial. -The Conditional Throughput still comes from a good long trial, -but a load higher than the Relevant Lower Bound may also compute to a good value.</t> - -</section> -</section> -</section> -<section anchor="trials-with-longer-duration"><name>Trials with Longer Duration</name> - -<t>If there are trial results with an intended duration larger -than the goal trial duration, the precise definitions -in Appendix A and Appendix B treat them in exactly the same way -as trials with duration equal to the goal trial duration.</t> - -<t>But in configurations with moderate (including 0.5) or small -Goal Exceed Ratio and small Goal Loss Ratio (especially zero), -bad trials with longer than goal durations may bias the search -towards the lower load values, as the noiseful end of the spectrum -gets a larger probability of causing the loss within the longer trials.</t> - - - - -</section> -</section> -<section anchor="iana-considerations"><name>IANA Considerations</name> - -<t>No requests of IANA.</t> - -</section> -<section anchor="security-considerations"><name>Security Considerations</name> - -<t>Benchmarking activities as described in this memo are limited to -technology characterization of a DUT/SUT using controlled stimuli in a -laboratory environment, with dedicated address space and the constraints -specified in the sections above.</t> - -<t>The benchmarking network topology will be an independent test setup and -MUST NOT be connected to devices that may forward the test traffic into -a production network or misroute traffic to the test management network.</t> - -<t>Further, benchmarking is performed on a "black-box" basis, relying -solely on measurements observable external to the DUT/SUT.</t> - -<t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for -benchmarking purposes. Any implications for network security arising -from the DUT/SUT SHOULD be identical in the lab and in production -networks.</t> - -</section> -<section anchor="acknowledgements"><name>Acknowledgements</name> - -<t>Some phrases and statements in this document were created -with help of Mistral AI (mistral.ai).</t> - -<t>Many thanks to Alec Hothan of the OPNFV NFVbench project for thorough -review and numerous useful comments and suggestions in the earlier versions of this document.</t> - -<t>Special wholehearted gratitude and thanks to the late Al Morton for his -thorough reviews filled with very specific feedback and constructive -guidelines. Thank you Al for the close collaboration over the years, -for your continuous unwavering encouragement full of empathy and -positive attitude. Al, you are dearly missed.</t> - -</section> -<section anchor="appendix-a-load-classification"><name>Appendix A: Load Classification</name> - -<t>This section specifies how to perform the load classification.</t> - -<t>Any intended load value can be classified, according to a given [Search Goal] (#Search-Goal).</t> - -<t>The algorithm uses (some subsets of) the set of all available trial results -from trials measured at a given intended load at the end of the search. -All durations are those returned by the Measurer.</t> - -<t>The block at the end of this appendix holds pseudocode -which computes two values, stored in variables named -<spanx style="verb">optimistic</spanx> and <spanx style="verb">pessimistic</spanx>.</t> - - -<t>The pseudocode happens to be a valid Python code.</t> - -<t>If values of both variables are computed to be true, the load in question -is classified as a lower bound according to the given Search Goal. -If values of both variables are false, the load is classified as an upper bound. -Otherwise, the load is classified as undecided.</t> - -<t>The pseudocode expects the following variables to hold values as follows:</t> - -<t><list style="symbols"> - <t><spanx style="verb">goal_duration_sum</spanx>: The duration sum value of the given Search Goal.</t> - <t><spanx style="verb">goal_exceed_ratio</spanx>: The exceed ratio value of the given Search Goal.</t> - <t><spanx style="verb">good_long_sum</spanx>: Sum of durations across trials with trial duration -at least equal to the goal final trial duration and with a Trial Loss Ratio -not higher than the Goal Loss Ratio.</t> - <t><spanx style="verb">bad_long_sum</spanx>: Sum of durations across trials with trial duration -at least equal to the goal final trial duration and with a Trial Loss Ratio -higher than the Goal Loss Ratio.</t> - <t><spanx style="verb">good_short_sum</spanx>: Sum of durations across trials with trial duration -shorter than the goal final trial duration and with a Trial Loss Ratio -not higher than the Goal Loss Ratio.</t> - <t><spanx style="verb">bad_short_sum</spanx>: Sum of durations across trials with trial duration -shorter than the goal final trial duration and with a Trial Loss Ratio -higher than the Goal Loss Ratio.</t> -</list></t> - -<t>The code works correctly also when there are no trial results at a given load.</t> - -<figure><sourcecode type="python"><![CDATA[ -balancing_sum = good_short_sum * goal_exceed_ratio / (1.0 - goal_exceed_ratio) -effective_bad_sum = bad_long_sum + max(0.0, bad_short_sum - balancing_sum) -effective_whole_sum = max(good_long_sum + effective_bad_sum, goal_duration_sum) -quantile_duration_sum = effective_whole_sum * goal_exceed_ratio -optimistic = effective_bad_sum <= quantile_duration_sum -pessimistic = (effective_whole_sum - good_long_sum) <= quantile_duration_sum -]]></sourcecode></figure> - -</section> -<section anchor="appendix-b-conditional-throughput"><name>Appendix B: Conditional Throughput</name> - -<t>This section specifies how to compute Conditional Throughput, as referred to in section [Conditional Throughput] (#Conditional-Throughput).</t> - -<t>Any intended load value can be used as the basis for the following computation, -but only the Relevant Lower Bound (at the end of the search) -leads to the value called the Conditional Throughput for a given Search Goal.</t> - -<t>The algorithm uses (some subsets of) the set of all available trial results -from trials measured at a given intended load at the end of the search. -All durations are those returned by the Measurer.</t> - -<t>The block at the end of this appendix holds pseudocode -which computes a value stored as variable <spanx style="verb">conditional_throughput</spanx>.</t> - - -<t>The pseudocode happens to be a valid Python code.</t> - -<t>The pseudocode expects the following variables to hold values as follows:</t> - -<t><list style="symbols"> - <t><spanx style="verb">goal_duration_sum</spanx>: The duration sum value of the given Search Goal.</t> - <t><spanx style="verb">goal_exceed_ratio</spanx>: The exceed ratio value of the given Search Goal.</t> - <t><spanx style="verb">good_long_sum</spanx>: Sum of durations across trials with trial duration -at least equal to the goal final trial duration and with a Trial Loss Ratio -not higher than the Goal Loss Ratio.</t> - <t><spanx style="verb">bad_long_sum</spanx>: Sum of durations across trials with trial duration -at least equal to the goal final trial duration and with a Trial Loss Ratio -higher than the Goal Loss Ratio.</t> - <t><spanx style="verb">long_trials</spanx>: An iterable of all trial results from trials with trial duration -at least equal to the goal final trial duration, -sorted by increasing the Trial Loss Ratio. -A trial result is a composite with the following two attributes available: <list style="symbols"> - <t><spanx style="verb">trial.loss_ratio</spanx>: The Trial Loss Ratio as measured for this trial.</t> - <t><spanx style="verb">trial.duration</spanx>: The trial duration of this trial.</t> - </list></t> -</list></t> - -<t>The code works correctly only when there if there is at least one -trial result measured at a given load.</t> - -<figure><sourcecode type="python"><![CDATA[ -all_long_sum = max(goal_duration_sum, good_long_sum + bad_long_sum) -remaining = all_long_sum * (1.0 - goal_exceed_ratio) -quantile_loss_ratio = None -for trial in long_trials: - if quantile_loss_ratio is None or remaining > 0.0: - quantile_loss_ratio = trial.loss_ratio - remaining -= trial.duration - else: - break -else: - if remaining > 0.0: - quantile_loss_ratio = 1.0 -conditional_throughput = intended_load * (1.0 - quantile_loss_ratio) -]]></sourcecode></figure> - -</section> - - - </middle> - - <back> - - -<references title='References' anchor="sec-combined-references"> - - <references title='Normative References' anchor="sec-normative-references"> - -&RFC1242; -&RFC2285; -&RFC2544; -&RFC8219; -&RFC9004; - - - </references> - - <references title='Informative References' anchor="sec-informative-references"> - -<reference anchor="TST009" target="https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf"> - <front> - <title>TST 009</title> - <author > - <organization></organization> - </author> - <date year="n.d."/> - </front> -</reference> -<reference anchor="FDio-CSIT-MLRsearch" target="https://csit.fd.io/cdocs/methodology/measurements/data_plane_throughput/mlr_search/"> - <front> - <title>FD.io CSIT Test Methodology - MLRsearch</title> - <author > - <organization></organization> - </author> - <date year="2023" month="October"/> - </front> -</reference> -<reference anchor="PyPI-MLRsearch" target="https://pypi.org/project/MLRsearch/1.2.1/"> - <front> - <title>MLRsearch 1.2.1, Python Package Index</title> - <author > - <organization></organization> - </author> - <date year="2023" month="October"/> - </front> -</reference> - - - </references> - -</references> - - -<?line 3102?> - - - - - </back> - -<!-- ##markdown-source: -H4sIAAAAAAAAA+S9+3MbV5Iu+Hv9FRXsmGiyB4Qo+THd2pnVypLVw9uWpZFo -a+/t7ehbBApEjQAUuqogmt7Y/30zv8w8J089QMp2xOzGnYiJlgmg6jzy5MnH -l1+en59nWVd1m/Jp/vqw6ar9psy/q9s2f1d0VZ2/L4tmsc6K6+um/ERf+e5d -K39Z1otdsaVfLZti1Z1XZbc6v97e3pxvN4185fziX7Jl0dFXnlw8+ZL+6/zx -v2RZtW+e5l1zaLsnFxd/uniSFU1ZPM3rfZvd3jzNvyl3i/W2aD5Wu5v8Qy3/ -++emPuyzj7dP88tdVza7sjt/yW/NFkX3NK92qzrLFvWSvvo0P7TnRbuoqmxf -Pc3p/36XL4od/bXMi6Yp7vLTapUXm01+V7Zned3k66Jd5+uyKbM87+rFU/6A -/tnWTdeUq/YpHrEsVwUtTkvfsM/vtvIx/2dWHLp13TzNcvzfuf5vTkOjb7ye -53+pd21X7Lq7XX1bLX4On8sKvi4WVflx8kt1Q9N6UbUL2o27tiu3bfio3BbV -5mm+/Sg//T8W/K35ot6Oj+THef623hQfe+//sSm6j3Xvo/vf+qnZ8y/cS7Nd -3WxJbD6VvBTvXr14/OTLJ/rPJ0/++JX986svv9R//vHJ4z/pP/90cUF/zXg3 -3UOu3l+RmMjKdkVzU9KGr7tu3z599Oj29nZedm01p7E+WpYb+knziP/w95v2 -0fevfjynHz+6uHj894s//Yn+l/7/i/nFl3P6w9cXj27av+tX6JNPF19cfHnx -eD/fL1fyKjkRJ/RxTp+f0B9fvazq8xfvL6/OwyEYH9airbr5ajmv6kcLOiXt -o21J0rGsN/XNHf27aA9NuS13XfuITkfx9/2m2JV/79Yk4zfr/aF7RAfo7/L8 -R8lYXr2kR+Y8gvyqbLv8dXxsfh5P5gl+FA7eF+ePL+gvb+/eXt438P3dXtZy -39T/WS66R+H7jx7Pn8wfp8MJH+b4cEavoPHs8rfF4mNxU9JRXZY/jQ0mOz8/ -z4vrtmuKRZdlV+uqzWmdDrwmOb16X7dlm5c/deWurUiw+dD9VcXmb3lcqFxf -f32X0fmsdqwpinxX3uZuwen0bzblclS3qSY7DTM5m8dlzItqi1dv6cnb6ufS -Xrc8NPzr3SxrD/s9qYl8a8/e8LMb9+yynWXFbpmXu3WxW5R5U7b0XfqffUlb -f11tqu4u5y/Q6dkXjf5lzotS0lJUpAjv6NtFS+tKx0IWhfWcW5AKoyyWS3o4 -/ZN+uFjznHc3pJn42U35j0OlIkcPLVv6B63I9R2+XH4qNgdMCAPpSLLo+Vm9 -IhW46m5JO59fFy19n7TurerjVhTC73ljixwC3PKg6/yGziAr26bNt3VT5qum -LJf1dubWlXb4U7WkHaYhV/zeYkPT362qG11Yugv4f1paX96GltU1aUR6b1jo -ds3r3jVVsaEplQ2tfLFkHdeV9L809npHu8HiLV+aZV29KfnxkJFF2XRFteNf -LmgxWFrpN/imblGb31Yd3QvVzRpPb3Ujdcv5Mbx4OiXemjC2m5oHhZ9/ou3j -r0a54GVi8d9Wy+WmzLL/++lT2nremv8nw1F5Tj/FCuZ/aYrtsr7d8e7S+pBu -62g56y1dF81H/mCGH+i3Oz5Ft3S/0UTs+tQnt/mu7vgp12X+qWqraxokTZ7H -35A00dW3lEt8jgeaNqALZfGxpteuaPVZvT8qHn355IsnX3zxpwt88ZIuUnoy -LxAJW0UrTkOgsSxnYbz0UhvY03sf/uTij3/86k9/vJCF+Ctdxn/jBXoUFyj7 -Xf720LB+gKy+X9T7Us+K/pm3MdEncjpI3BZNRfN/mBqYkYxE0R5ROV7BsLSy -flhCDOzQZO64vPzhivf9+X6/gTh8KnbVZlOMK7XrqiXVy8eAxm1PwyMyE03a -vLaktSNppcNE27lt6eI8z7+pdqwvdIxd8bHkydckfiz1fCLbcGj4oUs+Jaui -EbGijaSFp+fuugOduDuazoEVQhjanF5xBaGBPqH5Vnx25diYVsTG7IsDq/Br -WoKy3Okr6eE0WoyFRa/GBDZ9rcovee9nzWf9liS4au/SE8riv6HjjrNICpBW -jjQT6TfRACQOIupLNk8+QcrdMrPWY6XFUwp/1JnR6siRpceUP9ElRYvxc9nU -tE5kL7Wz/Jq+y3OodkuyY5s7GsmKf0tLR1+FumpFGrY8Rf76rt6d4xn8YHtx -mcvFJVpv5cdXyald0Ayb/HZtqxjUEy9PtaNz1lak9HadqF9/C5ho5HwO6AU0 -/+KaFoJ0IX0a9TFr/ESg2325qFbVQvaz3O439Z08clWbJtYLDeoFsvecP5lQ -0D29zPskkzHNzYrnPH+zl9tgczfLVV3TOf8sJT0PI5HJ8VAT5SzuTP5n1tEz -+fmyWq3KZqCmeUTPd6nMwZUo6OuLLi8Leo57HJ9KurF3C34Pj+Ny15beOFjU -RcOKGkaXiEf/GTP6S7OpaDK0TiQAdHfCK+EtoWO1wdZWW1oIO1Qyzm/1V/Zs -+qBm+wUv4V/RZ3QwyFDHCeOf/EBDeVU3dMxwgN7xWp++eneWFzTk4qdqe9jS -cq9wbHiLRCerHf+3/NSU1Bfzr+dPzniMkONiw5aSnA38p40JAk8aiZaQJBcC -vSzp2/Ru7IKX5XSTZbzvSr530zNMZwQHjGdO4ig3Mv5AEg2LIMiHSgc/6b3+ -BF+gDV1WbJ7KiVLrnFVb00LhQWkHdYvftOVGJ78uDw0NuVrgkmmLTyX0p5g+ -vFH0QjazxApY01pDU32qq2V+2O3KBW1N0VT0AlLcDUntNatcvive11u9zMhc -S04bxgkXaYNLh/R6PMvJ0ZWD3rC3tuufWTyFHNyik0dUdMrxiRz6ZUn2EZ0Q -esKBvksHqfyEKfBSJrYqlrimzdjQJQi7aHXoDqyctmzoydtoQnGMo7aezGpT -/gSVXe54i2mtgol9TXYG/5Ik+RN8Q9qFjl/X4sIpbm5Y7/m/z2EXjP+Erw3e -CZWw7MDCZ8Yo7QbPcFORKy4CFO7pWcaqnK8wkp/exYVx0I3flql5P89ewBgi -qSG9NjLQMBrozOGDYXfSVcofJU+O95C/FHBrTIyf9uH7mm5nWlCSrcSFqHbs -WZRLtRPr65Y0cHATnEU1Zyvscsl3yqqiz9/qNaOOHPkeex6pWVy9u0iUJ4vJ -odVpsKtBhnJVH2DKQ7TZV1JPxF1NFbtTLM18ydI14q0d1nRxQtMb6s2Z7He/ -IzuQXqEq+KWteGKWQ/F9R4eTtHvd0ID/Jgbqv7KZ/b+//svbJ/lff3z7t/zq -zcs3T+l59Ue5cuTWhRHRFxW6KkjSZXo05nb+r4/wsGMPJmubtxZG2UzsBtp4 -Oo8dLaqsuPrAJ9EzOYlPTgxpPhp0NEkz80o7Xw8ml16MZtLEDw97VZV85LLd -YXvNV9VKr1D+Kiwk3m5Wbdi9eqk/SM49beSa1c5uwd4tixkfcB54fGpfGNgA -VsuThJSOVLUSF5bGrEMTF3Zd2tgX8Ica0rXisbGLgEfNRVq3YiM7cxTWUPlT -uTiIDc63bdTppNTojIoc0YVzXdIAS3XXF4cGSzBq1Ts7jw4r3W+LTk9GzTce -XXcZCw2/7rxckQavMPhUiT6XQdCm8Vih/d0raH0X5Z7/97DhiEPBmnR1aNRB -i7o/u11X9BEfpE6vBjmjaiBAHnAhNHr4Wp1i9E3k7zO5yB+uPiuO6yxEAfK1 -ys6f7ETF7sKiQJyW3n6zwzVGjyi2JPHQV9gLvvP3vMzBBMrwEr6ZeUWLbsbH -qym7pubbkP4TV4vzctRvoTuubLbVrkx9nCzu37IuxQqXa5UjNTTmrt7vxbvW -OeO4D3zEWUY2mERCig07JXdkAHxiCaX7dAFLjp0WXJs8hKZYludkcWXmNY3d -L3IT09edNffv9S1bJzOxJlcHEuKvL85po2o+C2KBL+sDSUOb7jNGyNPbFmbn -trBwr2E11Pweu91pTw7XiE5XMIPEOXKjYF3K55DW/v0PV7qKMBVFVZGXcI4v -FC2sMBYndZF5/cwOXZKdwce9Jsmq2KChTT9sDpAPM0dpIaAn6SDt+XqPVtuo -ffp4/vgMXmX68gX5MvwduYl5NjYcGUN7dBAYAIemctZyMFk6i+U9eFxPzhJx -U+MN7ifUEI3rsJczVOwQ+2vgcNNnbCZgVIXForJBfG4mBh7/oSKdVkER+DHT -dsw4BSJ/w2/4Mtb5z/iltGg0xkuxrRdFm14W8ZUzfIGXWM14yAcLV823tWrp -TN5kPyfddkP3hb8uvIcNL6ZeZWzAlQ3MZ/yM17jeWzRPxk1yv1voxYIFY02Q -939JO1MfGt7blv7GorRoWJRZ8etgcCdtDiyMMJ/7L6LV+HPF9wFeYJOuMC95 -Jhy+c9LqrL3K3aeqqXesdzPS6Gw5y0Tt3sJksDP8Ibl9pKBn4dMt7ndSV0sO -pyG8kJU/0ZAqXNxstDW4eRdlVG/9IWe0xfKKsPLxK7W6TXxty3Il3odahuEl -qh7Y1L9h32GekRMJB5+vl1lGmnGnIa+wvaQaSyjnOvgn+Yu3P9CEG41QV2Rs -miDzFY1P8C7SMByQYqnq6ADkLW0u3Hoa5Us6Z6Q7OdCFsC5t4opOHDwI1kzF -QQSCpk3TwRKyLwULlK9+1uUV/YiuOk5E6Jr/vg2KwBkgczkDzg6emW+mz+Nl -jTqF9qkpadHUWCG5ZzHhKFY5E3uiWkn43f2cjwt987Cj0S/JQuAbnI/MLYua -D7RKuIIDWxVdtDusFU0Ej+eF4WPI9wwfoPKndXVN53+1OSy6g6yyt6yqjjyT -1QyXmGQa2owH0pQbOIcI4eLoQEmyQuDAAQfG2ESg+wnqWkN+sGzVeOKhcwhD -I6OQI9ZfJCtdSfeUO/dYXlxDrGNarHTGO3Rdiurjm5yvvbI5J3uSzAP6hK5T -MshncF/J1IDZI64Le3zwI2mhMxpSka/v9ryA5KrzABZ0OMmYZ+uF9QlHIVUv -yAxlJbEnJV/QOxH195yPsPeqK9P2IlJ4G8fhyYKqNmR10NqEJ5IJJ3cwyVe+ -J9OJBHNn3q+FB5biGByqlmyIqid18muyW/Lf44m/t5g7/zIomCxY02HR/ZYH -UVD3JGhnNZ7JwDywdyGroF+Qm+C5Woaw6ze2XP7hXv/T1lWI8rCpiqchhOW+ -LlkVlwmSd+qpEO1SRLXxnm8seAljD4MabltaKHNjF5pS10uBd1rEQ2Qp/JD9 -T12P6xLKpKNtpUtInqSbx2aDKAtbFBktnzVO9SCrQkaqaBz+F0coN0ty/DOc -eR4EBEQvYIiOPDc+Kf72mm/Ym011wypN7sQYZs4qjkTcsn0JpW/T6u8HnY7w -WbXrh9aCWsN+zvDrYPaKvWqDTddrc6ARVfydVjeoaw50O77ZsS20tAWyD5CH -4eU7Lg76YLl5VeEt7dLRTG4hP4UC6SnoPWeUyXaZJdPMFhv4fXXcMv7x2CjX -xZ6ci1aMloZO5eZOtkcCOzCf+6OlqYlMwjmTH/HKsKfCdjw2PO50A59RfkE3 -GudlR8fEO+FXD7ZUUG0ZuUW7m/5uPieFP/JkrPLEo9klO+w21UcedYj96Ftm -IhtwvMw7w2UqeY3krDKOJ4/PqRfkD8fgtwg0+TKSgNXlyH7YYWxm+aopdItb -MtF7v+eA+QImeKYmS/Cc6YVe5++WrZ+o3y+b9IyvJnLj6jZecLZiteSyP7Bj -i1c6K+mlqZ9rj1KiBeAQZbvGDQQJ5wBVtc1YJ9v9EQ3heMHAZOP/9KiQ6NFl -nXrGGlrgODGHgMzIqbzEFzhgTamWBN/lG8SmNZqa4YTTYf1U0L3EP0/vmv6w -2IjwF8UsExgAe6JAO7W/z50pCjUeHr0hu4wMzAOCI89ZJcu+35abzTmUcnPA -hR4U+kuxolWAyCjJDP7gN5DutW5T7srFx2iQscUw4zQ4qwmNiwVbLsN2TCgb -WbLpMzIuOiWkVTOk3vF3rrjoLv970ulNtWBjVEENbF3tFnczmDka2OpBIloa -vIbLELllW5dNwRCSDUGlGdZvdJYZrEADsagWDpfYYJeRJ9YhBhHCZC2hikBV -AaTHrpf8TeL8PqVoWBqy8z+Vcb93sswSHM5oWhzy3QaX9vN2JpOjGkMqMZLY -v/Q4r+dDa6YDhon+YnPDMd81OdM0B759Oo3FZ0n6asQlloADjZunCld+ZjpC -cqoZI6C6ksPaNLZFfbOrfnbBuNZis85zCEgY/o4ltjnGmZ3SbH7iJ9G6nB2T -hs5FazckrRuNUJO0kePCMYzUYuXAzrsBXumFzwGNx8wONzfsMyDNwj8fri5d -WPQoaJv/PLB7sSuzI9l62Bt0yPnp16WP3zFCwFIjGz0kdjfPs7d13cuc8PLt -kHfB1u/5C0lWCxdEXKdE3XclRwn41NbXQBEZymeALpCQrnnXUHUc/SyXtoPB -r7FXsQuC9QhYqRhgG40rJ9iu068vcon7mVfCoTK6CySbruIUDDqJ1XMs78yS -2r1DYPkxRQvF9D85OGIXsaAEj+ionVV17fix9Q5Dg+DEglR6SJ3EUDDpIM7s -J96DZcA20RoKAVML7gMXFJ4J2az1TTIv+R0Ow8BES7NM5uVxrgzxwYLxU3nB -Dl+IvyRaKxtxkcJdEjByATpCj8Xfbkn1w1xgh4DzgpnGZ+Hm8tAOfPOJyoP8 -JtuW5B8zAe14+VeVa5a1wkYk2+ewlBGxtyhpWcsMJ6uy5B9b/i5rjwXsJW7U -8al4/6Nu2JalbJtkQo6gYHZJ8vL5YlE3Bvg5dheYOzOWNiVXkDE5Za7xu9FL -Jjo0Th7o9pKQkr1+5BLDw9flZi/RF41FTqw/TSlJDu8YPNTpnX8P7FX3Qm6x -tvd8vWfUA66A8WOVhmkuS0aTWA7FG6S4GmUIO0mZS3peUXRw/e84nRZlI7Xw -ZxoAR86ePYWMgz0KkTgt5zdzMhJpfjdiMLJvvuSwB4ecMekzTQepYHJkQ+0N -zj9InB4JPPpJyfdxmpYThVB1LrHXT6OY6MekHv2FLGQSn6Xcdg4Qhu37vt6d -/w+GbjFmUPQw49rTYP6/uJ+dWc7D71rRCgKSD6MBbATh1GmaYcfgKJU/y3VI -PFyyDYzYa8hridhdidTLCd/hemyLO33P+4oPRCmh6jKkbAD5g+Jnv9TQ2QWf -RLrXsO7q57kcHJ0lem6OvKGATgvFeugFZxAsaABaz65e1Bs5JoArHToBq4pb -fJCLgT79uCOFzg8o2ALdhIXhIUmOpzCjN85XoiQCDKG5B6Mb8cORq5rTbXuk -sNhrnWUJFE/h2uoKMHbXJlU1BvcRWyV7XdORKCJchd8H7NtrUjbNLpm3otfi -FauqiJdBLI3oeWKJkEiNosWnIx7+W/brgpPMEOL8Sq4DWr9iyXvTsvLia0iv -lEKCT4g3BRMkQtdzS72bKbJYO/sA3n1ftzo3gWFNpUCKFKijQhPni0G+V/hj -o9e7Ww58H4IsPidiTcH7D7gujc3npqpMe7gnyQYmMa51gH0jSadhB3XKiqYL -AQKsilz/pgjZsEP0HwsJk2ZzJyi8ClDqsSim3Mf0l3WxbyUsZwuLjXJwY6zd -RsUewEEHRYbdiKUDuDov9jDrt0W8G338wPy43eib+KjxAZPt7kynshZVce+9 -nl78rrwhfYz56dtoSNWS75Ra6qYsrr2NpyFWQvDRSY+XoOFFHSN93vOrZJk5 -T36+4lDCkiNuo9eexsCSr+YWH28wbprAKwE8bHFhB3dJ8vAhAXU3WrIhRrsV -bJAnoUaQWMuGlq0ZPrDEUjQkCJ9YPwpsj5YhMVYVcyFLLTpv5xa2vxKcd7sf -tCpmBRsYKbB9xBrKgjWkZlAASSqa1TsD7h0Sddf8zsxZL8ADLElLFvuA7cZO -84ql2w44BvkNS8Yili7OogFjqc1I5XcsMj/L5G7UzS8/wfgUWzuMHVrb6epo -sOpMOc4NASKFB/+WzEMDDxkYP8HdyxbLQZbdDb5cAMrhhZJmZhNH7eGwDIOz -9Y1YJjoT/LC+/gTYG/uvXe2hbL1qEv4uVmONoADExx25cUkR5CcPKpqbZN5c -eqSt6Ix36udlH4AM0vXv+bSh2soyfOoiAGfFN0fmopgKJ5lpTBL58aZqPwJS -TwM4cF4KyIQ4mgWtYJbUDLiQDZtN+w2723Gjdnzvx8e1IyDiVtY9KhQ6XZxt -va7VcZSolljN7KVOwpAlF8d5u02xkExnBU/wp05cYUCG5VOJ/csgn3w50zwj -oDmpjyqGhMHaZhmHfQ5bUj13Id5Og7wz1Z2MzrkWsnsSOIjA+/gWGaBCgoYj -/Hr+hY2RJ9geFgrRwuk+x4FWrJUMV4CES670GRmxwBHZ0Oz9PBNhMKdTTHqB -GCYzszPkfkoS8K0oJANolFJ2YIGOLt7nOJin+CedjRjFSRflLKxbLima+MX+ -RTpPX1TEioNf9CKzbYavQWCSzioHw47EIpFVV/W/YIWs+oNBEtXuUGbuelFz -yMcTEYHunbt59jL44O24Ez5mmBpqISRoNdfAxhJ8xSI/7CraYitNoxtoiXt1 -fBDZpWRPTG8EV70XIpAyz38cCkFdyR2BpHcER7aZm4M3TESsTiPyJpQKpU60 -x0yezRQ1KXavxOUcUJ1fsCh9qponLBBziXn7hORg3qyjSIdVG00P+Omy7rzh -Msr8sOdIP6z1umk4XqcvwaqW2bTyincBI+7Lnb8RXPnAe18+oIBu0xIR0D1R -bhCBU7kAaRfrHacOsqWTLIcHscJTDieuS2xEAeu8WhwAnODocQw9CahUnZqp -McxHoNt/rmvJyuGtF39kR/GOZYpuo29ef/izKO5nCaqbIdf5j2+f0oXKQWHy -5N75QCuLzjvTgqJiH72XeDcKV/lB/Lubptiv22f5aUzgJQFbKeJCIP/QAnoL -lShANi3e9x9KVAeJxAUtkZgwqCEIOM0lNoBOh0Ik49vOxqHgZBG8oWv0U0U/ -yaa2Nm5+YWhJK+A2AJ6Dr2WFQPzJ0guwPE2ThERvtaNjKStJ4+Z/A12Sbjpu -qaJainKDQjm2+5mEGQ0uIHhoqWVcVmw58msZjbUrNvUNG15+0IDPiimHcH6y -U2b94eAxvBPL7Ita9EFqqrwWU8ih+hBDI5kTv19NLDOToLFfkPpualwNE7+S -yiOLVwPxzAu4jKB6fs5rmt/NkVc35blh8REgau4gRAL/4c/r5UGNTGyHhPPE -hnB/4OzFplpULNWAdLWp+XmqTqab1iU2/SyTcgC/krCW1QzNT91P3ohwnI2d -7ONFGVI28Uos+RtWAkDkCVZSt9NNZ8Zoja1CP/jy4qnx6dTL7kglRVhwErXB -nMVu7/3xY1nuBbRnYQITmEyuAXjaA6S5FamZlKGSRww3LzxhGEEKq9bQM9E0 -mMPa4KOKrB+KgTWuaBOS+gY5U8OpZadAwbGx597elN2h2bVnoYI1BKAVXmLl -R34kk4qH7RbVPOTi417opWIk4Ky6mO98DtyNLN08v2ymHgEQIJdMcryA7XAY -3RwJQlgqWnZN/xYYe5H58cnzA3hCyg43DAOOtxbQJgEuLlaF5AMWnL7SXxmE -W4oAJKLGZXqC/NdJscOodXvXvCyaE4cny/u1IzNib16tHgQyam/g+GMc4lMk -oQ1Zos5VWQjeFQk/tflOC5+dgU8e0pdnFkaQ9df4Q9VKOdPcFoxXs5VQqMRX -/Net8pGdcZa8aiVYJPWQJZCPcHet0USc5Kb3lDPxgl87X/U/xI5k93NSEIFu -LRymwDu7/wgPEESdGqGzSVMpZMIM+OHsHjKYBZGg+Y+Q06Dlevftf/xw+e7b -l2MKDBdEh4zCRsKZeFIvmiOhrjhguy2X1VboVhD7k2m2wygPKQkZj4Jz6Q7Q -zGHRukrtUPPNETkk13e0M0AKMbfCDfM58SIOx+FqUgu5wVrGeJupPwPAEAii -qoPnCG9g+EWn8jiOanBOxZvGEYRPNPyX/s6A7aKvim7kRQaXDw+KWG66ekKq -Dc453A2NMHq/nl7Ie9+VS/c990SFHusr9Q06SlKtFQfNEhkkAf/WCgyvyoar -M+kay9mFy08Siq0rYCgEpcNC9b1i30G0Reu8Y4PjpVSEnHCuvoMfQeqMfBxn -2SPzh5fwZdl7iWcq+oyXRNO3VacvInp4UnkPO+wRR15Tz2VcT/74FX9bmWni -E9xrZmbpwkXlbUUAPacdV4AUR4E0ayguIKAQ2XMcOfLSAmKx7SVhSfeUy0xv -VNracrvvVFdydBwkLn32EhQ5JG45xsSHrEEhEKaAwoN9JZVRNhvJhDh34mGu -EagqEF0QGALt0jx/rp6S+UjPY4KYXXVa6MXHnJFiliM4DdWtiAyTq842ypkM -WGlrULRX7T5yXtgqm3yVQAOcnbzjTGIdYSXmPWPvC18o+62rroiIFQwP4QSF -D9HqaB0gU99MpCfJTOM/Fg0uy1NxHHUNy+V5zbleiQuUS2REJyp5vyBnkl9c -Ps0RkkhWi4UzmKaavT3PH5Jmns/nblncDyU6c+oifI8HP3Sn5cgvv5xrnCqE -s5Tl7nw8nnlaqZvuGUDO7CcMUM6/vhB/KikbDUWjEptkeZknv/JVzfmp+bv/ -ONCiMn6JbJqiUQhs74cJhEd/e+yHU0v4Rf4qyTqWcSnuCZvmvbBpfurD0Phi -BRgIr6+O4bB79LCq2qNjTvkUgZx4GnaDhY9tjZMlh/BUT5yI8XQihBuai9oW -3dMTF9IwesG43c4QCc9/iCTeFrwQeOpMs7otU4jQd//PvPipao/tyRdBOKG1 -9xJulVizjkFNMf+NgAi17zCaQRHqwUxDEFIDRSxFzMOzixWKz3RUL7TAJP+u -FoKb0WP7Zfo92+KXDLwA80rCxhB++JgO3zfVsmrkv8G1VHBxeFhhcVnpqis2 -QsGCWCdWEVEr99DxxTq2uo/p7Zy6vwuvNVBni1iqFdeIV2Arcmn5Mb8ig+LX -r0gMLpNM2ukl/8/ZmR+Q/Oq9+9WT/I0jpclP3yQ/kvNpEfVycj++7n8zPKDH -iFMoyFpu0wn6m8f+Z40R6ZwdWdmLObnYgnSJmzUyc57xF+GrdTLz12++C+9o -SnK8HF5oVJjopT+CM0GCW4sSZqP8wM0/ugruIc+5LGZZ/ZS/OMPV1eq4RTWI -GWXwl4Am1Js+Fyh5VH4wcUdG+Me5bos+9cx+nqQc4oSezJl8pPfORKKZXdT9 -ghaTzwBkOX/PiOwztjck3tWalAtWuzVkI5AjbHxuafr0j8u3n77EkOgfX9tv -pvf6TzYp1mqt3+OBVH4lCoGNIqcJR8Six7306h3JwrFvcQU8CvyOfoee9D79 -TjoTukyutCo955Xrl6ifuhr1My1Sd09jete/5d88+ebz9mclMQurNuUU7f7L -f672X/N+CG9C2IAp28sZhy+V8UDIWRQcDM9uKZ+w8etj7jALxkN/nxeQ7A/j -VsF5eKGuz7PxFw3ec2mjPTD8GChKKRK+5LAE+Q351R25qc9AzoPcLNzRQF6i -ZCfzo0w2sKPFxQgBn8S5kLrBclvLOYdDaxAov5qcMtAsCszeU4D9mRnUWRRn -aoFzgtQ4hCS2OJXqeehaBWyoFBkARqJpu0pgkBf/MliJZLd+NKoZWoV7Izq6 -RHgcb4bqj+ARyYp6+1VDwBXH56ob5SYxo35imr8jDx88Gy/vuZ7oXCt/c/4D -sOo4w6f027MzJGjgIbWJp/lUCW1+HUPGQ9kx+NXhvD0FAtnxSAT1q09Kh4Dk -qIvczmVlXj5sZR5rzCFZmZcPWJlfQVxy7/RRYyhrd7g+jzkcqSOY4XkIbO7E -MKR/BoJHDk5MkcHBtcdgyK29swCv5kLYC4iAvrFQwQuLZYh3rJECLhJmSjhT -enaOeBOu5Abonys5Us+Xyzwuq5RJYRUQ9A9aYUqjv/6LetPfVD8XDSkmdmgr -Dqf9vhPwlpKk+Dy6OMjNasE374z/QTLxRw468D/pspNJIUgAz39T1+w2zRmM -aTgmfgGCoPys21I0nEUnUD2aPz0biwI4fXIl2AaAejmeE0NGrh4Twbk/Y2Wu -nh/RArDpkTqFca911j7JgovaMrjxPDzEl+oLP83m+SBH7wt4YjmAoK/y/FvO -TRgYAakhfgqv074qJbIaKN9ZTFeOWyIA1w0o7wECGa6L1OFJ35dQEXhisTWT -TMh0ijPdXatALThCw0htgVQjbVEL2k05xhS2fYKXg+Ac4fdAfCvRAKvDQhyf -T3qEsdvzGArJLIAbaAF6zvVZTnaPvvWk53KehBdLwvwkvhYO7BN+wrHXhuf5 -9+b5N021vImf8lPUiyhSJLmSZERbO9oFAXslOV9+xhL0fUpdFX5hoO8gl/J8 -GsZ36WxZDiAogtAJ0PpC0MKaqeYLHLBw5b0F2kaZ24prAYtlYpvw/s6l/MK8 -Fj0f6TorpHPt9N2S34rVXXMwWCFVuC7Etn+lD3RxW/o2+cvvDrs4VQOy8eKc -5R94z5DGcdIi1i5r57ZaslOviyHbaU6bPKN0z+DUHN0nrPs+lcnTTLIFiiTY -pp/L/qVztU6oiCPcA/qJbOyKWYmqIS2xJCo2G13LLQCClqUCuVQbmUy3uRTd -0ZX17tsXb16//vb7l9++HLttfpAQbBVpnI/bqpo/ZC5FzFoBz72gz8lwO6YI -El/p0imveJCVSapVWZvn33335kM/PRfhLLgnuNLJsTW50tk4WsmxAS/PiZ9Q -HpW//uH9FQiYimUZABih5sGjNbKrGHoEunoHnAZbJVVAi4OPT2tQmawg1PNn -WjORJOkvd3pfRfcoXtWzQfTcMv9ajyF+GlTPrrO0vQQhru+U8qxUOp9tQJZo -+CUbv7V6PqnUrzlfMS5cL+iAgSLJo6SrMQaoRWC4QTR31gPCSW0XpzVkFnr3 -90x8loGYlMtCRh20lwA/MS+XwFtCSBLT90pyCCSau3vfUZamRmpaCWvkKxZw -Cx+YhaA+t1TamZ5KNITPasPxC4nW4PkoLhdiu2/E5QNMu26sYCxLzJBoe0wE -QCeNEsGp40U16tZLkBVo0bfmprXRxFIJwCIYJuE6brPXz/+7cAZEIKiWguSo -WLxmTO5BcErWkGFXooKLV5SBMjKZGtzNl8oNJ7wUlrcP6g7gCGhpQcBi+/Vm -d9iatv8XOcChPYoUzTHAjYZhZPa9xFwfIADloQ1AzBCBPGi5epEguJL8sxM5 -juuO+EQRlDYQN/zRyVZPtKBr3TBN59riBUonMWSEEqXIk/CzFi4XKPbAceVd -vVa1xwgALRzjL+u+3hrYxcR/SIPMSGwmb0Lhamvh72kjehgyjyXqg51w+DmL -R0mrAxdaT4dnWj1Z5ngQ1eGlgU6nJbIjaYnZREYie0hGAiHRXxn7n/s8ukt8 -a9VWFiVADnqx1+RTDYfiVNyBoFbPlLuwQI3swMj43sfEjtsXSCTaJv2CLMsR -I2NU8IVDMoxeaMNvEEHj9VLZbpQTPxIFaTyQ0/qKvoRVHkQkwnXk9ovINsMK -mVmiQ6q6YHL0bL7EDYxCiY1hUqm91oCEScwUfhbnwmFISULadwCekwJDoZFK -08s9TS4sHKK6Yy1PUN9ZYH5BjwnRpZwJHnlu/4DSBL5/cxUsWS5oCBR8racB -HI2VFLhZb8kraDkDy/wQjRK47S0iqxHSTcnVE5UsQq19eI7K4of13bPckZ1h -7JFAJHRF0GgGWAJWqkqvqxvJ00kAGsELEddvhZj9UylnF0a0Qv92uc/Tu6IK -k1d55HtxmXq31nWa+GazEDH3KCzOjbRmK3axPUvD+snhuUwMbgGIX4P7KQpj -YQzICfU6+BlFlwhppCJCgvgwRFwwlnpM2NCwq96xnrOyVP91NEi/KstNuLAi -R4HeLTGWbuFxmvmpGFLnLbc2OrsnQP+TFMTdru+werelFsjykN9YX5U8f+y7 -mvWMH+PTIetp0VlZiTLBVTeelwaPMjzgZVhe954ncyE3KJdVEXv/6MFSdJK5 -o+KVHNppeDD4Jvmxl8JStRCC0QU3X0CZBcqdSMq8X+qdGSmKkaHOh/mh92Vk -R3cODV/sciLSXHBVk3b/HT45j5+c4xNJ0YHIln5t2om/b/8+C/AKnCwJahjD -akX2TTnANCVb/QKoSJ9JicguPBHb4OYh8Ulw8pTQpeGHEp4uAK5qe0kdXB/6 -DXZiHtKX4HtygxhCNldOqFg4UIL8lHUhrxU62SGyZoke8SXpEDGerWxEelIT -Qrf1P9tOSXyt6AR3sMpHZ5YQXACzlhT6DL9P3ssu208d2Vh7poPnLQgPiMw0 -hVVKd5C6NRIB2G4GZoNhXdgzuuC+auZq5Sd+Cx5fo9kTj1dRucGcE7JQGcO3 -u+W+rvhyQVZs8bEUXNBM0mp6WaB3joCGgmWFn6+Lzep8eaCj/RPdkftC+x2Q -HUVP2AkAt3ds6Jwy50PVbhVigyAWA41INArh9L6MPnwgEmm1AWE0TgKWn6VH -UoE4yT4OHU7zO+3YNWEZ3o92OIF1b4WBqMlhWnJFjTuadI1WbmgxTTyU5dfG -SDuxa9l4sHinmBHgyqMRbMhKUdvkXDb9ptifaK6T2ySV9PP86l35E3CSYPq3 -SgI/bw0wh11kfBWeomlI5sLeA5/PriGc7K70MdhQH6z5avcRyUg3otzGj+oV -j3qn55VEtAH2qh/SIbk8YdFX8hCawYlIuPAs0+Ijj2qt/4wUXlVaYInOm0Nl -8WRH7IFCnJcpUcF92Q2pGxoNceAT8wcHYPFeEpEjrdGpfZqNdK5z9mLPS5XS -iGIjXOM4g71kh11EPANt30fm6YkbaHsCQ7Lk3BX0QVoElUxJkedSFjf2CZfu -tGbmq09Si829iBHZ8QCSN4Ud+7xfPW62M203IM7LFrIzwIrlUvmikuCBYjml -y4fRVeFUgYVei23NXrBOQH07pV71a6jMnLAJhiCFqNe3Tb1iuqK+1Mine/l0 -SnIMg55QQbe6VEKjcczKnGWuzlULoI2usBd4qwS5dx1YF0YS4grT7FUYjDNd -54PAbSg3cPOElGoJ4NL4N4KkpFVghlmPT+Qykyb00GFtvL5Lrja3uqzwfDLS -STKoYTWQoywSs5F9TyrS+puebVkXXauLpd+uGjuDiltYFHuRMwg1QmtFNzbi -TDYrxJ3b5EsjMXE2WSKuZgXTRB2aSOeFlgZuCBkQtrJvUl+gK+6ku7cOcwuy -pQv8/t/f/PDdy1ylVbmZrQLdHeTMZ1XvR6mRlibXnE75cgSQNgKKfCiUDaVx -0kkP5QUjQ7XgrYuYzpSxTomYxb/o7vZlr7TpN8Ar/mKk4udiFEkruSkp3iW2 -x0oilNoHLSxWSIbSeGPxi9MOFrHRgLzjxEqFXT2WYepIqTae/moQZbb4DBAl -4+jlOG2qiOwLo+2MiDHHmY/ZBlfDFanbYi0azeeN6O34QPoUwiYHtQxalA8Q -x2BDnTrnnXo3SuQf+SzI9iwbNXm/4LKCwwYx4HYdp302m5Qp9ueZv6gii/Ms -Cc33XdbJMP0qASwLVVjRqwFc0eUm3ULgmCg12aUGBMl2vCnDrX0xv8B2Pp5f -zISHA4vExRBzjdS4GrdrrccPNM8hpKJ0h1pKsTooyba6X9eBPz0zCEHdMbt3 -7/dJGOC6dE9wGnwqE6FNkhPZo0vjZOqZJ+IHZCc+dXBTdsG7CFV9V+YT8w/R -EOYkBPQHG6Ih3hiwJUEU8wOR2uzUchK+Qe7Z+HEOzOEzcf7U8ANs34ouwKEU -W4mwqUAHKpsQF9gfFsdtYyCXFZnE3tOY7n8clBrDSqkkLKnt6pXMZCAGHomB -K/O6BCm/rjxHmTUnyS0AyLxmLhzzRjmANlZQN4jTvWXrhF0DRiLxmjyqe8EU -C8NxyKgXKjoa7MGPwM/FWkt7qqrvQMehutYeTNJzkHy8fQSPTLhwAmokGWhD -t8X22cO6R76xpNGqqBrQNTEL4aUkxLp176F4lq9gZIotZBXY5Zx5qkaxCS2T -DYaCqtXO3OKbfhMC0W1+Iq8/eZa/eH95Fagy2/zxBf2O+3EpeoV8atIjtCnJ -uB4K+/VQbImESzGPwhNWiky5rpd3z45GWV8bVLynEgQSddghMOLPkAStvw33 -9wjOwtg/Lt9arIcecBPb72orprbcXjMd4nQQ3GdlA8HkUOmPMUOGoiE2U8kv -ObRRLQ4OfN8leXxx8U/3/ki7QyYl4kV6MZsDUWygdoVZeiHjOlbM47OYw2Ie -7R/DmV+ubW562RGolcGqhRCgMweODX3q1h14nBPro75nr3ybYxb+osw0xnCX -XFsh5f3wHZNL2PUvRSxK0W5SwqrMnn18hzpxYW8yBSs8sFYq2aqpWilxpDYS -fK1iGZ7JZ/9OweYyJVfJ5+TIMiDpxdQbWZ2e0nh9Le+/vsbvjC/uvzEmlTE8 -tUSxCR7Y1OFSMkE9tXtmlRYhA84ytFCOI2WlcyyfvCD71qbkG3m5hN2t1J6H -X7x6Z0ElLvKXhMULlwCM1Zt4yqkyPYS79oo29JnEGiW9NtLcRA6OcPtE8/3B -gcwPSBjiHg3ROyawltaAooChWhCL1cyRUwITaSxZ3WuhvAP7w6dKMoJc5P+A -cQVOMImQ/DPdguX+n5nVRMb073QpoVPBrQE3NzHU+8yYhvInrRCu19KJBTcM -TfiLCyNDvO8yiMnco/iwMnzNI8UKqYsJ57/XCFCJlSPeJt4k8Ri6hjA9nKnq -+12MFK5KbRR02QtYOpIUxo3noUvt2MBd4zAXRYGrYtLdC/fqAQo8JZNrEvoQ -3vVWRrl5eki9LFK3iD/KPZ2F/mepdDit2elxqBbgGQMJcWhnrYzRSTj3YNS6 -skKk5Y6vD4eoNRcR+k1uSLbb7PgeYtt0n4yz5Josb+5TrYTpYFgmhcwtD9q1 -sFpwyE6wp9pcu8fFP8vof4PK4obzm14ckSvQJVpakV38XjgltXJGcA89JCMN -wOLx+SmfMtfPVaI0Byv4S+6N4YqB1EPaqRniDF0+FNkb9gFl18Ll1otPs37K -eAT0RTJFkZaPXSt9RA52UsAYTsnqFIYJLYAcvIAVGIiOQb7BCJEm4So8fjlF -HcYj9NO0Y2isD0iYDTiwNVMkGTKcHLnUZY2EiBvzjavCOUcwxbte30I3yw+7 -LdoYG9ciPkeOe3QTDWGCu/W6mqLn6C2AdityaEOk7UpDcNOrYPyP3D33qma5 -80bVsV6HEykJSY8HAqg0GZD1zdnZ9JLEdEXPIP2NM17K3PeQlJfN3OW8Lnt5 -ABYlBfR6Q+LUrpGzBK56dTSYLpq1utmxW64Sgm6ldyP5qyxRl6uCbOSKcXAk -ALOM2zntO5Fn6+w8OiVTrZrHiDFWbZM6fO9TTCKC5X55kCtpWRegYMBV8Fk6 -LQwHfH7Yy3KcZRaEvb7zHscw0xWw2J/J16A2ofYu9V+Ra9kKXgTgCFXUd598 -z0Srv/CYvXy6GHJMk75nyeo4qmlsXROmff6eu0Zveko1tugEpfuIjnXovf/m -dZgQ8aV0vdKjW0WC5jEgxJyM+Ww/gopIyyNdgSJiUA5yrNkIK912BaFcD9q4 -/1x95uaKGf9CKG9BkO5QVQzriUgqwI6h8baknQ+c+D0Xap6lxKcmFfbJLwop -5i6kqMCZQVgx/yVhxV8SHpwIDeJZR8KDJ/ddMUKjOnrFROLFcXK9Hl4izdeG -5KvXcL/ZpZEcgOOXRpfMxV0aoor62l6TpKwbQtJoLG8mPaI9aZ8ZzbgCOCJv -v58i3UPdxG9YNOSKvIOhhXrt0QKi9TiHsMu63qR1HQL/06CNkb1qLo+hO0BD -sfcRy9BNVp5KLMF3DYEIYgVeVdHJn/BAnwPmTydxYyoflnW/b8GI+9hjbAzZ -TqsEmsdiaGxoOoxYKomYirBU81SmB+4YIcF5dr4pdzfdOngZ/BJlq7NrXpqj -utU5c6PqVFHBnnDP8+9x7GkovFspOG+msToHzGmN/MBHlj37mnjFdrVIlwiO -K/jvOMZUa58am7OkdLmcTw0wa/WN/TQstJ6wQUe+gIB/NUwDaOQQZAh+XCV9 -g3hCg5svr2KckQMTfBiyRBv3iA9xgTl+adbF/J/n8p9nI+1uxsNd6vvZ4ESF -7dEjHgvHK5sNBSQuWzgfQRjfH7YPOhdJtUULdOSkgf+LT42XMmlXjy6pQoKg -hVTd8ASQfLmmvwXwo+xn7+UwVNo/XMy1gJ8JjTrMtdXoz2B90BWauwW5yObk -WZ1l4bIZjhODc9H94FQOuodAY5JNe/X+6uLiT7ma6jBb8Jdz/cuAnki4uF1P -mcxzIowMadC3xNWgQWUM10P2Twid71+R7NS3OmPBPeOoMklXyydeEEPa2oq2 -KdCEiCXtirSxIO/sEvoBHNLfsOHFy2J/P8ffz/F3WRzILcspx7F2Zn2HkzCZ -tho/B303dyDrzyfEnIvqduxI8UkBcbc2YcEK0tU2fmu4lJmXwWK8cVtyUzB6 -+86Kevt3xOB+6B2/aoUnMFO4BK8RDGYH1T1WFXvYk+8gEt84Xm9TVYaWTiaN -a0ilnbvNRQIR1CXxe4KpFid163gtZxm95dRTJuK9Y+2v1K/sM2N6UfgWrZE/ -QxgSpahdALm7zRYqfyoS/puKDB8KuYSS+g6z5+udsQEdGGX/PiGXRPgS1l1+ -aDlSGpl9sr8GANvzp1IJ8SJhjOczZ185f/5/PYUjdZ5+R0Fp4UnfPJ2415KH -fUMPc187d1ydAe2oveMhQGHoU7zDuSIBSISz49dwcgK65FVe6hW8KlaBZkEt -HzbPwobzQwZyJZ5PzOPCMxxo5cwFL/XVxtfFw5LkK/oAGY+/0F3xh5+nIr38 -f6iW3Xog+CKTh5CDjgdBBG0BGFVmZZmMY+lXY7YDrv5BcnhlZeGy8A8yEbDq -hkEvRjohwJ3ENtmtH/oB9Ps/XHlsMfcAIAWIziSiv9bAZRYLxsehj7hri94P -FMad8lZPhsmg7r7nJgmoipMg1kKLi2O6dEDDFg/Pr8E0o9Lgcovi3W1EbQw+ -1BZr+3oPrbVY10ykphT/IyFP+gJvm+9Opj2frJPlLYvNVPYIzOJV5GspugHm -9k5JOyBfURBDy8QedUGrnbvjN9HoJcw4TvZML0WuDFCJ2tX9K2ikB+nplWSi -20FQDS1ayCpLNJcnE+bgJuftUW84P1OavHjLjkAzPOHBRFikV0XSqi8d3ehZ -NiC5sZIdtFSVfyalJ+dHLLbzESfhfGAunY/dmm9GwsjnqXpJzn0/Lv78v6OK -QyC0DExPg+oQxVHIsSpLm7blRFORDr3gk5C6/81kvDY2z/TyKFZO3Q5iPix0 -1iBzW0lcXnXh9Z26bUKDy2KGMj3NZptX6qgbGUCebJ80LtFqOTHlcHHLpKsm -cPOpd5bVO69QokPo1WnPjBjJdZBoH6wNELeo1X6ImeE6xqg7AjVIjxhnfBf9 -VkCk+8taqYq45DM09L5IQdzIPQSD10rEfC4FPTSgY9tYcI1DC1x/V/bjNTaB -qBX8dgUeUpShBVaf4y6RtKtN3xrWiVVGv2VVT2/0P56sIgrrWbSCJ4lRhDT3 -f7UOvFj9/J483PoFJOrKRT6TYz1d7c0H/DB1iP9rT3j2KrH9rOAngECY6+MO -ObokxumKJrIChUbSQWU3bDwWFkyb0oSQzwDg/znVUaEVQWDjYpPQt43xIzlT -9QK+7skNFZYC7HkMXkvVUIG6/BibG8bHTvmStLjzXwef850Z/3j+RplSxAQ5 -LjkMYlENZziG0LuZ55PdP5+hIhrLfeanlnaBvT2U/fYsu09X8S6+Ln6C9zTX -zECEcqwaaTxLIgXbJxEksJptuV5vlhUbUjrLO62Iktd9BqV7qOF4IKG7mhJj -piAXEC466TTbLoBZrDqHXzrOb5NbIgFdDazOHbmEmOY8jgwBNo0nhE46rhbd -VZ9LddY8YgsfSmn9Guhsa4jNsnjJ27X7qIjteUDHcXLJUPRrmrqmOjiZqWm3 -D8+/H0vAjoG6QemsUmJwdzczzj1JvVmshjaiamt2+vKHq3MuaEYzm3ke+KaN -X4qjLDwPemzDNUdkGxf7VhG+inQKsJlrWl9Y9JHWprGSqFaDflsaL0uXwC9h -SPt0PapMzt/biKbIhy6PUAV/viPLlgHdG+alNgOb1/oluT7RVQf/pB3aRplP -wtHJDzs0iYP6oUX1uc/Izzj77atHIneGHtr2sKWVqX4Wi4GuvlRogrwIEuov -IO4NJe7QJY6Fo0Jyb0afcyvVND8/6kkZfgmB2UljxZiuuDACmR7pdycZ5zdW -hDHzrfD0pg7tBAPaPYTAkPgyJ0xKlV1XJjXL7ri4wTAmcuEgopU2SZZWnkz8 -oX/R8pVXfXhbPyBIC8LBwJmS6yzYtgMbe89Yw8FgGK99p9VL4WO5qdbMJjCs -pV7U2qbnTqalyHZZdQ37yFkaRogAykWod+CFKT64lrIUo0mRrqkWj+9b+bNQ -77Wrq5abSgvwgGmIIwMCDmDXHLZz67DFyENxeHr0ScWnuloy+bCt+qY8kY3X -olE2fXgf8SAZlnEo0/RYsypQvJRpa61GbIcNIQgtzu2XARvdf+Scyahu66Y1 -ykO1x3cSfXBLwsvEmVcOZrMEKv1K764TBtk06IBE5Tr0ejSAdZxnS7u/0Lbz -rXR0BpWmclhFRzG0AOMSvWpHp6tiUmfZlOu7zo6NNj7KUQqJh0soylKOSfhq -0FHZW0HWt15cC7vFJByB/GfoUabV6uqoTujGtG2W7wuVpX2hfBOvfg+v7FgA -Iv+3/OsL47Mci0aMfsEd7X/LL/5pLD4hH/hSj5SRQzjAQkv1vvjw+mcTc59Z -52v+PrbfeZ2yktWRpVOrXKbk7dDB3M+Mh5M9nc41Q8+EaQjKkQcSMx3DADNq -MqUZfXIKNTNHvj+QCW7u6HTIctLrFpZZKQNO5TksSYlYu+w+fi0HWr7hRzTI -yLxKi96LDdPdT3I/w74JyqJYZgvf9FCZvYrWdS+LBHJWRSVZVjkRb3bBayoi -mylis3Z0AACINLKZpmnTsnuuq/7GN37TjoM80U/cz8OyJJrVYEYcNbZaATsD -BZyk25g6yvQC9g5Pq437lb52NzPiVo/kQ+i9MWwI3wMl1ApfHYN7gK10vmp4 -zwAOs/t2UyaX7QaF7TtFzDNtSmgNd/IznbRqh4jF3YnUCDGdHZ6RNlC/r632 -jM4MxmqQFE4SWCGdNO8dGtqhO4N4Adwy4RZhBWbhB6RLjHd6TzRBb9jRjwA4 -hSfakVYZkVJ0V5Hy8yzAaWjWQP9ZfqRe4UHfkWt62qCDJPtMDXuV0FU/OimI -XLALQ6BzvZWhBiuBSSIhgX4XKCTgtqU0RBYxL1F2xk+e5dW8BH0wzM+gJYUH -S67OqzWfha6dcJOSMAWL3UlDW/vkxPZbTHV/gwRN2aOwf/qrVf7jJ79Q53/F -Sl/hHaKav7ZKJmR2wlEyLaNXAKzuIB/oblzvs4gjMZC+VDDTg6RQyk6jO7qZ -FKK1DNylIXTmzfGIvP6l04BAsx9ryIprRjyOSPqkOtAVm6lZF++RYJFqmvDO -59yft+NYl+6WKzYcOO0I2IUm0SxHMGnwRThAILa9YBMlz2roxG8E7Scsu3rI -0jrBGCQydHWkdlfw4q3R6Li8rtIUD/44XtTAIh6E1yWN9QUJ7ZGZddc29NCp -tz8E9/gr0+mFA5EmRYA+rISD7Z5EU3ldt2FJVqNjRe9WmOZDXAdqVHdxmT2Z -1Ko/7jlHlvliR/Jal1YviWgLzISG0Ke11qBma3sw2eTJ/GiuVTjEEKuKsdZz -tbwWUubM/QraVpTgoFm9Vt3206ja0bhNxL2tFdbUagTchRIyIDg8ZqUPkumD -COdDqUXB6nBNof/kBp6UXfUMqsBrr9bIWNhjJG849jVDn0uOM+gOX9agd5XH -DUxss67YAY+/lq73SH/+FtCQs8yyEGNIpCSZwAaZv1ZYXHCfAY4U09KJqISU -hxs+qbxJiU6oskRgPMOjLegsg29vRz5Aot1b1NUrxF2TphUOY2+SH5ovyxYZ -ivoUzr1YOqwKnGqdZXothTTvYlE3imwZqvKzrAcv5NS7yYUY4TQcMWIOexgs -RhDqW5dbcm+oWADMVuuvnxsWxPa1kc3hYIXv+dt5zPx7Vy7W5eIjkwwJSxnt -17PxwNjT/FI7EVzXyE2A3VHCane8UFFWFbI+YedcuhhfyPhIlFBX/QjuzZ/A -8GPg6lOFStf66G/w1VBChFwT0raasLo39vi6XFZ0d0eTWWKX8AzYxsVBOQwD -lFyWiKNkJvQH3qfdqGNolQjFiHcHWKtQWMtiGb9p0MOKJSzayWMjoZgfUJ0J -FACybQ8An05nWXUQdAYW2uwivUhyVDXvhSAXhHiMP40l+IOYKc8RPOw5B8fQ -RiWQoEtMnnyKQ9O1aUBSETkuaNkP2AzpHYyknSNVHZ2HHYP2gi8UwOrBT7RX -9Z0i3zVFN0G4E6Eeha9/IT1KOrYThKNmtan2bYxkKrt0sl0+dg/A1LhZnOJA -k8U15D5vjxSIaVk1tGUA8UtI0F0AbWjUKQHQhSYF5X38VB6+mEMiJ6KXT3kb -rovFR1GsTO5zNp/QKi84G6cV1YqC3alFi4fjMVgftGWzRUyLxaR/6VqiSeFB -qQgKxApUZiO8mVYa58wJjVFxI61NBb62qftb56Z+T3KYXeg7ubrSWf5vEife -lV735adwcnD8z5yvxF6IOjTpFIfByd+NIoaPmTjeSFMTByHyEQsnK7a1ZVOG -WK8x9TvL9LbuEuvhmFWUyhUZRb8NXvazjCLGImCqYq24qXaTSlbuLVs6hZHy -Ce/pZTe9efYhhK1C0G8Ww91efCSn2/5/y+ZJZiuUEP/1Ns8lX/MypDX9fzAx -PLhdyv5GNwUdo5yRMWIS0GgghofWM45JCh88EE39UbjBWtGMHNHaLe7OJlKQ -MWVN1q+OnOzusTLgpG0NR4xhTbiuasnxCQyYWBKZ+TPTXHda/LiLbUsYfcod -BSo710caZL+HlCaM8hpaeYDtdqS4AenAvqWXtt0KVlvibA20hwJvp/oaxsB2 -3Ptbi6qtRB8m2klwtPazgen4Cj5vU94A/Ov8c6sGUNiQb54xqqytEnd0HYH8 -5fqd63K8cgpQULJ3VyMILgGltAIrlvVKi0bHxP2qOZRa4CfFXmz3kTIWptXN -XSrU248WHi7LrdB/6HG4aQp2HeTmPRnXFl5TnMxclFOCMyGTx4YWjprUhStd -/5SkYvsstk8L2jN6xFCAP1L0lHG+p/NBOnwXQyMmYptxAdyldwI6PaKHtc+j -3CO4AUUI9IAb6bzfy7NeTYqRUmC0Yp0jsO1eMUp8Fnp81CGpfr2pyRqrUvv0 -t7d/k1vx/xfmb2rc/S9q/34P37VajrizbLG2IybrQNjHbNgJRrqhFTtRy5ui -JH+7IqmzDIkIml/RxdUZPX1TZs5kSHDYk7NXjSPsbXd7odAc4/iZ9WljOOyb -/ZJRaq7EASnlGpU6YSH1bz5JUQgzdWzROpv1Y70C0fo01tQwNXy08xtQhPH4 -xaQlZxtNZGsRlI6Bxgt3XkS0tK0VgzwYmlLaDcgekw4mc0mggSlrpGeBZbGH -SDGCN2E4le2WgLQLB2QDcBSbCzFkOrkC92VlocScXWnvOEh4Y8qEfL5c9vtG -ecaQNn/eRRnA7reGSxyH6ZJs+CTDNDGJ+9bIEX1AoujeGpx5NmoL8Q6M25DN -KJnWLJvQFxiXFwJvECWncsDsHznf5ITEPnN8TyH03QiqrEemlmnDILFwgT50 -qjhyS8WEzAjuUtFEeEaG6vJQuhLKet8NLdLwdCAaboX2j4laHNDQSvIumxGT -dpw3NXQ/pJkyEGDFTixAYcxaxI8E95A8zQpljrfmHK1WULNQw6L9ogXF7itT -87ANq6Rpwjj8ooPgQCuL4kn2coihTrAWjWHdVtztNOalulB7FSr16Mxyu4vW -gIPV6HpL6T6QkMMywOajOPDaDi70cwpPSjz7zlHXTJzq5+GeGd196QpT9tkC -ZfHFK5J6Bpf81/D0A3xDwd+pszdeqRgavLi1QEqJnKz9ptDG6sGo2bOr0lRo -wsigf7WbJhDScTgMO19UJGLr+nYCl8zg9fuby9I6cQ3qOWeF5vl3DBjna0aS -lAH2ySB7C+xN7MvuroMjU25aFvwbcgD5P8fPKDmBAccugtfUqMcaq3pNjRHW -Jb5KY5Jw39VgTt4Ayeeia0EM6XsW1df/ybV0WptNNn8/DZ6Q7RXTyXBfB9Rr -tDRon50OTFPl4bYODh3KX7iSg5XsMLEfK2E4hCVMkYK4OFoGdD84S45Mo52W -QvnEumi2A5l7nPQRqFqBE3JuarTgayJTN9ytU5jrtCN7cKyqwtlLHZbF98bL -gqwX+YANNzSI+pymzZje27rtmKpqSefq7aVj8+LmUtt6yUeAbaQHlqV8r4Tw -5XLQJSBZz/cpOA6IssLSVZzrGrtMjhhMg7Ktcc9mSI00ajwZPbDsSbrURuce -OmxNOh8jB9UOXCX4k4kSxofWJyLzaCyzUSbRhGooIoF/7r9q06YfHJWtBNmt -nRPAmVy+4xh31DHmRjKrTXEzcexIJOKqPedYFUeSD005mo+m9UJvJ2NxJOPx -pTe4AFdESwHtaAeOxhbp+/vYgbm6xcBQwEI9jXTrrhZG2dmrT5VWZWlTGbDI -aRCFizo6xuO1emk4CUcEkCWurVcdersVbtbSTKq9a7sSmGvRI2I/LMu2utlF -rm08ykFGAxm1/LrtmoM8EwEaa+stfT7fF1ilk7j2L/H0E98x1lMsGL0qfv0j -+ZMfax2Qqfypoisx7pKwE5k/LbgD7fcF+AMnj4/VPivLw7HGIfEZybKql6VX -PRumoP/TpYr79zRzKmPWO5mzsdaMvJXJq6SmRwvb+MuyiVI063pSqrzAXFML -kZ2qwEUpTZXkEiJD+TxSFfYaYHK0zXqxYYK8GoedmXQWc48vhXUemUA35Yor -CEJrsB5vHKzdKXtRHYs2P/HFy/TYNgzvxAAoqXvYU3okcfzy1qts1uALu2hD -mbKjjrQaUTmk4A9Jvtkr8lcrR7pv+Uq8kduj/+NZBk1H1jRi+cIGFP0k+zvL -6abY3Rxo/FZ0H/jzhzddHCqsrdAFui+X2tSdd0J9W+EhMPJZWBWRfRb/eeY8 -XU1ftnDK5Tfh29qQTa6nNj4zVlXLQ62gesrIvEo1nbbxbHtbwqV+UnhcCdVC -9TNL3GVEvMzU8rPuBPG+pDUh63jQtVO3JiDd7cIPpe5ps0tru3daVjiM/TJ4 -qf2G7pYjmB4+Ez4pgQ4Li7bD28M+P037ZQc+aPr1WfDoJKTBNrzx6Q262vZU -YMgVZWzY+lyRlNDwYu87ZdgIxuekTsaKjQVSq1pjCyLJMckmvGDBz+dU2joQ -7bGrv6y3ieYYbfF3MZfyhDuJKUqfca1GOMtuUBEj1H2HmxvjQT41ik0/7TN3 -9WXmPjKrAa+5uS3aWmvABzNSihdINruEByBzFyxUtU5VSv9bRSIg2MT6p4hq -bopo6JrugrUFLnocyIoEl+0h+d0xYlj6jyvaCyvWBCnMA0sDiwU6cok474tu -TSebTuPSElQOqUv6iXtkl2M85ZoxZoTYqmiyoBtDG0AtS/AtJ7SW17R3mh3+ -4Jt9iuQXaEk0G8HCsdFuPVGYECLSLmU+eZwMJtyZ6Tx2XaytXQ54D7lUzTSA -ntjbegBkRNO3YlnK07z8AbJr5d66ZmhOwTMo6dIK+iht0sWTElwGxyhUbGgR -7/KQgF9W7TnLKscJ1LURuyEoBxPCAa3LcW/ql94yk4QjmaQFyuXGyIdIojyz -dnCk+u1FZll9za19+9h+f/e46AKULSoTpSVUvK1GnMWJC6rPwud+yXUGvDG6 -rogTkV+xsESzm1Ib20730zu32n9MitLKfnyAWa/l2mL3DOc+pKzIX5r3t+v3 -bQhacYpia/X8g7Yz/OtsKUmEAycvfQ6q2NxwTGW9NYZkjDqkflETseRLsk+U -Wwzt8ACRj4bPKImc41CzxFkNSnBcDL3bvvcOo5jUFIFgv5BXS+qG3rmsrfqk -Zm6p8ThibalhcM8xyK2YgaOrOzFZaV0ysyeie+nseJwY258xsvK8hiWmOfKc -1VJQiD6WG0txRFWiVZDgN7Zo7pP1m9H1aqK/nuff2BVjLPvSmTm11tx6BEss -uCGz1Bg57Xkl4KML9vNZZqA0UISpZk6YZ4LZNr5M83RAYy0LUePJuvW2LD7K -bTfGC9YG5JN085VFRQ+evVCJDDNISu2EW4+fXk6yzqQdifKAcIlgHbsCyGy9 -p1dey1knOOG6iSd5wopvG+8L63XIU8UGfKzoxquM77sp3ciK/ISx+if56fqw -RWg3f3F5Jjr/RMArJ/zHE7hvJ2nsLNoerZPLebJr2tnXNuq6tFtkK/Y6jEhk -9cRZuzcKGmt0d3fcz1RC/0X7cbQLiRsJuw7GQpLY9QJ66Tv9hui3ckeLHXHx -hAS4GJdy3tXnwKeIUekaQvXvWCg8WqFNsUiRJ8KkjomOHtzHuWelnQyYpdZs -/kIr0Ok65iwJHRqyZOTsu1Yy0rg8plRjw7NNfaMkx8uSzibu85FFUhXQCNmn -iya0NIzWLHrnlwyv/N5ShWiKhVuEn8yK61WGQn19WuCt0SPRHG0yoH7hnyr6 -vnOEvndqloZrcEiaHxcqzd1AqReKeiWRyYp2cGb68SMojHiTxRZNw/x22yMd -G5ZwYh+skeOwP/ZguXsWGXhRes2cisDJtOxzyfFqiQgJMMPTTZxyn7yUguIs -ixsXhT1SJNjKW1CGsV7k+nCZFSJe43E8tgPI0dVTJekhBTkjmN6JBb+T2Odk -1tVo8L1qBQOU/Pf5n8Em30OU8A7RhvGup2ih5ypHUYgshzsys1M2hCSolaYn -6tXZVPCSA9BFde+B6PMO/s5DCF7oudJQWNrEZUW+b4sYhQ+3LJKfzPo0ndu6 -qz7p42hl2ZjXLLvrmlCSc7qsScOMansO6DOx1JLWaJD1Oh6h9yF6LrKbiE9W -Wve7ZHHQAHryDTwgLchPDVIFcAr3z7rYg90lMrXh99s7R+ADHEIsGy66h8XJ -j3fDdr/6UeC5rZpMQ0NfvJkS1dmfyk29F1uRE2jL8lw5ZBgXUHDsrMjf3tEm -7XL0ZWIeCPs1TFRGsC7La47G0L8OSoOuK1op5+j+YKBJ+uzt3dtLH6STF0pR -u0SoMqn1McDOmXQxDA9RAB7/lkcsxuca/rOmQexJZ+ihWH6qN59glHNkRFyp -19aaQsHMIcdvs8QrMR38as9IHQNf3dGCqpsnrEgYBr/q0Ir7KS2Mt3vauUA9 -YsUDfqZq1+L3tDSnCG0ZvppOvNs+G5hEp7mgotlyZ6XMmQ9WwH4qXQIEuqj1 -LxNEj+3ZLNuUBdaHWar3hTrjJBvVNCjvZXDwGKDDzy9/AoZgA/I9D0+neV38 -ceTobj9+oXB4iVhqDASgD52sJMs8k50hg8LX8To5Gx+kqDR/9bKqz1+8v7w6 -d2c2iBODHJ6G9iNs1irqpOqOHbEhHoEWmk8FPX8nvdN6nUgF0p26Kra1pCy6 -urkbZGn7bezeS5RT7q0tlxqDDq9cnoPpndZc6kFD5o29/EaMjhnezRw014IP -p1WWZsFmriMNsx05CXjUQPRmuY8lxg2b58+lXzPseIcoEgtYNUkMX7ElbSTP -QmwjLBLxhTgPYTyVKOFXL+dVLTcnTrQdTd5HOZGKSbH4cqqnyetlbk1h+8NQ -/qGN9Bx6FpbWisly5KaqQUpVCrjEDQjqXhbc8D75Zfd7WruPrlXuNd1raf+n -y2+vXo2lP9WPieZzEPQNxKwNchbKlvS25+KFDf2/Mbk5/m8IeHKFTCFbSHhF -IumFjv87bIisrPQARWefwZUyyIaxTmGgC9tDSF5EzbI6iNWyZSnQYKkdyEUt -OKJ97dC9IQFiqLXBCAYDmKjv/1hyGQKgAAvGw4hXovL9KEznUfq0R/2cZtsO -CnvcodXK16LaRrpMRx4FfLdgwgvlOZXOh3Y+Ag2lkojrARLNqPRQ8lguTttw -Kp/XVgwRMXhyYZC9FWGyjJqjlqoC9oiu/50wKBkDjYaByPbmZ+k7WtMa7nR8 -e/W+YoKqOTNUwV3Y71afGHf+ce5XoQoetEI9I6zgRCQLNmvJGEQk+lEWIUWb -MThZN4qdUFiEurExXwKdF/kpse60cbzm6PfqQoyhxIOuhVzppW0xpRlip0nj -gbmIEA/Qj7jEP5VWFsOzBWhgRa6Ooco4uZBuWxleVHFsDBl1KAmlc2Sopl3I -/sw0nVBX7je162dc7Xb1p1A8wvInaoAjIeLKG0+8V1SB7mFqtfVLWoBEX4LJ -IWFFZ7NjZmiZrZ7RLDbo0JCB77YaJxykalIlmSLkpAtLRwk/GvWa88htxMsJ -ZYi6Jbp7l5yp/DnQLMj60051UhlS9ylbw5MQpguP62olHVxtyp8sAwvO6GG7 -Bd0CcZhVqSEmL6eXRzDX5LelEK0ppjPOW92M1PmAWb4tdjursKbjBHlaFRV7 -EdfFRuomGUSviwMaTou6B1A/uQPRzVbxvba+sKIRNbwlAH6jleVP423kE2gS -P9gxgirJq0XLT9PDvDVSJT4DkzqY5YUrnPHPagPDdB4Ce7hxhWcznGf/bpWG -kRtAi8kzWUJfZKhfNDLT+F1d7qTYCqirfbFzXKQ+p9jhmm3TimiL4oQPk9K3 -CDLGDslD0R9uq40ROq4V2oAN0eAvRqCLZ8vWINliTTRLrVG1TlThIeuCrCJe -zx/UDYERknBB9lEDZCt+1ICkpTjDTKyxPTJCdz4yvw2FkuaIhQZlraowMldZ -F0oJnD0JahiKp5XXWHyUt4IJbDlaggnDEYpNjH1h+tg2JHQCnmczynwwxEP/ -INL+M+u3FBwW501x6mqegDa1Q2wpVxZoxWKTTeSzQnMv/Jc1PfAVm5nL+45U -PY/K0UDIvARWbco3UYBy59w1clCrX6PtLvmGhN+u5rVH0k1pkOtmqWsdREur -oicgsuBhE3t0Vx8pvbbiMjcEAx3QoVaGM3NsDRuXNs5SLGr28hBay6+Khebk -pqI8AWTjO+dlEQCZPFsqJP/qC8QiyEl7tpm1dOZbzDImqNysRNK41wsfAI0+ -BudToDphlvybLEySF/3QOsBACPsK4Y5Z9f31s3XFnXW0wFDY10eSsllM2UIq -kk2jG2WEW8QfjY8lQ6+gb4qW3RbOGfC6purnFAoHWqSvZXqFzGRLVcvlpjyb -8b205rIHGGa0aOxdGTWRA0+0yVmNrol0EgvqNqJCGDpJ61C3sr59gIjubTgC -dkq1NGUsB9nuyVDIyMZU9CHSkTVHhoN2juiAFbijZFc5oRKk0YCWrVwaoSb9 -CtpcAgIpZqRT9HaP9yrzRfd1wubg6TxDd1ifh9P7Pztd+qq+NIgkhZHt2dG2 -ssmaC0ULSmS5eJadFsuuGlWX+ipYXljmKw7q+obCwIvKLaMYwJQvwNXW7xTB -k1II7KR3bcICJt1ZQqtWUwm9Am0RawaMVxKhzIMNFEqqvzPRn6Djwefn9WqU -igdAhgr9PT5V0vy8ksC75ibT9s5JvZokZO7pUqCtpALlbExWsxBlMq/Cgobp -uiNN3lcpPk6QOb5XrbuPWZHNnWmS0MSWV13m5eTZzpk7y9ZLBhd5z6BpbRck -wt4r3Yg/WxeuuEpYEYIxasMGjmKkV3GbqxwFW7KWf995nRMK+XZFBCgC+RE6 -ibMLEQJuyqogU4dTvlOMROwxoP2/tIfICL1Fn47Y2CNlIxGelkeMJtlQ3gE3 -WI28aB2PNbOlazfcEprQYLeJT2EP9hOvD9h0o/IkHdF3LtiV6mfh1eLVG4rV -gE4kMldoSwikUhNDqcdtod8TRmvVGCF0F5udlNZ3RHxFNlJmcRw9/WAEiOjd -KLCmvu4KmJA6iq32QUtRWzFbaJsC/SzpvYVkXtiWzOR86yU0ZH7vCz2Hph21 -b3ieBH9qxwOZXFPCKcs1O/su7JyjgOmxw7izfhk4+1n5j8phYBGCq1SpRpOk -MtZZZ+KpraOorsnpT70+C2JKK1UoTZA1OPLtwqzmrH9DgQl7l0Q/ZRBDEtsJ -OeSlQdMGFE13eiSHPDbWJxA7A0lcRrM2fuTMGhzvwEllgJRYZ4zb1LHptAyv -DaVdoTA58N4Y6Y4+geysfQkLNa1Rfpa9rKWTBK+sOAGazvJzvy3MbC02Iz+R -ZFnqj7ehWZ5vm+BX6ln2ng4BM3CVJp4GpJTIXbr+RRcId1KvKOVVcTZym3J2 -iBZmK7nqDhZNRoTCvSVFovB8Y/HPYIJJn4g2K5pyCGNhh5KhWVqwyTw+Jecc -BW0b+nb2T6RZ/qExJ5dL6Dlip5rLZnDbd+vP7mJP88nSnzkeieRn+Hvo7Pzd -iGKsLC4gd67GY/WqHl4NGXbRjKzEaI+LWndhTQWYAlInttu3IF1j2Ys1RSYt -nqLqOD8Vwi/JKTnGS3WaEFPxsdUOG6Nl/ZahBZCkz57de1c2GFuMu0xRVSLQ -gSs9oV405kbPdRUrThLKQkfyKBhAXmL/S46zb7ScTcpBHkR7SRL7QdGgiUGj -Mjo1oQIsAUFnH30FamzgO4E04SweTYHwp1wjhSX6yfQn+74+tJs7s+RG4mFm -bvaikh80qiPNPzWbNwiA5OjUaUGDZBx5GIclZqzMl4nIncsUHD653Py5iQBs -aY2nrTJdTMu0xhXIBJgYjn2vu/rQoDiEgVr4DyuGkirArm7OlCvR6pi4vCEH -bxidLP4T2gztDdiBxHoKbhSLIG3bzOcDHWhiNxjAEs1ETgk02mJrZhdufqAl -Cy5uLzcKXIR9l1qZp9b/qgtUby35rEiMKd11ds3eujE6afNiWO6iSZGNTB0r -DvuNWTKhxzEDyDTKp6hqLQ1GGxwmMJJuvK4nTuxoTFsx1iRN6g7Uz3wpxTz0 -XdbF3Iyx2nH3Q1I776Vrzme04jn2VhDhjIQLQpLESrwgn1gaXmdPKYo8CdSz -WNc8IjZEUewiq03PLJo4OF6gsndTs3QvtcwqEftVxQQWyP8puBvCmhWjAYtQ -gVNJK56w74vCiofCn3iAmMd1qUkcfQY6vSGeCmMo6aVy2eH5PhCH+nopnwvh -7OwUBr2Fi0a+Ym1nztzBtshk9MpidJL/do6/nWWjUVjnKEovOTVNj1xs8+x/ -DEjZRXE4hhsMLV6x/RFmEgHrapcLmhxZAJbuNwXp2H7YAkrfFYRJABkAzB7V -9MX8Kwun4CqUmJq05lDYFkL4aI0u1OaW3tBj4mL8aIaqXk3SoF40/0joTEIT -Xehf2AvOzZJgJZ0w0a88vG4dytlDy3AcjF4+JHS5cZ4pMkVCXMoPXYvtEGsa -VGitIR73LBg2fi80peEnGjIb9sdzNIXTy0iyGoVRCzMELcZRzkzgb6GJu6Ql -n9Ilce8Etlz3sProSVoyaHawFGAZhnqCn195prVKMRQxBmRslHKLCw5ajdT9 -YGG/rcZo4gGK3i2pepixY3Vm7aJTH99+M+vFuGKCNrstqpjiY10u7MWSC7Qm -ut0E0tNyAlafunXLHoy6ELWJtZw04kNgUjRbIPxUorjpO0ca6cZo0DLp6aoX -EgQxU232LS4B3nnb1ajVwmdB9M40yw0akyuRLXecQronHoYAS7hD5ba4UKkg -zrNXY/2ktP8bUxCw+SF2u3wq4hzPePI4zssFtXeKeGFoPXsWIj8TIyHrIXSw -TCyNAOXZddWi2hdcTxl9HGZVaQEd0TSj65ZBR9ImAj01jHOeBRN0ZClk0yDC -e+OlWiXPtw6Ad7AQ1C0wfSNtHa0HJCPiJ8k8+NeSN5c2WMwIN5p3yJSIVlSk -BjcYCpy017P0j6W+XP8QL9DHrhzhuvS3C02dbhdtr7Q0AgvfsNa8waGmUn+f -prFcBkvXLD/elf7ouMfxFJurBcEEWiMYFzaJcVuJKo8Zz9TV7hGkWobJs5H4 -xTJjMsYf28kwKNIkvKg8GyvYBKlIjEontuyvypkgdV3Q7di1azK7ZjG+jHpW -J6JZymZdjFwMI742zL74y35LAvEtD9akfMqJZYeJjWWYeQJA7a0ak0xquEeC -QxPpCA4sj+EbL9WeE+yf8eI6VDIDWenkGqYkZrtAPymkWeQUpNjGYwdjWJ5/ -QCMdHLSDnE1X46mgO88vMDQig5+HOY+RTqVoyx9aK4J8/PhP+cfyjuELIOd3 -kI+y1HMXIckCUeZmT01ZAwFyaeFFJs/tnEW007Ms2b/rktZXwbZIvzGYMyDr -+FWC8Sy062GlLHlQCbI+rIuk6qcITypzLcpLU2ozUamKTsHCmi3bK6FVIqVO -O6PL725rBoCj5TOPb6O1fLCJbaRbWoUINFfWKIDNqymmtqHsBWa6/C9lqdgC -qX8QmWmBTAj4bq0cuu5T5D+nIzEbkx29m9LyClfmFK/IlIRByaVQu0vX47lc -jwrLuF+6rpId7pFfkZ5sy+EK3rt63PrXMXdPL+aMY97wta9LaRtddZtykub4 -A4ffCzajk9vC3iTOMSCxVqhk5gSbq0zgMj7ecZNXKyMjN0kX+nb32DcUe8H3 -Vag7ybwh5cpQXc5nYM2GvqPLklubAqc7zBiK19WgyjBmc7re1SbpS00rhdZj -MfE6G04kM/FzUtAthGOez/Xv21gCJzhkRLE7z6U8187nydULJEq/av/JH7/6 -W7ggv5h/PX8MZ0sEod++I/Bf0gVD66PcXdXPLIRYaWstyAiaJLgWg0S7pVuJ -7K89rzy4fy7GYJgUoarQogsHpQUCMJYBPpVE3abacmA+Bprm2beKARGli6Zy -CmXS6Lv7tjg7wmRDK0ECWbYxAy35c1ekyNYj3pjQdDlI62rf6pDURfv+8gVt -iOC8+QzwZi4WpIfo5WBoYukTjpC69QX0H3f8MuW9WfYzN8Z6M7dOu75Ti05T -+VUM08lzQ+RHS+tZkZEztmtptFiGpgz5/iI/iat7QgLBCS0NJvt9Zq6j4kb5 -lv2GVAizPr64yF/TksxGhhehS/Bc4lnRYMs/AfTPYck18oB+G/jLDH7ir/lf -huELJcpSxzD/6uLJVxgIh2/HmxGd6UqGQ6CHTyJRipbrxzThlictSZNAmWxh -XJV59kEVcBEPVQAIWXlwPOGzPowOia6pGSTBTcOEAxDB91/6vpkPN447Az6V -M2riK1VQbDSvVvEAuKX8SKFCap69VMCi9JWpQgA2YN28Asv+Gqf13scFWImE -j86Tj5RlrvpNe1LEQIUT9FCxDjkTIVOMAQdmYEKzLskA4OAvyZEYdpTYVYty -E5owWLZDZCjqHGlBPEGtD+dMGYsCj9iISzXnTqHFgxvkFAr2mBYWqz4nqWLf -sAscuSlahfckjaO1a5Anu0OcTWF+k5juSDmW90YkmQsu/Jmj9uPaEQ7XWTlL -avt8QAbyyCSFMA7B2aAdzjd1/dGqm9RiGKYTffoRydpgI/T7r8XHnM188Mxh -taohdie+Mn7PaO6nBnK/ufqdTE3OMWv0G7qCxUVlh1arU5ZaOXqpFYzsIojT -WHUhu+jGOs9Pv7n6YHQ08p3+fLRXp3QeSArN5PVW0sT3LENMxKwSzY3fgoL3 -DDyhYSejYj2J+yfapim1wiHsNTwqqaQTtgGHbp5pulny1Em5SoBio0mCLTue -k7TN4P+XjUXRSv6qX6CokQCFHuzA87Wg3yoFQGp1tdBBLcDNee8SqCyECgde -fs29YBg+LkVV5I/rwaDT/z8dr8bf3Xr8W7Be/46J/gFPOn1M6uw8OPt/58n8 -Hcf47H+OydfLN99/m//4FvW5f3ma//t2S1MPrW9mk9oF9TjgTrCYMrejcQgE -SQ8OwxtShK02EuffuW9BgetLIYeNgrKZamo5yA1baR88aXZ8/aE7Daa7Ip1I -SrhS3BEWxGZZl9rpjgHI14cbTVSY31MBM8mqs2aGXf7RCxh7ofhsLDJm5jH8 -0ab6VPnuXCHZUSl6UB0eWoQTn104sa7I+OWuDunY1risUpNvqimHZwTHtYdY -Cef0HR24UnexGxywxzLdt1anWggsnVPktw0iw2xqkdDwgXj9F9JCTKhsLONV -i3C4HP0zLX11uubp2efGHGSxWEmfKIVHoLlqT4Kve1tq6voTCP0+9e4SNRSk -OHmm91E0KfWikJuJpTvf13sOqsbtC+iPw25TfSzRRZmLsbkN97Ls6fpjet4z -RwGquao6yFzMdaaY7v2h4TKGNpS1NpFeY/RGk+qsFj0Z+8RC/hbyrRxCgcJM -kT9H7t3A6qetkx4YaOEp3TTFfs03SOuysKsDMKV0JCVyKhGWGIMLPwQGO7RA -ZQQcKN6lTXvVBUoEiwjSezjgEC/xsZG6c2JgO0cz4KN1+UD+HhpAG+uXMtIE -ldPDRtHFR7TR2gxJVJlPy91hbBsT2MqjtPx1fGzf11acHlzYIrTIg7N/yjom -BEyAFFT1jq6A+Jg/kYI/iXv3zCa1laC2R0VU0N983kLXp2MSF1H0h70JztAI -DYlytWIYiKJ6VjVoOaiwYjBZuHAtAS1wgU8JeSstNsIsZzrj0SH0q+zohQEn -5V46+QSXwjiyGuIDqFHu3O3QWjIF0hXHnwN337Dkit7oDWumyTjgYg98O58H -3TlLEhqd9K3bltEycK0854Mt6pH/jmwQmt6IamQGmz5CcCgFbyQJrybWlEDV -KP9FVOmW0ULTalez0aIBrphs1SO2mfK5qeigVSiiN8YlxCi4mFQ8cf6Vr/aL -FYI+gjgz4EKXVHTEWDg597MsVEKkEYtOehDOtSWBdMVG3AXgTe0JEhhj2wxp -L/SdNmLabYHvOObbJEKqLZ3D30IKM3YYR62U4goCoyJjQu1K0XL8XSC7DEDG -VYxNuCHs9DxqGZ1ZRlWkkJVwij6lK7Wv1LWegVkGzDyPD02pIee0D1zQz+15 -i+UoKG7uO4G5q4CNtZSxozdgsvCs21gEXoxpwDa0ErNINMoQwUmxEf4OjVIe -dpgCKDM5td5n5eUMeBU6tktgc1O3Iai/ODSIWI3aIvQ+WY1o0zie5bY7bBl6 -cNgzvjGxOyVAD+ikX8BTLXHgyByQfmdCqSiI1QeNO5sat8MAx8agZo0ZFow3 -C5E2uLDy7MDHx6W/hR9vlqzUZ+EmSUT+nRYSucVCaWSxda3xNZd9HBC7AMA/ -xZLgyaZBOmrticUP4/icskB75uehScpip22qdtoQ7roqBjVOnrLCLEaRhU/F -zQEWfSrcCBfJ9aYztLIoQABmWo7osRyudaGLgLG3S7JsXI6SNd2DfEbZeqrA -XaFk2qK/RpivJQr2QMs7RvNb9HC3+m2W+h1cI7VG7ut3alWMMR48oEgSK1rq -ZeRtkvA3dqRkr5lgazTpH5wg1mDfvP7w57l0UeYVtt6rdJFpT+jU/hvSj70v -luzVT44aDlD507q6rqwTQkvz5LQip6FWZlHGxwajMcVPpwx1ySEYPqPwLSc1 -CtBjaaabEeav9PdkB7ThYPXIs3bRLel1eNd+4IUlMcF63cdmDZ94tT60/eMS -4gy6W0KQVofm2Ulue2SQjJVhRtCh0XGvYI2McBCrGnkKLj9rpGS12ynT5fDB -ciJBq5Mk6z0S6lpxEA6H1GNmm4pP9Gi1Wo1TcrsuDePlcutwH29OHa/v9qx9 -2qoVwjvxpAVEnZPGkpCkdegKvXkMP9dfyPyDxivomdFDDHXXDDwrpRAt2fxn -n4kFiHacZ2p+zxbcePdANiHk2R6SFaAQ+rj5QzhsUZCsvBusojM0zoyMNS62 -qJWy/i+LMN4R8JjcAwlDcTgVrr40PiOzhfWtldMSqn5E9ogLVDjuK/jDAYwJ -VChGFwuDQrvr1LD2VDIZl7NOw9SEeVsZYpjckUbNG25cuhaGYqmpdhxLElZQ -h8KSrD5GDCgnNmZY+q0lk/BZYVczhkNhRcJ8i1n4JzNu2dGOhoUxfAE/kB3S -OyOxik189Ebkd/yivJ69A3ekJu1YiIPksbLuTVClCbY0GLxoycFBnAdesJHg -anZcIvihtRQmKWTorTPa3qvRxlNxfz+3v59lju5ZyV3aZFNiEIGTLnxcuXAs -QyYAgc8JQUUxBXfx3ipDsud40Oqh8ZQg7OaRTwyYp43SI+/eH/6AS0UC7X/4 -Azb7D3+4LuKfnmbZOf3pG/sT/UWrOWLByzW4U3zbZC1kJB2JNoDtQ2tUcn5q -P3X4QOQJD/PPYTZxnFakjS5Vmi2VcSPDhGM6tvHoDeIOiSehM+JfR/GNakRW -JgKNPCJ32SKmBeb5m5h9xyMiwYd3OpwLccVIswF3jRG59qwTT/Ba5JBNFNhm -Qy4PiedE9KVisDShUnS+J5RUpvtXmL4fLVJrNZIuNIh5aFyM1mScR93RJYPo -GTo9aVmLpMxQpBBGG6sVJG8Uxxt65ZTdTN4ADiqwo+J5xsZRNeNgCEUCccNe -ei+Cmejx4SgC+mZL7wkIAZjTYrOdRRzEyoUzLLPjqoPi+DNdsdHV5E6iwYwq -w/K4ghOUDbqdjE/GLtG7w1akVWv+fcIpoLZ5mItWTxXcwKFigj1ttjjo4+B/ -vi2Q2JNnYGxQYw23YfXfA1cxG95wc/mxtDrnS7r5dq0cooYEr+baHpoYG2iS -vp3Et4ReD/Zya//j/cojZY5Fq3nZsNpjKzpkqQdRlcTakf8aab77Cw6LJNR/ -uwPDz/v1h4af8qsPzlQn4d/oOGGmv/JIafZk/FhNjf+3PWwQgM8/cDbyhxw6 -TRf+6oM3vSC/4jhqFvTeI/mQvtTG3BirHHhV6L3MeYR6o7rZg1g35LnIjn7I -o0/+cW7PPAGiayMh89uGzh9nLNm+E1fnZM8wFvqf+Iu5PPqcruOP1Z6rdp5C -Vzy+uDiPQ431crxUi1L+rL8EnoB/zszTM+t5yyIg3jwbDjdsyX/EPf6PmY4l -HXbgwioSHbMXO+Ei/1f657/mj/m3u/l8ru9ed92+ffroEf0tTGBeNzeP+L8e -/YeBTBLZPB8U6gZ0OPSZDSuIzVObdHFTPksp+nvbsv34BLuCE/Lj26f5c8n7 -Bsy+IpDQsrpEy3da1Xn+ouDQwm1pqCL0anSGXdAc6N/hR4/VQZbTbYyss+3e -s2S0z1tl/jF673FdcCvAX24kovGK/sarxrpfrUQO2k7h1YUCOK6mvWfD5QY0 -atgVycsC2YMQXsL2vGBIdGg5Eer9enGd/wXXYILmGra9fZV1/Ui51wCdO15d -1kr+65pzjOhpWlnF5dBMupq49eQPoUZb6uWuR+lMfEnjTCtsAFIV15N/MVN0 -RV4BcV70yUpwdmA/TNgaQvRg9dVgx5ZBxwH7ItvRsc0DP5JeF658XkpCGSQs -6xD2t6sPkr27rUNx69Due8EdBPpR9DRmiQDi0vNgnRx5yYlot5c1c7AobVb4 -UGI0FQPbItAjPEySbHo4Jtr/IXuqzvYoPPw0ZOSTfVLJUsKvM6t6HTOF36xn -vlN8oitTq/i3Hsy9AKMP6zvDtj7j5ooVimcBpODVbivhqNS25EL2hoeFNTbr -04prm/yrC85w3rfuV2vLGvHk5HHdnYnkhPJx3IMpEn/E/wgVNjC+LTrMTVTm -+ff1rbYXaofoUteWMkQ2UnH+Tcd+7x5dtiFriyYmbaRHM55Qn8ZT0OXYGJ5x -KUpbz1ARhQOIB4vtyc90ISA8xsJA9EOj7kR4R+AWGsXvYM7SrcL2QwSdtYov -s4QDMEV1QIi1Ac2jhAv4+Uw6kaOFhfyczLgKWL+YcFixEUrmCUtpA8J0Zuuu -uNOMQ8WhfAqx1LaeRcH963TxxES9ROztpYWBAhBXD6zXU6KMLSXG5T6wfoZ1 -kOi0daV+hcZf38qHHNCTyJ9niAqxP/mjRv/yfyODFIQZD2UpCfNqOPjPuuv8 -CHRRoS5CxjauncS7jv6ITum9pJrcnAZTCoMfMgOslT7RjzS/46DDee5qMLeV -5LE8JFiMeofBi8yTgnQGicCRM0vT5quYX2UXpmSmFVej7+ZMvAYJKotKFzsZ -NEBdUwaKzAOLRM9v7l2jry7+aWKVKrkylRqJB/zGU+5YA88AAg3LxDkpNuVk -j/jhAcIo0y4Hlgo7J0sOFTz4aRMWlzxebw7BsOL+lyybtTe4d4smPXDTuTwk -Dhj1JsIDeN5ayMdNfJZmvxJbypdJ/LK4Vz+Fy8pYZ6FLcYJVoFmc+JUQO0gS -vpHKonTd7NDNU4soQhSj4Cj9TaxuIykREnn5OlqNKUbaZwYV2TNYtJAcjH+n -5X02la2OsNlYwBYznlwcnVSFjJFrPDAT/iFYtHKlyWFjzfzMvztRJ1sBEBu9 -YMz9Tealf8enbctd+LJ766wAPx4WSE6KKzOmtL0KKeSFd6GUmcvRs+xNt/a5 -ofGHibTim219aCT9HEqxLboe7DhaAB1qzP+qlEmoLWIXWDxG21AzijH2aRxp -0ygsNgyUg+MSLCdLT22VJfG6X89/uQs1CBGioD+vtAJBegzhEZLj07hsRMAI -Gc5+v7nTFF1wGUI/r1C3oDA6bQFzXPkki1NzORndRuXg9auHE9E/UOYDy63A -ozZW6MZ8P4CfF5/urD+hoD50wgbjMPqik8FKnETLTNqblcIfIlro+9AtewYL -6BmAGeEhQAiTwHL7wUpMTiR72U9hI1TGEkjdBrWbOh9+xSwQF6rRq6QPan/F -eAWwOdKuXpqLcSBF7O52Ue/LSFgxFXigw628WXKzvReGcLtnH+rsjuzT91qK -ZI2wEpaZoHpiJdH4+MCp1wdnwXSW4Ar7wIBqcRI1H7ChrhXCxQuVDdDI0/Tt -PDSAHxrNb4ZGwr5rq5mxojiC88WZrj4qfmjazMatmhiEmIrLuEzEqUZElE2J -e5VkGnM5GqWprPKfzSfyWWlH61Yz/oDo4Ta0TzilLlzhKXs6iHp4SZwRwXPC -Y5wF+uDnPL64+CfrBIbmXty2EF64ZnxA1cqabzfkckd3BsXCMBMLX7NuGd2q -3Rcnwob3Q9Pm+0gonR7aJvGnDHj3AS+jJwihE3w2U9+u93fW0jzDTPRSf2Yn -CH+idkLo+hUFwDGT+MqxCkoQYu3uvCEU9nuWx61O63ZCxFUAC2U8Jn5VZlYC -k/AdglElQEv6Gbb+V+OuAKgvSLOiBXWLD9SGWIzNS4lCZoGEeSC1s/FRR6BJ -rzAkPZvht9WQBiyDtuuPYETgjwyB9/5zh9FjjPhcM1wHrHr2V62c2Nm/cPV6 -t8bEsH7xcn7W2HpLevRCuyRfX4MtItAnOvQTANKrn4xKQGCF2vVUAjrxGs6f -c+bIxWn4SWDatt4Y/AjcxSeX4uqcPCBe5hnLwP2T0FUpZ2MDRBAOrQR0pAFB -r0pLKtqnRKNaDXfA8ux9ABfu7URl0EDJDjDEpyPewz6IIYq9EPPBaJG3Rwoo -X9euSzLbnLjHJKEyDmoq3H1pDvmxyxaQ+WE6Js3q85dAoXv/8wTPfd8D8S33 -RA178zOWQS/qW/P95tAmv+o9LEzfPdCacq4G3wjO0GjMxT32dl1vSv/M8Acu -QTm0wyf3hqUBLPcEKRdpO7Ko+4fcT7h3rwwGsmeI9wOeExbODUURMcdeoSba -kcGO8qBMa6LjxJPhhcemFXXgL31rqg3P8283bXnsB6FbktpMaWnNsbEaJeCm -jAE1bam2O8pVExGUYkqZHyFuSw8DCVsmK8bVPVrNcZUSyhpXR1zeLCW4mlzn -4yw3Utedn1pnKK1XFLFSTPjZvV7VgJoSax7ikFEeLQXBuzCyCXQRvWXkVGmI -4LpRz+baSB/cJTGhemOSF8uRiooRa6dNeZj50kkNB4G994lIWHQ+EXsye5Vk -zH114MwN6eQnaZUzB6PxrLCSD4+OXL4ah1UHUZUzOc6Ti6pZAdMoVXcR7tqV -EsiwgGpu+DY0nU2ynNfktq6Meto5tRrKs0uNZXtwoxgC6FnSRnvNFBwg5Aue -gLbMWB+24lgoc+HNgXapDb0vpYZHohyN1bgPoei/Q0aH5s+ngi4j2z9r7gzO -Dcy0ajT0k6+UmNOKhxDQCXTb6Hw1To/9VjuCvleuuWtUNGp8mwNkfExApQwc -8w72dp/y2T+ESa2AtKjqJTOdC4IC/kiCy7YUSPTkrBXVNe9gCWAbwq6VAAvE -DxPvCv6r3NSelttRioybHAZ1HhMjeUf5U8fYQPGhZs5r3nNG9y54Z8l7haxJ -rsTkNAhnGqvXYP7dljrRicDJ92+ukhYH9r2sf78wnrPdV+pcFtf0pyIwctaD -kYxwHINM5n6F2RmJGHkU3dqMp9t1iMDO82pORq9iACJxitwYTdV+1LCgSZBV -rUvpF47todkZCpeWUZ6BdMu44gxE9lxi6Zd0uDkiUdrRyhJDd5neAqA8l+rH -0NK+atFLwa7VsQAHzugVuQM513fEE2q9i2zysTe7FJm7kQlOuy+JIzqoVYvf -GO3atJcDn4UM71GBs2L92UAgJTArRB284kavlU0Y1PccCRL5MUn7bJnPxuR9 -pB+gUKAYujKR+ixcHbEEZ35ETPork0nh5WeLgjH1h/xtjwO9resdQqAQmG+K -DQ/WC4zUJnLBaAHicD7PTNdg6YquHlPZRzF6iHOBYciYCEfyY4hyhjr2a3Bh -Vwww16jefUBA/xJpdjGU3NA2Kmk7rZYfc5Ep7ztq9TThkLT6sG3WlFJE8Wpt -+tk4GeJiXS4+GjU70yAaxmmg6x73dN3JfWt7EpKFCRDXYpt0FG8leWgsQ8zC -+YDiuDwWx2koADk8Kelk4Hn0DCY0Iogf0MALSewhEEDrmTqj4LdeknqYOXKY -jV9tI/fa/9ve1TS3kRzZO35FB+dC2CBH9o4Pq13tBE2FwowdfYRErWO9sSE1 -gCbZSxCNQAPk8DK/fSvfy8yq6g+QY+vgwxwcYwpAdXVVVlZ+vneZHnB1CcMg -xGgVryI8rV6ivEPzAMVNFcwSMaPZY4OWLWhl0UIjdyoAxvxL3Xlo/LTc9R6h -xqBYEk0DQAYYy/+t+qunKmDPxuMQb6NUqOPXiwNmhHWe6hWwbjLCekNui8bq -wIVlT1enEP4VkjPSfSiGf9GWVwreSGQ5cibpwqm9xlnzlmGaC/MeMCy1mCRv -wmWHbgq/s6kXt23OgkEzfsTInzxh3jvDWG2MlglA0URJRjrd8LQ4wK+8bgY5 -U5Ie9HUlZDt1eHZd6Z7BBJH36rPzKi+vh27Uf6CvsHOaB+duiOyLIoBlvTTv -TZaECrxNgJlG/WuC49YpgEAfYcODY5edEJXxVDBmVC62YmociJFJq2v1QIFR -RtOkBxpbyadrfEwy7vpwhgVjaO7vD/OBwqYf6xscMDsaAxFDYi09PVb/nA1F -C9PRpBYqBuH6CjX/fbufp8R1nEz/R72GqXknrtNjwls0JsY7GzafqwGi1nHY -/k97SyeWSCdymK1n8urZlwcm2badqF78R6hA7bibZXHDbolUVy5QriLsG8// -WYxFprNiIrRWRH8V6dETVlyV28oVPH39mPCbGGBgGtBTUKOOXu9ykjIOQXjN -/hl5qIyy1LlvI9PnWDjgIg+P8zpQrQhz1zIFjYIxKTa+80tOhn3Fy+iXWQV0 -6buYrK0h1SlGPW22WbiF14tkjfqFAUwe0JoPrsuAjiIOTLx6FEylyVW+PXhZ -hb8BNrJSCkzhiDMe16xSpu74JgS+R9+X5qXT8SPEbbXS9q0x1rmJM4nH3ijJ -wFUHnMVPBCxsM0DAPC+kLJDZJ8mp9jrHHNitox/q1vDZ7G7LT7p4qiugo0os -laA/4TV3qHDbNdmOg56pp5c5TY5BesGENrdsR8wo7Z4R20SQK1Z7UrvGahy3 -ZrPoN1GKJo7nNvK2JkXZRJGT77BWAdn4EA+9eg/BshMzFJKY0MMVxxF9iuaE -U/aY2E0POIDQTrnC5G5LvGDeNisHZrLMY66PJ534QN+V61yT70HeQDTzAU3U -MQYYNRw4yP8GO3ffapVotsrxLIDdNJ5jrS6JkckBkDHnPSYUGuwQK5AG0knC -cKPi+A9sSyw1BczhQcI2i9ggMNBBfxoLbTwBrVMruq+SF/BEjHyXpl7kawZW -SJMSAtL3MOpQJ0I5gMfv/IJeqNLYsAZO0ckw/MQIXkwxXFg3ebmtOo4/9dp6 -INXQrdwgs2RmHM/MeehC2ohf4bW5Z0UChiOluggdy09B8WykGu4EBT8KXmPy -Rj6pzLAYmFGkQOp4JSQDMmV1HCtvXpz+aepEvZO+NQjmYJz2LqfwcVICKtVh -07C30f9kICWyYHK20eZFwLa20j2creBQSaWvMQ0+VNuU6Ml1S8oYbdeJ9exO -gqZsIxhuGh8J3zSGb47fttZhx79j2HcwGP25dReJELkpPnz1M0moBE2vuq9J -v1JV9JqMoQrfHiDcA3wrUA8jjqXg9C6Ejhgd+Zttvaj6gGyia8JOInvRhVq8 -WlU/qwbo/9DDXUqQBq1kZp217AN6bzqAA9c2PaDAT395//mn18YQ3id0G4Cp -Awj9TNEe5WAGtdbspaSf5tx9Uy+HhuqPNHeLhl6UiJ9BJuNEDeWVht/LMlZU -DoTBdI28qsx7HQPhe0ZVc0Q9V9NQ2zHWxd1jcV9XD6z8/RTLeiToUxz9CkHx -XuVnyMaR5Zojx53RG4z8gndu8neLZgcLByKqFvbzGs1iGQ5JihSm6Zb6DtEg -UCNKkT+O5zxIwo0sovwhtdTNspGYRaV0ERdsxpFIOWHjpYQp1TeRTNUyPWTq -1g5IWwxr8SsVhO5NTQa9Oomg627UJBFaNz8W77dSI3YBjDS/+tFtQhSIkb7Q -fiXff374oXifFnLLFSV0iPP/E5cIyxzrxVVrfFecKw6u5VDaHw9RPRjZGrY4 -7QoBM+AxsA/u0YDmpf7TmZexy5mRiBruclBnoLs0S7bpqGEpfpSuGFBKPIeE -4pJDY9/yeHoKZyu/15xFnD4OqrTeOGXQB63uL17i9i8uRSTcAJBB5LTFynyv -6AewImEtnzNpcm45yyrpIBlGx1IV2nR1b/4OkQvWT/VHPOfZuv4wgtoU8eXo -elsvjzxbbyxH7Hd/emA7U37HA2idOjkc5ut1tbSUWk3GMiXKWzRl8B4Leboe -TD3WzIC/e/2RCOEQsA/hrxRNMoIturrC9ehAASfN9kTkgmRy05RNPPIfg/CO -rAiU2pva6+aAbJImiQBwI6SCXH81Bq4IudYEG4e1MO61ZUA1N/VVmBwiArIE -8tI49KcHMEry7YPH2jLPJ49IQO1dBwvDpODP82Ww494jYvYvoN6qk3CJIKEj -M4EcGj0l6BSTDdHGt5k7Xcl7tXa4jlC+IvavhXhXj0echbjMApxjWU0/OzCh -VODFumXlpVEJzmwxt4T9SdEl2ApFu1ALZ4J8WM9LmDByCUZ30VaVkybaHhbH -9DXUgls0QSm2m4Y/iCs7HeDbybLDye77/fIcPZB0lS2N4Edko90F+88ULqxQ -Jf96kOk855i/RkuERlUMR9SwKK+aYBEBDnfdZx/5URF33so7nZMoXYqlzj5c -SMIWNWnOVfiWPio+NYAf8eyOYMMRgusOHK+IER3F51BhyPc/vjn/wx9/+CMk -IohTOCMbFYUOfZJw5L3+fKnrKz9FFVi1jrAlD2VN0l51N6UmozW7Dga3WWXw -DJKRGk3QoMQNXMz3pFgoyutrMJOBRuzghIbv7P6l/VdFiREyQK7QEhFyMWDg -Hfs13b+beoldUbiGgmsbrNBfwYDdLIlPlVUT4F9pz+0qUz5Sba5LAKMvzeUJ -xsX6UTKZpyK0YPR14idACzDgwZHelo9z2KKleatAHwsKoNr2bql34DcPi5pg -1reEOxCuAIcFIYwS8lKVagXO1tNggKmX7lPBtXcMhzhsbjnGSKf19GlV/19g -Px3l5GvtkdLliMCheVBeSZuRDlcQwgoKRwyPRstkr/xwoPDGdZjJa52ukOpe -WLb5PEe4zRi/BpcVunuZ1ZWiTgZ7UYDZcbZUBr8bsoUMl6pNDqBkcIeDxt7A -nZP7Gh5TvxNWKU9I7gy/Wj9wl0k57uzltXWQrk6j91XQUPJ7tAoxUru0iCnl -tGGPVIdDQmGtVbfvtACkNLq/EYEyc5dX2Rgif9iA26radNetu17e9aEzlQyd -hE436WrGUPhACM+O+iAnJMZkg7buBl+tU9aZ5nnVM8NCXN/stJ3a7YNYijUU -MSKvVzT5OsnhCI3FatsNrvWMaQaBOkpXFkItQUw2wK4AoddKG+bCKkW4Ia1Q -t86Dov5dceGLmreSWjAeX3xjyW5n3+n0nc6Gj7UivZAVAxFr+JyseJDoPOEK -NGTM96xsdp8tAt098WJ7IbTpV9ZMa3U2avpIMUa8GNVU0tLEbSk5Fn2GXN1X -5XZndEGDuWBc/EKDe2yRNnR1SqXvuppGgEjYaJuwOvXCDLWqs4cuAixmZe2s -ipJOiaatc8AnLAk3AuHfGwmkCckwGOSYbhJFQGHclaZojNdhGvXfaxCHi/k5 -6cs7zf+eFpaQ81BSx0DodtXalFY7KMEuvRp+GVGxI9udRkpgN8uE5D3CqsnU -K5YPGXNmMEKBISSXWluBOlMGcr4dwqfLfhLGSGZ2feNwm5FPO0yU5Ts1LKR9 -3d4ko0ByuJrtLslUqhGsKz6SFbgpFfXR6BUl58IC3WrtHQmsqI0B2kGYUKpp -j3wllwd35x41MHwFK7k65us7TcqqmtoehF8vqiCAflTfDpzQxJ4E+Ka5h7Iy -jxqE9qOzAaFOB4I2bfNXQqLgUgNYV/ZNGAj+9cWLH/7X9MUaak7iL4t6u9jf -KRXEzM1XtxNeq4CgrL3a7dUwilxTMh6kJ1fu8l6gHsJTbmq9fB8Myle0FXe+ -bqMYUGqkjdi/Fz5+3cF0vVoJc6SZIr1jR+SXx03VxodI7sXq6psmI3FlVFpi -hRJeb2vfKb/Mq7XsaauRZFQS4N+RibP9RFSYYr9pxHRm3kLEhxEDpY0GjL+Z -Ygs1M0oLLA4cYb6hUz8RIjKcMki/3nIr4upy6x0vJazbySccqXCoy0USsNRT -WUnGFQA1TVGFFWkeUdmuB4yBUanaeBRMovmqXNyezJufs/Cp2jZKdLMQf5BZ -RmDGy4qCQCoVo10lhs9u+xjV6Mde8Oo8DV4NHx3DhMh04yxXdql+PR1Q0YJU -bIQuY2a6MWsk6fdwxVmNKswPpXUiCUU4W7gR2s7O7dcMTaiJP3Dtom8Hl66f -kZ318rgRpO7zDIJMHqCrIfvHyz8fSjAb600fFsYWwlUzNbfCB+R3UpZClZsd -dbKoBfSwWrYBEVdEvu1QUAymbBwRRWnb8nJCuzEVUDWZZfTkDL9z5CrQIN4x -9sxJ1tH8I6PFJrxp8iYDADOHU8Z8SuKKSQFCjlfTse8TV0tqSDWVzww2s58G -RWVB15SU2Wp8b6SXSr9vgSpYRgdmyXsLOzvkcwSJt6b1i87H4EjRg3TYm/NI -OgR0v2siuXhYi0WacOvUIrWtbJdKQ07/lrx69RgkXVCRGUpPue8Q/5Nb2UQk -Mn4brZJV1cDXuGlWy4QIEZdcZAInEqkM8BMTvOWycc9KS0K0NkPar672NIgX -KySamnX+egwNanXEOPGepR/KZRVVYoYcHGT5XZDlv4ksS4JcTXIyfkvYYL9x -vF43zpNMutuuQzrQ61f8uCStqokMuU2aVAb3PatjKzcok5ocDzGZC8qWDILj -mBDpfZNY1xdrhSuRoCLTZh+1daNzRXOjW+G0hAE4IhV2Vuf7GhTAWueQV0IO -a5aZ2yb+M5RZgI5txNEeKQp3j4qTSd6RRELOFxU3Y9eEhYLxE2MktpdXsWiB -flVydYzB4SPRcuXE924VJBCuzlQoislUjhGGH4xrxaWyKgDYtVo00Rml50eO -KCPRVYAQVF12bN7ZdJQCkjazcxSble5SMuvgldX5hagqu3TKWCtstQRcN+t2 -2oXFKi7O3p2JPCGuoBfB5F2D4AByUWEP5DugA/tUhamIUuz+4M9pulpO7T2x -28o2+jZe1XlX3TX0yeq7mvVwk121uFlLhvsxcfIirl4p5tn3IipU9wuL7C/l -2g4Wbo2I7WRVzoXTQMjIqvV9vW3WjMLRIxBnng2rDEwHWSsXlbcSeKgsHN4Y -unXeBWN0EKQT7fnIsvRr4qaFt9nwRQxOndS+nnpAqp6OijSvv/386RLtpHPM -YO0lglJEs6iSHl8NW/FgE6mVlhiaA0sx5IWJTZbM5iLWXt0GBbGLdpupBhkh -HLmgWe8I2oefSOcBr4xZ/np1CswO2oQjt7ePRH3WwTmTnkWBh2mDMiAkcFIH -GGRpDsmmI2AFQZyO7m94/CcaKQylidSKIGlxjawTfFvbFhOLDJA+THKSzd15 -qwtjC1Y9RwVoq9WaeEtHobyG94XYY3Qa827+VQJG5Vx1b7IREx0acLnF2eJ2 -3TwEob3WKDD7Cjc3W7ReI8Iba17suDjFJqrPF+y5Jtcw0gASgxaOkzCVs4vi -+I7//7Ss5Zp6KzaAWIq3cAvPguESzH2Yjqp/33949+a/ivA/rJjM3UsvzIWb -hIuqrh4Khf4KKn/vXGuqS3T2xJGuHfqusqpcQ/1pPdvjWMpxx4ERcRN+gdIr -US67/dLOp70C1zrI89mqeBsuNG2rD0NO3OXkfCUdBhWBtQJankeVroIyn5eL -WyPoZZNk0LgToAGshC/5VOht1rdASz9b+XXAVgLxI6lsoKLuNeb4GGYfLGL5 -LsCOlPwQC7Z+ELxWlMpZmRf2FX1gAvcqWCE3cC4nQVwZJwp3BVYhiO5qhqmU -IK9EgknAXIgvkZRavhwma7xMU2ym3rTdwnnJxwpkhflLTk6fkq3fTTsrulR5 -aCT4n4QYU9oq+SdZOVWdlqtrYXa+uRPhCh4SAjNAicM9NFVF7NwR5X1Zrxic -zZpmeWzHAZHy11BvLa2k1KaOs1XGL4ViKdn74Pzst+toy1oe2W6FVSOC1RlW -onW2R7TwN221D6cgeOMTI8nVwhbBr7OKz3YnHfFynIycKBgAwSVYTr5GGJSv -kOOvCRTN16H6zb9qUl2hYz3fmgCqnKKAiXyzctoiF4lzI61L5O7Efn02OqYQ -Y9COjaiSYVkeDLpdPjv8PsnrB5OGmNc0maUajRVMbGlpCuv/ofWaR1cRcpR+ -X4UAzWvdIJn44AQfOKy603qOMZtI4NM31EsAFMxfSwI/PAYBWpOoACXZGgsI -K4GQYHxFIpZBHDwquN1nKEWRZ2ByGOIoP4+wi3EMkgN5+uRkrsonQJK6WE6d -zrDnISslK0jPTMniHJU0Tiq8ihwjr4lysH00QH4V0/+LHd4v7f7uK3mmMmTG -hN54cFF8IAacvuCnOlC/2eXJgZrlFwmi6mw+dVsgtbA2LR7vZeO9gqdfBT8Y -GESkmn1LFPTocofR0DTVaUDoQzyFuc/Lf7apP2vaWHK4vf/IxC2l+TTg0rdc -7n+eaT895Ut4T0tW6basRGOEJc0WxYqzPAbRgyqc/PLLL8UGunLiXVSyFMWr -It/S4ndF73wW3xfHfzh9UZz0P5pOKoMa+YI1xpCpcBe/Dz7Rz8cvTl8AlyV5 -0kmRTSUdCnarDia/zk56GLH30FnRU0/TieFmZP8cRhx6zsBrTxJQtFf9Rxb/ -/qoYfMIkxa97VRwPPe6kyF5pOj5Y2LnMGv3zyzGwuScMUqv0HAl1ib73SD2g -LHyoX88C87R1i55f7+trtcwsv5pSzD50eCGYOxqgPx6zO6cTaed1X8fm4b3c -I2kFZt2G7p7fTOvEtC51QdWsDnvqdu3XpDb7S6zNHrSkzy/FmDm/PGUYX9gR -2IqNHhIkUpXX4VuYyKgbiXbyyDzjm3wbA/nXsyl9cxP5N5vwN5vwG9uEmDIn -FWZ9FlSWlB5YzZ4hKedpoG/5DpJ6iNSVnfqo7kvJ+T3LkTJRySvqTCJECaxs -PA8SPUhZJEyVvxRFFFaAiH2StMmEt/tsoNaZiud1V+t2nmYj2ZvpOJ0tMw1t -Pxy1FQmglCDzjfBfT7LlGAe8Tq3IsK/RIDMjraMMZkXXbkslfDph6aMs8Ksi -G+93B+xNt5Licoefv5PXwJISO3BdJFLJ3oLw9kO/DWshPyaVns3nP4pgr1pL -QjH4u1dFd9f963GcE/tWVv4tHDBx8HkQ19tJ/Lf66tdOJKzVZPgGCx+a/fAF -9oOv7MBIU9qaJydimC9uB+5o9jICZm4VzNv8Ika3Cb9xLt9oT4v3a+CCCD2h -hKfX1YDngtyWXKHLxlJBA/0sGPxjtQZmBttXvg/G1sn3ZqPizkNdh7cs1Vvv -aBjukPmXtEPG4PM6uWHLMgFK67qRQM6uuTvZb0ZnyeE+rwXU0XMy0VZod49S -PjjfhiUOpuKMKTP5r74JeTkUCE7QLadPPOpsuYzd4KtaovqeCj8PS1ULoVY4 -z5fV9o7ko9ry+Kz+wHJT7zJGrnfVA4ZqX1rGApsq9pfVgDFb4JViYrpfaDXQ -fUOwBaXDXOTDa5kFtXzY37ta09TGs4ruFsRaP745b2XUSoHifKTKKkvIOO8C -Uu9W46KlDVhXLKPahNerZW9QnM8aCHT1SUp9ZVsIs47Z07OL6bCJ9v/XgXit -I+IBAA== - ---> - -</rfc> - diff --git a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.md b/docs/ietf/draft-ietf-bmwg-mlrsearch-08.md index eb2a218bb8..387ff4dba8 100644 --- a/docs/ietf/draft-ietf-bmwg-mlrsearch-07.md +++ b/docs/ietf/draft-ietf-bmwg-mlrsearch-08.md @@ -2,8 +2,8 @@ title: Multiple Loss Ratio Search abbrev: MLRsearch -docname: draft-ietf-bmwg-mlrsearch-07 -date: 2024-07-18 +docname: draft-ietf-bmwg-mlrsearch-08 +date: 2024-08-28 ipr: trust200902 area: ops |