fix(methodology): Clean-up and expand MLR outline

Change-Id: I0fe3e23d8cdbbaa687fd40537a582345ba354df4
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

1 files changed, 198 insertions, 151 deletions

diff --git a/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md b/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md
index 3bf004e277..938666aa64 100644
--- a/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md
+++ b/docs/content/methodology/measurements/data_plane_throughput/mlr_search.md
@@ -39,7 +39,7 @@ PDR:
 ## Test Report Information
 
 The MLRsearch specification requires (or recommends) the test report to contain
-some information, here is the summary.
+specific information, here is the summary.
 
 CSIT uses TRex as the traffic generator, over SSH, bash and Python layers
 that add roughly 0.5 second delay between trials.
@@ -52,6 +52,11 @@ TRex uses multiple worker threads (typically 8) so the traffic
 can be bursty on small time scales, but even the small buffers on DUT side
 usually make this effect invisible.
 
+All traffic profiles generate ethernet frames that carry IPv4 or IPv6 packets,
+in minority of tests also containing VLAN tags (dot1q or dot1ad).
+For convenience, the rest of this description uses "packet" where MLRsearch
+specification expects "frame".
+
 TRex is usually precise in the number of packets sent,
 but in high-performance setups it may show "duration stretching"
 where it takes considerably longer than the intended duration
@@ -66,289 +71,331 @@ and the results are presented as aggregate values, e.g. east-west plust west-eas
 Other specifics of traffic profiles are too numerous to be listed here,
 CSIT code is the authoritative documentation.
 
-Max load is computed from known values of per-direction NIC limits,
-usually large packets are limited by media bandwidth and small frames
-are limited by intrinsid packets-per-second (pps) limits.
+Max load is computed from known values of per-direction NIC limits.
+Usually, large packet traffic is limited by media bandwidth,
+and small frame traffic is limited by per-port packets-per-second (pps) limits.
 Min load is set to 9.001 kpps to ensure enough packets for latency histogram.
 
 If min load is classified as an upper bound for the NDR goal,
 the test fails immediatelly (not searching for PDR).
-Also, if the search takes too long, the test fails firhout a result.
+Also, if the search takes too long, the test fails fithout a result.
 If max load is classified as a lower bound, this situation is reported
-as zero-width irregular result, usually not distinguished from regular results.
+as zero-width irregular result, otherwise not distinguished
+from regular results in the frontend presentation.
 
 Relevant lower bound and relevant upper bound are recorded together with
 conditional throughput for both goals, but only the conditional throughput
-is presented in our WebUI frontend.
+values are is presented in our WebUI frontend.
 
-TRex uses lightweight way to count forwarded packets,
+TRex uses a lightweight way to count forwarded packets,
 so it does not identify duplicate and reordered packets.
 Some testbeds contain a link for TRex self-test, the results of such "trex" tests
 are measured as separate tests, but they are not influencing real SUT tests.
 
-As different test require different SUT setups, those are lightly documented
+As different tests require different SUT setups, those are lightly documented
 on suite descriptions, but CSIT code is the authoritative documentation.
 
-<!--
-TODO: Are all requirements addressed in "Deviations from RFC 2544"?
-
-Compliant in principle, but:
-+ trial wait times
-+ trial time overhead to effdur
-+ start+sleep+stop
-+ 10us buffer
-+ TRex is bursty in principle but not in practice
-+ test report shows aggregate values
-+ traffic profile details are in code only
-+ fails on timeout or irregular NDR
-- intermediate phases increase dursum
-+ min load 9kbps per direction for latency
-+ max load by nic known pps and bps limits
-+ hitting that max load is treated as regular result of zero width
-+ only conditional throughputs are processed
-- (at least PDR is "less discrete")
-+ relevant bounds are also stored
-+ duplicate and reordered packets are not detected
-+ self-test is done for repeatability reasons
-- (not for max load reasons)
-+ SUT config is in code
-+ (only partly in suite documentation)
-
-TODO: Update the above when the below progresses.
--->
-
 ## Heuristics
 
-MRR and receive rate at MRR load are used as initial guesses for the search.
+MLRsearch specification giver large freedom for implementations.
+The CSIT implementation applies many heuristic in order to save search time.
+Here is a short summary.
 
-All previously measured trials (except the very first one which acts
-as a warm-up) are taken into consideration.
-
-At the start of the search, a discrete set of possible loads is pre-computed
+Before the search starts, a discrete set of possible loads is pre-computed
 and used to avoid rounding errors.
 
+The very first trial is done at max load, but acts as a warm-up,
+its results are ignored. All other 1-second trial results
+are used for load classification.
+
+MRR (forwarding rate at max load) and the forwarding rate at MRR load
+are used as initial candidate loads at the start of regular search.
+
 The specified search goals are treated as "final targets",
-but receded by "intermediate targets" of smaller duration sum
-and larger goal width. This allows the algorithm to quickly converge
+but preceded by "intermediate targets" of smaller duration sum (4.58s and 1s)
+and larger goal width (double and quadruple, respectively).
+This allows the algorithm to quickly converge
 towards the "interesting region" where full duration sum is needed.
 
 Generally, smaller candidate loads are measured first.
-For more tricks (uneven splits and multiple selection strategies)
+For more tricks (e.g. uneven splits and multiple selection strategies)
 see the source of the Python implementation.
 
 # Additional details
 
 There will be future documents linked from here.
+
 For now, there is only an outline of future (ambitious) content.
+Items in parenthesis are hints on what content will be in the parent subsection.
 
-## History of early versions in CSIT?
 
-### MLRsearch as a class of algorithms
+* History of early versions in CSIT
 
-(mutually incompatible)
+  * MLRsearch as a class of algorithms
 
-### Example tests benefiting from different goals?
+    * (Older Python library versions were mutually incompatible,
+      both witch each other and with early versions of MLRsearch IETF document.)
 
-## Design principles
+  * Example tests benefiting from different goals?
 
-### Independence of components
+* Design principles
 
-### Implementation freedom
+  * Independence of components
 
-#### Optional and implementation-required inputs
+    * (Programming languages, asynchronous calls via Manager.)
 
-#### Reasonable default values
+  * Implementation freedom
 
-#### Better outputs in future
+    * Optional and implementation-required inputs
 
-#### "allowed if makes worse" principle
+    * Reasonable default values
 
-### Follow intuition, avoid surprises
+    * Better outputs in future
 
-### Usage
+    * "allowed if makes worse" principle
 
-(anomaly detection in trending, comparison tables with low stdev for release)
+      * (Some requirements ensure fairness when benchmarking is done
+        by business competition. They are not needed and just waste time
+        if benchmarking is done in "home lab". For example,
+        decreasing wait time can only hurt results by turning bigger latency
+        into frame loss.)
 
-### Max load and min load
+  * Follow intuition, avoid surprises
 
-### Size of loss
+  * Usage
 
-(does not matter, only binary low-loss vs high-loss)
+    * (anomaly detection in trending, comparison tables with low stdev for release)
 
-### Goals used
+  * Max load and min load
 
-### Simulator
+  * Size of loss
 
-(PLRsearch fitting functions, exotic goals)
+    * (does not matter, only binary low-loss vs high-loss)
 
-(Example of time savings between RFC2544 and CSIT goal at the same accuracy?)
+  * Goals used
 
-### Long trials vs many trials
+    * (Why 50% exceed ratio is so good.)
 
-### Conservativeness
+  * Simulator
 
-### Fail fast
+    * (PLRsearch fitting functions, exotic goals)
 
-### Timeout
+    * (Example of time savings between RFC2544 and CSIT goal at the same accuracy?)
 
-## Measurer questions
+  * Long trials vs many trials
 
-### Capabilities
+    * (Many trials offer better flexibility and avoid issues with handling
+      long-vs-short results fairly.)
 
-(Traffic profiles specific to TRex, TG TA and Yang)
+    * (Mention reconfiguration tests still use long trial with zero tolerance?)
 
-### Self test
+  * Conservativeness
 
-### Warm-up
+    * (Some performance bugs manifest as infrequent big loss spikes.)
 
-### Time overhead
+    * (Is conservativeness still as important when exceed ratio is 50%?)
 
-### Predicting offered count
+  * Fail fast
 
-### Duration stretching
+    * (If min load is an upper bound for one goal,
+       bail out to save time on broken tests.)
 
-(start+sleep+stop)
+  * Timeout
 
-### Burstiness
+    * (If SUT behavior suddenly becomes erratic, it could prolong the duration
+      of a "job run" that tests many different SUT settings and traffic profiles.
+      To control such duration spikes, CSIT test fails if search result
+      is not found reasonably fast.)
 
-### Negative loss
+* Measurer questions
 
-### Aggregate limits
+  * Capabilities
 
-(RX+TX, sum over ports, number of queues, CPU limits, baseline vs burst in cloud)
+    * (Traffic profiles specific to TRex, TG TA and Yang)
 
-### Other Oload issues
+  * Self test
 
-(duplex and other interferences; DUT-DUT links with encapsulation overhead)
+  * Warm-up
 
-## Test report
+  * Time overhead
 
-### Definition
+  * Predicting offered count
 
-### Alternative units
+  * Duration stretching
 
-#### Unidirectional vs bidirectional
+    * (start+sleep+stop)
 
-#### Bandwidth
+  * Burstiness
 
-## Heuristics
+  * Negative loss
+
+    * (Typically, misconfigured SUT can produce small number of unexpected
+      additional packets, for example with IPv6 autonegotiation not disabled.)
+
+    * (Implementations can decide to either allow or treat as artificial losses.)
+
+    * (Rarely, excess of packets may signal wait times between trials is too low
+      and SUT buffers too high.)
+
+  * Aggregate limits
+
+    * (RX+TX, sum over ports, number of queues, CPU limits, baseline vs burst in cloud)
+
+  * Other Oload issues
+
+    * (duplex and other interferences; DUT-DUT links with encapsulation overhead)
+
+* Test report
+
+  * Definition
+
+  * Alternative units
+
+    * Unidirectional vs bidirectional
+
+    * Bandwidth
+
+* Heuristics
+
+  * FRMOL and FRFRMOL
+
+  * Intermediate targets
+
+  * Relative width
+
+  * Discrete loads
+
+  * Expansion coefficient
+
+  * Uneven splits
+
+  * Selector strategies
+
+  * Candidate ordering
+
+* DUT behaviors
+
+  * Periodic interrupts
 
-### FRMOL and FRFRMOL
+  * More details on distribution of big and small loss spikes
 
-### Intermediate phases
+    * (performance spectrum as a probabilistic distribution over trial forwarding rate)
 
-### Relative width
+    * (trial results as small population)
 
-### Discrete loads
+    * (median and other quantiles, "touching" quantiles)
 
-### Expansion coefficient
+  * Exceed probability
 
-### Uneven splits
+    * (load regions, common patterns seen in practice)
 
-### Selector strategies
+  * Large buffers
 
-### Candidate ordering
+  * Performance decrease due to resource leaks
 
-## DUT behaviors
+  * Energy mode switching can cause loss inversion?
 
-### Periodic interrupts
+* Correctness
 
-### More details on distribution of big and small loss spikes
+  * Balancing sum from short trials
 
-(performance spectrum as a probabilistic distribution over trial forwarding rate)
+  * Optimistic and pessimistic estimates
 
-(trial results as small population)
+  * Load is eventually classified
 
-(median and other quantiles, "touching" quantiles)
+  * Gaming is possible but slow
 
-### Exceed probability
+  * Brittle heuristics
 
-(load regions, common patterns seen in practice)
+  * Goal ordering
 
-### Large buffers
+  * Discouraged goals
 
-### Performance decrease due to resource leaks
+    * Goal Width > Goal Loss Ratio
 
-### Energy mode switching can cause loss inversion?
+    * Goal Duration Sum < Goal Final Trial Duration
 
-## Correctness
+    * Incomparable goals
 
-### Balancing sum from short trials
+      * (worst case: slow race to bottom)
 
-### Optimistic and pessimistic estimates
+  * When a load can become undecided again?
 
-### Load is eventually classified
+* Related test procedures
 
-### Gaming is possible but slow
+  * Latency
 
-### Brittle heuristics
+  * Passive Telemetry
 
-### Goal ordering
+  * Active Telemetry
 
-### Discouraged goals
+  * Comparison with FRMOL
 
-#### Goal Width > Goal Loss Ratio
+  * Comparison with PLRsearch
 
-#### Goal Duration Sum value lower than Goal Final Trial Duration
+* Beyond frames
 
-#### Incomparable goals
+  * Transactions
 
-(worst case: slow race to bottom)
+  * Fragmentation
 
-### When a load can become undecided again?
+  * Throttled TCP
 
-## Related test procedures
+  * Fixed scale
 
-### Latency
+    * (NAT performance depends on session table size.)
 
-### Passive Telemetry
+    * (Trials should be long enough to hit all sessions?)
 
-### Active Telemetry
+  * Ramp-up
 
-### Comparison with FRMOL
+    * (NAT slow-fast path example: Before testing fast path,
+      we need a slow enough trial to ensure SUT started tracking all sessions.)
 
-### Comparison with PLRsearch
+    * (Session counting as verification ramp-up was successful.)
 
-## Beyond frames
+    * (Issues with sessions timing out before real search starts or during search.)
 
-### Transactions
+    * (Heuristics to avoid ramp-ups, e.g. when observing zero loss
+      on large enough trial.)
 
-### Fragmentation
+  * Reset
 
-### Throttled TCP
+    * (NAT slow-fast path example: When testing slow path, session table
+      needs to be explicitly dropped.)
 
-### Ramp-up
+  * Bisecting for telemetry thresholds
 
-### Fixed scale
+  * Bisecting for B2B burst size
 
-### Reset
+* Future improvements
 
-### Bisecting for telemetry thresholds
+  * Return trials at relevant bounds
 
-### Bisecting for B2B burst size
+  * Allow flipping the conservativeness?
 
-## Future improvements
+    * (return the larger load when Loss Inversion happens)
 
-### Return trials at relevant bounds
+  * Short high-loss trials to affect Conditional Throughput?
 
-### Allow flipping the conservativeness?
+  * Multiple runs between hard SUT resets
 
-(return the larger load when Loss Inversion happens)
+    * (Example: RSS key randomized at start but not between trials.)
 
-### Short high-loss trials to affect Conditional Throughput?
+  * Duration sum based on misclassification probability
 
-### Multiple runs between hard SUT resets
+    * (the idea is to decide early on one-sided results and late on balanced results)
 
-### Duration sum based on misclassification probability
+    * (needs a prior on exceed probability distribution; and error/time balance)
 
-(needs a prior on exceed probability distribution; and error/time balance)
+  * Heavy loss should be worse than narrow loss
 
-### Heavy loss should be worse than narrow loss
+    * (larger discussion on similarities with SLA)
 
-### Predict goodput based on loss and latency
+  * Predict goodput based on loss and latency?
 
-## Examples?
+* Examples?
 
-(take a real run and discuss heuristic decisions?)
+  * (take a real run and discuss heuristic decisions?)
 
-## Summarize how MLRsearch addressed the Identified Problems?
+* Summarize how MLRsearch addressed the Identified Problems