Report: Add perpatch_performance_tests to methodology

- It was removed from trending pages.

Change-Id: I876ef1a6a92322ab99b8f7d06bf94f29f2784333
Signed-off-by: Tibor Frank <tifrank@cisco.com>

3 files changed, 249 insertions, 0 deletions

diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index 255fd15aa3..a2b1b7b2cc 100644
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -30,3 +30,4 @@ Test Methodology
     methodology_vpp_device_functional
     methodology_autogen
     methodology_aws/index
+    methodology_rca/index
diff --git a/docs/report/introduction/methodology_rca/index.rst b/docs/report/introduction/methodology_rca/index.rst
new file mode 100644
index 0000000000..82c88a3ef9
--- /dev/null
+++ b/docs/report/introduction/methodology_rca/index.rst
@@ -0,0 +1,6 @@
+Root Cause Analysis
+-------------------
+
+.. toctree::
+
+    methodology_perpatch_performance_tests
diff --git a/docs/report/introduction/methodology_rca/methodology_perpatch_performance_tests.rst b/docs/report/introduction/methodology_rca/methodology_perpatch_performance_tests.rst
new file mode 100644
index 0000000000..a3fb4c5938
--- /dev/null
+++ b/docs/report/introduction/methodology_rca/methodology_perpatch_performance_tests.rst
@@ -0,0 +1,242 @@
+Per-patch performance tests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Updated for CSIT git commit id: 72b45cfe662107c8e1bb549df71ba51352a898ee.
+
+A methodology similar to trending analysis is used for comparing performance
+before a DUT code change is merged. This can act as a verify job to disallow
+changes which would decrease performance without a good reason.
+
+Existing jobs
+`````````````
+
+VPP is the only project currently using such jobs.
+They are not started automatically, must be triggered on demand.
+They allow full tag expressions, but some tags are enforced (such as MRR).
+
+There are jobs available for multiple types of testbeds,
+based on various processors.
+Their Gerrit triggers words are of the form "perftest-{node_arch}"
+where the node_arch combinations currently supported are:
+2n-clx, 2n-dnv, 2n-skx, 2n-tx2, 2n-zn2, 3n-dnv, 3n-skx, 3n-tsh.
+
+Test selection
+--------------
+
+..
+    TODO: Majority of this section is also useful for CSIT verify jobs. Move it somewhere.
+
+Gerrit trigger line without any additional arguments selects
+a small set of test cases to run.
+If additional arguments are added to the Gerrit trigger, they are treated
+as Robot tag expressions to select tests to run.
+While very flexible, this method of test selection also allows the user
+to accidentally select too high number of tests, blocking the testbed for days.
+
+What follows is a list of explanations and recommendations
+to help users to select the minimal set of tests cases.
+
+Verify cycles
+_____________
+
+When Gerrit schedules multiple jobs to run for the same patch set,
+it waits until all runs are complete.
+While it is waiting, it is possible to trigger more jobs
+(adding runs to the set Gerrit is waiting for), but it is not possible
+to trigger more runs for the same job, until Gerrit is done waiting.
+After Gerrit is done waiting, it becames possible to trigger
+the same job again.
+
+Example. User triggers one set of tests on 2n-skx and immediately
+also triggers other set of tests on 3n-skx. Then the user notices
+2n-skx run end early because of a typo in tag expression.
+When the user tries to re-trigger 2n-skx (with fixed tag expression),
+that comment gets ignored by Jenkins.
+Only when 3n-skx job finishes, the user can trigger 2n-skx.
+
+One comment many jobs
+_____________________
+
+In the past, the CSIT code which parses for perftest trigger comments
+was buggy, which lead to bad behavior (as in selection all performance test,
+because "perftest" is also a robot tag) when user included multiple
+perftest trigger words in the same comment.
+
+The worst bugs were fixed since then, but it is still recommended
+to use just one trigger word per Gerrit comment, just to be safe.
+
+Multiple test cases in run
+__________________________
+
+While Robot supports OR operator, it does not support parentheses,
+so the OR operator is not very useful. It is recommended
+to use space instead of OR operator.
+
+Example template:
+perftest-2n-skx {tag_expression_1} {tag_expression_2}
+
+See below for more concrete examples.
+
+Suite tags
+__________
+
+Traditionally, CSIT maintains broad Robot tags that can be used to select tests,
+for details on existing tags, see
+`CSIT Tags <https://github.com/FDio/csit/blob/master/docs/tag_documentation.rst>`_.
+
+But it is not recommended to use them for test selection,
+as it is not that easy to determine how many test cases are selected.
+
+The recommended way is to look into CSIT repository first,
+and locate a specific suite the user is interested in,
+and use its suite tag. For example, "ethip4-ip4base" is a suite tag
+selecting just one suite in CSIT git repository,
+avoiding all scale, container, and other simialr variants.
+
+Note that CSIT uses "autogen" code generator,
+so the robot running in Jenkins has access to more suites
+than visible just by looking into CSIT git repository,
+so suite tag is not enough to select even the intended suite,
+and user still probably wants to narrow down
+to a single test case within a suite.
+
+Fully specified tag expressions
+_______________________________
+
+Here is one template to select a single test case:
+{test_type}AND{nic_model}AND{nic_driver}AND{cores}AND{frame_size}AND{suite_tag}
+where the variables are all lower case (so AND operator stands out).
+
+Currently only one test type is supported by the performance comparison jobs:
+"mrr".
+The nic_driver options depend on nic_model. For Intel cards "drv_avf" (AVF plugin)
+and "drv_vfio_pci" (DPDK plugin) are popular, for Mellanox "drv_rdma_core".
+Currently, the performance using "drv_af_xdp" is not reliable enough, so do not use it
+unless you are specifically testing for AF_XDP.
+
+The most popular nic_model is "nic_intel-xxv710", but that is not available
+on all testbed types.
+It is safe to use "1c" for cores (unless you are suspection multi-core performance
+is affected differently) and "64b" for frame size ("78b" for ip6
+and more for dot1q and other encapsulated traffic;
+"1518b" is popular for ipsec and other payload-bound tests).
+
+As there are more test cases than CSIT can periodically test,
+it is possible to encounter an old test case that currently fails.
+To avoid that, you can look at "job spec" files we use for periodic testing,
+for example `this one <https://github.com/FDio/csit/blob/master/docs/job_specs/report_iterative/2n-skx/vpp-mrr-00.md>`_.
+
+..
+    TODO: Explain why "periodic" job spec link lands at report_iterative.
+
+Shortening triggers
+___________________
+
+Advanced users may use the following tricks to avoid writing long trigger comments.
+
+Robot supports glob matching, which can be used to select multiple suite tags at once.
+
+Not specifying one of 6 parts of the recommended expression pattern
+will select all available options. For example not specifying nic_driver
+for nic_intel-xxv710 will select all 3 applicable drivers.
+You can use NOT operator to reject some options (e.g. NOTdrv_af_xdp),
+but beware, with NOT the order matters:
+tag1ANDtag2NOTtag3 is not the same as tag1NOTtag3ANDtag2,
+the latter is evaluated as tag1AND(NOT(tag3ANDtag2)).
+
+Beware when not specifying nic_model. As a precaution,
+CSIT code will insert the defailt NIC model for the tetsbed used.
+Example: Specifying drv_rdma_core without specifying nic_model
+will fail, as the default nic_model is nic_intel-xxv710
+which does not support RDMA core driver.
+
+Complete example
+________________
+
+A user wants to test a VPP change which may affect load balance whith bonding.
+Searching tag documentation for "bonding" finds LBOND tag and its variants.
+Searching CSIT git repository (directory tests/) finds 8 suite files,
+all suited only for 3-node testbeds.
+All suites are using vhost, but differ by the forwarding app inside VM
+(DPDK or VPP), by the forwarding mode of VPP acting as host level vswitch
+(MAC learning or cross connect), and by the number of DUT1-DUT2 links
+available (1 or 2).
+
+As not all NICs and testbeds offer enogh ports for 2 parallel DUT-DUT links,
+the user looks at `testbed specifications <https://github.com/FDio/csit/tree/master/topologies/available>`_
+and finds that only x710 NIC on 3n-skx testbed matches the requirements.
+Quick look into the suites confirm the smallest frame size is 64 bytes
+(despite DOT1Q robot tag, as the encapsulation does not happen on TG-DUT links).
+It is ok to use just 1 physical core, as 3n-skx has hyperthreading enabled,
+so VPP vswitch will use 2 worker threads.
+
+The user decides the vswitch forwarding mode is not important
+(so choses cross connect as that has less CPU overhead),
+but wants to test both NIC drivers (not AF_XDP), both apps in VM,
+and both 1 and 2 parallel links.
+
+After shortening, this is the trigger comment fianlly used:
+perftest-3n-skx mrrANDnic_intel-x710AND1cAND64bAND?lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm*NOTdrv_af_xdp
+
+Basic operation
+```````````````
+
+The job builds VPP .deb packages for both the patch under test
+(called "current") and its parent patch (called "parent").
+
+For each test (from a set defined by tag expression),
+both builds are subjected to several trial measurements (BMRR).
+Measured samples are grouped to "parent" sequence,
+followed by "current" sequence. The same Minimal Description Length
+algorithm as in trending is used to decide whether it is one big group,
+or two smaller gropus. If it is one group, a "normal" result
+is declared for the test. If it is two groups, and current average
+is less then parent average, the test is declared a regression.
+If it is two groups and current average is larger or equal,
+the test is declared a progression.
+
+The whole job fails (giving -1) if some trial measurement failed,
+or if any test was declared a regression.
+
+Temporary specifics
+```````````````````
+
+The Minimal Description Length analysis is performed by
+CSIT code equivalent to jumpavg-0.1.3 library available on PyPI.
+
+In hopes of strengthening of signal (code performance) compared to noise
+(all other factors influencing the measured values), several workarounds
+are applied.
+
+In contrast to trending, trial duration is set to 10 seconds,
+and only 5 samples are measured for each build.
+Both parameters are set in ci-management.
+
+This decreases sensitivity to regressions, but also decreases
+probability of false positives.
+
+Console output
+``````````````
+
+The following information as visible towards the end of Jenkins console output,
+repeated for each analyzed test.
+
+The original 5 values are visible in order they were measured.
+The 5 values after processing are also visible in output,
+this time sorted by value (so people can see minimum and maximum).
+
+The next output is difference of averages. It is the current average
+minus the parent average, expressed as percentage of the parent average.
+
+The next three outputs contain the jumpavg representation
+of the two groups and a combined group.
+Here, "bits" is the description length; for "current" sequence
+it includes effect from "parent" average value
+(jumpavg-0.1.3 penalizes sequences with too close averages).
+
+Next, a sentence describing which grouping description is shorter,
+and by how much bits.
+Finally, the test result classification is visible.
+
+The algorithm does not track test case names,
+so test cases are indexed (from 0).