diff options
Diffstat (limited to 'docs/content/methodology/per_patch_testing.md')
| -rw-r--r-- | docs/content/methodology/per_patch_testing.md | 77 |
1 files changed, 38 insertions, 39 deletions
diff --git a/docs/content/methodology/per_patch_testing.md b/docs/content/methodology/per_patch_testing.md index a64a52caf6..6ae40a13dc 100644 --- a/docs/content/methodology/per_patch_testing.md +++ b/docs/content/methodology/per_patch_testing.md @@ -5,7 +5,7 @@ weight: 5 # Per-patch Testing -Updated for CSIT git commit id: 72b45cfe662107c8e1bb549df71ba51352a898ee. +Updated for CSIT git commit id: d8ec3f8673346c0dc93e567159771f24c1bf74fc. A methodology similar to trending analysis is used for comparing performance before a DUT code change is merged. This can act as a verify job to disallow @@ -13,15 +13,15 @@ changes which would decrease performance without a good reason. ## Existing jobs -VPP is the only project currently using such jobs. They are not started automatically, must be triggered on demand. -They allow full tag expressions, but some tags are enforced (such as MRR). +They allow full tag expressions, all types of perf tests are supported. There are jobs available for multiple types of testbeds, based on various processors. Their Gerrit triggers words are of the form "perftest-{node_arch}" where the node_arch combinations currently supported are: -2n-clx, 2n-tx2, 2n-zn2, 3n-tsh. +2n-icx, 2n-clx, 2n-spr, 2n-zn2, 3n-icx, 3n-tsh, 3n-alt, 2n-tx2, 3n-snr, +3na-spr, 3nb-spr. ## Test selection @@ -47,10 +47,10 @@ the same job again. Example. User triggers one set of tests on 2n-icx and immediately also triggers other set of tests on 3n-icx. Then the user notices -2n-icx run end early because of a typo in tag expression. -When the user tries to re-trigger 2n-icx (with fixed tag expression), -that comment gets ignored by Jenkins. -Only when 3n-icx job finishes, the user can trigger 2n-icx. +2n-icx run ended early because of a typo in tag expression. +When the user tries to re-trigger 2n-icx (with a fixed tag expression), +that comment is ignored by Jenkins. +Only when 3n-icx job finishes, the user can trigger 2n-icx again. ### One comment many jobs @@ -65,8 +65,8 @@ to use just one trigger word per Gerrit comment, just to be safe. ### Multiple test cases in run While Robot supports OR operator, it does not support parentheses, -so the OR operator is not very useful. It is recommended -to use space instead of OR operator. +so the OR operator is not very useful. +It is recommended to use space instead of OR operator. Example template: perftest-2n-icx {tag_expression_1} {tag_expression_2} @@ -88,10 +88,9 @@ avoiding all scale, container, and other simialr variants. Note that CSIT uses "autogen" code generator, so the robot running in Jenkins has access to more suites -than visible just by looking into CSIT git repository, -so suite tag is not enough to select even the intended suite, -and user still probably wants to narrow down -to a single test case within a suite. +than visible just by looking into CSIT git repository. +Thus, suite tag is not enough to select precisely the intended suite, +and user is encouraged to narrow down to a single test case within a suite. ### Fully specified tag expressions @@ -99,25 +98,26 @@ Here is one template to select a single test case: {test_type}AND{nic_model}AND{nic_driver}AND{cores}AND{frame_size}AND{suite_tag} where the variables are all lower case (so AND operator stands out). -Currently only one test type is supported by the performance comparison jobs: -"mrr". +The fastest and the most widely used type of performance test is "mrr". +As an alternative, "ndrpdr" focuses on small losses (ax opposed to max load), +but takes longer to finish. The nic_driver options depend on nic_model. For Intel cards "drv_avf" (AVF plugin) and "drv_vfio_pci" (DPDK plugin) are popular, for Mellanox -"drv_rdma_core". Currently, the performance using "drv_af_xdp" is not reliable +"drv_mlx5_core". Currently, the performance using "drv_af_xdp" is not reliable enough, so do not use it unless you are specifically testing for AF_XDP. -The most popular nic_model is "nic_intel-xxv710", but that is not available +The most popular nic_model is "nic_intel-e810cq", but that is not available on all testbed types. -It is safe to use "1c" for cores (unless you are suspection multi-core +It is safe to use "1c" for cores (unless you are suspecting multi-core performance is affected differently) and "64b" for frame size ("78b" for ip6 and more for dot1q and other encapsulated traffic; -"1518b" is popular for ipsec and other payload-bound tests). +"1518b" is popular for ipsec and other CPU-bound tests). As there are more test cases than CSIT can periodically test, it is possible to encounter an old test case that currently fails. To avoid that, you can look at "job spec" files we use for periodic testing, for example -[this one](https://github.com/FDio/csit/blob/master/resources/job_specs/report_iterative/2n-icx/vpp-mrr-00.md). +[this one](https://raw.githubusercontent.com/FDio/csit/master/resources/job_specs/report_iterative/2n-spr/vpp-mrr-00.md). ### Shortening triggers @@ -129,16 +129,16 @@ once. Not specifying one of 6 parts of the recommended expression pattern will select all available options. For example not specifying nic_driver -for nic_intel-xxv710 will select all 3 applicable drivers. -You can use NOT operator to reject some options (e.g. NOTdrv_af_xdp), -but beware, with NOT the order matters: +for nic_intel-e810cq will select all 3 applicable drivers. +You can use NOT operator to reject some options (e.g. NOTdrv_af_xdp). +Beware, with NOT the order matters: tag1ANDtag2NOTtag3 is not the same as tag1NOTtag3ANDtag2, the latter is evaluated as tag1AND(NOT(tag3ANDtag2)). Beware when not specifying nic_model. As a precaution, CSIT code will insert the defailt NIC model for the tetsbed used. Example: Specifying drv_rdma_core without specifying nic_model -will fail, as the default nic_model is nic_intel-xxv710 +will fail, as the default nic_model is nic_intel-e810cq which does not support RDMA core driver. ### Complete example @@ -155,7 +155,7 @@ available (1 or 2). As not all NICs and testbeds offer enogh ports for 2 parallel DUT-DUT links, the user looks at [testbed specifications](https://github.com/FDio/csit/tree/master/topologies/available) -and finds that only xxv710 NIC on 3n-icx testbed matches the requirements. +and finds that only e810xxv NIC on 3n-icx testbed matches the requirements. Quick look into the suites confirm the smallest frame size is 64 bytes (despite DOT1Q robot tag, as the encapsulation does not happen on TG-DUT links). It is ok to use just 1 physical core, as 3n-icx has hyperthreading enabled, @@ -167,15 +167,15 @@ but wants to test both NIC drivers (not AF_XDP), both apps in VM, and both 1 and 2 parallel links. After shortening, this is the trigger comment fianlly used: -perftest-3n-icx mrrANDnic_intel-x710AND1cAND64bAND?lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm*NOTdrv_af_xdp +perftest-3n-icx mrrANDnic_intel-e810cqAND1cAND64bAND?lbvpplacp-dot1q-l2xcbase-eth-2vhostvr1024-1vm\*NOTdrv_af_xdp ## Basic operation The job builds VPP .deb packages for both the patch under test (called "current") and its parent patch (called "parent"). -For each test (from a set defined by tag expression), -both builds are subjected to several trial measurements (BMRR). +For each test (from the set defined by tag expressions), +both builds are subjected to several trial measurements (in case of MRR). Measured samples are grouped to "parent" sequence, followed by "current" sequence. The same Minimal Description Length algorithm as in trending is used to decide whether it is one big group, @@ -185,19 +185,21 @@ is less then parent average, the test is declared a regression. If it is two groups and current average is larger or equal, the test is declared a progression. -The whole job fails (giving -1) if some trial measurement failed, -or if any test was declared a regression. +The whole job fails (giving -1) if any test was declared a regression. +If a test fails, a fake result values are used, +so it is possible to use the job fo verify current fixes a test failing in parent +(if a test is not fixed, it is treated as a regression). ## Temporary specifics The Minimal Description Length analysis is performed by -CSIT code equivalent to jumpavg-0.1.3 library available on PyPI. +CSIT code equivalent to jumpavg-0.4.1 library available on PyPI. In hopes of strengthening of signal (code performance) compared to noise (all other factors influencing the measured values), several workarounds are applied. -In contrast to trending, trial duration is set to 10 seconds, +In contrast to trending, MRR trial duration is set to 10 seconds, and only 5 samples are measured for each build. Both parameters are set in ci-management. @@ -209,8 +211,8 @@ probability of false positives. The following information as visible towards the end of Jenkins console output, repeated for each analyzed test. -The original 5 values are visible in order they were measured. -The 5 values after processing are also visible in output, +The original 5 values (or 1 for non-mrr) are visible in order they were measured. +The values after processing are also visible in output, this time sorted by value (so people can see minimum and maximum). The next output is difference of averages. It is the current average @@ -220,11 +222,8 @@ The next three outputs contain the jumpavg representation of the two groups and a combined group. Here, "bits" is the description length; for "current" sequence it includes effect from "parent" average value -(jumpavg-0.1.3 penalizes sequences with too close averages). +(jumpavg-0.4.1 penalizes sequences with too close averages). Next, a sentence describing which grouping description is shorter, and by how much bits. Finally, the test result classification is visible. - -The algorithm does not track test case names, -so test cases are indexed (from 0). |