diff options
Diffstat (limited to 'docs/job_specs/perf_tests_job_specs.md')
| -rw-r--r-- | docs/job_specs/perf_tests_job_specs.md | 139 |
1 files changed, 0 insertions, 139 deletions
diff --git a/docs/job_specs/perf_tests_job_specs.md b/docs/job_specs/perf_tests_job_specs.md deleted file mode 100644 index 1f34c34b03..0000000000 --- a/docs/job_specs/perf_tests_job_specs.md +++ /dev/null @@ -1,139 +0,0 @@ -## Performance Tests Job Specifications - -## Scope - -Following FD.io CSIT test job specifications are provided in this note: - -- Select list of performance tests for daily trending and extensive per - release test coverage (behaviour repeatibility verification, graphs, - analytics and comparisons). -- Full list of performance tests for per release results reporting. - -Next sections list test selection criteria for each group, with testbed -environments listed in round brackets (...). - -## Select List Test Jobs - -### NICs - -- Primary: xxv710 (2n-skx, 3n-skx), x553 (3n-tsh). -- Secondary: x710 (2n-skx, 3n-skx). - -### Test Suites - -- Primary: all forwarding baseline, all forwarding scale, some feature - spot checks. -- Secondary: some forwarding base, forwarding maximum scale. - -### Frame Sizes - -- 64B: ip4, ip4_tunnels, l2, vts, container_memif, vm_vhost. -- 78B: ip6, ip6_tunnels, srv6. -- imix: crypto, nfv_density. - -### Processor Cores - -- Cores: 1c, 2c, 4c. - -## Full List Test Jobs -### NICs - -- Primary: xxv710 (2n-skx, 3n-skx), x553 (3n-tsh). -- Secondary: x710 (skx). - -### Test Suites - -- Primary: all tests. -- Secondary: some forwarding base, forwarding maximum scale. - -### Frame Sizes - -- 64B: ip4, ip4_tunnels, l2, vts, container_memif, vm_vhost, crypto. -- 78B: ip6, ip6_tunnels, srv6. -- imix: all including crypto and nfv_density. -- 1518B: ip4, ip4_tunnels, ip6, ip6_tunnels, srv6, l2, vts, - container_memif, vm_vhost, crypto. -- 9000B: ip4, ip4_tunnels, ip6, ip6_tunnels, srv6, l2, vts, - container_memif, vm_vhost, crypto. - - no vic1227, no vic1385 due to lack of support for 9000B. - -### Processor Cores - - - Cores: 1c, 2c, 4c. - -## Test Job Definitions -### Test Suite Allocation per Job - -In order to avoid multi-day jobs executing the tests, following is a -simple approach to split tests across exclusive job executions: - -- Separate jobs per (testbed environment, nic model, test-directory) -- Each job executes tests for: - - All specified frame sizes. - - All specified cores. - -### Report Jobs - -Patches defining RF Tag Selectors for each group are: - -- select-list: https://gerrit.fd.io/r/c/csit/+/21146 - - Execution frequency: 10 times for report. -- full-list: https://gerrit.fd.io/r/c/csit/+/21438 - - Incomplete definition. - - Execution frequency: Once for report. -- nfv_density: https://gerrit.fd.io/r/c/csit/+/21361 - - Execution frequency: 5..10 times for report. -- tcp: https://gerrit.fd.io/r/c/csit/+/21456 - - Execution frequency: 10 times for report. - -### Daily Trending Jobs - -Frequency of executing daily trending jobs differs between the -development cycle and report generation cycle. In the latter case the -frequency gets reduced in order to allocated more of a (limited) -physical testbed resource to tests required for report and reduce the -time required to complete data collection for report. - -- Daily tests - - Development cycle: - - Frequency: vpp select-list MRR tests twice a day. - - Times (UTC): every day starting at 02:00 and 14:00. - - Duration: jobs are monitored to last no longer than 12 hrs. - - Report cycle: - - Frequency: vpp select-list MRR tests once a day. - - Times (UTC): every Monday and Thursday starting at 02:00. - - Duration: jobs are monitored to last no longer than 12 hrs. -- Weekly tests - - Development cycle: - - Frequency: dpdk tests and vpp select-list NDR/PDR (MLRsearch) tests once a week. - - Times (UTC): every Sunday at 02:00. - - Duration: jobs are monitored to last no longer than 24 hrs. - - Report cycle: - - Same as in development cycle. - -## Exceptions -### Tests Failing Due to Known Issues - -- KernelVM: 9000B frames are failing - - https://jira.fd.io/browse/CSIT-1532 - -## TODOs - -- Add test types - - mlrsearch - - mrr - - plrsearch - - http/tcp - - nfvdensity - - vpp-ip4 - - vpp-ipsec - -- Analyse trending testbed load - PM, MK - - 2x 3n-skx - - tunnel tests only - - 4x 2n-skx - -- Future - - Add weekly ndrpdr jobs and report-like comparisons to previous release - -## END OF DOCUMENT
\ No newline at end of file |