aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMaciek Konstantynowicz <mkonstan@cisco.com>2019-08-25 21:11:02 +0100
committerMaciek Konstantynowicz <mkonstan@cisco.com>2019-08-27 12:16:38 +0000
commita0465dc55b2f7edfa86e7803704bdb1415fd2b8e (patch)
treeb7a07e832651705c4e5b45fe29f7dc686c142c59
parent6cdc335b42c3fa4c4c69ec0b20314df02118e4b4 (diff)
docs: added csit tests job specs.
Change-Id: I271b1c700f72204dcf03bee6f2cd30b25ca5663e Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
-rw-r--r--docs/job_specs/perf_tests_job_specs.md141
1 files changed, 141 insertions, 0 deletions
diff --git a/docs/job_specs/perf_tests_job_specs.md b/docs/job_specs/perf_tests_job_specs.md
new file mode 100644
index 0000000000..f151ae8afd
--- /dev/null
+++ b/docs/job_specs/perf_tests_job_specs.md
@@ -0,0 +1,141 @@
+## Performance Tests Job Specifications
+
+## Scope
+
+Following FD.io CSIT test job specifications are provided in this note:
+
+- Select list of performance tests for daily trending and extensive per
+ release test coverage (behaviour repeatibility verification, graphs,
+ analytics and comparisons).
+- Full list of performance tests for per release results reporting.
+
+Next sections list test selection criteria for each group, with testbed
+environments listed in round brackets (...).
+
+## Select List Test Jobs
+
+### NICs
+
+- Primary: xxv710 (2n-skx, 3n-skx), xl710 (3n-hsw), x553 (3n-tsh).
+- Secondary: x710 (2n-skx, 3n-skx).
+
+### Test Suites
+
+- Primary: all forwarding baseline, all forwarding scale, some feature
+ spot checks.
+- Secondary: some forwarding base, forwarding maximum scale.
+
+### Frame Sizes
+
+- 64B: ip4, ip4_tunnels, l2, vts, container_memif, vm_vhost.
+- 78B: ip6, ip6_tunnels, srv6.
+- imix: crypto, nfv_density.
+
+### Processor Cores
+
+- Cores: 1c, 2c, 4c.
+
+## Full List Test Jobs
+### NICs
+
+- Primary: xxv710 (2n-skx, 3n-skx), xl710 (3n-hsw), x553 (3n-tsh).
+- Secondary: x520 (3n-hsw), x710 (skx, hsw), vic1227 (3n-hsw), vic1385
+ (3n-hsw).
+
+### Test Suites
+
+- Primary: all tests.
+- Secondary: some forwarding base, forwarding maximum scale.
+
+### Frame Sizes
+
+- 64B: ip4, ip4_tunnels, l2, vts, container_memif, vm_vhost, crypto.
+- 78B: ip6, ip6_tunnels, srv6.
+- imix: all including crypto and nfv_density.
+- 1518B: ip4, ip4_tunnels, ip6, ip6_tunnels, srv6, l2, vts,
+ container_memif, vm_vhost, crypto.
+- 9000B: ip4, ip4_tunnels, ip6, ip6_tunnels, srv6, l2, vts,
+ container_memif, vm_vhost, crypto.
+ - no vic1227, no vic1385 due to lack of support for 9000B.
+
+### Processor Cores
+
+ - Cores: 1c, 2c, 4c.
+
+## Test Job Definitions
+### Test Suite Allocation per Job
+
+In order to avoid multi-day jobs executing the tests, following is a
+simple approach to split tests across exclusive job executions:
+
+- Separate jobs per (testbed environment, nic model, test-directory)
+- Each job executes tests for:
+ - All specified frame sizes.
+ - All specified cores.
+
+### Report Jobs
+
+Patches defining RF Tag Selectors for each group are:
+
+- select-list: https://gerrit.fd.io/r/c/csit/+/21146
+ - Execution frequency: 10 times for report.
+- full-list: https://gerrit.fd.io/r/c/csit/+/21438
+ - Incomplete definition.
+ - Execution frequency: Once for report.
+- nfv_density: https://gerrit.fd.io/r/c/csit/+/21361
+ - Execution frequency: 5..10 times for report.
+- tcp: https://gerrit.fd.io/r/c/csit/+/21456
+ - Execution frequency: 10 times for report.
+
+### Daily Trending Jobs
+
+Frequency of executing daily trending jobs differs between the
+development cycle and report generation cycle. In the latter case the
+frequency gets reduced in order to allocated more of a (limited)
+physical testbed resource to tests required for report and reduce the
+time required to complete data collection for report.
+
+- Daily tests
+ - Development cycle:
+ - Frequency: vpp select-list MRR tests twice a day.
+ - Times (UTC): every day starting at 02:00 and 14:00.
+ - Duration: jobs are monitored to last no longer than 12 hrs.
+ - Report cycle:
+ - Frequency: vpp select-list MRR tests once a day.
+ - Times (UTC): every Monday and Thursday starting at 02:00.
+ - Duration: jobs are monitored to last no longer than 12 hrs.
+- Weekly tests
+ - Development cycle:
+ - Frequency: dpdk tests and vpp select-list NDR/PDR (MLRsearch) tests once a week.
+ - Times (UTC): every Sunday at 02:00.
+ - Duration: jobs are monitored to last no longer than 24 hrs.
+ - Report cycle:
+ - Same as in development cycle.
+
+## Exceptions
+### Tests Failing Due to Known Issues
+
+- KernelVM: 9000B frames are failing
+ - https://jira.fd.io/browse/CSIT-1532
+
+## TODOs
+
+- Add test types
+ - mlrsearch
+ - mrr
+ - plrsearch
+ - http/tcp
+ - nfvdensity
+ - vpp-ip4
+ - vpp-ipsec
+
+- Analyse trending testbed load - PM, MK
+ - 3x 3n-hsw
+ - 2x 3n-skx
+ - tunnel tests only
+ - 4x 2n-skx
+
+- Future
+ - Add weekly ndrpdr jobs and report-like comparisons to previous release
+
+## END OF DOCUMENT \ No newline at end of file