1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
|
---
title: "Suite Generation"
weight: 5
---
# Suite Generation
CSIT uses robot suite files to define tests. However, not all suite files
available for Jenkins jobs (or manually started bootstrap scripts) are present
in CSIT git repository. They are generated only when needed.
## Autogen Library
There is a code generation layer implemented as Python library called "autogen",
called by various bash scripts.
It generates the full extent of CSIT suites, using the ones in git as templates.
## Sources
The generated suites (and their contents) are affected by multiple information
sources, listed below.
### Git Suites
The suites present in git repository act as templates for generating suites.
One of autogen design principles is that any template suite should also act
as a full suite (no placeholders).
In practice, autogen always re-creates the template suite with exactly
the same content, it is one of checks that autogen works correctly.
### Regenerate Script
Not all suites present in CSIT git repository act as template for autogen.
The distinction is on per-directory level. Directories with
`regenerate_testcases.py` script usually consider all suites as templates
(unless possibly not included by the glob patten in the script).
The script also specifies minimal frame size, indirectly, by specifying protocol
(protocol "ip4" is the default, leading to 64B frame size).
### Constants
Values in `Constants.py` are taken into consideration when generating suites.
The values are mostly related to different NIC models and NIC drivers.
### Python Code
Python code in `resources/libraries/python/autogen` contains several other
information sources.
#### Testcase Templates
The test case part of template suite is ignored, test case lines
are created according to text templates in `Testcase.py` file.
#### Testcase Argument Lists
Each testcase template has different number of "arguments", e.g. values
to put into various placeholders. Different test types need different
lists of the argument values, the lists are in `regenerate_glob` method
in `Regenerator.py` file.
#### Iteration Over Values
Python code detects the test type (usually by substrings of suite file name),
then iterates over different quantities based on type.
For example, only ndrpdr suite templates generate other types (mrr and soak).
#### Hardcoded Exclusions
Some combinations of values are known not to work, so they are excluded.
Examples: Density tests for too much CPUs; IMIX for ASTF.
## Non-Sources
Some information sources are available in CSIT repository,
but do not affect the suites generated by autogen.
### Testbeds
Overall, no information visible in topology yaml files is taken into account
by autogen.
#### Testbed Architecture
Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT.
#### Testbed Size
Historically, 2-node and 3-node suites have diferent names, and while
most of the code is common, the differences are not always simple enough.
Autogen treat 2-node and 3-node suites as independent templates.
TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node
testbeds, so they support all 3 robot tags.
They are also detected and treated differently by autogen,
mainly because they need different testcase arguments (no CPU count).
Autogen does nothing specifically related to the fact they should run
only in testbeds/NICs with TG-TG line available.
#### Other Topology Info
Some bonding tests need two (parallel) links between DUTs. Autogen does not
care, as suites are agnostic. Robot tag marks the difference, but the link
presence is not explicitly checked.
### Job specs
Information in job spec files depend on generated suites (not the other way).
Autogen should generate more suites, as job spec is limited by time budget.
More suites should be available for manually triggered verify jobs,
so autogen covers that.
### Bootstrap Scripts
Historically, bootstrap scripts perform some logic,
perhaps adding exclusion options to Robot invocation
(e.g. skipping testbed+NIC combinations for tests that need parallel links).
Once again, the logic here relies on what autogen generates,
autogen does not look into bootstrap scripts.
|
