diff options
| -rw-r--r-- | docs/report/introduction/methodology.rst | 1 | ||||
| -rw-r--r-- | docs/report/introduction/methodology_autogen.rst | 139 |
2 files changed, 140 insertions, 0 deletions
diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst index 3fcd5bea8d..255fd15aa3 100644 --- a/docs/report/introduction/methodology.rst +++ b/docs/report/introduction/methodology.rst @@ -28,4 +28,5 @@ Test Methodology methodology_lxc_drc_container_memif methodology_nfv_service_density methodology_vpp_device_functional + methodology_autogen methodology_aws/index diff --git a/docs/report/introduction/methodology_autogen.rst b/docs/report/introduction/methodology_autogen.rst new file mode 100644 index 0000000000..1bf5e9e5ea --- /dev/null +++ b/docs/report/introduction/methodology_autogen.rst @@ -0,0 +1,139 @@ +.. _autogen_methodology: + +Suite Generation +^^^^^^^^^^^^^^^^ + +CSIT uses robot suite files to define tests. +However, not all suite files available for Jenkins jobs +(or manually started bootstrap scripts) are present in CSIT git repository. +They are generated only when needed. + +Autogen Library +~~~~~~~~~~~~~~~ + +There is a code generation layer implemented as Python library called "autogen", +called by various bash scripts. + +It generates the full extent of CSIT suites, using the ones in git as templates. + +Sources +~~~~~~~ + +The generated suites (and their contents) are affected by multiple information +sources, listed below. + +Git Suites +---------- + +The suites present in git repository act as templates for generating suites. +One of autogen design principles is that any template suite should also act +as a full suite (no placeholders). + +In practice, autogen always re-creates the template suite with exactly +the same content, it is one of checks that autogen works correctly. + +Regenerate Script +----------------- + +Not all suites present in CSIT git repository act as template for autogen. +The distinction is on per-directory level. Directories with +regenerate_testcases.py script usually consider all suites as templates +(unless possibly not included by the glob patten in the script). + +The script also specifies minimal frame size, indirectly, by specifying protocol +(protocol "ip4" is the default, leading to 64B frame size). + +Constants +--------- + +Values in Constants.py are taken into consideration when generating suites. +The values are mostly related to different NIC models and NIC drivers. + +Python Code +----------- + +Python code in resources/libraries/python/autogen contains several other +information sources. + +Testcase Templates +__________________ + +The test case part of template suite is ignored, test case lines +are created according to text templates in Testcase.py file. + +Testcase Argument Lists +_______________________ + +Each testcase template has different number of "arguments", e.g. values +to put into various placeholders. Different test types need different +lists of the argument values, the lists are in regenerate_glob method +in Regenerator.py file. + +Iteration Over Values +_____________________ + +Python code detects the test type (usually by substrings of suite file name), +then iterates over different quantities based on type. +For example, only ndrpdr suite templates generate other types (mrr and soak). + +Hardcoded Exclusions +____________________ + +Some combinations of values are known not to work, so they are excluded. +Examples: Density tests for too much CPUs; IMIX for ASTF. + +Non-Sources +~~~~~~~~~~~ + +Some information sources are available in CSIT repository, +but do not affect the suites generated by autogen. + +Testbeds +-------- + +Overall, no information visible in topology yaml files is taken into account +by autogen. + +Testbed Architecture +____________________ + +Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT. + +Testbed Size +____________ + +Historically, 2-node and 3-node suites have diferent names, and while +most of the code is common, the differences are not always simple enough. +Autogen treat 2-node and 3-node suites as independent templates. + +TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node +testbeds, so they support all 3 robot tags. +They are also detected and treated differently by autogen, +mainly because they need different testcase arguments (no CPU count). +Autogen does nothing specifically related to the fact they should run +only in testbeds/NICs with TG-TG line available. + +Other Topology Info +___________________ + +Some bonding tests need two (parallel) links between DUTs. +Autogen does not care, as suites are agnostic. +Robot tag marks the difference, but the link presence is not explicitly checked. + +Job specs +--------- + +Information in job spec files depend on generated suites (not the other way). +Autogen should generate more suites, as job spec is limited by time budget. +More suites should be available for manually triggered verify jobs, +so autogen covers that. + +Bootstrap Scripts +----------------- + +Historically, bootstrap scripts perform some logic, +perhaps adding exclusion options to Robot invocation +(e.g. skipping testbed+NIC combinations for tests that need parallel links). + +Once again, the logic here relies on what autogen generates, +autogen does not look into bootstrap scripts. |