diff options
authorVratko Polak <vrpolak@cisco.com>2022-06-28 12:36:17 +0200
committerVratko Polak <vrpolak@cisco.com>2022-06-30 08:22:36 +0000
commit44ff812b59a8cdcb3ac97cbda528e84e74b3547f (patch)
parent9e22f65d5e9f7045625a8d6bf79ce75d836e1447 (diff)
feat(methodology): add methodology for autogen
Change-Id: I416c4120f2d7e05582a1f4fc529dfc86a09a7788 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2 files changed, 140 insertions, 0 deletions
diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index 3fcd5bea8d..255fd15aa3 100644
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -28,4 +28,5 @@ Test Methodology
+ methodology_autogen
diff --git a/docs/report/introduction/methodology_autogen.rst b/docs/report/introduction/methodology_autogen.rst
new file mode 100644
index 0000000000..1bf5e9e5ea
--- /dev/null
+++ b/docs/report/introduction/methodology_autogen.rst
@@ -0,0 +1,139 @@
+.. _autogen_methodology:
+Suite Generation
+CSIT uses robot suite files to define tests.
+However, not all suite files available for Jenkins jobs
+(or manually started bootstrap scripts) are present in CSIT git repository.
+They are generated only when needed.
+Autogen Library
+There is a code generation layer implemented as Python library called "autogen",
+called by various bash scripts.
+It generates the full extent of CSIT suites, using the ones in git as templates.
+The generated suites (and their contents) are affected by multiple information
+sources, listed below.
+Git Suites
+The suites present in git repository act as templates for generating suites.
+One of autogen design principles is that any template suite should also act
+as a full suite (no placeholders).
+In practice, autogen always re-creates the template suite with exactly
+the same content, it is one of checks that autogen works correctly.
+Regenerate Script
+Not all suites present in CSIT git repository act as template for autogen.
+The distinction is on per-directory level. Directories with
+regenerate_testcases.py script usually consider all suites as templates
+(unless possibly not included by the glob patten in the script).
+The script also specifies minimal frame size, indirectly, by specifying protocol
+(protocol "ip4" is the default, leading to 64B frame size).
+Values in Constants.py are taken into consideration when generating suites.
+The values are mostly related to different NIC models and NIC drivers.
+Python Code
+Python code in resources/libraries/python/autogen contains several other
+information sources.
+Testcase Templates
+The test case part of template suite is ignored, test case lines
+are created according to text templates in Testcase.py file.
+Testcase Argument Lists
+Each testcase template has different number of "arguments", e.g. values
+to put into various placeholders. Different test types need different
+lists of the argument values, the lists are in regenerate_glob method
+in Regenerator.py file.
+Iteration Over Values
+Python code detects the test type (usually by substrings of suite file name),
+then iterates over different quantities based on type.
+For example, only ndrpdr suite templates generate other types (mrr and soak).
+Hardcoded Exclusions
+Some combinations of values are known not to work, so they are excluded.
+Examples: Density tests for too much CPUs; IMIX for ASTF.
+Some information sources are available in CSIT repository,
+but do not affect the suites generated by autogen.
+Overall, no information visible in topology yaml files is taken into account
+by autogen.
+Testbed Architecture
+Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT.
+Testbed Size
+Historically, 2-node and 3-node suites have diferent names, and while
+most of the code is common, the differences are not always simple enough.
+Autogen treat 2-node and 3-node suites as independent templates.
+TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node
+testbeds, so they support all 3 robot tags.
+They are also detected and treated differently by autogen,
+mainly because they need different testcase arguments (no CPU count).
+Autogen does nothing specifically related to the fact they should run
+only in testbeds/NICs with TG-TG line available.
+Other Topology Info
+Some bonding tests need two (parallel) links between DUTs.
+Autogen does not care, as suites are agnostic.
+Robot tag marks the difference, but the link presence is not explicitly checked.
+Job specs
+Information in job spec files depend on generated suites (not the other way).
+Autogen should generate more suites, as job spec is limited by time budget.
+More suites should be available for manually triggered verify jobs,
+so autogen covers that.
+Bootstrap Scripts
+Historically, bootstrap scripts perform some logic,
+perhaps adding exclusion options to Robot invocation
+(e.g. skipping testbed+NIC combinations for tests that need parallel links).
+Once again, the logic here relies on what autogen generates,
+autogen does not look into bootstrap scripts.