feat(methodology): add methodology for autogen

Change-Id: I416c4120f2d7e05582a1f4fc529dfc86a09a7788
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

2 files changed, 140 insertions, 0 deletions

diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index 3fcd5bea8d..255fd15aa3 100644
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -28,4 +28,5 @@ Test Methodology
     methodology_lxc_drc_container_memif
     methodology_nfv_service_density
     methodology_vpp_device_functional
+    methodology_autogen
     methodology_aws/index
diff --git a/docs/report/introduction/methodology_autogen.rst b/docs/report/introduction/methodology_autogen.rst
new file mode 100644
index 0000000000..1bf5e9e5ea
--- /dev/null
+++ b/docs/report/introduction/methodology_autogen.rst
@@ -0,0 +1,139 @@
+.. _autogen_methodology:
+
+Suite Generation
+^^^^^^^^^^^^^^^^
+
+CSIT uses robot suite files to define tests.
+However, not all suite files available for Jenkins jobs
+(or manually started bootstrap scripts) are present in CSIT git repository.
+They are generated only when needed.
+
+Autogen Library
+~~~~~~~~~~~~~~~
+
+There is a code generation layer implemented as Python library called "autogen",
+called by various bash scripts.
+
+It generates the full extent of CSIT suites, using the ones in git as templates.
+
+Sources
+~~~~~~~
+
+The generated suites (and their contents) are affected by multiple information
+sources, listed below.
+
+Git Suites
+----------
+
+The suites present in git repository act as templates for generating suites.
+One of autogen design principles is that any template suite should also act
+as a full suite (no placeholders).
+
+In practice, autogen always re-creates the template suite with exactly
+the same content, it is one of checks that autogen works correctly.
+
+Regenerate Script
+-----------------
+
+Not all suites present in CSIT git repository act as template for autogen.
+The distinction is on per-directory level. Directories with
+regenerate_testcases.py script usually consider all suites as templates
+(unless possibly not included by the glob patten in the script).
+
+The script also specifies minimal frame size, indirectly, by specifying protocol
+(protocol "ip4" is the default, leading to 64B frame size).
+
+Constants
+---------
+
+Values in Constants.py are taken into consideration when generating suites.
+The values are mostly related to different NIC models and NIC drivers.
+
+Python Code
+-----------
+
+Python code in resources/libraries/python/autogen contains several other
+information sources.
+
+Testcase Templates
+__________________
+
+The test case part of template suite is ignored, test case lines
+are created according to text templates in Testcase.py file.
+
+Testcase Argument Lists
+_______________________
+
+Each testcase template has different number of "arguments", e.g. values
+to put into various placeholders. Different test types need different
+lists of the argument values, the lists are in regenerate_glob method
+in Regenerator.py file.
+
+Iteration Over Values
+_____________________
+
+Python code detects the test type (usually by substrings of suite file name),
+then iterates over different quantities based on type.
+For example, only ndrpdr suite templates generate other types (mrr and soak).
+
+Hardcoded Exclusions
+____________________
+
+Some combinations of values are known not to work, so they are excluded.
+Examples: Density tests for too much CPUs; IMIX for ASTF.
+
+Non-Sources
+~~~~~~~~~~~
+
+Some information sources are available in CSIT repository,
+but do not affect the suites generated by autogen.
+
+Testbeds
+--------
+
+Overall, no information visible in topology yaml files is taken into account
+by autogen.
+
+Testbed Architecture
+____________________
+
+Historically, suite files are agnostic to testbed architecture, e.g. ICX or ALT.
+
+Testbed Size
+____________
+
+Historically, 2-node and 3-node suites have diferent names, and while
+most of the code is common, the differences are not always simple enough.
+Autogen treat 2-node and 3-node suites as independent templates.
+
+TRex suites are intended for a 1-node circuit of otherwise 2-node or 3-node
+testbeds, so they support all 3 robot tags.
+They are also detected and treated differently by autogen,
+mainly because they need different testcase arguments (no CPU count).
+Autogen does nothing specifically related to the fact they should run
+only in testbeds/NICs with TG-TG line available.
+
+Other Topology Info
+___________________
+
+Some bonding tests need two (parallel) links between DUTs.
+Autogen does not care, as suites are agnostic.
+Robot tag marks the difference, but the link presence is not explicitly checked.
+
+Job specs
+---------
+
+Information in job spec files depend on generated suites (not the other way).
+Autogen should generate more suites, as job spec is limited by time budget.
+More suites should be available for manually triggered verify jobs,
+so autogen covers that.
+
+Bootstrap Scripts
+-----------------
+
+Historically, bootstrap scripts perform some logic,
+perhaps adding exclusion options to Robot invocation
+(e.g. skipping testbed+NIC combinations for tests that need parallel links).
+
+Once again, the logic here relies on what autogen generates,
+autogen does not look into bootstrap scripts.