tests: make tests less make dependent

Implement command line argument parsing instead of passing arguments via
environment variables. Add script for running tests without having to
invoke make. Deprecate running tests via make.

Type: improvement
Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e
Signed-off-by: Klement Sekera <ksekera@cisco.com>
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

1 files changed, 82 insertions, 95 deletions

diff --git a/docs/developer/build-run-debug/testing_vpp.rst b/docs/developer/build-run-debug/testing_vpp.rst
index ca9a09efb71..28ddb21b122 100644
--- a/docs/developer/build-run-debug/testing_vpp.rst
+++ b/docs/developer/build-run-debug/testing_vpp.rst
@@ -43,98 +43,85 @@ Current “make test-help” output:
 
 ::
 
-     $ make test-help
-      test                   - build and run (basic) functional tests
-      test-debug             - build and run (basic) functional tests (debug build)
-      test-all               - build and run functional and extended tests
-      test-all-debug         - build and run functional and extended tests (debug build)
-      retest                 - run functional tests
-      retest-debug           - run functional tests (debug build)
-      retest-all             - run functional and extended tests
-      retest-all-debug       - run functional and extended tests (debug build)
-      test-cov               - generate code coverage report for test framework
-      test-gcov                      - build and run functional tests (gcov build)
-      test-wipe              - wipe (temporary) files generated by unit tests
-      test-wipe-cov          - wipe code coverage report for test framework
-      test-wipe-doc          - wipe documentation for test framework
-      test-wipe-papi         - rebuild vpp_papi sources
-      test-wipe-all          - wipe (temporary) files generated by unit tests, docs, and coverage
-      test-shell             - enter shell with test environment
-      test-shell-debug       - enter shell with test environment (debug build)
-      test-checkstyle        - check PEP8 compliance for test framework
-      test-refresh-deps      - refresh the Python dependencies for the tests
-
-     Arguments controlling test runs:
-      V=[0|1|2]              - set test verbosity level
-                               0=ERROR, 1=INFO, 2=DEBUG
-      TEST_JOBS=[<n>|auto]   - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
-      MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
-      CACHE_OUTPUT=[0|1]     - cache VPP stdout/stderr and log as one block after test finishes (default: 1)
-      FAILFAST=[0|1]         - fail fast if 1, complete all tests if 0
-      TIMEOUT=<timeout>      - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
-      RETRIES=<n>            - retry failed tests <n> times
-      DEBUG=<type>           - set VPP debugging kind
-         DEBUG=core          - detect coredump and load it in gdb on crash
-         DEBUG=gdb           - allow easy debugging by printing VPP PID
-                               and waiting for user input before running
-                               and tearing down a testcase
-         DEBUG=gdbserver     - run gdb inside a gdb server, otherwise
-                               same as above
-         DEBUG=attach        - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
-
-      STEP=[yes|no]          - ease debugging by stepping through a testcase
-      SANITY=[yes|no]        - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes)
-      EXTENDED_TESTS=[1|y]   - used by '[re]test-all' & '[re]test-all-debug' to run extended tests
-      TEST=<filter>          - filter the set of tests:
-         by file-name        - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
-         by file-suffix      - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
-         by wildcard         - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
-                               e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
-                                    TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
-                                    TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
-                                    TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
-                                    TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
-
-      VARIANT=<variant>      - specify which march node variant to unit test
-                               e.g. VARIANT=skx test the skx march variants
-                               e.g. VARIANT=icl test the icl march variants
-
-      COREDUMP_SIZE=<size>   - pass <size> as unix { coredump-size <size> } argument to vpp
-                               e.g. COREDUMP_SIZE=4g
-                                    COREDUMP_SIZE=unlimited
-      COREDUMP_COMPRESS=1    - compress core files if not debugging them
-      EXTERN_TESTS=<path>    - path to out-of-tree test_<name>.py files containing test cases
-      EXTERN_PLUGINS=<path>  - path to out-of-tree plugins to be loaded by vpp under test
-      EXTERN_COV_DIR=<path>  - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
-
-      PROFILE=1              - enable profiling of test framework via cProfile module
-      PROFILE_SORT_BY=opt    - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
-      PROFILE_OUTPUT=file    - output profiling info to file - use absolute path (default: stdout)
-
-      TEST_DEBUG=1           - turn on debugging of the test framework itself (expert)
-
-      SKIP_AARCH64=1         - skip tests that are failing on the ARM platorm in FD.io CI
-
-      RND_SEED=seed          - Seed RND with given seed
-
-     Starting VPP in GDB for use with DEBUG=attach:
-
-      test-start-vpp-in-gdb       - start VPP in gdb (release)
-      test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
-
-     Arguments controlling VPP in GDB runs:
-
-      VPP_IN_GDB_TMP_DIR     - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb)
-      VPP_IN_GDB_NO_RMDIR=0  - don't remove existing tmp dir but fail instead
-      VPP_IN_GDB_CMDLINE=1   - add 'interactive' to VPP arguments to run with command line
-
-     Creating test documentation
-      test-doc               - generate documentation for test framework
-      test-wipe-doc          - wipe documentation for test framework
-
-     Creating test code coverage report
-      test-cov               - generate code coverage report for test framework
-      test-wipe-cov          - wipe code coverage report for test framework
-
-     Verifying code-style
-      test-checkstyle        - check PEP8 compliance
+    $ make test-help
+    Running tests:
+
+     test                   - build and run (basic) functional tests
+     test-debug             - build and run (basic) functional tests (debug build)
+     test-all               - build and run functional and extended tests
+     test-all-debug         - build and run functional and extended tests (debug build)
+     retest                 - run functional tests
+     retest-debug           - run functional tests (debug build)
+     retest-all             - run functional and extended tests
+     retest-all-debug       - run functional and extended tests (debug build)
+     test-cov               - generate code coverage report for test framework
+     test-gcov                      - build and run functional tests (gcov build)
+     test-wipe              - wipe (temporary) files generated by unit tests
+     test-wipe-cov          - wipe code coverage report for test framework
+     test-wipe-papi         - rebuild vpp_papi sources
+     test-wipe-all          - wipe (temporary) files generated by unit tests, and coverage
+     test-shell             - enter shell with test environment
+     test-shell-debug       - enter shell with test environment (debug build)
+     test-checkstyle        - check PEP8 compliance for test framework
+     test-refresh-deps      - refresh the Python dependencies for the tests
+
+    Arguments controlling test runs:
+
+     V=[0|1|2]              - set test verbosity level
+                              0=ERROR, 1=INFO, 2=DEBUG
+     TEST_JOBS=[<n>|auto]   - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
+     MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
+     CACHE_OUTPUT=[0|n|no]  - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes)
+     FAILFAST=[1|y|yes]     - fail fast if 1, otherwise complete all tests
+     TIMEOUT=<timeout>      - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
+     RETRIES=<n>            - retry failed tests <n> times
+     DEBUG=<type>           - set VPP debugging kind
+        DEBUG=core          - detect coredump and load it in gdb on crash
+        DEBUG=gdb           - allow easy debugging by printing VPP PID
+                              and waiting for user input before running
+                              and tearing down a testcase
+        DEBUG=gdbserver     - run gdb inside a gdb server, otherwise
+                              same as above
+        DEBUG=attach        - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
+     STEP=[1|y|yes]         - enable stepping through a testcase (for testcase debugging)
+     SANITY=[0|n|no]        - disable sanity import of vpp-api/sanity vpp run before running tests
+     EXTENDED_TESTS=[1|y|yes] - run extended tests
+     TEST=<filter>          - filter the set of tests:
+        by file-name        - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
+        by file-suffix      - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
+        by wildcard         - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
+                              e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
+                                   TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
+                                   TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
+                                   TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
+                                   TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
+     VARIANT=<variant>      - specify which march node variant to unit test
+                              e.g. VARIANT=skx test the skx march variants
+                              e.g. VARIANT=icl test the icl march variants
+     COREDUMP_SIZE=<size>   - pass <size> as unix { coredump-size <size> } argument to vpp
+                              e.g. COREDUMP_SIZE=4g
+                                   COREDUMP_SIZE=unlimited
+     COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them
+     EXTERN_TESTS=<path>    - path to out-of-tree test_<name>.py files containing test cases
+     EXTERN_PLUGINS=<path>  - path to out-of-tree plugins to be loaded by vpp under test
+     EXTERN_COV_DIR=<path>  - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
+     PROFILE=[1|y|yes]      - enable profiling of test framework via cProfile module
+     PROFILE_SORT_BY=opt    - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
+     PROFILE_OUTPUT=file    - output profiling info to file - use absolute path (default: stdout)
+     TEST_DEBUG=[1|y|yes]   - enable debugging of the test framework itself (expert)
+     API_FUZZ=[1|y|yes]     - enable VPP api fuzz testing
+     RND_SEED=<seed>        - Seed RND with given seed
+
+    Starting VPP in GDB for use with DEBUG=attach:
+
+     test-start-vpp-in-gdb       - start VPP in gdb (release)
+     test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
+
+    Creating test code coverage report:
+
+     test-cov               - generate code coverage report for test framework
+     test-wipe-cov          - wipe code coverage report for test framework
+
+    Verifying code-style:
+
+     test-checkstyle        - check PEP8 compliance