diff options
Diffstat (limited to 'docs/developer/build-run-debug/testing_vpp.rst')
| -rw-r--r-- | docs/developer/build-run-debug/testing_vpp.rst | 177 |
1 files changed, 82 insertions, 95 deletions
diff --git a/docs/developer/build-run-debug/testing_vpp.rst b/docs/developer/build-run-debug/testing_vpp.rst index ca9a09efb71..28ddb21b122 100644 --- a/docs/developer/build-run-debug/testing_vpp.rst +++ b/docs/developer/build-run-debug/testing_vpp.rst @@ -43,98 +43,85 @@ Current “make test-help” output: :: - $ make test-help - test - build and run (basic) functional tests - test-debug - build and run (basic) functional tests (debug build) - test-all - build and run functional and extended tests - test-all-debug - build and run functional and extended tests (debug build) - retest - run functional tests - retest-debug - run functional tests (debug build) - retest-all - run functional and extended tests - retest-all-debug - run functional and extended tests (debug build) - test-cov - generate code coverage report for test framework - test-gcov - build and run functional tests (gcov build) - test-wipe - wipe (temporary) files generated by unit tests - test-wipe-cov - wipe code coverage report for test framework - test-wipe-doc - wipe documentation for test framework - test-wipe-papi - rebuild vpp_papi sources - test-wipe-all - wipe (temporary) files generated by unit tests, docs, and coverage - test-shell - enter shell with test environment - test-shell-debug - enter shell with test environment (debug build) - test-checkstyle - check PEP8 compliance for test framework - test-refresh-deps - refresh the Python dependencies for the tests - - Arguments controlling test runs: - V=[0|1|2] - set test verbosity level - 0=ERROR, 1=INFO, 2=DEBUG - TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1) - MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto) - CACHE_OUTPUT=[0|1] - cache VPP stdout/stderr and log as one block after test finishes (default: 1) - FAILFAST=[0|1] - fail fast if 1, complete all tests if 0 - TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600) - RETRIES=<n> - retry failed tests <n> times - DEBUG=<type> - set VPP debugging kind - DEBUG=core - detect coredump and load it in gdb on crash - DEBUG=gdb - allow easy debugging by printing VPP PID - and waiting for user input before running - and tearing down a testcase - DEBUG=gdbserver - run gdb inside a gdb server, otherwise - same as above - DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb) - - STEP=[yes|no] - ease debugging by stepping through a testcase - SANITY=[yes|no] - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes) - EXTENDED_TESTS=[1|y] - used by '[re]test-all' & '[re]test-all-debug' to run extended tests - TEST=<filter> - filter the set of tests: - by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py - by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py - by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*' - e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name - TEST='bfd.*.*' is equivalent to above example of filter by file-suffix - TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class - TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase - TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes - - VARIANT=<variant> - specify which march node variant to unit test - e.g. VARIANT=skx test the skx march variants - e.g. VARIANT=icl test the icl march variants - - COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp - e.g. COREDUMP_SIZE=4g - COREDUMP_SIZE=unlimited - COREDUMP_COMPRESS=1 - compress core files if not debugging them - EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases - EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test - EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report - - PROFILE=1 - enable profiling of test framework via cProfile module - PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime) - PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout) - - TEST_DEBUG=1 - turn on debugging of the test framework itself (expert) - - SKIP_AARCH64=1 - skip tests that are failing on the ARM platorm in FD.io CI - - RND_SEED=seed - Seed RND with given seed - - Starting VPP in GDB for use with DEBUG=attach: - - test-start-vpp-in-gdb - start VPP in gdb (release) - test-start-vpp-debug-in-gdb - start VPP in gdb (debug) - - Arguments controlling VPP in GDB runs: - - VPP_IN_GDB_TMP_DIR - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb) - VPP_IN_GDB_NO_RMDIR=0 - don't remove existing tmp dir but fail instead - VPP_IN_GDB_CMDLINE=1 - add 'interactive' to VPP arguments to run with command line - - Creating test documentation - test-doc - generate documentation for test framework - test-wipe-doc - wipe documentation for test framework - - Creating test code coverage report - test-cov - generate code coverage report for test framework - test-wipe-cov - wipe code coverage report for test framework - - Verifying code-style - test-checkstyle - check PEP8 compliance + $ make test-help + Running tests: + + test - build and run (basic) functional tests + test-debug - build and run (basic) functional tests (debug build) + test-all - build and run functional and extended tests + test-all-debug - build and run functional and extended tests (debug build) + retest - run functional tests + retest-debug - run functional tests (debug build) + retest-all - run functional and extended tests + retest-all-debug - run functional and extended tests (debug build) + test-cov - generate code coverage report for test framework + test-gcov - build and run functional tests (gcov build) + test-wipe - wipe (temporary) files generated by unit tests + test-wipe-cov - wipe code coverage report for test framework + test-wipe-papi - rebuild vpp_papi sources + test-wipe-all - wipe (temporary) files generated by unit tests, and coverage + test-shell - enter shell with test environment + test-shell-debug - enter shell with test environment (debug build) + test-checkstyle - check PEP8 compliance for test framework + test-refresh-deps - refresh the Python dependencies for the tests + + Arguments controlling test runs: + + V=[0|1|2] - set test verbosity level + 0=ERROR, 1=INFO, 2=DEBUG + TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1) + MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto) + CACHE_OUTPUT=[0|n|no] - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes) + FAILFAST=[1|y|yes] - fail fast if 1, otherwise complete all tests + TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600) + RETRIES=<n> - retry failed tests <n> times + DEBUG=<type> - set VPP debugging kind + DEBUG=core - detect coredump and load it in gdb on crash + DEBUG=gdb - allow easy debugging by printing VPP PID + and waiting for user input before running + and tearing down a testcase + DEBUG=gdbserver - run gdb inside a gdb server, otherwise + same as above + DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb) + STEP=[1|y|yes] - enable stepping through a testcase (for testcase debugging) + SANITY=[0|n|no] - disable sanity import of vpp-api/sanity vpp run before running tests + EXTENDED_TESTS=[1|y|yes] - run extended tests + TEST=<filter> - filter the set of tests: + by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py + by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py + by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*' + e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name + TEST='bfd.*.*' is equivalent to above example of filter by file-suffix + TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class + TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase + TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes + VARIANT=<variant> - specify which march node variant to unit test + e.g. VARIANT=skx test the skx march variants + e.g. VARIANT=icl test the icl march variants + COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp + e.g. COREDUMP_SIZE=4g + COREDUMP_SIZE=unlimited + COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them + EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases + EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test + EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report + PROFILE=[1|y|yes] - enable profiling of test framework via cProfile module + PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime) + PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout) + TEST_DEBUG=[1|y|yes] - enable debugging of the test framework itself (expert) + API_FUZZ=[1|y|yes] - enable VPP api fuzz testing + RND_SEED=<seed> - Seed RND with given seed + + Starting VPP in GDB for use with DEBUG=attach: + + test-start-vpp-in-gdb - start VPP in gdb (release) + test-start-vpp-debug-in-gdb - start VPP in gdb (debug) + + Creating test code coverage report: + + test-cov - generate code coverage report for test framework + test-wipe-cov - wipe code coverage report for test framework + + Verifying code-style: + + test-checkstyle - check PEP8 compliance |