1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
Testing VPP
===========
As of this writing, the vpp source tree includes over 1,000 unit test
vectors. Best practices prior to pushing patches for code review: make
sure that all of the “make test” test vectors pass.
We attempt to maintain the top-level “make test-help” command so that it
accurately describes all of the “make test” options.
Examples
--------
Basic test run, all test vectors, single-vpp instance, optimized image:
::
$ make test
10-way parallel basic test run:
::
$ make TEST_JOBS=10 test
Run a specific test suite (mpls, in this case):
::
$ make TEST=test_mpls test
Run a specific test suite, debug image, pause prior to running the test
suite; attach to the vpp image in gdb:
::
$ make TEST=xxx DEBUG=gdb test-debug
Detailed Documentation
----------------------
Current “make test-help” output:
::
$ make test-help
Running tests:
test - build and run (basic) functional tests
test-debug - build and run (basic) functional tests (debug build)
test-all - build and run functional and extended tests
test-all-debug - build and run functional and extended tests (debug build)
retest - run functional tests
retest-debug - run functional tests (debug build)
retest-all - run functional and extended tests
retest-all-debug - run functional and extended tests (debug build)
test-cov - generate code coverage report for test framework
test-gcov - build and run functional tests (gcov build)
test-wipe - wipe (temporary) files generated by unit tests
test-wipe-cov - wipe code coverage report for test framework
test-wipe-papi - rebuild vpp_papi sources
test-wipe-all - wipe (temporary) files generated by unit tests, and coverage
test-shell - enter shell with test environment
test-shell-debug - enter shell with test environment (debug build)
test-checkstyle - check PEP8 compliance for test framework
test-refresh-deps - refresh the Python dependencies for the tests
Arguments controlling test runs:
V=[0|1|2] - set test verbosity level
0=ERROR, 1=INFO, 2=DEBUG
TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
CACHE_OUTPUT=[0|n|no] - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes)
FAILFAST=[1|y|yes] - fail fast if 1, otherwise complete all tests
TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
RETRIES=<n> - retry failed tests <n> times
DEBUG=<type> - set VPP debugging kind
DEBUG=core - detect coredump and load it in gdb on crash
DEBUG=gdb - allow easy debugging by printing VPP PID
and waiting for user input before running
and tearing down a testcase
DEBUG=gdbserver - run gdb inside a gdb server, otherwise
same as above
DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
STEP=[1|y|yes] - enable stepping through a testcase (for testcase debugging)
SANITY=[0|n|no] - disable sanity import of vpp-api/sanity vpp run before running tests
EXTENDED_TESTS=[1|y|yes] - run extended tests
TEST=<filter> - filter the set of tests:
by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
VARIANT=<variant> - specify which march node variant to unit test
e.g. VARIANT=skx test the skx march variants
e.g. VARIANT=icl test the icl march variants
COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp
e.g. COREDUMP_SIZE=4g
COREDUMP_SIZE=unlimited
COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them
EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases
EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test
EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
PROFILE=[1|y|yes] - enable profiling of test framework via cProfile module
PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout)
TEST_DEBUG=[1|y|yes] - enable debugging of the test framework itself (expert)
API_FUZZ=[1|y|yes] - enable VPP api fuzz testing
RND_SEED=<seed> - Seed RND with given seed
Starting VPP in GDB for use with DEBUG=attach:
test-start-vpp-in-gdb - start VPP in gdb (release)
test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
Creating test code coverage report:
test-cov - generate code coverage report for test framework
test-wipe-cov - wipe code coverage report for test framework
Verifying code-style:
test-checkstyle - check PEP8 compliance
|
