1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
Testing VPP
===========
As of this writing, the vpp source tree includes over 1,000 unit test
vectors. Best practices prior to pushing patches for code review: make
sure that all of the “make test” test vectors pass.
We attempt to maintain the top-level “make test-help” command so that it
accurately describes all of the “make test” options.
Examples
--------
Basic test run, all test vectors, single-vpp instance, optimized image:
::
$ make test
10-way parallel basic test run:
::
$ make TEST_JOBS=10 test
Run a specific test suite (mpls, in this case):
::
$ make TEST=test_mpls test
Run a specific test suite, debug image, pause prior to running the test
suite; attach to the vpp image in gdb:
::
$ make TEST=xxx DEBUG=gdb test-debug
Detailed Documentation
----------------------
Current “make test-help” output:
::
$ make test-help
test - build and run (basic) functional tests
test-debug - build and run (basic) functional tests (debug build)
test-all - build and run functional and extended tests
test-all-debug - build and run functional and extended tests (debug build)
retest - run functional tests
retest-debug - run functional tests (debug build)
retest-all - run functional and extended tests
retest-all-debug - run functional and extended tests (debug build)
test-cov - generate code coverage report for test framework
test-gcov - build and run functional tests (gcov build)
test-wipe - wipe (temporary) files generated by unit tests
test-wipe-cov - wipe code coverage report for test framework
test-wipe-doc - wipe documentation for test framework
test-wipe-papi - rebuild vpp_papi sources
test-wipe-all - wipe (temporary) files generated by unit tests, docs, and coverage
test-shell - enter shell with test environment
test-shell-debug - enter shell with test environment (debug build)
test-checkstyle - check PEP8 compliance for test framework
test-refresh-deps - refresh the Python dependencies for the tests
Arguments controlling test runs:
V=[0|1|2] - set test verbosity level
0=ERROR, 1=INFO, 2=DEBUG
TEST_JOBS=[<n>|auto] - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
CACHE_OUTPUT=[0|1] - cache VPP stdout/stderr and log as one block after test finishes (default: 1)
FAILFAST=[0|1] - fail fast if 1, complete all tests if 0
TIMEOUT=<timeout> - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
RETRIES=<n> - retry failed tests <n> times
DEBUG=<type> - set VPP debugging kind
DEBUG=core - detect coredump and load it in gdb on crash
DEBUG=gdb - allow easy debugging by printing VPP PID
and waiting for user input before running
and tearing down a testcase
DEBUG=gdbserver - run gdb inside a gdb server, otherwise
same as above
DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
STEP=[yes|no] - ease debugging by stepping through a testcase
SANITY=[yes|no] - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes)
EXTENDED_TESTS=[1|y] - used by '[re]test-all' & '[re]test-all-debug' to run extended tests
TEST=<filter> - filter the set of tests:
by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
by wildcard - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
VARIANT=<variant> - specify which march node variant to unit test
e.g. VARIANT=skx test the skx march variants
e.g. VARIANT=icl test the icl march variants
COREDUMP_SIZE=<size> - pass <size> as unix { coredump-size <size> } argument to vpp
e.g. COREDUMP_SIZE=4g
COREDUMP_SIZE=unlimited
COREDUMP_COMPRESS=1 - compress core files if not debugging them
EXTERN_TESTS=<path> - path to out-of-tree test_<name>.py files containing test cases
EXTERN_PLUGINS=<path> - path to out-of-tree plugins to be loaded by vpp under test
EXTERN_COV_DIR=<path> - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
PROFILE=1 - enable profiling of test framework via cProfile module
PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout)
TEST_DEBUG=1 - turn on debugging of the test framework itself (expert)
SKIP_AARCH64=1 - skip tests that are failing on the ARM platorm in FD.io CI
RND_SEED=seed - Seed RND with given seed
Starting VPP in GDB for use with DEBUG=attach:
test-start-vpp-in-gdb - start VPP in gdb (release)
test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
Arguments controlling VPP in GDB runs:
VPP_IN_GDB_TMP_DIR - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb)
VPP_IN_GDB_NO_RMDIR=0 - don't remove existing tmp dir but fail instead
VPP_IN_GDB_CMDLINE=1 - add 'interactive' to VPP arguments to run with command line
Creating test documentation
test-doc - generate documentation for test framework
test-wipe-doc - wipe documentation for test framework
Creating test code coverage report
test-cov - generate code coverage report for test framework
test-wipe-cov - wipe code coverage report for test framework
Verifying code-style
test-checkstyle - check PEP8 compliance
|
