summaryrefslogtreecommitdiffstats
path: root/docs/developer/build-run-debug/testing_vpp.rst
blob: ca9a09efb71439d5ea7c80d5d85315950173e5b9 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
Testing VPP
===========

As of this writing, the vpp source tree includes over 1,000 unit test
vectors. Best practices prior to pushing patches for code review: make
sure that all of the “make test” test vectors pass.

We attempt to maintain the top-level “make test-help” command so that it
accurately describes all of the “make test” options.

Examples
--------

Basic test run, all test vectors, single-vpp instance, optimized image:

::

     $ make test

10-way parallel basic test run:

::

     $ make TEST_JOBS=10 test

Run a specific test suite (mpls, in this case):

::

     $ make TEST=test_mpls test

Run a specific test suite, debug image, pause prior to running the test
suite; attach to the vpp image in gdb:

::

     $ make TEST=xxx DEBUG=gdb test-debug

Detailed Documentation
----------------------

Current “make test-help” output:

::

     $ make test-help
      test                   - build and run (basic) functional tests
      test-debug             - build and run (basic) functional tests (debug build)
      test-all               - build and run functional and extended tests
      test-all-debug         - build and run functional and extended tests (debug build)
      retest                 - run functional tests
      retest-debug           - run functional tests (debug build)
      retest-all             - run functional and extended tests
      retest-all-debug       - run functional and extended tests (debug build)
      test-cov               - generate code coverage report for test framework
      test-gcov                      - build and run functional tests (gcov build)
      test-wipe              - wipe (temporary) files generated by unit tests
      test-wipe-cov          - wipe code coverage report for test framework
      test-wipe-doc          - wipe documentation for test framework
      test-wipe-papi         - rebuild vpp_papi sources
      test-wipe-all          - wipe (temporary) files generated by unit tests, docs, and coverage
      test-shell             - enter shell with test environment
      test-shell-debug       - enter shell with test environment (debug build)
      test-checkstyle        - check PEP8 compliance for test framework
      test-refresh-deps      - refresh the Python dependencies for the tests

     Arguments controlling test runs:
      V=[0|1|2]              - set test verbosity level
                               0=ERROR, 1=INFO, 2=DEBUG
      TEST_JOBS=[<n>|auto]   - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
      MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
      CACHE_OUTPUT=[0|1]     - cache VPP stdout/stderr and log as one block after test finishes (default: 1)
      FAILFAST=[0|1]         - fail fast if 1, complete all tests if 0
      TIMEOUT=<timeout>      - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
      RETRIES=<n>            - retry failed tests <n> times
      DEBUG=<type>           - set VPP debugging kind
         DEBUG=core          - detect coredump and load it in gdb on crash
         DEBUG=gdb           - allow easy debugging by printing VPP PID
                               and waiting for user input before running
                               and tearing down a testcase
         DEBUG=gdbserver     - run gdb inside a gdb server, otherwise
                               same as above
         DEBUG=attach        - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)

      STEP=[yes|no]          - ease debugging by stepping through a testcase
      SANITY=[yes|no]        - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes)
      EXTENDED_TESTS=[1|y]   - used by '[re]test-all' & '[re]test-all-debug' to run extended tests
      TEST=<filter>          - filter the set of tests:
         by file-name        - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
         by file-suffix      - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
         by wildcard         - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
                               e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
                                    TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
                                    TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
                                    TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
                                    TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes

      VARIANT=<variant>      - specify which march node variant to unit test
                               e.g. VARIANT=skx test the skx march variants
                               e.g. VARIANT=icl test the icl march variants

      COREDUMP_SIZE=<size>   - pass <size> as unix { coredump-size <size> } argument to vpp
                               e.g. COREDUMP_SIZE=4g
                                    COREDUMP_SIZE=unlimited
      COREDUMP_COMPRESS=1    - compress core files if not debugging them
      EXTERN_TESTS=<path>    - path to out-of-tree test_<name>.py files containing test cases
      EXTERN_PLUGINS=<path>  - path to out-of-tree plugins to be loaded by vpp under test
      EXTERN_COV_DIR=<path>  - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report

      PROFILE=1              - enable profiling of test framework via cProfile module
      PROFILE_SORT_BY=opt    - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
      PROFILE_OUTPUT=file    - output profiling info to file - use absolute path (default: stdout)

      TEST_DEBUG=1           - turn on debugging of the test framework itself (expert)

      SKIP_AARCH64=1         - skip tests that are failing on the ARM platorm in FD.io CI

      RND_SEED=seed          - Seed RND with given seed

     Starting VPP in GDB for use with DEBUG=attach:

      test-start-vpp-in-gdb       - start VPP in gdb (release)
      test-start-vpp-debug-in-gdb - start VPP in gdb (debug)

     Arguments controlling VPP in GDB runs:

      VPP_IN_GDB_TMP_DIR     - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb)
      VPP_IN_GDB_NO_RMDIR=0  - don't remove existing tmp dir but fail instead
      VPP_IN_GDB_CMDLINE=1   - add 'interactive' to VPP arguments to run with command line

     Creating test documentation
      test-doc               - generate documentation for test framework
      test-wipe-doc          - wipe documentation for test framework

     Creating test code coverage report
      test-cov               - generate code coverage report for test framework
      test-wipe-cov          - wipe code coverage report for test framework

     Verifying code-style
      test-checkstyle        - check PEP8 compliance