docs: document "make test-help"

Type: docs

Signed-off-by: Dave Barach <dave@barachs.net>
Change-Id: I67a582b7bf16c3fe307941973e983a0fe28cb221

3 files changed, 208 insertions, 66 deletions

diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst
index 04d704319b2..9b9a67fcf21 100644
--- a/docs/gettingstarted/developers/index.rst
+++ b/docs/gettingstarted/developers/index.rst
@@ -20,6 +20,7 @@ The Developers section covers the following areas:
 
    building
    running_vpp
+   testing_vpp
    gdb_examples
    add_plugin
    add_plugin_goapi
diff --git a/docs/gettingstarted/developers/testing_vpp.md b/docs/gettingstarted/developers/testing_vpp.md
new file mode 100644
index 00000000000..4e4ca1aa30a
--- /dev/null
+++ b/docs/gettingstarted/developers/testing_vpp.md
@@ -0,0 +1,141 @@
+Testing VPP
+===========
+
+As of this writing, the vpp source tree includes over 1,000 unit test
+vectors. Best practices prior to pushing patches for code review: make
+sure that all of the "make test" test vectors pass.
+
+We attempt to maintain the top-level "make test-help" command so that it
+accurately describes all of the "make test" options.
+
+Examples
+--------
+
+Basic test run, all test vectors, single-vpp instance, optimized image:
+
+```
+  $ make test
+```
+
+10-way parallel basic test run:
+
+```
+  $ make TEST_JOBS=10 test
+```
+
+Run a specific test suite (mpls, in this case):
+
+```
+  $ make TEST=test_mpls test
+```
+
+Run a specifc test suite, debug image, pause prior to running the test
+suite; attach to the vpp image in gdb:
+
+```
+  $ make TEST=xxx DEBUG=gdb test-debug
+```
+
+Detailed Documentation
+----------------------
+
+Current "make test-help" output:
+
+
+```
+  $ make test-help
+   test                   - build and run (basic) functional tests
+   test-debug             - build and run (basic) functional tests (debug build)
+   test-all               - build and run functional and extended tests
+   test-all-debug         - build and run functional and extended tests (debug build)
+   retest                 - run functional tests
+   retest-debug           - run functional tests (debug build)
+   retest-all             - run functional and extended tests
+   retest-all-debug       - run functional and extended tests (debug build)
+   test-cov               - generate code coverage report for test framework
+   test-gcov                      - build and run functional tests (gcov build)
+   test-wipe              - wipe (temporary) files generated by unit tests
+   test-wipe-cov          - wipe code coverage report for test framework
+   test-wipe-doc          - wipe documentation for test framework
+   test-wipe-papi         - rebuild vpp_papi sources
+   test-wipe-all          - wipe (temporary) files generated by unit tests, docs, and coverage
+   test-shell             - enter shell with test environment
+   test-shell-debug       - enter shell with test environment (debug build)
+   test-checkstyle        - check PEP8 compliance for test framework
+   test-refresh-deps      - refresh the Python dependencies for the tests
+
+  Arguments controlling test runs:
+   V=[0|1|2]              - set test verbosity level
+                            0=ERROR, 1=INFO, 2=DEBUG
+   TEST_JOBS=[<n>|auto]   - use at most <n> parallel python processes for test execution, if auto, set to number of available cpus (default: 1)
+   MAX_VPP_CPUS=[<n>|auto]- use at most <n> cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto)
+   CACHE_OUTPUT=[0|1]     - cache VPP stdout/stderr and log as one block after test finishes (default: 1)
+   FAILFAST=[0|1]         - fail fast if 1, complete all tests if 0
+   TIMEOUT=<timeout>      - fail test suite if any single test takes longer than <timeout> (in seconds) to finish (default: 600)
+   RETRIES=<n>            - retry failed tests <n> times
+   DEBUG=<type>           - set VPP debugging kind
+      DEBUG=core          - detect coredump and load it in gdb on crash
+      DEBUG=gdb           - allow easy debugging by printing VPP PID
+                            and waiting for user input before running
+                            and tearing down a testcase
+      DEBUG=gdbserver     - run gdb inside a gdb server, otherwise
+                            same as above
+      DEBUG=attach        - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb)
+
+   STEP=[yes|no]          - ease debugging by stepping through a testcase
+   SANITY=[yes|no]        - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes)
+   EXTENDED_TESTS=[1|y]   - used by '[re]test-all' & '[re]test-all-debug' to run extended tests
+   TEST=<filter>          - filter the set of tests:
+      by file-name        - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py
+      by file-suffix      - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py
+      by wildcard         - wildcard filter is <file>.<class>.<test function>, each can be replaced by '*'
+                            e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name
+                                 TEST='bfd.*.*' is equivalent to above example of filter by file-suffix
+                                 TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class
+                                 TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase
+                                 TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes
+
+   VARIANT=<variant>      - specify which march node variant to unit test
+                            e.g. VARIANT=skx test the skx march variants
+                            e.g. VARIANT=icl test the icl march variants
+
+   COREDUMP_SIZE=<size>   - pass <size> as unix { coredump-size <size> } argument to vpp
+                            e.g. COREDUMP_SIZE=4g
+                                 COREDUMP_SIZE=unlimited
+   COREDUMP_COMPRESS=1    - compress core files if not debugging them
+   EXTERN_TESTS=<path>    - path to out-of-tree test_<name>.py files containing test cases
+   EXTERN_PLUGINS=<path>  - path to out-of-tree plugins to be loaded by vpp under test
+   EXTERN_COV_DIR=<path>  - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report
+
+   PROFILE=1              - enable profiling of test framework via cProfile module
+   PROFILE_SORT_BY=opt    - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime)
+   PROFILE_OUTPUT=file    - output profiling info to file - use absolute path (default: stdout)
+
+   TEST_DEBUG=1           - turn on debugging of the test framework itself (expert)
+
+   SKIP_AARCH64=1         - skip tests that are failing on the ARM platorm in FD.io CI
+
+   RND_SEED=seed          - Seed RND with given seed
+
+  Starting VPP in GDB for use with DEBUG=attach:
+
+   test-start-vpp-in-gdb       - start VPP in gdb (release)
+   test-start-vpp-debug-in-gdb - start VPP in gdb (debug)
+
+  Arguments controlling VPP in GDB runs:
+
+   VPP_IN_GDB_TMP_DIR     - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb)
+   VPP_IN_GDB_NO_RMDIR=0  - don't remove existing tmp dir but fail instead
+   VPP_IN_GDB_CMDLINE=1   - add 'interactive' to VPP arguments to run with command line
+
+  Creating test documentation
+   test-doc               - generate documentation for test framework
+   test-wipe-doc          - wipe documentation for test framework
+
+  Creating test code coverage report
+   test-cov               - generate code coverage report for test framework
+   test-wipe-cov          - wipe code coverage report for test framework
+
+  Verifying code-style
+   test-checkstyle        - check PEP8 compliance
+```
diff --git a/docs/usecases/container_test.md b/docs/usecases/container_test.md
index dc45aabe57b..ad0bc2ea098 100644
--- a/docs/usecases/container_test.md
+++ b/docs/usecases/container_test.md
@@ -64,9 +64,9 @@ Create three network segments
 Aka, linux bridges.
 
 ```
-    # lxc network create dora
+    # lxc network create respond
     # lxc network create internet
-    # lxc network create swan
+    # lxc network create initiate
 ```
 
 We'll explain the test topology in a bit. Stay tuned.
@@ -95,12 +95,12 @@ containers, run vpp without installing it, etc.
       eth2:
         name: eth2
         nictype: bridged
-        parent: dora
+        parent: respond
         type: nic
       eth3:
         name: eth3
         nictype: bridged
-        parent: swan
+        parent: initiate
         type: nic
       root:
         path: /
@@ -142,10 +142,10 @@ and host configuration:
     - none
 ```
 
-Repeat the process with the "dora" and "swan" networks, using these
+Repeat the process with the "respond" and "initiate" networks, using these
 configurations:
 
-### dora network configuration
+### respond network configuration
 
 ```
     config:
@@ -155,7 +155,7 @@ configurations:
       ipv6.address: none
       ipv6.nat: "false"
     description: ""
-    name: dora
+    name: respond
     type: bridge
     used_by:
     managed: true
@@ -163,7 +163,7 @@ configurations:
     locations:
     - none
 ```
-### swan network configuration
+### initiate network configuration
 
 ```
     config:
@@ -173,7 +173,7 @@ configurations:
       ipv6.address: none
       ipv6.nat: "false"
     description: ""
-    name: swan
+    name: initiate
     type: bridge
     used_by:
     managed: true
@@ -191,14 +191,14 @@ build vpp, ssh into the container, edit source code, run gdb, etc.
 Make sure that e.g. public key auth ssh works.
 
 ```
-    # lxd launch ubuntu:18.04 dora
+    # lxd launch ubuntu:18.04 respond
     <spew>
-    # lxc exec dora bash
-    dora# cd /scratch/my-vpp-workspace
-    dora# apt-get install make ssh
-    dora# make install-dep
-    dora# exit
-    # lxc stop dora
+    # lxc exec respond bash
+    respond# cd /scratch/my-vpp-workspace
+    respond# apt-get install make ssh
+    respond# make install-dep
+    respond# exit
+    # lxc stop respond
 ```
 
 Mark the container image privileged. If you forget this step, you'll
@@ -206,7 +206,7 @@ trip over a netlink error (-11) aka EAGAIN when you try to roll in the
 vpp configurations.
 
 ```
-    # lxc config set dora security.privileged "true"
+    # lxc config set respond security.privileged "true"
 ```
 
 Duplicate the "master" container image
@@ -216,10 +216,10 @@ To avoid having to configure N containers, be sure that the master
 container image is fully set up before you help it have children:
 
 ```
-    # lxc copy dora dorahost
-    # lxc copy dora swan
-    # lxc copy dora swanhost
-    # lxc copy dora dhcpserver    # optional, to test ipv6 prefix delegation
+    # lxc copy respond respondhost
+    # lxc copy respond initiate
+    # lxc copy respond initiatehost
+    # lxc copy respond dhcpserver    # optional, to test ipv6 prefix delegation
 ```
 
 Install handy script
@@ -250,7 +250,7 @@ Here's the script:
     #!/bin/bash
 
     set -u
-    export containers="dora dorahost swan swanhost dhcpserver"
+    export containers="respond respondhost initiate initiatehost dhcpserver"
 
     if [ x$1 = "x" ] ; then
         echo missing command
@@ -292,16 +292,16 @@ Finally, we're ready to describe a test topology. First, a picture:
        v                             |                          v
       eth0                           |                         eth0
     +------+ eth1                                       eth1 +------+
-    | dora | 10.26.88.100 <= internet bridge => 10.26.88.101 | swan |
+    | respond | 10.26.88.100 <= internet bridge => 10.26.88.101 | initiate |
     +------+                                                 +------+
       eth2 / bvi0 10.166.14.2        |       10.219.188.2 eth3 / bvi0
        |                             |                          |
-       | ("dora" bridge)             |          ("swan" bridge) |
+       | ("respond" bridge)             |          ("initiate" bridge) |
        |                             |                          |
        v                             |                          v
       eth2 10.166.14.3               |           eth3 10.219.188.3
     +----------+                     |                   +----------+
-    | dorahost |                     |                   | dorahost |
+    | respondhost |                     |                   | respondhost |
     +----------+                     |                   +----------+
       eth0 (management lan) <========+========> eth0 (management lan)
 ```
@@ -309,14 +309,14 @@ Finally, we're ready to describe a test topology. First, a picture:
 ### Test topology discussion
 
 This topology is suitable for testing almost any tunnel encap/decap
-scenario.  The two containers "dorahost" and "swanhost" are end-stations
-connected to two vpp instances running on "dora" and "swan".
+scenario.  The two containers "respondhost" and "initiatehost" are end-stations
+connected to two vpp instances running on "respond" and "initiate".
 
 We leverage the Linux end-station network stacks to generate traffic
 of all sorts.
 
-The so-called "internet" bridge models the public internet. The "dora" and
-"swan" bridges connect vpp instances to local hosts
+The so-called "internet" bridge models the public internet. The "respond" and
+"initiate" bridges connect vpp instances to local hosts
 
 End station configs
 -------------------
@@ -325,14 +325,14 @@ The end-station Linux configurations set up the eth2 and eth3 ip
 addresses shown above, and add tunnel routes to the opposite
 end-station networks.
 
-### dorahost configuration
+### respondhost configuration
 
 ```
     ifconfig eth2 10.166.14.3/24 up
     route add -net 10.219.188.0/24 gw 10.166.14.2
 ```
 
-### swanhost configuration
+### initiatehost configuration
 
 ```
     sudo ifconfig eth3 10.219.188.3/24 up
@@ -343,9 +343,9 @@ VPP configs
 -----------
 
 Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in
-the "dora" config.
+the "respond" config.
 
-### dora configuration
+### respond configuration
 
 ```
     set term pag off
@@ -356,10 +356,10 @@ the "dora" config.
     set int ip6 table host-eth1 0
     set int state host-eth1 up
 
-    comment { default route via swan }
+    comment { default route via initiate }
     ip route add 0.0.0.0/0 via 10.26.68.101
 
-    comment { "dora-private-net" }
+    comment { "respond-private-net" }
     create host-interface name eth2
     bvi create instance 0
     set int l2 bridge bvi0 1 bvi
@@ -376,16 +376,16 @@ the "dora" config.
     comment { nat44 untranslated subnet 10.219.188.0/24 }
 
     comment { responder profile }
-    ikev2 profile add swan
-    ikev2 profile set swan udp-encap
-    ikev2 profile set swan auth rsa-sig cert-file /scratch/setups/doracert.pem
-    set ikev2 local key /scratch/setups/swankey.pem
-    ikev2 profile set swan id local fqdn swan.barachs.net
-    ikev2 profile set swan id remote fqdn broiler2.barachs.net
-    ikev2 profile set swan traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
-    ikev2 profile set swan traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
+    ikev2 profile add initiate
+    ikev2 profile set initiate udp-encap
+    ikev2 profile set initiate auth rsa-sig cert-file /scratch/setups/respondcert.pem
+    set ikev2 local key /scratch/setups/initiatekey.pem
+    ikev2 profile set initiate id local fqdn initiator.my.net
+    ikev2 profile set initiate id remote fqdn responder.my.net
+    ikev2 profile set initiate traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
+    ikev2 profile set initiate traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
     create ipip tunnel src 10.26.68.100 dst 10.26.68.101
-    ikev2 profile set swan tunnel ipip0
+    ikev2 profile set initiate tunnel ipip0
 
     comment { ipv6 prefix delegation }
     ip6 nd address autoconfig host-eth1 default-route
@@ -400,21 +400,21 @@ the "dora" config.
     ip route add 10.219.188.0/24 via ipip0
 ```
 
-### swan configuration
+### initiate configuration
 
 ```
     set term pag off
 
     comment { "internet" }
     create host-interface name eth1
-    comment { set dhcp client intfc host-eth1 hostname swan }
+    comment { set dhcp client intfc host-eth1 hostname initiate }
     set int ip address host-eth1 10.26.68.101/24
     set int state host-eth1 up
 
     comment { default route via "internet gateway" }
     comment { ip route add 0.0.0.0/0 via 10.26.68.1 }
 
-    comment { "swan-private-net" }
+    comment { "initiate-private-net" }
     create host-interface name eth3
     bvi create instance 0
     set int l2 bridge bvi0 1 bvi
@@ -430,24 +430,24 @@ the "dora" config.
     comment { nat44 untranslated subnet 10.166.14.0/24 }
 
     comment { initiator profile }
-    ikev2 profile add dora
-    ikev2 profile set dora udp-encap
-    ikev2 profile set dora auth rsa-sig cert-file /scratch/setups/swancert.pem
-    set ikev2 local key /scratch/setups/dorakey.pem
-    ikev2 profile set dora id local fqdn broiler2.barachs.net
-    ikev2 profile set dora id remote fqdn swan.barachs.net
+    ikev2 profile add respond
+    ikev2 profile set respond udp-encap
+    ikev2 profile set respond auth rsa-sig cert-file /scratch/setups/initiatecert.pem
+    set ikev2 local key /scratch/setups/respondkey.pem
+    ikev2 profile set respond id local fqdn responder.my.net
+    ikev2 profile set respond id remote fqdn initiator.my.net
 
-    ikev2 profile set dora traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
-    ikev2 profile set dora traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
+    ikev2 profile set respond traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
+    ikev2 profile set respond traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
 
-    ikev2 profile set dora responder host-eth1 10.26.68.100
-    ikev2 profile set dora ike-crypto-alg aes-cbc 256  ike-integ-alg sha1-96  ike-dh modp-2048
-    ikev2 profile set dora esp-crypto-alg aes-cbc 256  esp-integ-alg sha1-96  esp-dh ecp-256
-    ikev2 profile set dora sa-lifetime 3600 10 5 0
+    ikev2 profile set respond responder host-eth1 10.26.68.100
+    ikev2 profile set respond ike-crypto-alg aes-cbc 256  ike-integ-alg sha1-96  ike-dh modp-2048
+    ikev2 profile set respond esp-crypto-alg aes-cbc 256  esp-integ-alg sha1-96  esp-dh ecp-256
+    ikev2 profile set respond sa-lifetime 3600 10 5 0
 
     create ipip tunnel src 10.26.68.101 dst 10.26.68.100
-    ikev2 profile set dora tunnel ipip0
-    ikev2 initiate sa-init dora
+    ikev2 profile set respond tunnel ipip0
+    ikev2 initiate sa-init respond
 
     set int mtu packet 1390 ipip0
     set int unnum ipip0 use host-eth1
@@ -463,13 +463,13 @@ mentioned. These certificates are used in the ikev2 key exchange.
 Here's how to generate the certificates:
 
 ```
-    openssl req -x509 -nodes -newkey rsa:4096 -keyout dorakey.pem -out doracert.pem -days 3560
-    openssl x509 -text -noout -in doracert.pem
-    openssl req -x509 -nodes -newkey rsa:4096 -keyout swankey.pem -out swancert.pem -days 3560
-    openssl x509 -text -noout -in swancert.pem
+    openssl req -x509 -nodes -newkey rsa:4096 -keyout respondkey.pem -out respondcert.pem -days 3560
+    openssl x509 -text -noout -in respondcert.pem
+    openssl req -x509 -nodes -newkey rsa:4096 -keyout initiatekey.pem -out initiatecert.pem -days 3560
+    openssl x509 -text -noout -in initiatecert.pem
 ```
 
-Make sure that the "dora" and "swan" configurations point to the certificates.
+Make sure that the "respond" and "initiate" configurations point to the certificates.
 
 DHCPv6 server setup
 -------------------