From 9292c32b4579a1da2e851fe5ca7de3415b45fc07 Mon Sep 17 00:00:00 2001 From: Dave Barach Date: Tue, 20 Apr 2021 07:52:38 -0400 Subject: docs: document "make test-help" Type: docs Signed-off-by: Dave Barach Change-Id: I67a582b7bf16c3fe307941973e983a0fe28cb221 --- docs/gettingstarted/developers/index.rst | 1 + docs/gettingstarted/developers/testing_vpp.md | 141 ++++++++++++++++++++++++++ docs/usecases/container_test.md | 132 ++++++++++++------------ 3 files changed, 208 insertions(+), 66 deletions(-) create mode 100644 docs/gettingstarted/developers/testing_vpp.md (limited to 'docs') diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst index 04d704319b2..9b9a67fcf21 100644 --- a/docs/gettingstarted/developers/index.rst +++ b/docs/gettingstarted/developers/index.rst @@ -20,6 +20,7 @@ The Developers section covers the following areas: building running_vpp + testing_vpp gdb_examples add_plugin add_plugin_goapi diff --git a/docs/gettingstarted/developers/testing_vpp.md b/docs/gettingstarted/developers/testing_vpp.md new file mode 100644 index 00000000000..4e4ca1aa30a --- /dev/null +++ b/docs/gettingstarted/developers/testing_vpp.md @@ -0,0 +1,141 @@ +Testing VPP +=========== + +As of this writing, the vpp source tree includes over 1,000 unit test +vectors. Best practices prior to pushing patches for code review: make +sure that all of the "make test" test vectors pass. + +We attempt to maintain the top-level "make test-help" command so that it +accurately describes all of the "make test" options. + +Examples +-------- + +Basic test run, all test vectors, single-vpp instance, optimized image: + +``` + $ make test +``` + +10-way parallel basic test run: + +``` + $ make TEST_JOBS=10 test +``` + +Run a specific test suite (mpls, in this case): + +``` + $ make TEST=test_mpls test +``` + +Run a specifc test suite, debug image, pause prior to running the test +suite; attach to the vpp image in gdb: + +``` + $ make TEST=xxx DEBUG=gdb test-debug +``` + +Detailed Documentation +---------------------- + +Current "make test-help" output: + + +``` + $ make test-help + test - build and run (basic) functional tests + test-debug - build and run (basic) functional tests (debug build) + test-all - build and run functional and extended tests + test-all-debug - build and run functional and extended tests (debug build) + retest - run functional tests + retest-debug - run functional tests (debug build) + retest-all - run functional and extended tests + retest-all-debug - run functional and extended tests (debug build) + test-cov - generate code coverage report for test framework + test-gcov - build and run functional tests (gcov build) + test-wipe - wipe (temporary) files generated by unit tests + test-wipe-cov - wipe code coverage report for test framework + test-wipe-doc - wipe documentation for test framework + test-wipe-papi - rebuild vpp_papi sources + test-wipe-all - wipe (temporary) files generated by unit tests, docs, and coverage + test-shell - enter shell with test environment + test-shell-debug - enter shell with test environment (debug build) + test-checkstyle - check PEP8 compliance for test framework + test-refresh-deps - refresh the Python dependencies for the tests + + Arguments controlling test runs: + V=[0|1|2] - set test verbosity level + 0=ERROR, 1=INFO, 2=DEBUG + TEST_JOBS=[|auto] - use at most parallel python processes for test execution, if auto, set to number of available cpus (default: 1) + MAX_VPP_CPUS=[|auto]- use at most cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto) + CACHE_OUTPUT=[0|1] - cache VPP stdout/stderr and log as one block after test finishes (default: 1) + FAILFAST=[0|1] - fail fast if 1, complete all tests if 0 + TIMEOUT= - fail test suite if any single test takes longer than (in seconds) to finish (default: 600) + RETRIES= - retry failed tests times + DEBUG= - set VPP debugging kind + DEBUG=core - detect coredump and load it in gdb on crash + DEBUG=gdb - allow easy debugging by printing VPP PID + and waiting for user input before running + and tearing down a testcase + DEBUG=gdbserver - run gdb inside a gdb server, otherwise + same as above + DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb) + + STEP=[yes|no] - ease debugging by stepping through a testcase + SANITY=[yes|no] - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes) + EXTENDED_TESTS=[1|y] - used by '[re]test-all' & '[re]test-all-debug' to run extended tests + TEST= - filter the set of tests: + by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py + by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py + by wildcard - wildcard filter is .., each can be replaced by '*' + e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name + TEST='bfd.*.*' is equivalent to above example of filter by file-suffix + TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class + TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase + TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes + + VARIANT= - specify which march node variant to unit test + e.g. VARIANT=skx test the skx march variants + e.g. VARIANT=icl test the icl march variants + + COREDUMP_SIZE= - pass as unix { coredump-size } argument to vpp + e.g. COREDUMP_SIZE=4g + COREDUMP_SIZE=unlimited + COREDUMP_COMPRESS=1 - compress core files if not debugging them + EXTERN_TESTS= - path to out-of-tree test_.py files containing test cases + EXTERN_PLUGINS= - path to out-of-tree plugins to be loaded by vpp under test + EXTERN_COV_DIR= - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report + + PROFILE=1 - enable profiling of test framework via cProfile module + PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime) + PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout) + + TEST_DEBUG=1 - turn on debugging of the test framework itself (expert) + + SKIP_AARCH64=1 - skip tests that are failing on the ARM platorm in FD.io CI + + RND_SEED=seed - Seed RND with given seed + + Starting VPP in GDB for use with DEBUG=attach: + + test-start-vpp-in-gdb - start VPP in gdb (release) + test-start-vpp-debug-in-gdb - start VPP in gdb (debug) + + Arguments controlling VPP in GDB runs: + + VPP_IN_GDB_TMP_DIR - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb) + VPP_IN_GDB_NO_RMDIR=0 - don't remove existing tmp dir but fail instead + VPP_IN_GDB_CMDLINE=1 - add 'interactive' to VPP arguments to run with command line + + Creating test documentation + test-doc - generate documentation for test framework + test-wipe-doc - wipe documentation for test framework + + Creating test code coverage report + test-cov - generate code coverage report for test framework + test-wipe-cov - wipe code coverage report for test framework + + Verifying code-style + test-checkstyle - check PEP8 compliance +``` diff --git a/docs/usecases/container_test.md b/docs/usecases/container_test.md index dc45aabe57b..ad0bc2ea098 100644 --- a/docs/usecases/container_test.md +++ b/docs/usecases/container_test.md @@ -64,9 +64,9 @@ Create three network segments Aka, linux bridges. ``` - # lxc network create dora + # lxc network create respond # lxc network create internet - # lxc network create swan + # lxc network create initiate ``` We'll explain the test topology in a bit. Stay tuned. @@ -95,12 +95,12 @@ containers, run vpp without installing it, etc. eth2: name: eth2 nictype: bridged - parent: dora + parent: respond type: nic eth3: name: eth3 nictype: bridged - parent: swan + parent: initiate type: nic root: path: / @@ -142,10 +142,10 @@ and host configuration: - none ``` -Repeat the process with the "dora" and "swan" networks, using these +Repeat the process with the "respond" and "initiate" networks, using these configurations: -### dora network configuration +### respond network configuration ``` config: @@ -155,7 +155,7 @@ configurations: ipv6.address: none ipv6.nat: "false" description: "" - name: dora + name: respond type: bridge used_by: managed: true @@ -163,7 +163,7 @@ configurations: locations: - none ``` -### swan network configuration +### initiate network configuration ``` config: @@ -173,7 +173,7 @@ configurations: ipv6.address: none ipv6.nat: "false" description: "" - name: swan + name: initiate type: bridge used_by: managed: true @@ -191,14 +191,14 @@ build vpp, ssh into the container, edit source code, run gdb, etc. Make sure that e.g. public key auth ssh works. ``` - # lxd launch ubuntu:18.04 dora + # lxd launch ubuntu:18.04 respond - # lxc exec dora bash - dora# cd /scratch/my-vpp-workspace - dora# apt-get install make ssh - dora# make install-dep - dora# exit - # lxc stop dora + # lxc exec respond bash + respond# cd /scratch/my-vpp-workspace + respond# apt-get install make ssh + respond# make install-dep + respond# exit + # lxc stop respond ``` Mark the container image privileged. If you forget this step, you'll @@ -206,7 +206,7 @@ trip over a netlink error (-11) aka EAGAIN when you try to roll in the vpp configurations. ``` - # lxc config set dora security.privileged "true" + # lxc config set respond security.privileged "true" ``` Duplicate the "master" container image @@ -216,10 +216,10 @@ To avoid having to configure N containers, be sure that the master container image is fully set up before you help it have children: ``` - # lxc copy dora dorahost - # lxc copy dora swan - # lxc copy dora swanhost - # lxc copy dora dhcpserver # optional, to test ipv6 prefix delegation + # lxc copy respond respondhost + # lxc copy respond initiate + # lxc copy respond initiatehost + # lxc copy respond dhcpserver # optional, to test ipv6 prefix delegation ``` Install handy script @@ -250,7 +250,7 @@ Here's the script: #!/bin/bash set -u - export containers="dora dorahost swan swanhost dhcpserver" + export containers="respond respondhost initiate initiatehost dhcpserver" if [ x$1 = "x" ] ; then echo missing command @@ -292,16 +292,16 @@ Finally, we're ready to describe a test topology. First, a picture: v | v eth0 | eth0 +------+ eth1 eth1 +------+ - | dora | 10.26.88.100 <= internet bridge => 10.26.88.101 | swan | + | respond | 10.26.88.100 <= internet bridge => 10.26.88.101 | initiate | +------+ +------+ eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0 | | | - | ("dora" bridge) | ("swan" bridge) | + | ("respond" bridge) | ("initiate" bridge) | | | | v | v eth2 10.166.14.3 | eth3 10.219.188.3 +----------+ | +----------+ - | dorahost | | | dorahost | + | respondhost | | | respondhost | +----------+ | +----------+ eth0 (management lan) <========+========> eth0 (management lan) ``` @@ -309,14 +309,14 @@ Finally, we're ready to describe a test topology. First, a picture: ### Test topology discussion This topology is suitable for testing almost any tunnel encap/decap -scenario. The two containers "dorahost" and "swanhost" are end-stations -connected to two vpp instances running on "dora" and "swan". +scenario. The two containers "respondhost" and "initiatehost" are end-stations +connected to two vpp instances running on "respond" and "initiate". We leverage the Linux end-station network stacks to generate traffic of all sorts. -The so-called "internet" bridge models the public internet. The "dora" and -"swan" bridges connect vpp instances to local hosts +The so-called "internet" bridge models the public internet. The "respond" and +"initiate" bridges connect vpp instances to local hosts End station configs ------------------- @@ -325,14 +325,14 @@ The end-station Linux configurations set up the eth2 and eth3 ip addresses shown above, and add tunnel routes to the opposite end-station networks. -### dorahost configuration +### respondhost configuration ``` ifconfig eth2 10.166.14.3/24 up route add -net 10.219.188.0/24 gw 10.166.14.2 ``` -### swanhost configuration +### initiatehost configuration ``` sudo ifconfig eth3 10.219.188.3/24 up @@ -343,9 +343,9 @@ VPP configs ----------- Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in -the "dora" config. +the "respond" config. -### dora configuration +### respond configuration ``` set term pag off @@ -356,10 +356,10 @@ the "dora" config. set int ip6 table host-eth1 0 set int state host-eth1 up - comment { default route via swan } + comment { default route via initiate } ip route add 0.0.0.0/0 via 10.26.68.101 - comment { "dora-private-net" } + comment { "respond-private-net" } create host-interface name eth2 bvi create instance 0 set int l2 bridge bvi0 1 bvi @@ -376,16 +376,16 @@ the "dora" config. comment { nat44 untranslated subnet 10.219.188.0/24 } comment { responder profile } - ikev2 profile add swan - ikev2 profile set swan udp-encap - ikev2 profile set swan auth rsa-sig cert-file /scratch/setups/doracert.pem - set ikev2 local key /scratch/setups/swankey.pem - ikev2 profile set swan id local fqdn swan.barachs.net - ikev2 profile set swan id remote fqdn broiler2.barachs.net - ikev2 profile set swan traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 - ikev2 profile set swan traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 + ikev2 profile add initiate + ikev2 profile set initiate udp-encap + ikev2 profile set initiate auth rsa-sig cert-file /scratch/setups/respondcert.pem + set ikev2 local key /scratch/setups/initiatekey.pem + ikev2 profile set initiate id local fqdn initiator.my.net + ikev2 profile set initiate id remote fqdn responder.my.net + ikev2 profile set initiate traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 + ikev2 profile set initiate traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 create ipip tunnel src 10.26.68.100 dst 10.26.68.101 - ikev2 profile set swan tunnel ipip0 + ikev2 profile set initiate tunnel ipip0 comment { ipv6 prefix delegation } ip6 nd address autoconfig host-eth1 default-route @@ -400,21 +400,21 @@ the "dora" config. ip route add 10.219.188.0/24 via ipip0 ``` -### swan configuration +### initiate configuration ``` set term pag off comment { "internet" } create host-interface name eth1 - comment { set dhcp client intfc host-eth1 hostname swan } + comment { set dhcp client intfc host-eth1 hostname initiate } set int ip address host-eth1 10.26.68.101/24 set int state host-eth1 up comment { default route via "internet gateway" } comment { ip route add 0.0.0.0/0 via 10.26.68.1 } - comment { "swan-private-net" } + comment { "initiate-private-net" } create host-interface name eth3 bvi create instance 0 set int l2 bridge bvi0 1 bvi @@ -430,24 +430,24 @@ the "dora" config. comment { nat44 untranslated subnet 10.166.14.0/24 } comment { initiator profile } - ikev2 profile add dora - ikev2 profile set dora udp-encap - ikev2 profile set dora auth rsa-sig cert-file /scratch/setups/swancert.pem - set ikev2 local key /scratch/setups/dorakey.pem - ikev2 profile set dora id local fqdn broiler2.barachs.net - ikev2 profile set dora id remote fqdn swan.barachs.net + ikev2 profile add respond + ikev2 profile set respond udp-encap + ikev2 profile set respond auth rsa-sig cert-file /scratch/setups/initiatecert.pem + set ikev2 local key /scratch/setups/respondkey.pem + ikev2 profile set respond id local fqdn responder.my.net + ikev2 profile set respond id remote fqdn initiator.my.net - ikev2 profile set dora traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 - ikev2 profile set dora traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 + ikev2 profile set respond traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 + ikev2 profile set respond traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 - ikev2 profile set dora responder host-eth1 10.26.68.100 - ikev2 profile set dora ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048 - ikev2 profile set dora esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256 - ikev2 profile set dora sa-lifetime 3600 10 5 0 + ikev2 profile set respond responder host-eth1 10.26.68.100 + ikev2 profile set respond ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048 + ikev2 profile set respond esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256 + ikev2 profile set respond sa-lifetime 3600 10 5 0 create ipip tunnel src 10.26.68.101 dst 10.26.68.100 - ikev2 profile set dora tunnel ipip0 - ikev2 initiate sa-init dora + ikev2 profile set respond tunnel ipip0 + ikev2 initiate sa-init respond set int mtu packet 1390 ipip0 set int unnum ipip0 use host-eth1 @@ -463,13 +463,13 @@ mentioned. These certificates are used in the ikev2 key exchange. Here's how to generate the certificates: ``` - openssl req -x509 -nodes -newkey rsa:4096 -keyout dorakey.pem -out doracert.pem -days 3560 - openssl x509 -text -noout -in doracert.pem - openssl req -x509 -nodes -newkey rsa:4096 -keyout swankey.pem -out swancert.pem -days 3560 - openssl x509 -text -noout -in swancert.pem + openssl req -x509 -nodes -newkey rsa:4096 -keyout respondkey.pem -out respondcert.pem -days 3560 + openssl x509 -text -noout -in respondcert.pem + openssl req -x509 -nodes -newkey rsa:4096 -keyout initiatekey.pem -out initiatecert.pem -days 3560 + openssl x509 -text -noout -in initiatecert.pem ``` -Make sure that the "dora" and "swan" configurations point to the certificates. +Make sure that the "respond" and "initiate" configurations point to the certificates. DHCPv6 server setup ------------------- -- cgit 1.2.3-korg