summaryrefslogtreecommitdiffstats
path: root/test/Makefile
AgeCommit message (Collapse)AuthorFilesLines
2024-01-21build: Explicitly use gmakeTom Jones1-2/+2
VPP requires GNU Make to build, on GNU systems (such as Debian), GNU Make is installed as 'make', typically with a symlink from 'gmake'. On other systems (such as FreeBSD), 'make' is a BSD Make derriviative and GNU Make is installed a 'gmake'. Use $(MAKE) variable for make calls from within Makefiles. This variable is set to the path of the calling make program, i.e., /usr/local/bin/gmake on a bsd system. This is the recommended way to call make from Makefiles in the GNU Make documentation. Type: improvement Change-Id: Id9162a34a0f8358f22090718087918dae31c0fce Signed-off-by: Tom Jones <thj@freebsd.org>
2024-01-18tests: preload api filesMaxime Peim1-1/+7
When sanity test is not done, API files are not loaded until the first test case is run. Hence, it is not possible to use enums, etc. outside of a test class. By preloading API files before running any tests, it prevents its issue. Type: fix Change-Id: I8730150374e6c5f8d6933ec037811372ac2a8da0 Signed-off-by: Maxime Peim <mpeim@cisco.com>
2023-11-27tests: Excluded some components from LCOVadrianvillin1-0/+3
Type: test Change-Id: I387611cfc361e664eb1e78ae669ba4e3901c4fe6 Signed-off-by: adrianvillin <avillin@cisco.com>
2023-11-03tests: refactor asf framework codeDave Wallace1-2/+7
- Make framework.py classes a subset of asfframework.py classes - Remove all packet related code from asfframework.py - Add test class and test case set up debug output to log - Repatriate packet tests from asf to test directory - Remove non-packet related code from framework.py and inherit them from asfframework.py classes - Clean up unused import variables - Re-enable BFD tests on Ubuntu 22.04 and fix intermittent test failures in echo_looped_back testcases (where # control packets verified but not guaranteed to be received during test) - Re-enable Wireguard tests on Ubuntu 22.04 and fix intermittent test failures in handshake ratelimiting testcases and event testcase - Run Wiregard testcase suites solo - Improve debug output in log.txt - Increase VCL/LDP post sleep timeout to allow iperf server to finish cleanly. - Fix pcap history files to be sorted by suite and testcase and ensure order/timestamp is correct based on creation in the testcase. - Decode pcap files for each suite and testcase for all errors or if configured via comandline option / env var - Improve vpp corefile detection to allow complete corefile generation - Disable vm vpp interfaces testcases on debian11 - Clean up failed unittest dir when retrying failed testcases and unify testname directory and failed linknames into framwork functions Type: test Change-Id: I0764f79ea5bb639d278bf635ed2408d4d5220e1e Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-10-03build: add ability to disable some plugins from packaging and testsAndrew Yourtchenko1-1/+9
When custom-packaging the VPP artifacts, it can be useful to exclude some of the core plugins from packaging/testing, for some reasons. A removal of a plugin(s) from the worktree needs to be tracked as a separate change, and thus is tricky from the maintenance point of view. This change adds the ability to "pretend they do not exist" - plugins which are added to the comma-separated environment variable "VPP_EXCLUDED_PLUGINS" will not be added to the build process and not packaged. The tests do not have the 1:1 relationship as plugins, so they might need to be modified separately. This change includes some of these modifications as an example. Type: feature Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: Id31562d00a01ced1acbb4996a633517cbd6f09d8
2023-08-16tests docs: update python3 venv packagesDave Wallace1-4/+4
- Package update performed by 1. updating pip, pip-tools, setuptools 2. 'make test-refresh-deps' on ubuntu 22.04 3. fixing 'make test' and 'make docs' issues on ubuntu 22.04 4. 'make test-refresh-deps' on ubuntu 20.04 - Add dependency for 'make test-refresh-deps' to insure python venv is set up. - Update of python formatter, black, caused reformating of 41 python code files. Type: make Change-Id: I7cafdf4b5189065ac57cb6b254937f6e0897a924 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-08-11tests: filter coverage report outputDave Wallace1-0/+3
- Remove test code & non-vpp code from coverage report - Remove driver/hardware vpp code which cannot be tested in 'make test' from coverage report Type: fix Change-Id: I04b50c14bc3437b845f2afafae47297189e61e3f Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-08-08tests: fix setting gcov flag for test-cov targetDave Wallace1-1/+1
Type: fix Change-Id: I3c663babe4f32f5d2870265336b5d272cc029ce7 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-06-22tests docs: fix lcov code coverage report generationDave Wallace1-14/+34
- Updated/rebased version of https://gerrit.fd.io/r/c/vpp/+/34199 Type: test Change-Id: I43913ecfd11a4578bdb10c4be76253fe38d57976 Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-04-25tests: fix test-help formattingKlement Sekera1-1/+1
Type: improvement Change-Id: Ib7703359b998456bff88caee88c2734c7724bc09 Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2023-04-25tests: support multiple filter expressionsKlement Sekera1-46/+159
Support multiple comma-delimited filter expressions, e.g. to run both bfd and ip4 tests, it's now possible to do: make test TEST=bfd,ip4 Same goes for wildcards, e.g.: make test TEST=bfd,..test_longest_prefix_match,..test_icmp_error Type: improvement Change-Id: I0cceaa443cb612dca955f301c7407959f9a71a6e Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2022-12-07tests: multiple apidir locationsOle Troan1-1/+6
To support testing of external plugins, add support to the test framework and PAPI for specifying a list of locations to look for api.json files. Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I128a306e3c091dc8ef994801b1470b82d2f4595d Signed-off-by: Ole Troan <ot@cisco.com>
2022-08-16tests: move "venv" to "build-root" directory from "test" directorySaima Yunus1-2/+3
Type: refactor - refactored VPP test code to remove "ignore_path" variable from "discover_tests" function and "run_test" code - configured VPP test makefile, config file, and 'run.sh' shell script to move "venv" directory from "test" dir to "build-root" dir Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com> Change-Id: Id2beecbb99f24ce13ed118a1869c5adbef247e50
2022-08-04tests: run a test inside a QEMU VMNaveen Joy1-0/+1
Use the script test/run.py to run a test named test_vm_tap inside a QEMU VM. The run script builds out a virtual env, launches a light weight QEMU VM, mounts host directories, starts VPP inside the VM and runs the test. The test named test_vm_tap, creates two tap v2 interfaces in separate Linux namespaces and using iPerf, streams traffic between the VM and VPP. All data files are stored in the directory named /tmp/vpp-vm-tests. To clean up, use the make test-wipe command. Usage: test/run.py --vm --debug --test test_vm_tap Type: improvement Change-Id: I4425dbef52acee1e5b8af5acaa169b89a2c0f171 Signed-off-by: Naveen Joy <najoy@cisco.com>
2022-05-11tests: fix checkstyle-pythonKlement Sekera1-1/+1
Add --check to black to pass back error value and fail checkstyle if a reformat would occur. Type: fix Fixes: d9b0c6fbf7aa5bd9af84264105b39c82028a4a29 Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I9a1fbe224929fc461ff833a589f73ca06e7cc9d6
2022-05-10tests: replace pycodestyle with blackKlement Sekera1-33/+25
Drop pycodestyle for code style checking in favor of black. Black is much faster, stable PEP8 compliant code style checker offering also automatic formatting. It aims to be very stable and produce smallest diffs. It's used by many small and big projects. Running checkstyle with black takes a few seconds with a terse output. Thus, test-checkstyle-diff is no longer necessary. Expand scope of checkstyle to all python files in the repo, replacing test-checkstyle with checkstyle-python. Also, fixstyle-python is now available for automatic style formatting. Note: python virtualenv has been consolidated in test/Makefile, test/requirements*.txt which will eventually be moved to a central location. This is required to simply the automated generation of docker executor images in the CI. Type: improvement Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8 Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-03-22tests: fix DEBUG=attach functionalityKlement Sekera1-2/+3
Make make test-start-vpp-in-gdb work again. Fix incorrect temp directory when using DEBUG=attach. Type: fix Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec Change-Id: Ie98b637acbbe0221606ccdc7b54f63885e5951a4 Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2022-02-17tests: make tests less make dependentKlement Sekera1-61/+123
Implement command line argument parsing instead of passing arguments via environment variables. Add script for running tests without having to invoke make. Deprecate running tests via make. Type: improvement Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-02-07tests: Update python packagesDave Wallace1-2/+5
- pip == 22.0.3 - pip-tools == 6.5.0 - setuptools == 60.7.1 (now pinned in test/Makefile) - upgrade packages in requirements-3.txt - install iperf3 for 'make test TEST=vcl' Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I1bd85f10fb4f6ba87b9bc1267905e5f1b8eb16de
2022-02-05tests: fix test-checkstyle-diff if no .py changedKlement Sekera1-2/+2
Type: fix Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I6d23ff14380b0154f617b3ab7cd1acd5aa45106a
2022-01-31tests: correct python usage in virtualenvKlement Sekera1-9/+9
Use 'python3' for all python calls in virtualenv. Using $(PYTHON_INTERP) (set to $(PYTHON)) is incorrect when $(PYTHON) contains absolute path (e.g. '/usr/bin/pypy3') as it uses interpreter outside of virtualenv. Type: fix Change-Id: I19878360cfc1ab6a591e09a079b0b69c1a1a58cf Signed-off-by: Klement Sekera <ksekera@cisco.com>
2022-01-31ip: reassembly - add a way to disable for forusKlement Sekera1-2/+2
Add API to disable full reassembly of "forus" packets. Mark packets passing through ip[4|6]-local nodes with a new buffer flag and check for that flag in reassembly. Enable IP6 "forus" full reassembly by default to be consistent with existing IP4 setting. Type: improvement Change-Id: I7067792fcd4304182654237968e4c4d9293c6143 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-11-02tests: update python packagesDave Wallace1-2/+2
- Also fix docs requirements and venv cleanup required for docker executor building Type: test Change-Id: I839423224d95c45af42f459217887f4201cbb11a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-10-13docs: better docs, mv doxygen to sphinxNathan Skrzypczak1-23/+2
This patch refactors the VPP sphinx docs in order to make it easier to consume for external readers as well as VPP developers. It also makes sphinx the single source of documentation, which simplifies maintenance and operation. Most important updates are: - reformat the existing documentation as rst - split RELEASE.md and move it into separate rst files - remove section 'events' - remove section 'archive' - remove section 'related projects' - remove section 'feature by release' - remove section 'Various links' - make (Configuration reference, CLI docs, developer docs) top level items in the list - move 'Use Cases' as part of 'About VPP' - move 'Troubleshooting' as part of 'Getting Started' - move test framework docs into 'Developer Documentation' - add a 'Contributing' section for gerrit, docs and other contributer related infos - deprecate doxygen and test-docs targets - redirect the "make doxygen" target to "make docs" Type: refactor Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com> Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-08-13tests docs: upgrade python packagesDave Wallace1-2/+2
- Upgrade python package requirements for test & docs - Clean up docs generation warnings - Consolidate python requirements for docs in test requirements specs. - Upgrade pip Type: make Change-Id: I74a3924b43ed93d15b32ec9f6fc41ed1ba95b69b Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-05-26tests: Fix 'test-wipe'Neale Ranns1-1/+0
Type: fix Error: vagrant@ ~/vpp (make-test-wipe) $ make test-wipe make[1]: Entering directory '/home/vagrant/vpp/test' make[2]: Entering directory '/home/vagrant/vpp/test' make[2]: *** ext: No such file or directory. Stop. make[2]: Leaving directory '/home/vagrant/vpp/test' make[1]: *** [Makefile:239: wipe] Error 2 make[1]: Leaving directory '/home/vagrant/vpp/test' make: *** [Makefile:440: test-wipe] Error 2 Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: I3615a962882d5d56c394aa628e9942fed07cde84
2021-05-26tests: move vapi C/C++ test files into src/Damjan Marion1-5/+1
Type: improvement Change-Id: Iaad004661db1e9076e6c27227878f1612fa7289f Signed-off-by: Damjan Marion <damarion@cisco.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-05-13tests: move test source to vpp/testDave Wallace1-29/+12
- Generate copyright year and version instead of using hard-coded data Type: refactor Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I6058f5025323b3aa483f5df4a2c4371e27b5914e
2021-04-16tests: cpus awarenessKlement Sekera1-1/+2
Introduce MAX_CPUS parameters to control maximum number of CPUs used by VPP(s) during testing, with default value 'auto' corresponding to all CPUs available. Calculate test CPU requirements by taking into account the number of workers, so a test requires 1 (main thread) + # of worker CPUs. When running tests, keep track of both running test jobs (controlled by TEST_JOBS parameter) and free CPUs. This then causes two limits in the system - to not exceed number of jobs in parallel but also to not exceed number of CPUs available. Skip tests which require more CPUs than are available in system (or more than MAX_CPUS) and print a warning message. Type: improvement Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-12tests: support attaching to existing vppKlement Sekera1-25/+28
Introduce a new option DEBUG=attach to run a test against existing already running vpp. A new target 'make test-start-gdb' will spawn VPP in gdb for this purpose. Customization options explained in test-help. Type: improvement Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-31tests: call venv module directlyKlement Sekera1-1/+1
This avoids ModuleNotFoundError: No module named 'virtualenv.seed.embed.via_app_data' https://github.com/pypa/virtualenv/issues/1873 Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I860cc8c4ad2ba509815c5e7cf02fa5e01ee2ed63
2021-03-25tests: introduce test-checkstyle-diffKlement Sekera1-2/+17
Make test-checkstyle-diff is a new target which checks PEP8 compliance only for changed files. This makes it faster to execute and also more readable as most of the time, only changed files will fail. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I71baca76ab3a21a7a3790617cbfb0d48aacbd9ec
2021-03-24tests: add vpp test dirs to PYTHONPATHDave Wallace1-4/+10
- Required to relocate test infra modules to be co-located in feature test directory. - Remove/regenerate build-root/build-test/src/ soft links when running make test to allow re-running 'make test' after moving source files. - Suppress annoying 'ls' error message when gathering test source files caused by 'c' test application directory being named 'test'. - Use PYTHONPYCACHEPREFIX to generate all compiled source under build-root/build-test. This only works for python 3.8 or greater, but existing 'make test-wipe' target cleans up in-situ __pycache__ directories for older python versions. Type: fix Change-Id: If3c3ff0c58fb26a17395babc34783e06b1dd84e7 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-18tests: remove shm files created by vpp on resetKlement Sekera1-0/+1
Even though we don't use shm transport anymore, vpp still populates shm. Add cleanup before running tests to avoid the nuisance of running out of space on /dev/shm. Type: fix Fixes: eb506becd038f081bc7ac340412d3685425bb8b4 Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I2ef87e991223ee60892d56af1482025d91e77d46
2021-03-17tests: remove shm handlingKlement Sekera1-2/+0
Not used anymore with socket transport. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
2021-03-15tests: use socket transport instead of shared memoryOle Troan1-2/+0
Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I9e65c94a5a05047a5104e9361ea36eac77b40442 Signed-off-by: Ole Troan <ot@cisco.com>
2021-01-08tests: Use soft links for the test file stagingNeale Ranns1-2/+3
Type: fix with hard links this sequence doesn't work echo "GARBAGE" >> test/vpp_ipsec.py git checkout test/vpp_ipsec.py also blow away the staging diretory with 'test-wipe' just to be thorough Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: Ic38d7ec514102759f2eafead59c7f56c5b39f33d
2020-12-22tests: fix gathering of test src filesDave Wallace1-1/+1
- Use hard links instead of soft links so that python module import works correctly for relocated test/vpp_*.py modules. Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: If9c031bf0918344dedd4dee5f9336e869590b0ae
2020-08-31vpp_config: update node variants to skx and iclRay Kinsella1-2/+2
Node variant names have changed to skx and icl. Type: fix Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I5be832aa59e4e440df53869077d801c011e3defa
2020-06-24tests: fix deps pinning / patchingAloys Augustin1-5/+3
This ensures we don't recompute the requirements-{2,3}.txt on each make test run, and skips patching scapy if it is already patched instead of failing. Change-Id: I3da57182ae49f3dd04db139d96734a5d145fedff Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-05-25tests: update pip and pip-toolsAloys Augustin1-4/+5
This fixes an issue where the pinned requirements file can be modified when running the tests. Change-Id: Ic89d1844d1fd8d00f62211a9b051a26ac34ee316 Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-04-28vlib: startup multi-arch variant configurationRay Kinsella1-0/+4
Support for startup node multi-arch variant selection through startup.conf. This is to facilitate unit, functional testing and benchmarking of non-default multi-arch variant node code path. Also added parameters to make test, to specific using multi-arch variants in unit testing. Type: improvement Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
2020-04-07tests: pin sphinx and sphinx-rtd-themeAloys Augustin1-3/+1
Add these two packages to requirements.txt so that their version and the version of their dependencies are pinned to limit the risk of unexpected breakage. Change-Id: If330404f2e840af3d2628f997ce406cd14e7e128 Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-02-07tests: support python 3.8Ole Troan1-2/+2
Make test framework python3 version independence. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-0/+2
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-18build tests: fail test run if we can't patch scapyPaul Vinciguerra1-1/+1
Type: fix Change-Id: I694e44f449863b3edbbc8b6095e94f770dc20330 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-14tests: changes for scapy 2.4.3 migrationsnaramre1-1/+1
Type: fix Change-Id: I7e041b666dabd90df23a920a1f1d99db4c10ddfe Signed-off-by: snaramre <snaramre@cisco.com>
2019-12-04build: fix make test with distributed srcDave Wallace1-102/+157
This patch addresses the functionality that is missing for distributed make test source files. In addition it extends the concept of colocating test source code with functional source code (eg. src/vcl/test). It also cleans up deficiencies in the make test makefiles. NOTE: Due to the way sphinx document tools work, all test, all of the make test python code is gathered using soft links into the directory: .../build-root/build-test/src Change summary: - Remove 'make test' help details from main makefile help output to reduce clutter and redundant help output - Move all generated build output to .../build-root/build-test - Move test_vcl.py as first usecase for distributed core feature make test code - Add test-wipe-all target and fix wipe targets to remove all generated files - Fix 'make test-doc' to generate module documentation for all source files - Remove unused targets in test/doc/Makefile - Fix 'make test-shell' - Fix test/ext Makefile to not generate bogus output Type: fix Fixes: a43c93f8554ad7418e31be3791b3fb71232f60ac Change-Id: Icc2ddb81d474081c3ede4548ecd7a0624651f62d Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-11-27misc: add address sanitizer heap instrumentationBenoît Ganne1-0/+3
Introduce AddressSanitizer support: https://github.com/google/sanitizers/ This starts with heap instrumentation. vlib_buffer, bihash and stack instrumentation should follow. Type: feature Change-Id: I7f20e235b2f79db72efd0e756f22c75f717a9884 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-11-05tests: support setting random seedKlement Sekera1-0/+2
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>