summaryrefslogtreecommitdiffstats
path: root/test/Makefile
AgeCommit message (Collapse)AuthorFilesLines
2023-04-25tests: fix test-help formattingKlement Sekera1-1/+1
Type: improvement Change-Id: Ib7703359b998456bff88caee88c2734c7724bc09 Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2023-04-25tests: support multiple filter expressionsKlement Sekera1-46/+159
Support multiple comma-delimited filter expressions, e.g. to run both bfd and ip4 tests, it's now possible to do: make test TEST=bfd,ip4 Same goes for wildcards, e.g.: make test TEST=bfd,..test_longest_prefix_match,..test_icmp_error Type: improvement Change-Id: I0cceaa443cb612dca955f301c7407959f9a71a6e Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2022-12-07tests: multiple apidir locationsOle Troan1-1/+6
To support testing of external plugins, add support to the test framework and PAPI for specifying a list of locations to look for api.json files. Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I128a306e3c091dc8ef994801b1470b82d2f4595d Signed-off-by: Ole Troan <ot@cisco.com>
2022-08-16tests: move "venv" to "build-root" directory from "test" directorySaima Yunus1-2/+3
Type: refactor - refactored VPP test code to remove "ignore_path" variable from "discover_tests" function and "run_test" code - configured VPP test makefile, config file, and 'run.sh' shell script to move "venv" directory from "test" dir to "build-root" dir Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com> Change-Id: Id2beecbb99f24ce13ed118a1869c5adbef247e50
2022-08-04tests: run a test inside a QEMU VMNaveen Joy1-0/+1
Use the script test/run.py to run a test named test_vm_tap inside a QEMU VM. The run script builds out a virtual env, launches a light weight QEMU VM, mounts host directories, starts VPP inside the VM and runs the test. The test named test_vm_tap, creates two tap v2 interfaces in separate Linux namespaces and using iPerf, streams traffic between the VM and VPP. All data files are stored in the directory named /tmp/vpp-vm-tests. To clean up, use the make test-wipe command. Usage: test/run.py --vm --debug --test test_vm_tap Type: improvement Change-Id: I4425dbef52acee1e5b8af5acaa169b89a2c0f171 Signed-off-by: Naveen Joy <najoy@cisco.com>
2022-05-11tests: fix checkstyle-pythonKlement Sekera1-1/+1
Add --check to black to pass back error value and fail checkstyle if a reformat would occur. Type: fix Fixes: d9b0c6fbf7aa5bd9af84264105b39c82028a4a29 Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I9a1fbe224929fc461ff833a589f73ca06e7cc9d6
2022-05-10tests: replace pycodestyle with blackKlement Sekera1-33/+25
Drop pycodestyle for code style checking in favor of black. Black is much faster, stable PEP8 compliant code style checker offering also automatic formatting. It aims to be very stable and produce smallest diffs. It's used by many small and big projects. Running checkstyle with black takes a few seconds with a terse output. Thus, test-checkstyle-diff is no longer necessary. Expand scope of checkstyle to all python files in the repo, replacing test-checkstyle with checkstyle-python. Also, fixstyle-python is now available for automatic style formatting. Note: python virtualenv has been consolidated in test/Makefile, test/requirements*.txt which will eventually be moved to a central location. This is required to simply the automated generation of docker executor images in the CI. Type: improvement Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8 Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-03-22tests: fix DEBUG=attach functionalityKlement Sekera1-2/+3
Make make test-start-vpp-in-gdb work again. Fix incorrect temp directory when using DEBUG=attach. Type: fix Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec Change-Id: Ie98b637acbbe0221606ccdc7b54f63885e5951a4 Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2022-02-17tests: make tests less make dependentKlement Sekera1-61/+123
Implement command line argument parsing instead of passing arguments via environment variables. Add script for running tests without having to invoke make. Deprecate running tests via make. Type: improvement Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-02-07tests: Update python packagesDave Wallace1-2/+5
- pip == 22.0.3 - pip-tools == 6.5.0 - setuptools == 60.7.1 (now pinned in test/Makefile) - upgrade packages in requirements-3.txt - install iperf3 for 'make test TEST=vcl' Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I1bd85f10fb4f6ba87b9bc1267905e5f1b8eb16de
2022-02-05tests: fix test-checkstyle-diff if no .py changedKlement Sekera1-2/+2
Type: fix Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I6d23ff14380b0154f617b3ab7cd1acd5aa45106a
2022-01-31tests: correct python usage in virtualenvKlement Sekera1-9/+9
Use 'python3' for all python calls in virtualenv. Using $(PYTHON_INTERP) (set to $(PYTHON)) is incorrect when $(PYTHON) contains absolute path (e.g. '/usr/bin/pypy3') as it uses interpreter outside of virtualenv. Type: fix Change-Id: I19878360cfc1ab6a591e09a079b0b69c1a1a58cf Signed-off-by: Klement Sekera <ksekera@cisco.com>
2022-01-31ip: reassembly - add a way to disable for forusKlement Sekera1-2/+2
Add API to disable full reassembly of "forus" packets. Mark packets passing through ip[4|6]-local nodes with a new buffer flag and check for that flag in reassembly. Enable IP6 "forus" full reassembly by default to be consistent with existing IP4 setting. Type: improvement Change-Id: I7067792fcd4304182654237968e4c4d9293c6143 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-11-02tests: update python packagesDave Wallace1-2/+2
- Also fix docs requirements and venv cleanup required for docker executor building Type: test Change-Id: I839423224d95c45af42f459217887f4201cbb11a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-10-13docs: better docs, mv doxygen to sphinxNathan Skrzypczak1-23/+2
This patch refactors the VPP sphinx docs in order to make it easier to consume for external readers as well as VPP developers. It also makes sphinx the single source of documentation, which simplifies maintenance and operation. Most important updates are: - reformat the existing documentation as rst - split RELEASE.md and move it into separate rst files - remove section 'events' - remove section 'archive' - remove section 'related projects' - remove section 'feature by release' - remove section 'Various links' - make (Configuration reference, CLI docs, developer docs) top level items in the list - move 'Use Cases' as part of 'About VPP' - move 'Troubleshooting' as part of 'Getting Started' - move test framework docs into 'Developer Documentation' - add a 'Contributing' section for gerrit, docs and other contributer related infos - deprecate doxygen and test-docs targets - redirect the "make doxygen" target to "make docs" Type: refactor Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com> Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-08-13tests docs: upgrade python packagesDave Wallace1-2/+2
- Upgrade python package requirements for test & docs - Clean up docs generation warnings - Consolidate python requirements for docs in test requirements specs. - Upgrade pip Type: make Change-Id: I74a3924b43ed93d15b32ec9f6fc41ed1ba95b69b Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-05-26tests: Fix 'test-wipe'Neale Ranns1-1/+0
Type: fix Error: vagrant@ ~/vpp (make-test-wipe) $ make test-wipe make[1]: Entering directory '/home/vagrant/vpp/test' make[2]: Entering directory '/home/vagrant/vpp/test' make[2]: *** ext: No such file or directory. Stop. make[2]: Leaving directory '/home/vagrant/vpp/test' make[1]: *** [Makefile:239: wipe] Error 2 make[1]: Leaving directory '/home/vagrant/vpp/test' make: *** [Makefile:440: test-wipe] Error 2 Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: I3615a962882d5d56c394aa628e9942fed07cde84
2021-05-26tests: move vapi C/C++ test files into src/Damjan Marion1-5/+1
Type: improvement Change-Id: Iaad004661db1e9076e6c27227878f1612fa7289f Signed-off-by: Damjan Marion <damarion@cisco.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-05-13tests: move test source to vpp/testDave Wallace1-29/+12
- Generate copyright year and version instead of using hard-coded data Type: refactor Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I6058f5025323b3aa483f5df4a2c4371e27b5914e
2021-04-16tests: cpus awarenessKlement Sekera1-1/+2
Introduce MAX_CPUS parameters to control maximum number of CPUs used by VPP(s) during testing, with default value 'auto' corresponding to all CPUs available. Calculate test CPU requirements by taking into account the number of workers, so a test requires 1 (main thread) + # of worker CPUs. When running tests, keep track of both running test jobs (controlled by TEST_JOBS parameter) and free CPUs. This then causes two limits in the system - to not exceed number of jobs in parallel but also to not exceed number of CPUs available. Skip tests which require more CPUs than are available in system (or more than MAX_CPUS) and print a warning message. Type: improvement Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-12tests: support attaching to existing vppKlement Sekera1-25/+28
Introduce a new option DEBUG=attach to run a test against existing already running vpp. A new target 'make test-start-gdb' will spawn VPP in gdb for this purpose. Customization options explained in test-help. Type: improvement Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-31tests: call venv module directlyKlement Sekera1-1/+1
This avoids ModuleNotFoundError: No module named 'virtualenv.seed.embed.via_app_data' https://github.com/pypa/virtualenv/issues/1873 Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I860cc8c4ad2ba509815c5e7cf02fa5e01ee2ed63
2021-03-25tests: introduce test-checkstyle-diffKlement Sekera1-2/+17
Make test-checkstyle-diff is a new target which checks PEP8 compliance only for changed files. This makes it faster to execute and also more readable as most of the time, only changed files will fail. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I71baca76ab3a21a7a3790617cbfb0d48aacbd9ec
2021-03-24tests: add vpp test dirs to PYTHONPATHDave Wallace1-4/+10
- Required to relocate test infra modules to be co-located in feature test directory. - Remove/regenerate build-root/build-test/src/ soft links when running make test to allow re-running 'make test' after moving source files. - Suppress annoying 'ls' error message when gathering test source files caused by 'c' test application directory being named 'test'. - Use PYTHONPYCACHEPREFIX to generate all compiled source under build-root/build-test. This only works for python 3.8 or greater, but existing 'make test-wipe' target cleans up in-situ __pycache__ directories for older python versions. Type: fix Change-Id: If3c3ff0c58fb26a17395babc34783e06b1dd84e7 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-18tests: remove shm files created by vpp on resetKlement Sekera1-0/+1
Even though we don't use shm transport anymore, vpp still populates shm. Add cleanup before running tests to avoid the nuisance of running out of space on /dev/shm. Type: fix Fixes: eb506becd038f081bc7ac340412d3685425bb8b4 Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I2ef87e991223ee60892d56af1482025d91e77d46
2021-03-17tests: remove shm handlingKlement Sekera1-2/+0
Not used anymore with socket transport. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
2021-03-15tests: use socket transport instead of shared memoryOle Troan1-2/+0
Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I9e65c94a5a05047a5104e9361ea36eac77b40442 Signed-off-by: Ole Troan <ot@cisco.com>
2021-01-08tests: Use soft links for the test file stagingNeale Ranns1-2/+3
Type: fix with hard links this sequence doesn't work echo "GARBAGE" >> test/vpp_ipsec.py git checkout test/vpp_ipsec.py also blow away the staging diretory with 'test-wipe' just to be thorough Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: Ic38d7ec514102759f2eafead59c7f56c5b39f33d
2020-12-22tests: fix gathering of test src filesDave Wallace1-1/+1
- Use hard links instead of soft links so that python module import works correctly for relocated test/vpp_*.py modules. Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: If9c031bf0918344dedd4dee5f9336e869590b0ae
2020-08-31vpp_config: update node variants to skx and iclRay Kinsella1-2/+2
Node variant names have changed to skx and icl. Type: fix Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I5be832aa59e4e440df53869077d801c011e3defa
2020-06-24tests: fix deps pinning / patchingAloys Augustin1-5/+3
This ensures we don't recompute the requirements-{2,3}.txt on each make test run, and skips patching scapy if it is already patched instead of failing. Change-Id: I3da57182ae49f3dd04db139d96734a5d145fedff Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-05-25tests: update pip and pip-toolsAloys Augustin1-4/+5
This fixes an issue where the pinned requirements file can be modified when running the tests. Change-Id: Ic89d1844d1fd8d00f62211a9b051a26ac34ee316 Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-04-28vlib: startup multi-arch variant configurationRay Kinsella1-0/+4
Support for startup node multi-arch variant selection through startup.conf. This is to facilitate unit, functional testing and benchmarking of non-default multi-arch variant node code path. Also added parameters to make test, to specific using multi-arch variants in unit testing. Type: improvement Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
2020-04-07tests: pin sphinx and sphinx-rtd-themeAloys Augustin1-3/+1
Add these two packages to requirements.txt so that their version and the version of their dependencies are pinned to limit the risk of unexpected breakage. Change-Id: If330404f2e840af3d2628f997ce406cd14e7e128 Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-02-07tests: support python 3.8Ole Troan1-2/+2
Make test framework python3 version independence. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-0/+2
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-18build tests: fail test run if we can't patch scapyPaul Vinciguerra1-1/+1
Type: fix Change-Id: I694e44f449863b3edbbc8b6095e94f770dc20330 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-14tests: changes for scapy 2.4.3 migrationsnaramre1-1/+1
Type: fix Change-Id: I7e041b666dabd90df23a920a1f1d99db4c10ddfe Signed-off-by: snaramre <snaramre@cisco.com>
2019-12-04build: fix make test with distributed srcDave Wallace1-102/+157
This patch addresses the functionality that is missing for distributed make test source files. In addition it extends the concept of colocating test source code with functional source code (eg. src/vcl/test). It also cleans up deficiencies in the make test makefiles. NOTE: Due to the way sphinx document tools work, all test, all of the make test python code is gathered using soft links into the directory: .../build-root/build-test/src Change summary: - Remove 'make test' help details from main makefile help output to reduce clutter and redundant help output - Move all generated build output to .../build-root/build-test - Move test_vcl.py as first usecase for distributed core feature make test code - Add test-wipe-all target and fix wipe targets to remove all generated files - Fix 'make test-doc' to generate module documentation for all source files - Remove unused targets in test/doc/Makefile - Fix 'make test-shell' - Fix test/ext Makefile to not generate bogus output Type: fix Fixes: a43c93f8554ad7418e31be3791b3fb71232f60ac Change-Id: Icc2ddb81d474081c3ede4548ecd7a0624651f62d Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-11-27misc: add address sanitizer heap instrumentationBenoît Ganne1-0/+3
Introduce AddressSanitizer support: https://github.com/google/sanitizers/ This starts with heap instrumentation. vlib_buffer, bihash and stack instrumentation should follow. Type: feature Change-Id: I7f20e235b2f79db72efd0e756f22c75f717a9884 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-11-05tests: support setting random seedKlement Sekera1-0/+2
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-10-28tests: switch test framework to python3 by defaultOle Troan1-6/+6
Type: make Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I84d32f022d9dc555837c8916feba04a224cd262a
2019-10-22tests: DEBUG=gdb[server]-all to debug worker(s)Dave Wallace1-0/+4
Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I0264451632c1ce780b38a2c15a7e34350fc6d521
2019-10-09build: clean up make help outputDave Wallace1-3/+3
This change is designed to help the uninformed find the right way to run extended tests by using the test-all[-debug] targets. 'make test EXTENDED_TESTS=y' fails to build as it has a dependency on 'vom-install' which is conveniently included in test-all[-debug]. - clarify test-all[-debug] description and make test-help description - Also align indentation of make help output Type: style Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: Ief54cc8a5af68c052aacb0d660237c5eb63451b5
2019-09-23tests: fix test-checkstyle to check plugin testsDave Wallace1-1/+1
Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I93eb0ae4338247fa2479f8e419483d1593436dc7
2019-09-02tests: run vapi tests by defaultKlement Sekera1-3/+1
Type: fix Change-Id: I7b190507446fca745fc4bc6abbdeed48f55abdb9 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-08-22tests: move plugin tests to src/plugins/*/testDave Wallace1-1/+8
- Relocate plugin tests for 'make test' into src/plugins/*/test so that plugin test cases are co-located with the plugin source code. Type: refactor Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I503e6a43528e14981799b735fa65674155713f67 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-07-28build: add option to wipe/rebuild papiPaul Vinciguerra1-0/+6
Previously, the only option was to wipe/rebuild all the test dependencies. Type: make Change-Id: Ia95d6e800f67bef033dcf614dbfa249c3e43159d Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-23tests: pin pip versionAloys Augustin1-0/+4
New pip releases can break pip-tools. This commit pins the version of pip used in the test virtualenv to prevent uncontrolled breakage. This fixes the current issue in make test: TypeError: __init__() got an unexpected keyword argument 'index_urls' Change-Id: I3b7ecb31e651401ada0d357e2bf093c91c934565 Type: feature Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2019-06-27tests: pin python dependenciesAloys Augustin1-17/+38
This commit ensures that the tests always run with the exact same version for all the Python dependencies. It uses pip-tools to achieve that. Our top-level dependencies are specified in the requirements.txt file. From this file, pip-tools generates the requirements-{2,3}.txt file, which pins all the versions of all the recursive dependencies, and is used to install the packages in the test virtualenv. To change or add a top-level dependency, update requirements.txt and run make test as usual with python2 and python3. The requirements-{2,3}.txt file will be updated and you can verify that nothing breaks. Then add all requirements* files in your commit. To refresh the python packages (i.e. get new versions of the recursive dependencies, or of the dependencies that are not pinned in requirements.txt), just run: PYTHON=python2.7 make test-refresh-deps PYTHON=python3.6 make test-refresh-deps and this will update the requirements-{2,3}.txt files. Ideally we should run this after each release. Type: make Change-Id: Ic533de3d06ec4019ff38f5231208da6f1025bfc7 Signed-off-by: Aloys Augustin <aloaugus@cisco.com>