aboutsummaryrefslogtreecommitdiffstats
path: root/test/Makefile
AgeCommit message (Collapse)AuthorFilesLines
2021-04-12tests: support attaching to existing vppKlement Sekera1-25/+28
Introduce a new option DEBUG=attach to run a test against existing already running vpp. A new target 'make test-start-gdb' will spawn VPP in gdb for this purpose. Customization options explained in test-help. Type: improvement Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-31tests: call venv module directlyKlement Sekera1-1/+1
This avoids ModuleNotFoundError: No module named 'virtualenv.seed.embed.via_app_data' https://github.com/pypa/virtualenv/issues/1873 Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I860cc8c4ad2ba509815c5e7cf02fa5e01ee2ed63
2021-03-25tests: introduce test-checkstyle-diffKlement Sekera1-2/+17
Make test-checkstyle-diff is a new target which checks PEP8 compliance only for changed files. This makes it faster to execute and also more readable as most of the time, only changed files will fail. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I71baca76ab3a21a7a3790617cbfb0d48aacbd9ec
2021-03-24tests: add vpp test dirs to PYTHONPATHDave Wallace1-4/+10
- Required to relocate test infra modules to be co-located in feature test directory. - Remove/regenerate build-root/build-test/src/ soft links when running make test to allow re-running 'make test' after moving source files. - Suppress annoying 'ls' error message when gathering test source files caused by 'c' test application directory being named 'test'. - Use PYTHONPYCACHEPREFIX to generate all compiled source under build-root/build-test. This only works for python 3.8 or greater, but existing 'make test-wipe' target cleans up in-situ __pycache__ directories for older python versions. Type: fix Change-Id: If3c3ff0c58fb26a17395babc34783e06b1dd84e7 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-18tests: remove shm files created by vpp on resetKlement Sekera1-0/+1
Even though we don't use shm transport anymore, vpp still populates shm. Add cleanup before running tests to avoid the nuisance of running out of space on /dev/shm. Type: fix Fixes: eb506becd038f081bc7ac340412d3685425bb8b4 Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I2ef87e991223ee60892d56af1482025d91e77d46
2021-03-17tests: remove shm handlingKlement Sekera1-2/+0
Not used anymore with socket transport. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
2021-03-15tests: use socket transport instead of shared memoryOle Troan1-2/+0
Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I9e65c94a5a05047a5104e9361ea36eac77b40442 Signed-off-by: Ole Troan <ot@cisco.com>
2021-01-08tests: Use soft links for the test file stagingNeale Ranns1-2/+3
Type: fix with hard links this sequence doesn't work echo "GARBAGE" >> test/vpp_ipsec.py git checkout test/vpp_ipsec.py also blow away the staging diretory with 'test-wipe' just to be thorough Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: Ic38d7ec514102759f2eafead59c7f56c5b39f33d
2020-12-22tests: fix gathering of test src filesDave Wallace1-1/+1
- Use hard links instead of soft links so that python module import works correctly for relocated test/vpp_*.py modules. Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: If9c031bf0918344dedd4dee5f9336e869590b0ae
2020-08-31vpp_config: update node variants to skx and iclRay Kinsella1-2/+2
Node variant names have changed to skx and icl. Type: fix Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I5be832aa59e4e440df53869077d801c011e3defa
2020-06-24tests: fix deps pinning / patchingAloys Augustin1-5/+3
This ensures we don't recompute the requirements-{2,3}.txt on each make test run, and skips patching scapy if it is already patched instead of failing. Change-Id: I3da57182ae49f3dd04db139d96734a5d145fedff Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-05-25tests: update pip and pip-toolsAloys Augustin1-4/+5
This fixes an issue where the pinned requirements file can be modified when running the tests. Change-Id: Ic89d1844d1fd8d00f62211a9b051a26ac34ee316 Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-04-28vlib: startup multi-arch variant configurationRay Kinsella1-0/+4
Support for startup node multi-arch variant selection through startup.conf. This is to facilitate unit, functional testing and benchmarking of non-default multi-arch variant node code path. Also added parameters to make test, to specific using multi-arch variants in unit testing. Type: improvement Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
2020-04-07tests: pin sphinx and sphinx-rtd-themeAloys Augustin1-3/+1
Add these two packages to requirements.txt so that their version and the version of their dependencies are pinned to limit the risk of unexpected breakage. Change-Id: If330404f2e840af3d2628f997ce406cd14e7e128 Type: fix Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2020-02-07tests: support python 3.8Ole Troan1-2/+2
Make test framework python3 version independence. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-0/+2
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-18build tests: fail test run if we can't patch scapyPaul Vinciguerra1-1/+1
Type: fix Change-Id: I694e44f449863b3edbbc8b6095e94f770dc20330 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-14tests: changes for scapy 2.4.3 migrationsnaramre1-1/+1
Type: fix Change-Id: I7e041b666dabd90df23a920a1f1d99db4c10ddfe Signed-off-by: snaramre <snaramre@cisco.com>
2019-12-04build: fix make test with distributed srcDave Wallace1-102/+157
This patch addresses the functionality that is missing for distributed make test source files. In addition it extends the concept of colocating test source code with functional source code (eg. src/vcl/test). It also cleans up deficiencies in the make test makefiles. NOTE: Due to the way sphinx document tools work, all test, all of the make test python code is gathered using soft links into the directory: .../build-root/build-test/src Change summary: - Remove 'make test' help details from main makefile help output to reduce clutter and redundant help output - Move all generated build output to .../build-root/build-test - Move test_vcl.py as first usecase for distributed core feature make test code - Add test-wipe-all target and fix wipe targets to remove all generated files - Fix 'make test-doc' to generate module documentation for all source files - Remove unused targets in test/doc/Makefile - Fix 'make test-shell' - Fix test/ext Makefile to not generate bogus output Type: fix Fixes: a43c93f8554ad7418e31be3791b3fb71232f60ac Change-Id: Icc2ddb81d474081c3ede4548ecd7a0624651f62d Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-11-27misc: add address sanitizer heap instrumentationBenoît Ganne1-0/+3
Introduce AddressSanitizer support: https://github.com/google/sanitizers/ This starts with heap instrumentation. vlib_buffer, bihash and stack instrumentation should follow. Type: feature Change-Id: I7f20e235b2f79db72efd0e756f22c75f717a9884 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-11-05tests: support setting random seedKlement Sekera1-0/+2
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-10-28tests: switch test framework to python3 by defaultOle Troan1-6/+6
Type: make Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I84d32f022d9dc555837c8916feba04a224cd262a
2019-10-22tests: DEBUG=gdb[server]-all to debug worker(s)Dave Wallace1-0/+4
Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I0264451632c1ce780b38a2c15a7e34350fc6d521
2019-10-09build: clean up make help outputDave Wallace1-3/+3
This change is designed to help the uninformed find the right way to run extended tests by using the test-all[-debug] targets. 'make test EXTENDED_TESTS=y' fails to build as it has a dependency on 'vom-install' which is conveniently included in test-all[-debug]. - clarify test-all[-debug] description and make test-help description - Also align indentation of make help output Type: style Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: Ief54cc8a5af68c052aacb0d660237c5eb63451b5
2019-09-23tests: fix test-checkstyle to check plugin testsDave Wallace1-1/+1
Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I93eb0ae4338247fa2479f8e419483d1593436dc7
2019-09-02tests: run vapi tests by defaultKlement Sekera1-3/+1
Type: fix Change-Id: I7b190507446fca745fc4bc6abbdeed48f55abdb9 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-08-22tests: move plugin tests to src/plugins/*/testDave Wallace1-1/+8
- Relocate plugin tests for 'make test' into src/plugins/*/test so that plugin test cases are co-located with the plugin source code. Type: refactor Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I503e6a43528e14981799b735fa65674155713f67 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-07-28build: add option to wipe/rebuild papiPaul Vinciguerra1-0/+6
Previously, the only option was to wipe/rebuild all the test dependencies. Type: make Change-Id: Ia95d6e800f67bef033dcf614dbfa249c3e43159d Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-23tests: pin pip versionAloys Augustin1-0/+4
New pip releases can break pip-tools. This commit pins the version of pip used in the test virtualenv to prevent uncontrolled breakage. This fixes the current issue in make test: TypeError: __init__() got an unexpected keyword argument 'index_urls' Change-Id: I3b7ecb31e651401ada0d357e2bf093c91c934565 Type: feature Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2019-06-27tests: pin python dependenciesAloys Augustin1-17/+38
This commit ensures that the tests always run with the exact same version for all the Python dependencies. It uses pip-tools to achieve that. Our top-level dependencies are specified in the requirements.txt file. From this file, pip-tools generates the requirements-{2,3}.txt file, which pins all the versions of all the recursive dependencies, and is used to install the packages in the test virtualenv. To change or add a top-level dependency, update requirements.txt and run make test as usual with python2 and python3. The requirements-{2,3}.txt file will be updated and you can verify that nothing breaks. Then add all requirements* files in your commit. To refresh the python packages (i.e. get new versions of the recursive dependencies, or of the dependencies that are not pinned in requirements.txt), just run: PYTHON=python2.7 make test-refresh-deps PYTHON=python3.6 make test-refresh-deps and this will update the requirements-{2,3}.txt files. Ideally we should run this after each release. Type: make Change-Id: Ic533de3d06ec4019ff38f5231208da6f1025bfc7 Signed-off-by: Aloys Augustin <aloaugus@cisco.com>
2019-05-09add mactime plugin unit / code coverage testsDave Barach1-0/+1
The unit and code coverage tests are boring. The rest of the patch involves test and packet-generator infra cleanups. Teach the "make test-xxx" family of targets to set the api test plugin path correctly, to make "binary-api <api-message-name> <args>" debug CLI commands work correctly in the "make test" environment. Unfortunately involves both the top-level and test Makefiles. Add a minor pg cli feature, a CLI to manually set s->sw_if_index[VLIB_TX]. Consider the case where one configures an interface with both a device-input and an output feature. To test the output feature using the pg, it's necessary to inject packets into the interface output node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set correctly. For example: packet-generator new { name tx limit 15 size 128-128 interface local0 # rx: device input feature not configured on local0 tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX] node loop0-output data { hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405 } } Fix a longstanding bug in the packet generator stream setup. Remove kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places] instead of using the stream value s->sw_if_index[VLIB_TX], and setting THAT datum correctly. Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed Signed-off-by: Dave Barach <dave@barachs.net>
2019-05-08make test: introduce COREDUMP_COMPRESS optionKlement Sekera1-0/+1
setting COREDUMP_COMPRESS=y (or =1) indicates that any core file is to be compressed by the test framework unless DEBUG=core is set. Change-Id: Ib0d6d30a7398dbeb6ab5e5dbb411bf5c641b5206 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-04-23API: Python and Unix domain socket improvementOle Troan1-0/+2
Handle the case where buffer overflows. Then SOCK_SEQPACKET assumption that multiple API messages are not returned by recv() is broken. Use SOCK_STREAM for API exchanges instead. Add support for running tests over sockets. make test SOCKET=1 Change-Id: Ibe5fd69b1bf617de4c7ba6cce0a7c2b3f97a2821 Signed-off-by: Ole Troan <ot@cisco.com>
2019-03-19Revert "tests: Deprecate the use of CLI commands in tests."Neale Ranns1-1/+1
This reverts commit e2e8f02165f27ae04e6d20ce0c628e29271a8db6. Change-Id: Ie23dcb8a5b3e96f77e0d117a6887c43064c79669 Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-18tests: Deprecate the use of CLI commands in tests.Paul Vinciguerra1-1/+1
Since the stated policy regarding CLI commands is: "Note that the debug CLI is a developer's tool - no warranty express or implied - and that we may choose not to fix debug CLI bugs.", this change emits a deprecation warning whenever a test case calls a CLI command. Change-Id: I91b30e86ae1b2fca36732837f36cdda762cdf458 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-26make test: Stop downloading pip.Paul Vinciguerra1-6/+2
Use the installed version of pip. If a newer version of pip is needed, it can be specified in requirements.txt. This is to improve idempotence by providing some control over upstream changes. Change-Id: I07b7651f8ddba6005bf85d11fef9e3343ee2dcd2 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-14make test: make test-ext working againKlement Sekera1-6/+6
Change-Id: Ib30d0c6b8794488119f258f7617fce494aa28f1f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-31VTL: Makefile: Don't install pip twice.Paul Vinciguerra1-1/+0
Pip is installed via virtualenv. No need to reinstall it again. Change-Id: I45bea1f9db841ca4309b763ab852623180937727 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-18VTL: Fix sphinx doc generation.Paul Vinciguerra1-1/+1
Remove unnecessary envvar magic for sphinx builds. Just provide the proper mock objects. Change-Id: I151513eb47d2f71cf221651145dbcdc153448772 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-17VTL: Move doc generation and linting to python3.Paul Vinciguerra1-5/+6
* running linting/pycodestyle under python3 provides the additional benefit of access to the python3 AST. Change-Id: I659dbd2733471285867be95eec078f7415aa0bcd Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-05VPP-1508: Python3: Support simultaneous p2/p3 venvs.Paul Vinciguerra1-1/+1
Change-Id: I24cf3e390f6f0b98be939ef717048742d3f183d6 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-18VPP-1508: Have test/Makefile honor PYTHON= envvar.Paul Vinciguerra1-8/+8
Change-Id: Iff71a5982b73c97e3f546beb1cf52935a267f5f5 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-06Skip failing ARM testcases for CIjuraj.linkes1-0/+2
There are a few tests that fail on ARM and thus we can't enable testing in CI. Skip the failing tests until they're fixed so that we catch new failures in CI (when we enable testing for the remaining tests). Change-Id: Ie896ef5c449ef965029633e38d317a8d5ac26352 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-29make test: create virtualenv under /test/Klement Sekera1-45/+47
instead of using build-root, use /test/venv directory for virtualenv similarly, don't pollute build-root with test-built binaries Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-26VPP-1508 Add support for environment markers.Paul Vinciguerra1-1/+1
Add the ability to specify a specific python library version based on the interpreter/platform/etc. Change-Id: I027acdf22ad839b5cff63b319f0aa100b0f336c8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-24VPP-1505 VPP-1508 Remove dep.: py_lispnetworkingPaul Vinciguerra1-1/+1
test_lisp.py No longer needs the external dependency to run. py27 runtests: commands[6] | stestr --test-path ./test run --slowest test_lisp ============================================================================== Basic LISP test ============================================================================== {0} test.test_lisp.TestLisp.test_lisp_basic_encap [0.487960s] ... ok ====== Totals ====== Ran: 1 tests in 18.0000 sec. - Passed: 1 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 0.4880 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:00:00.487960 Test id Runtime (s) --------------------------------------------- ----------- test.test_lisp.TestLisp.test_lisp_basic_encap 0.488 ______________________________________________________ summary ______________________________________________________ py27: commands succeeded congratulations :) vagrant@vpp:/vpp$ Change-Id: I35b35f8ddaed79066b3ed62fa0460c9ea83273c2 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-23make test: remove test-ext, fix test*-all targetsKlement Sekera1-5/+4
build extended binaries, vom & friends when test*-all targets are run, remove unneeded test-ext target (which is semantically same as test-debug-all) Change-Id: Id710e061fdc93e286a4ed91be6a0cdcf75a2dafd Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-22Add RFC5424 syslog protocol support (VPP-1139)Matus Fabian1-1/+1
Syslog protocol logging transport event messages across network over UDP protocol based on RFC5426. Change-Id: Ica74b40bcc2e6d0fbd41e9bf78e76395fbabab3c Signed-off-by: Matus Fabian <matfabia@cisco.com>
2018-08-23CSIT-1139: Implement parallel test executionjuraj.linkes1-5/+29
The implementation of parallel test execution in VPP Test Framework. - VPPTestCase test methods are grouped together - tests are running in separate processes - VPP instances spawned by tests are assigned to different cores - output from these processes is redirected through pipes and printed out testcase by testcase - TEST_JOBS env var is used to specify the number of parallel processes - improved test summary - a bit of code cleanup Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-17Add psutil to testing virtualenvjuraj.linkes1-1/+1
Change-Id: I650c9f226bcdc6328a7b4ff2dc4bd20cea8a512a Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>