aboutsummaryrefslogtreecommitdiffstats
path: root/test/run_tests.py
AgeCommit message (Collapse)AuthorFilesLines
2023-05-20tests: support for expected failuresKlement Sekera1-64/+61
- Add support for @unittest.expectedFailure decorator. Type: improvement Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I761751cda505e962225dc680b97c1fffa96f5176 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2023-04-25tests: fix parallel runs skipping some testsKlement Sekera1-3/+5
Fix corner case when a test would be skipped if it was not possible to start it due to insufficient cpus available in the middle of the loop. Type: fix Change-Id: Ie4580685ff55688375d649d7009131d9fe1e4f33 Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2023-04-25tests: support multiple filter expressionsKlement Sekera1-17/+39
Support multiple comma-delimited filter expressions, e.g. to run both bfd and ip4 tests, it's now possible to do: make test TEST=bfd,ip4 Same goes for wildcards, e.g.: make test TEST=bfd,..test_longest_prefix_match,..test_icmp_error Type: improvement Change-Id: I0cceaa443cb612dca955f301c7407959f9a71a6e Signed-off-by: Klement Sekera <klement.sekera@gmail.com>
2022-08-16tests: move "venv" to "build-root" directory from "test" directorySaima Yunus1-2/+1
Type: refactor - refactored VPP test code to remove "ignore_path" variable from "discover_tests" function and "run_test" code - configured VPP test makefile, config file, and 'run.sh' shell script to move "venv" directory from "test" dir to "build-root" dir Signed-off-by: Saima Yunus <yunus.saima.234@gmail.com> Change-Id: Id2beecbb99f24ce13ed118a1869c5adbef247e50
2022-05-24tests: fix default failed dir settingKlement Sekera1-4/+2
When running tests via run.sh, default setting of None would cause failed directory symlink to appear in vpp workspace with an ugly name. This patch places the symlink in temporary directory. Type: fix Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: Ic1715eba7ac1f82f71855e2aeb9b659d27bbb3af
2022-05-10tests: replace pycodestyle with blackKlement Sekera1-206/+305
Drop pycodestyle for code style checking in favor of black. Black is much faster, stable PEP8 compliant code style checker offering also automatic formatting. It aims to be very stable and produce smallest diffs. It's used by many small and big projects. Running checkstyle with black takes a few seconds with a terse output. Thus, test-checkstyle-diff is no longer necessary. Expand scope of checkstyle to all python files in the repo, replacing test-checkstyle with checkstyle-python. Also, fixstyle-python is now available for automatic style formatting. Note: python virtualenv has been consolidated in test/Makefile, test/requirements*.txt which will eventually be moved to a central location. This is required to simply the automated generation of docker executor images in the CI. Type: improvement Change-Id: I022a326603485f58585e879ac0f697fceefbc9c8 Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-04-29tests: fix handling failed test caseKlement Sekera1-1/+2
Add missing parameter where required. Type: fix Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I8cd7c31848836e3233cb79d1dd21884167db4354
2022-03-27tests: fix core file messageDmitry Valter1-3/+4
Prevent crashing on nonexistent VPP binary path class member when creating testsuite core message. Type: fix Fixes: b23ffd7ef216463c35b75c831e6a27e58971f4ec Signed-off-by: Dmitry Valter <d-valter@yandex-team.ru> Change-Id: Ib9b3dc8c69317e6561e5404bbdcbf672e417cbcd
2022-02-17tests: make tests less make dependentKlement Sekera1-71/+39
Implement command line argument parsing instead of passing arguments via environment variables. Add script for running tests without having to invoke make. Deprecate running tests via make. Type: improvement Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-04-16tests: cpus awarenessKlement Sekera1-94/+176
Introduce MAX_CPUS parameters to control maximum number of CPUs used by VPP(s) during testing, with default value 'auto' corresponding to all CPUs available. Calculate test CPU requirements by taking into account the number of workers, so a test requires 1 (main thread) + # of worker CPUs. When running tests, keep track of both running test jobs (controlled by TEST_JOBS parameter) and free CPUs. This then causes two limits in the system - to not exceed number of jobs in parallel but also to not exceed number of CPUs available. Skip tests which require more CPUs than are available in system (or more than MAX_CPUS) and print a warning message. Type: improvement Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-12tests: support attaching to existing vppKlement Sekera1-4/+9
Introduce a new option DEBUG=attach to run a test against existing already running vpp. A new target 'make test-start-gdb' will spawn VPP in gdb for this purpose. Customization options explained in test-help. Type: improvement Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-17tests: remove shm handlingKlement Sekera1-17/+4
Not used anymore with socket transport. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I7061048dee21ce02637e34f73fb9984bd241f7df
2021-02-10nat: fix EI hairpinning thread safetyKlement Sekera1-0/+2
Avoid doing inter-thread reads without locks by doing a handoff before destination address rewrite. Destination address is read from a session which is possibly owned by a different thread. By splitting the work in two parts with a handoff in the middle, we can do both in a thread safe way. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I1c50d188393a610f5564fa230c75771a8065f273
2021-01-22tests: add generalized tags for tests, use them for run-solo testsAndrew Yourtchenko1-5/+5
We have accumulated several scenarios in prod or wishlists where it would be useful to have a general infra to say yes/no about a certain test, and potentially make decisions based on that, for example: - runs solo (aka 'time-dependent') - (wishlist) part of quick smoke-test set - (wishlist) intermittent failure unrelated to timing - (wishlist) test broken with a multi-worker config in vpp Refactor the current "run-solo" code to allow for this extension. Type: test Change-Id: Ia5b3810e57c0543753c8e0dc4dc0cfb4a30b36ac Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-01-20tests: do not skip solo-run tests if running within a single job and all ↵Andrew Yourtchenko1-8/+7
remaining tests are solo-run There was a logic error, which results in not running solo tests if TEST_JOBS=1, and we have finished running the regular test and the remaining tests are all solo-run. They all get put into that pile, but the new runners never start. Solution: check the pile of solo-run jobs unconditionally. Thanks to Klement Sekera for spotting the issue. Type: fix Change-Id: Iaecacbe4de44050cd8d0ff2d524559df99f2bae4 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-08-27tests: "force solo" testcase supportAndrew Yourtchenko1-6/+46
Some of the tests are time-sensitive, and at present require a non-trivial modification in order to run at high concurrency. Without these modifications, they intermittently fail, and require the test retries. Rather than setting them to the extended tests and forgetting about them, put them into a "solo" set, which gets run in a single-threaded mode after the rest of the tests are done. Mark a few of the tests that showed errors during TEST_JOBS=48 as forced-solo. Also, give a better diagnostic if the testcase misses a docstring needed to represent it in the diagnostic outputs. Type: fix Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-01-16tests: fix TypeError in run_tests.pyPaul Vinciguerra1-4/+5
Traceback (most recent call last): File "run_tests.py", line 886, in <module> exit_code, suites = parse_results(results) File "run_tests.py", line 732, in parse_results results_per_suite.print_results() File "run_tests.py", line 683, in print_results result.get_testcase_names(failed_test_id) TypeError: 'NoneType' object is not iterable Type: test Change-Id: I2c5ae0b9e11be6bfb4490d9ce057db0b0a84c9bf Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-5/+0
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-11-05misc: Fix python scripts shebang lineRenato Botelho do Couto1-1/+1
Type: fix Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python' as a valid shebang line. It requires scripts to explicitly chose between python2 or python3. Change all to use python3 as suggested by Paul Vinciguerra. Depends-On: https://gerrit.fd.io/r/23170 Signed-off-by: Renato Botelho do Couto <renato@netgate.com> Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
2019-11-05tests: support setting random seedKlement Sekera1-0/+5
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-11-05tests: avoid test runner hanging on child test process joinAndrew Yourtchenko1-1/+13
In parallel test, the single process is spawning a bunch of child processes running the tests, and communicates to them. When the child process signals that it has finished, the parent calls child.join(). Sometimes this join never returns. The result is a lot of defunct python processes, and the test run just hangs. I have seen this failure intermittently a fair bit in a busy containerized environment, and by chance, consistently reproduced it on a Thinkpad X280 with 8G of RAM and Ubuntu 19.04, which allowed to diagnose it. Type: test Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: If0a3110fc2d23e73d77c310d61c3ea90a2b53610
2019-10-29tests: fix TEST= filtering for python3Klement Sekera1-1/+1
Type: fix Change-Id: Iceb60e8a507120434cfc68758476ff5351b05f58 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-10-21classify: tests to support python3Ole Troan1-1/+1
Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I8af6ee6e5239f0836978baa063a18a01c610279f
2019-07-24tests: add cpu/shm info to test runner outputPaul Vinciguerra1-1/+13
Type: test Depends-on: https://gerrit.fd.io/r/#/c/20484/ Change-Id: I140132cfcc4347035fe2bb9919f8e7923342940f Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09tests: treat all truthy env vars the same wayPaul Vinciguerra1-5/+4
Introduce a new class, that returns the truthiness of a env var. Since an environment variable is just a string, it would normally be true if not unset. The new class returns true when the env var is set to a string that would be considered true. Type: test Depends-on: https://gerrit.fd.io/r/20484 Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-03tests: fix FORCE_FOREGROUNDjuraj.linkes1-1/+7
Type: fix Change-Id: I145f9984c1d944d5b7de0487e8b41ae4fdb95b6d Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-05-08make test: introduce COREDUMP_COMPRESS optionKlement Sekera1-13/+16
setting COREDUMP_COMPRESS=y (or =1) indicates that any core file is to be compressed by the test framework unless DEBUG=core is set. Change-Id: Ib0d6d30a7398dbeb6ab5e5dbb411bf5c641b5206 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-03-18tests: Speedup tests by 30%.Paul Vinciguerra1-0/+1
Baseline: 2598325153 function calls (2598296959 primitive calls) in 2257.698 seconds Post-changeset: 1495427 function calls (1467233 primitive calls) in 1572.080 seconds Change-Id: I191b68b29c9e0f19964bf0b8879ddf357c9cbd5c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07make-test: Python pep8 compliance for sequencesNaveen Joy1-13/+13
For sequences, (strings, lists, tuples), use the fact that empty sequences are false. A tiny change towards getting full pep8 compliance, which will look good. Yes: if not seq: if seq: No: if len(seq): if not len(seq): (https://www.python.org/dev/peps/pep-0008/) Change-Id: I6b565bef424f81afc66aba3b0041a5029a9f8954 Signed-off-by: Naveen Joy <najoy@cisco.com>
2019-02-26make test: Add exception handling around subprocess.Paul Vinciguerra1-3/+13
This cleans up exception catching to identify oserrors. By raising the specific exception closer to the offending call, we get additional stack history and can add clearer error logging to assist in troubleshooting. Change-Id: I592e4d46844b822a816485bf54910f8daed92088 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-11Fix interactive testsjuraj.linkes1-2/+4
When running tests interactively (i.e. not using child process) the tests are split into small suites each containing tests from a VPPTestCase, but only the first suite is run. Fix this by creatin a new suite with all of these small suites in it and running that suite. Change-Id: I0236f4a980d01765eb993f41d54d98640bb2cd39 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-12-20Tests: Cleanup @skip decorator.Paul Vinciguerra1-1/+1
The runnning environment is static as of module load time, so only evalute the conditions once at module load time. Track-by: VPP-1518 Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-14Fix test resultsjuraj.linkes1-19/+23
In a rare case a failure can occur in a VppTestCase which hasn't been part of the original TestSuite, such as in memif testcases. Fix the reporting after test run in these cases. Change-Id: Ib2a34fc195179605c22b95bd0e2c7cebca17012f Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-12-10Test framework: StringIO fixes for Python3Ole Troan1-1/+5
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-07Test reader thread hardeningjuraj.linkes1-103/+111
The test reader thread could get stuck in case an unexpected Exception was raised during test runs in parent thread. Catch the Exception, close the thread and raise the Exception to avoid this. Change-Id: I3a9298d593c334d96d04b6207d604b684572a2ac Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-30make test: print TEST= values for failed testsKlement Sekera1-4/+4
This prints the format strings required to rerun the failed test next to the failures. Change-Id: I973d8ae025f026129826356bce265b1e70086d2f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-29make test: create virtualenv under /test/Klement Sekera1-4/+5
instead of using build-root, use /test/venv directory for virtualenv similarly, don't pollute build-root with test-built binaries Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-26Split non-parallel testsuitejuraj.linkes1-19/+8
Split one big suite into smaller suites when not running tests in parallel. This results in all tests being executed in one iteration. Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-22Revert "Fix instance method call in test/framework.py"Ole Trøan1-2/+6
This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968. Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71 Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-22Fix instance method call in test/framework.pyPaul Vinciguerra1-6/+2
Instance method breaking encapsulation by calling a global fn. Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-09Fix logging into multiple filesjuraj.linkes1-1/+1
When running tests in one process, only one logger was used and each testcase added its own file handler, which resulted in logs appearing in multiple files. Fix this by restoring the creation of new loggers for each testcase and only reuse the stream handler from parent process. Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-10-10Setup, teardown, DEBUG=core, FAILFAST fixesjuraj.linkes1-130/+227
- Fixed a bug when an error occuring in tearDownClass would not result in test being marked as failed - Improved test results reporting in cases when an error occurs in setUpClass and tearDownClass - Fixed DEBUG=core when the core is produced in setUpClass or tearDownClass - Reworked DEBUG=core to always be handled after all tests have been executed - Fixed FAILFAST=1 for parallel test runs Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-10-03test framework: add shell-style wildcard matching to filenamesAndrew Yourtchenko1-2/+5
Currently, one can either use "*" to denote all filenames, or to give the specific file name to run the tests in. This commit adds the possibility to run all tests matching the shell wildcard, e.g. TEST="test_acl_plugin*" will execute all ACL plugin testcases. Change-Id: I9048a601958947c7b757c3dfd57b19cdd8a1e3c0 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2018-09-20VPP-1421: Reworked results gatheringjuraj.linkes1-150/+170
The previous version sent the whole VPPTestResult object through pipe, which uses pickle to transfer objects. Pickle does not support sending any arbitrary objects and was causing issues. Now just a tuple of (test_id, result) is sent. Change-Id: I3a3a9e6f1b9ac9b05889babfc1f7560c7ac4471c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-18fix TEST_JOBS=auto max jobs detectionKlement Sekera1-1/+1
Change-Id: I63f747e92d239e878550392df24fb7aac2b5a3e3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-09-06Fix test summary and retriesjuraj.linkes1-22/+35
There was an issue when tests crashed and weren't properly retried. Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-30Fix hanging test runner when child process diesjuraj.linkes1-50/+59
When fixing the test summary after a test run doesn't finish properly I introduced a bug where child process which died would leave the whole run hanging. This patch fixed the bug while still having the correct test summary. Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-25Fix logging to file and summary for crashed testsjuraj.linkes1-84/+59
Logging was not being logged to log.txt Crashed tests were not reported properly when running tests in one process Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-23CSIT-1139: Implement parallel test executionjuraj.linkes1-158/+631
The implementation of parallel test execution in VPP Test Framework. - VPPTestCase test methods are grouped together - tests are running in separate processes - VPP instances spawned by tests are assigned to different cores - output from these processes is redirected through pipes and printed out testcase by testcase - TEST_JOBS env var is used to specify the number of parallel processes - improved test summary - a bit of code cleanup Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-06-27make test: easy profiling via make test PROFILE=1Klement Sekera1-1/+7
Change-Id: Ib845578485f523b7f14e98c83d05f78db382ecde Signed-off-by: Klement Sekera <ksekera@cisco.com>