summaryrefslogtreecommitdiffstats
path: root/test/run_tests.py
AgeCommit message (Collapse)AuthorFilesLines
2021-01-22tests: add generalized tags for tests, use them for run-solo testsAndrew Yourtchenko1-5/+5
We have accumulated several scenarios in prod or wishlists where it would be useful to have a general infra to say yes/no about a certain test, and potentially make decisions based on that, for example: - runs solo (aka 'time-dependent') - (wishlist) part of quick smoke-test set - (wishlist) intermittent failure unrelated to timing - (wishlist) test broken with a multi-worker config in vpp Refactor the current "run-solo" code to allow for this extension. Type: test Change-Id: Ia5b3810e57c0543753c8e0dc4dc0cfb4a30b36ac Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-01-20tests: do not skip solo-run tests if running within a single job and all ↵Andrew Yourtchenko1-8/+7
remaining tests are solo-run There was a logic error, which results in not running solo tests if TEST_JOBS=1, and we have finished running the regular test and the remaining tests are all solo-run. They all get put into that pile, but the new runners never start. Solution: check the pile of solo-run jobs unconditionally. Thanks to Klement Sekera for spotting the issue. Type: fix Change-Id: Iaecacbe4de44050cd8d0ff2d524559df99f2bae4 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-08-27tests: "force solo" testcase supportAndrew Yourtchenko1-6/+46
Some of the tests are time-sensitive, and at present require a non-trivial modification in order to run at high concurrency. Without these modifications, they intermittently fail, and require the test retries. Rather than setting them to the extended tests and forgetting about them, put them into a "solo" set, which gets run in a single-threaded mode after the rest of the tests are done. Mark a few of the tests that showed errors during TEST_JOBS=48 as forced-solo. Also, give a better diagnostic if the testcase misses a docstring needed to represent it in the diagnostic outputs. Type: fix Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-01-16tests: fix TypeError in run_tests.pyPaul Vinciguerra1-4/+5
Traceback (most recent call last): File "run_tests.py", line 886, in <module> exit_code, suites = parse_results(results) File "run_tests.py", line 732, in parse_results results_per_suite.print_results() File "run_tests.py", line 683, in print_results result.get_testcase_names(failed_test_id) TypeError: 'NoneType' object is not iterable Type: test Change-Id: I2c5ae0b9e11be6bfb4490d9ce057db0b0a84c9bf Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-5/+0
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-11-05misc: Fix python scripts shebang lineRenato Botelho do Couto1-1/+1
Type: fix Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python' as a valid shebang line. It requires scripts to explicitly chose between python2 or python3. Change all to use python3 as suggested by Paul Vinciguerra. Depends-On: https://gerrit.fd.io/r/23170 Signed-off-by: Renato Botelho do Couto <renato@netgate.com> Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
2019-11-05tests: support setting random seedKlement Sekera1-0/+5
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-11-05tests: avoid test runner hanging on child test process joinAndrew Yourtchenko1-1/+13
In parallel test, the single process is spawning a bunch of child processes running the tests, and communicates to them. When the child process signals that it has finished, the parent calls child.join(). Sometimes this join never returns. The result is a lot of defunct python processes, and the test run just hangs. I have seen this failure intermittently a fair bit in a busy containerized environment, and by chance, consistently reproduced it on a Thinkpad X280 with 8G of RAM and Ubuntu 19.04, which allowed to diagnose it. Type: test Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: If0a3110fc2d23e73d77c310d61c3ea90a2b53610
2019-10-29tests: fix TEST= filtering for python3Klement Sekera1-1/+1
Type: fix Change-Id: Iceb60e8a507120434cfc68758476ff5351b05f58 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-10-21classify: tests to support python3Ole Troan1-1/+1
Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I8af6ee6e5239f0836978baa063a18a01c610279f
2019-07-24tests: add cpu/shm info to test runner outputPaul Vinciguerra1-1/+13
Type: test Depends-on: https://gerrit.fd.io/r/#/c/20484/ Change-Id: I140132cfcc4347035fe2bb9919f8e7923342940f Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09tests: treat all truthy env vars the same wayPaul Vinciguerra1-5/+4
Introduce a new class, that returns the truthiness of a env var. Since an environment variable is just a string, it would normally be true if not unset. The new class returns true when the env var is set to a string that would be considered true. Type: test Depends-on: https://gerrit.fd.io/r/20484 Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-03tests: fix FORCE_FOREGROUNDjuraj.linkes1-1/+7
Type: fix Change-Id: I145f9984c1d944d5b7de0487e8b41ae4fdb95b6d Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-05-08make test: introduce COREDUMP_COMPRESS optionKlement Sekera1-13/+16
setting COREDUMP_COMPRESS=y (or =1) indicates that any core file is to be compressed by the test framework unless DEBUG=core is set. Change-Id: Ib0d6d30a7398dbeb6ab5e5dbb411bf5c641b5206 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-03-18tests: Speedup tests by 30%.Paul Vinciguerra1-0/+1
Baseline: 2598325153 function calls (2598296959 primitive calls) in 2257.698 seconds Post-changeset: 1495427 function calls (1467233 primitive calls) in 1572.080 seconds Change-Id: I191b68b29c9e0f19964bf0b8879ddf357c9cbd5c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07make-test: Python pep8 compliance for sequencesNaveen Joy1-13/+13
For sequences, (strings, lists, tuples), use the fact that empty sequences are false. A tiny change towards getting full pep8 compliance, which will look good. Yes: if not seq: if seq: No: if len(seq): if not len(seq): (https://www.python.org/dev/peps/pep-0008/) Change-Id: I6b565bef424f81afc66aba3b0041a5029a9f8954 Signed-off-by: Naveen Joy <najoy@cisco.com>
2019-02-26make test: Add exception handling around subprocess.Paul Vinciguerra1-3/+13
This cleans up exception catching to identify oserrors. By raising the specific exception closer to the offending call, we get additional stack history and can add clearer error logging to assist in troubleshooting. Change-Id: I592e4d46844b822a816485bf54910f8daed92088 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-11Fix interactive testsjuraj.linkes1-2/+4
When running tests interactively (i.e. not using child process) the tests are split into small suites each containing tests from a VPPTestCase, but only the first suite is run. Fix this by creatin a new suite with all of these small suites in it and running that suite. Change-Id: I0236f4a980d01765eb993f41d54d98640bb2cd39 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-12-20Tests: Cleanup @skip decorator.Paul Vinciguerra1-1/+1
The runnning environment is static as of module load time, so only evalute the conditions once at module load time. Track-by: VPP-1518 Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-14Fix test resultsjuraj.linkes1-19/+23
In a rare case a failure can occur in a VppTestCase which hasn't been part of the original TestSuite, such as in memif testcases. Fix the reporting after test run in these cases. Change-Id: Ib2a34fc195179605c22b95bd0e2c7cebca17012f Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-12-10Test framework: StringIO fixes for Python3Ole Troan1-1/+5
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-07Test reader thread hardeningjuraj.linkes1-103/+111
The test reader thread could get stuck in case an unexpected Exception was raised during test runs in parent thread. Catch the Exception, close the thread and raise the Exception to avoid this. Change-Id: I3a9298d593c334d96d04b6207d604b684572a2ac Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-30make test: print TEST= values for failed testsKlement Sekera1-4/+4
This prints the format strings required to rerun the failed test next to the failures. Change-Id: I973d8ae025f026129826356bce265b1e70086d2f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-29make test: create virtualenv under /test/Klement Sekera1-4/+5
instead of using build-root, use /test/venv directory for virtualenv similarly, don't pollute build-root with test-built binaries Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-26Split non-parallel testsuitejuraj.linkes1-19/+8
Split one big suite into smaller suites when not running tests in parallel. This results in all tests being executed in one iteration. Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-22Revert "Fix instance method call in test/framework.py"Ole Trøan1-2/+6
This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968. Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71 Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-22Fix instance method call in test/framework.pyPaul Vinciguerra1-6/+2
Instance method breaking encapsulation by calling a global fn. Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-09Fix logging into multiple filesjuraj.linkes1-1/+1
When running tests in one process, only one logger was used and each testcase added its own file handler, which resulted in logs appearing in multiple files. Fix this by restoring the creation of new loggers for each testcase and only reuse the stream handler from parent process. Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-10-10Setup, teardown, DEBUG=core, FAILFAST fixesjuraj.linkes1-130/+227
- Fixed a bug when an error occuring in tearDownClass would not result in test being marked as failed - Improved test results reporting in cases when an error occurs in setUpClass and tearDownClass - Fixed DEBUG=core when the core is produced in setUpClass or tearDownClass - Reworked DEBUG=core to always be handled after all tests have been executed - Fixed FAILFAST=1 for parallel test runs Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-10-03test framework: add shell-style wildcard matching to filenamesAndrew Yourtchenko1-2/+5
Currently, one can either use "*" to denote all filenames, or to give the specific file name to run the tests in. This commit adds the possibility to run all tests matching the shell wildcard, e.g. TEST="test_acl_plugin*" will execute all ACL plugin testcases. Change-Id: I9048a601958947c7b757c3dfd57b19cdd8a1e3c0 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2018-09-20VPP-1421: Reworked results gatheringjuraj.linkes1-150/+170
The previous version sent the whole VPPTestResult object through pipe, which uses pickle to transfer objects. Pickle does not support sending any arbitrary objects and was causing issues. Now just a tuple of (test_id, result) is sent. Change-Id: I3a3a9e6f1b9ac9b05889babfc1f7560c7ac4471c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-18fix TEST_JOBS=auto max jobs detectionKlement Sekera1-1/+1
Change-Id: I63f747e92d239e878550392df24fb7aac2b5a3e3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-09-06Fix test summary and retriesjuraj.linkes1-22/+35
There was an issue when tests crashed and weren't properly retried. Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-30Fix hanging test runner when child process diesjuraj.linkes1-50/+59
When fixing the test summary after a test run doesn't finish properly I introduced a bug where child process which died would leave the whole run hanging. This patch fixed the bug while still having the correct test summary. Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-25Fix logging to file and summary for crashed testsjuraj.linkes1-84/+59
Logging was not being logged to log.txt Crashed tests were not reported properly when running tests in one process Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-23CSIT-1139: Implement parallel test executionjuraj.linkes1-158/+631
The implementation of parallel test execution in VPP Test Framework. - VPPTestCase test methods are grouped together - tests are running in separate processes - VPP instances spawned by tests are assigned to different cores - output from these processes is redirected through pipes and printed out testcase by testcase - TEST_JOBS env var is used to specify the number of parallel processes - improved test summary - a bit of code cleanup Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-06-27make test: easy profiling via make test PROFILE=1Klement Sekera1-1/+7
Change-Id: Ib845578485f523b7f14e98c83d05f78db382ecde Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-25make test: fix RETRIES when setUpClass throwsKlement Sekera1-1/+4
This change adds handling of special case when setUpClass throws. In this case TestResults receives a mock object called _ErrorHolder. By parsing its description, we find test class name and use it to lookup the test class in test suite to be able to add it to the list of failures for re-running. Change-Id: I656f21e38aa450fc567cdcbcf6e586967f947a64 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-31make test: print a warning in case a core_pattern contains a filter programAndrew Yourtchenko1-0/+2
The default config on Ubuntu 16.04.4 desktop results in truncated cores when running make test which coredumps. Uninstalling the filter program (apport) makes the corefiles normal size. Print a warning about that fact, so the others potentially affected didn't have to wonder. Change-Id: Iba4b0a2765a25100d6e24fd7f4de0e0339efd835 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2018-03-24make test: enhance core-file informationKlement Sekera1-15/+20
Change-Id: I1283960a9a49f6d70b9d7b7793cfb345c22ccdea Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-23make test: code cleanupKlement Sekera1-5/+8
Change-Id: Ic689de569e5b6e6209d16d6acdb13c489daca1f5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-14make test: early core detection, code cleanupKlement Sekera1-2/+24
Change-Id: I503d69f902f1c8d3107fb3f2b9f31b0083ebb747 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-14make test: handle multiple failures in one caseKlement Sekera1-1/+4
Change-Id: I5451ddec4bfdede12653415f90bf991e81494033 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-17make test: detect child crashKlement Sekera1-7/+17
This change causes parent process to detect child crash in seconds instead of waiting for timeout to pass. Change-Id: Ib45e86b7fc97e687d99c554be4212aaeea781dcc Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-10make test: add RETRIES optionKlement Sekera1-8/+53
Change-Id: Ibe31e932bc997f0101a8947e01df90a90d1f100f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-03make test: Copy api_post_mortem.$$ file tmp test dir for archiving.Dave Wallace1-2/+8
Change-Id: I4baf89ef383dbc2f309081a6b56b13ebcb8fc2df Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-30make test: Create link to failed test dir on timeout.Dave Wallace1-0/+6
- Also change default coredump configuration from "coredump-size unlimited" to "full-coredump" Change-Id: Iefedc2636f2d9696b7575b34e91dd7be49f601fa Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-08-25make test: separate test discovery codeKlement Sekera1-34/+11
Separating test discovery code to it's own script file has the advantage of easily doing e.g. listing of all existing tests. Change-Id: I80dc280263cc7e33e7e13cb0d48b39bf08ece24d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-12make test: fix broken DEBUG=gdb* optionsKlement Sekera1-42/+50
Change-Id: I5d80982eeab78a629760f567eda3b1539d96e3a8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-10make test: bump default test timeout to 10 minutesKlement Sekera1-1/+1
Change-Id: I25d88966376d712ff61f29227a45880a59e8ecf2 Signed-off-by: Klement Sekera <ksekera@cisco.com>