summaryrefslogtreecommitdiffstats
path: root/test/Makefile
AgeCommit message (Collapse)AuthorFilesLines
2018-12-18VPP-1508: Have test/Makefile honor PYTHON= envvar.Paul Vinciguerra1-8/+8
Change-Id: Iff71a5982b73c97e3f546beb1cf52935a267f5f5 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-06Skip failing ARM testcases for CIjuraj.linkes1-0/+2
There are a few tests that fail on ARM and thus we can't enable testing in CI. Skip the failing tests until they're fixed so that we catch new failures in CI (when we enable testing for the remaining tests). Change-Id: Ie896ef5c449ef965029633e38d317a8d5ac26352 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-29make test: create virtualenv under /test/Klement Sekera1-45/+47
instead of using build-root, use /test/venv directory for virtualenv similarly, don't pollute build-root with test-built binaries Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-26VPP-1508 Add support for environment markers.Paul Vinciguerra1-1/+1
Add the ability to specify a specific python library version based on the interpreter/platform/etc. Change-Id: I027acdf22ad839b5cff63b319f0aa100b0f336c8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-24VPP-1505 VPP-1508 Remove dep.: py_lispnetworkingPaul Vinciguerra1-1/+1
test_lisp.py No longer needs the external dependency to run. py27 runtests: commands[6] | stestr --test-path ./test run --slowest test_lisp ============================================================================== Basic LISP test ============================================================================== {0} test.test_lisp.TestLisp.test_lisp_basic_encap [0.487960s] ... ok ====== Totals ====== Ran: 1 tests in 18.0000 sec. - Passed: 1 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 0.4880 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:00:00.487960 Test id Runtime (s) --------------------------------------------- ----------- test.test_lisp.TestLisp.test_lisp_basic_encap 0.488 ______________________________________________________ summary ______________________________________________________ py27: commands succeeded congratulations :) vagrant@vpp:/vpp$ Change-Id: I35b35f8ddaed79066b3ed62fa0460c9ea83273c2 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-23make test: remove test-ext, fix test*-all targetsKlement Sekera1-5/+4
build extended binaries, vom & friends when test*-all targets are run, remove unneeded test-ext target (which is semantically same as test-debug-all) Change-Id: Id710e061fdc93e286a4ed91be6a0cdcf75a2dafd Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-22Add RFC5424 syslog protocol support (VPP-1139)Matus Fabian1-1/+1
Syslog protocol logging transport event messages across network over UDP protocol based on RFC5426. Change-Id: Ica74b40bcc2e6d0fbd41e9bf78e76395fbabab3c Signed-off-by: Matus Fabian <matfabia@cisco.com>
2018-08-23CSIT-1139: Implement parallel test executionjuraj.linkes1-5/+29
The implementation of parallel test execution in VPP Test Framework. - VPPTestCase test methods are grouped together - tests are running in separate processes - VPP instances spawned by tests are assigned to different cores - output from these processes is redirected through pipes and printed out testcase by testcase - TEST_JOBS env var is used to specify the number of parallel processes - improved test summary - a bit of code cleanup Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-17Add psutil to testing virtualenvjuraj.linkes1-1/+1
Change-Id: I650c9f226bcdc6328a7b4ff2dc4bd20cea8a512a Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-07-22make test-ext: a new test target that builds VOM (existing one no longer do)Neale Ranns1-1/+4
Change-Id: If590e90b2a1c3b884863ec15a92ea4353ac9eb7c Signed-off-by: Neale Ranns <nranns@cisco.com>
2018-06-28make test: fix profilingKlement Sekera1-1/+1
Change-Id: Iad45ca8c167a017029b20ddd0a0b59087fa69498 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-27make test: easy profiling via make test PROFILE=1Klement Sekera1-1/+18
Change-Id: Ib845578485f523b7f14e98c83d05f78db382ecde Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-05-02Scapy; update to release 2.4.0Neale Ranns1-1/+1
Change-Id: Ibcae49ab106efa9cd0ff60be60904ac5a2fc0b65 Signed-off-by: Neale Ranns <neale.ranns@cisco.com>
2018-04-10test: Fix issues with new version of pycodestyle (VPP-1232)Chris Luke1-1/+1
- Ignore warnings W504 (newline after binary operator) which otherwise occurs a significant number of times. - Fix two instances of lines >79 chars. Change-Id: I8cef56f8afc237187995e638e610c8c0554e2bb5 Signed-off-by: Chris Luke <chrisy@flirble.org>
2018-04-03Makefile: Alter VPP_PYTHON_PREFIX for preloading depsEd Kern1-0/+2
Allow setting of VPP_PYTHON_PREFIX to alternate location so the python prereqs can be installed into base image Also added test-dep trigger to isolate dependency install from actual test run Change-Id: Ia80f5dbf71bc24eb46cd6586bcadd474ef822704 Signed-off-by: Ed Kern <ejk@cisco.com>
2018-03-24make test: use vpp-dev repo for py-lispnetworkingKlement Sekera1-1/+1
Change-Id: Ia87d55cad1d466a4cc624f06abbc322cddb5608c Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-19Scapy upgrade to 2.4.0.rc5Neale Ranns1-2/+2
- many of the patches fd.io applies in test/patches/2.3.3 are now upstreamed in 2.4 - 2.4 adds support for IGMPv3 which is my main motivation for the upgrade Change-Id: If2c0a524e3cba320b4a5d8cd07817c6ea2bf0c5a Signed-off-by: Neale Ranns <nranns@cisco.com>
2018-03-15IPIP: Linter on test_*.pyOle Troan1-1/+1
Change-Id: Icd491e382a2da0e5d55a673458605bc37e39b669 Signed-off-by: Ole Troan <ot@cisco.com>
2018-03-07test: disable reassembly tests and system modification scriptDamjan Marion1-1/+0
One of basic requirements from day one for maek test is that it should run unpriviledged, so modifying system paremeters is unacceptable. Disabling script and reassembly tests as they need to be done in different way. Change-Id: I063c73eb82f2ebd7499e8adb1574a9fd2475fb5b Signed-off-by: Damjan Marion <damarion@cisco.com>
2018-03-02make test: add "EXTENDED_TESTS" to help output.Dave Wallace1-0/+1
Change-Id: I89540e93f3c8839bf6fc4363f4736e9c01d91927 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2018-03-016RD: Rewritten 6RD RFC5969 support.Ole Troan1-1/+1
Change-Id: Ic30fbcb2630f39e45345d7215babf5d7ed4b33a0 Signed-off-by: Ole Troan <ot@cisco.com>
2018-02-26Python: addressing PEP8 deprecationMarco Varlese1-2/+2
PEP8 has been deprecated and python users are being asked to migrate to pycodestyle. Change-Id: I52d5f7b2bf72156216a9966e8322ec58763f24d4 Signed-off-by: Marco Varlese <marco.varlese@suse.com>
2018-02-22make test: build test/ext when doing coverageKlement Sekera1-1/+1
This makes VAPI/VOM tests part of code coverage report. Change-Id: I66511b0a59880c2a64daecd37088268bdf5e1269 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-17make test: add test framework debuggingKlement Sekera1-1/+10
New option TEST_DEBUG=1 turns on test framework debugging, which currently consists of printing difference in allocated objects/memory and also creates reference graphs for any unfreed VppPapiProvider/VPP objects - these take a lot of memory and thus should be freed regularly. Change-Id: I29db0c1341009d4b5c5df9222d14f3095883fd0f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-06BIER: fix support for longer bit-string lengthsNeale Ranns1-1/+1
Change-Id: I2421197b76be58099e5f8ed5554410adff202109 Signed-off-by: Neale Ranns <neale.ranns@cisco.com>
2018-02-01IPv4/6 reassemblyKlement Sekera1-1/+2
Change-Id: Ic5dcadd13c88b8a5e7896dab82404509c081614a Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-01-25make test: verify that /dev/shm is big enoughKlement Sekera1-0/+1
Verify that /dev/shm size is >= 512M, which should be enough for `make test' needs. If the verification fails, try to enlarge it automatically. This helps avoid docker vpp/make test crashes (seen as SIGBUS). Change-Id: I4e23d59ff8bf8befc320fa229fb6c9bfeb809a8f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-12-15ESP_AH_test_automation_scripts rev1“mystarrocks”1-1/+1
Change-Id: I941abdc4a02e4c52c66b9d299e380b27caca7c1d Signed-off-by: “mystarrocks” <mystarrocks@yahoo.com>
2017-11-06make test: add CACHE_OUTPUT optionKlement Sekera1-0/+1
This option allows one to toggle whether captured VPP stdout/stderr output is printed inline (each line logged immediately after being collected). Default setting is 0/no (behaviour same as before). This allows one to see the output in case of vpp crashing during test run even if test framework gets stuck waiting for shared memory mutex (which crashed vpp can no longer unlock). Complete stdout/stderr output is always printed as a block to the logger after test finishes. Change-Id: If3645a9b85562dba26ed9c87daa27ea5ef34f2e5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-06make test: fix DEBUG=core errorKlement Sekera1-0/+2
Change-Id: I703b2866a1607d6a2fad215e90b5cf2d7afdfd0d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-19make test: control sanity via SANITY=yes|no optionKlement Sekera1-2/+11
Allow skipping sanity when using DEBUG=gdb/gdbserver options to speed up development process by supplying SANITY=no option to make test. Change-Id: I6e619229f6866b22bdeb75654b4e750232ae22bd Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-10make test: add RETRIES optionKlement Sekera1-0/+1
Change-Id: Ibe31e932bc997f0101a8947e01df90a90d1f100f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-09-30make test: archive failed test data with build logs.Dave Wallace1-2/+1
- Fix invocation of compress_failed.sh - Fix compress_failed to copy compressed results files to $WORKSPACE/archives and return failure exit code. Failed test case data will be copied to logs.fd.io and found in the archives/<make test data dir>-FAILED directory in the build log link in the vpp-verify-master-ubuntu1604 jenkins job page. For example: https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1604/7353/archives/ Change-Id: Ife9a0737115e69c0a8441e3bb0133af1528d909b Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-27make test: clean ext binaries when doing test-wipeKlement Sekera1-0/+1
Change-Id: I9f5212ee670ea91c6b35f1406c256d0687b9c6b5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-09-19Add new C APIKlement Sekera1-1/+5
Change-Id: I717ce3cd7c867c155de149ec56623269d26d0ff7 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-17make test: collect symlinks to failed testsKlement Sekera1-1/+6
Compress files in temporary directories of failed tests and symlink the directories under /tmp/vpp-failed-unittests location - preparation for jenkins archivation. Automatically cleanup the directory at start of test run. The compression is performed only when environment variable COMPRESS_FAILED_TEST_LOGS is set to one of "yes", "y", "1". This is set in verify target, but left unset by default, so when invoking make test by hand, files won't be compressed. Change-Id: I84c8f1c6aa79aa9c0b753357022b1f195f17a283 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-14make test: properly handle ctrl-cKlement Sekera1-1/+9
Change-Id: Iab88886ebc1582626813777ea45ce97fc8e36443 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-10make test: detect hung testsKlement Sekera1-0/+1
Run tests in a forked process with a set of pipes to communicate keep-alives and overall result. This allows us to detect when e.g. vpp dies mid-API call causing the test to hang waiting for response (which will never come since vpp died). Support setting a (per test case) TIMEOUT make test option to set timeout, with a default timeout of 120 seconds. Example - fail the test suite if any test-case fails to finish within 300s: make TIMEOUT=300 test Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-07make test: kill all remaining subprocesses on exitKlement Sekera1-1/+1
This change introduces a wrapper script which kills all processes in the same process group as itself (with the exception of the script). Using this script to run the unit tests should prevent stale processes left behind in some cases (e.g. when test framework crashes). Change-Id: If3b9201c06b87fa6be095721436893207d09b5e4 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-06-07make test: improve debugabilityKlement Sekera1-1/+1
Introduce faulthandler to print stack trace to stderr on python crash. Don't disable automatic garbage collection to decrease the chances of running out of memory. Change-Id: I6927a5f6ea9569735d084d4ed3d258950a400d74 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-06-01make test: add suddenly missing dependency (six module)Klement Sekera1-1/+1
Change-Id: I121eaee6b752a6b3bfe1d5ecf803f67147c77beb Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-05-11make test: support coverage for out-of-tree pluginsKlement Sekera1-0/+6
Change-Id: Ibd7828b1e1c699630ff450122d8aa317b1c9da26 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-19make test: python interpreter customizationvagrant1-6/+18
Allow using custom python interpreter via 'PYTHON' env variable. E.g. env PYTHON=pypy make test. Get latest pip automatically to properly function in environments, where old pip is available and a proxy is required. This allows testing pypy in a centos vagrant-provided vm. Change-Id: I67a658fc927303468cc67f0ac192317ca2907625 Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Damjan Marion <damarion@cisco.com>
2017-04-10make test: automatic "vpp finishes startup" checkKlement Sekera1-0/+5
Add code which checks if vpp doesn't crash/exit immediately after startup to aid debugging stuff like mistyped graph node name or so. Refuse to run tests if the vpp is unable to start, complain loudly and print vpp's stderr at critical log level if this happens to make spotting these problems in jenkins easy. Change-Id: I40d3fbd05c822c0534713bae6bef05ecfb0e0c1d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-22make test: support out-of-tree pluginsKlement Sekera1-0/+4
usage: env EXTERN_PLUGINS=/path/to/plugins make test Change-Id: I8eece726dfafeff1cffd921c1e18cd3eb7eb64ed Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-22make test: support out-of-tree testsKlement Sekera1-3/+8
env EXTERN_TESTS="/path/to/extra/tests" make test causes to run the default test set and tests collected from test_*.py files under subtree specified in EXTERN_TESTS. Change-Id: I58c5471dd6010730278a5b47d4318737d920bc28 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-15Python API: Change from cPython to CFFI.Ole Troan1-1/+1
Change-Id: I03e52466fb3f909ae52b8fba601168f3eadbd972 Signed-off-by: Ole Troan <ot@cisco.com>
2017-03-09make test: automatic sanity checkKlement Sekera1-2/+14
Check if vpp_papi is importable before running the tests to avoid confusing python crashes. Change-Id: I6adf406e353bf381d590f2ef988a1ea79b95cf37 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-09make test: add make test-shell[-debug] targetsKlement Sekera1-0/+13
This starts a bash with the same environment as the test framework uses, allowing easy debugging. Change-Id: I956deda913b73dae5b1e1976417834ae4731f88a Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-08make test: split into basic and extended testsKlement Sekera1-2/+4
Implement plumbing to allow decorating tests as extended, e.g.: @unittest.skipUnless(running_extended_tests(), "part of extended tests") both methods and classes can be decorated this way. Change make test and make test-debug to run only non-extended tests. Introduce make test-all and make test-all-debug to run the full suite. Run full suite as part of make verify. Decorate most BFD tests as extended. Change-Id: I3bc64f59e9fe238f7f767d7e043dc165d03e9dfa Signed-off-by: Klement Sekera <ksekera@cisco.com>