summaryrefslogtreecommitdiffstats
path: root/test/Makefile
AgeCommit message (Collapse)AuthorFilesLines
2018-01-25make test: verify that /dev/shm is big enoughKlement Sekera1-0/+1
Verify that /dev/shm size is >= 512M, which should be enough for `make test' needs. If the verification fails, try to enlarge it automatically. This helps avoid docker vpp/make test crashes (seen as SIGBUS). Change-Id: I4e23d59ff8bf8befc320fa229fb6c9bfeb809a8f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-12-15ESP_AH_test_automation_scripts rev1“mystarrocks”1-1/+1
Change-Id: I941abdc4a02e4c52c66b9d299e380b27caca7c1d Signed-off-by: “mystarrocks” <mystarrocks@yahoo.com>
2017-11-06make test: add CACHE_OUTPUT optionKlement Sekera1-0/+1
This option allows one to toggle whether captured VPP stdout/stderr output is printed inline (each line logged immediately after being collected). Default setting is 0/no (behaviour same as before). This allows one to see the output in case of vpp crashing during test run even if test framework gets stuck waiting for shared memory mutex (which crashed vpp can no longer unlock). Complete stdout/stderr output is always printed as a block to the logger after test finishes. Change-Id: If3645a9b85562dba26ed9c87daa27ea5ef34f2e5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-06make test: fix DEBUG=core errorKlement Sekera1-0/+2
Change-Id: I703b2866a1607d6a2fad215e90b5cf2d7afdfd0d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-19make test: control sanity via SANITY=yes|no optionKlement Sekera1-2/+11
Allow skipping sanity when using DEBUG=gdb/gdbserver options to speed up development process by supplying SANITY=no option to make test. Change-Id: I6e619229f6866b22bdeb75654b4e750232ae22bd Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-10make test: add RETRIES optionKlement Sekera1-0/+1
Change-Id: Ibe31e932bc997f0101a8947e01df90a90d1f100f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-09-30make test: archive failed test data with build logs.Dave Wallace1-2/+1
- Fix invocation of compress_failed.sh - Fix compress_failed to copy compressed results files to $WORKSPACE/archives and return failure exit code. Failed test case data will be copied to logs.fd.io and found in the archives/<make test data dir>-FAILED directory in the build log link in the vpp-verify-master-ubuntu1604 jenkins job page. For example: https://logs.fd.io/production/vex-yul-rot-jenkins-1/vpp-verify-master-ubuntu1604/7353/archives/ Change-Id: Ife9a0737115e69c0a8441e3bb0133af1528d909b Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-27make test: clean ext binaries when doing test-wipeKlement Sekera1-0/+1
Change-Id: I9f5212ee670ea91c6b35f1406c256d0687b9c6b5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-09-19Add new C APIKlement Sekera1-1/+5
Change-Id: I717ce3cd7c867c155de149ec56623269d26d0ff7 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-17make test: collect symlinks to failed testsKlement Sekera1-1/+6
Compress files in temporary directories of failed tests and symlink the directories under /tmp/vpp-failed-unittests location - preparation for jenkins archivation. Automatically cleanup the directory at start of test run. The compression is performed only when environment variable COMPRESS_FAILED_TEST_LOGS is set to one of "yes", "y", "1". This is set in verify target, but left unset by default, so when invoking make test by hand, files won't be compressed. Change-Id: I84c8f1c6aa79aa9c0b753357022b1f195f17a283 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-14make test: properly handle ctrl-cKlement Sekera1-1/+9
Change-Id: Iab88886ebc1582626813777ea45ce97fc8e36443 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-10make test: detect hung testsKlement Sekera1-0/+1
Run tests in a forked process with a set of pipes to communicate keep-alives and overall result. This allows us to detect when e.g. vpp dies mid-API call causing the test to hang waiting for response (which will never come since vpp died). Support setting a (per test case) TIMEOUT make test option to set timeout, with a default timeout of 120 seconds. Example - fail the test suite if any test-case fails to finish within 300s: make TIMEOUT=300 test Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-07make test: kill all remaining subprocesses on exitKlement Sekera1-1/+1
This change introduces a wrapper script which kills all processes in the same process group as itself (with the exception of the script). Using this script to run the unit tests should prevent stale processes left behind in some cases (e.g. when test framework crashes). Change-Id: If3b9201c06b87fa6be095721436893207d09b5e4 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-06-07make test: improve debugabilityKlement Sekera1-1/+1
Introduce faulthandler to print stack trace to stderr on python crash. Don't disable automatic garbage collection to decrease the chances of running out of memory. Change-Id: I6927a5f6ea9569735d084d4ed3d258950a400d74 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-06-01make test: add suddenly missing dependency (six module)Klement Sekera1-1/+1
Change-Id: I121eaee6b752a6b3bfe1d5ecf803f67147c77beb Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-05-11make test: support coverage for out-of-tree pluginsKlement Sekera1-0/+6
Change-Id: Ibd7828b1e1c699630ff450122d8aa317b1c9da26 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-19make test: python interpreter customizationvagrant1-6/+18
Allow using custom python interpreter via 'PYTHON' env variable. E.g. env PYTHON=pypy make test. Get latest pip automatically to properly function in environments, where old pip is available and a proxy is required. This allows testing pypy in a centos vagrant-provided vm. Change-Id: I67a658fc927303468cc67f0ac192317ca2907625 Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Damjan Marion <damarion@cisco.com>
2017-04-10make test: automatic "vpp finishes startup" checkKlement Sekera1-0/+5
Add code which checks if vpp doesn't crash/exit immediately after startup to aid debugging stuff like mistyped graph node name or so. Refuse to run tests if the vpp is unable to start, complain loudly and print vpp's stderr at critical log level if this happens to make spotting these problems in jenkins easy. Change-Id: I40d3fbd05c822c0534713bae6bef05ecfb0e0c1d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-22make test: support out-of-tree pluginsKlement Sekera1-0/+4
usage: env EXTERN_PLUGINS=/path/to/plugins make test Change-Id: I8eece726dfafeff1cffd921c1e18cd3eb7eb64ed Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-22make test: support out-of-tree testsKlement Sekera1-3/+8
env EXTERN_TESTS="/path/to/extra/tests" make test causes to run the default test set and tests collected from test_*.py files under subtree specified in EXTERN_TESTS. Change-Id: I58c5471dd6010730278a5b47d4318737d920bc28 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-15Python API: Change from cPython to CFFI.Ole Troan1-1/+1
Change-Id: I03e52466fb3f909ae52b8fba601168f3eadbd972 Signed-off-by: Ole Troan <ot@cisco.com>
2017-03-09make test: automatic sanity checkKlement Sekera1-2/+14
Check if vpp_papi is importable before running the tests to avoid confusing python crashes. Change-Id: I6adf406e353bf381d590f2ef988a1ea79b95cf37 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-09make test: add make test-shell[-debug] targetsKlement Sekera1-0/+13
This starts a bash with the same environment as the test framework uses, allowing easy debugging. Change-Id: I956deda913b73dae5b1e1976417834ae4731f88a Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-08make test: split into basic and extended testsKlement Sekera1-2/+4
Implement plumbing to allow decorating tests as extended, e.g.: @unittest.skipUnless(running_extended_tests(), "part of extended tests") both methods and classes can be decorated this way. Change make test and make test-debug to run only non-extended tests. Introduce make test-all and make test-all-debug to run the full suite. Run full suite as part of make verify. Decorate most BFD tests as extended. Change-Id: I3bc64f59e9fe238f7f767d7e043dc165d03e9dfa Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-06make test: tell vpp to set coredump sizeKlement Sekera1-0/+3
Unless overridden by COREDUMP_SIZE env variable, tell VPP to set coredump size to unlimited, otherwise use $COREDUMP_SIZE as the argument. Change-Id: Ia2a6508207c66a171b33d272c820b1deb4a83e82 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-06make test: don't run if other vpp process runsKlement Sekera1-2/+22
Check if there are existing vpp processes before running the test suite and refuse to run if there are. This prevents the removal of other test suite temporary files and also makes sure that if the machine is loaded by (zombie) vpp processes, interactive tests (like bfd) won't fail. Change-Id: I88a74098188cb3f51966de5db19d7f80f39e51e2 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-21Add basic 4o4 LISP unit testFilip Tehlar1-1/+1
Change-Id: I2d812153d7afe7980346382b525af89b3c47e796 Signed-off-by: Filip Tehlar <ftehlar@cisco.com>
2017-02-14make test: improve stabilityKlement Sekera1-2/+9
Disable automatic garbage collection and run it manually before running each test case to minimize stalls. Improve vpp subprocess cleanup. Reduce helper thread count to one and properly clean that thread once it's not needed. Change-Id: I3ea78ed9628552b5ef3ff29cc7bcf2d3fc42f2c3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-06make test: improve test filteringKlement Sekera1-2/+10
Implement fine-grained test filtering by supporting more complicated filters beside the original file name suffix filter. Change-Id: If5a166d08cffe8c58cc6cf174e6df861c34dbaa6 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-06make test: fix dependenciesKlement Sekera1-2/+2
checkstyle - doesn't need scapy/pexpect, remove it doc - scapy wasn't patched properly, fix it Change-Id: I65202cb14edeb239d21ce10f17d9b4fccce43d62 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-05Test checksyle for IP mcast testsNeale Ranns1-1/+1
Change-Id: I20b61b7e63f74b4656d84717633e06646514c5eb Signed-off-by: Neale Ranns <nranns@cisco.com>
2017-01-27Fix make test scapy python patch issue, VPP-615Dave Wallace1-0/+1
Change-Id: I1e6798905f0049c793224a2c880f3b2cfadcf1e7 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-01-27Add sphinx read the doc theme.Tibor1-1/+1
Change-Id: Iade7c42081378bf211b81ad29b2898518442d2ff Signed-off-by: Tibor <tifrank@cisco.com>
2017-01-17make test: add checkstyle targetKlement Sekera1-0/+17
Change-Id: I59d3c3bc77474c96e1d6fa51811c1b13fb9a6c5b Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-01-11make test: improve documentation and PEP8 complianceKlement Sekera1-2/+2
Change-Id: Ib4f0353aab6112fcc3c3d8f0bcbed5bc4b567b9b Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-28test: fix paths after source code moveDamjan Marion1-1/+1
Change-Id: I3100260bb697f9af2f73ad3777e8b27069c5babe Signed-off-by: Damjan Marion <damarion@cisco.com>
2016-12-14make test: improve virtualenv patching processKlement Sekera1-10/+17
Change-Id: I5041bdc733c4da633d83f60f7db7747b4d0b8894 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-09Add make test code coverage reporting using gcovJuraj Sloboda1-4/+25
Change-Id: Ia8247841fdbe76e1d888aab49ae213b4216af273 Signed-off-by: Juraj Sloboda <jsloboda@cisco.com>
2016-11-22GRE tests and fixesNeale Ranns1-4/+26
Change-Id: I234240e9bdd4b69ad64a17b1449ae1e81c0edaca Signed-off-by: Neale Ranns <nranns@cisco.com>
2016-11-09Improve test framework documentationKlement Sekera1-4/+4
Change-Id: I06f0cbbbdd29e04a07f1db6807b3e16f1d41e8d2 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-10-31add vpp debugging support to test frameworkKlement Sekera1-7/+43
improve test documentation Change-Id: Ia9678aa2532ecb4cb33736aedb4a31aa3f2a3f93 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-10-26refactor test frameworkKlement Sekera1-2/+21
Change-Id: I31da3b1857b6399f9899276a2d99cdd19436296c Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Matej Klotton <mklotton@cisco.com> Signed-off-by: Jan Gelety <jgelety@cisco.com> Signed-off-by: Juraj Sloboda <jsloboda@cisco.com>
2016-10-03test: new test infrastructureDamjan Marion1-0/+3
Change-Id: I73ca19c431743f6b39669c583d9222a6559346ef Signed-off-by: Jan Gelety <jgelety@cisco.com> Signed-off-by: Juraj Sloboda <jsloboda@cisco.com> Signed-off-by: Stefan Kobza <skobza@cisco.com> Signed-off-by: Matej Klotton <mklotton@cisco.com> Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com> Signed-off-by: Damjan Marion <damarion@cisco.com>