aboutsummaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2018-09-06Fix test summary and retriesjuraj.linkes1-2/+2
There was an issue when tests crashed and weren't properly retried. Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-05STATS: Update Prometheus portOle Troan1-1/+1
Update temporary port allocation with fixed allocated from: https://github.com/prometheus/prometheus/wiki/Default-port-allocations Change-Id: I99a7e069fb95d00884458aeacaba06e4713fbb76 Signed-off-by: Ole Troan <ot@cisco.com>
2018-09-02STATS: Python binding to access VPP statistics and counters.Ole Troan1-1/+5
from vpp_papi.vpp_stats import VPPStats s = VPPStats(socketname='/var/run/stats.sock') c = s.ls('/if/rx') counters = s.dump(c) print(s.set_error_str()) Change-Id: I203ebe60b0c9ee5742aadc737c0f29051757959d Signed-off-by: Ole Troan <ot@cisco.com>
2018-08-30Fix hanging test runner when child process diesjuraj.linkes1-2/+12
When fixing the test summary after a test run doesn't finish properly I introduced a bug where child process which died would leave the whole run hanging. This patch fixed the bug while still having the correct test summary. Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-25Fix logging to file and summary for crashed testsjuraj.linkes1-0/+2
Logging was not being logged to log.txt Crashed tests were not reported properly when running tests in one process Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-23CSIT-1139: Implement parallel test executionjuraj.linkes1-132/+72
The implementation of parallel test execution in VPP Test Framework. - VPPTestCase test methods are grouped together - tests are running in separate processes - VPP instances spawned by tests are assigned to different cores - output from these processes is redirected through pipes and printed out testcase by testcase - TEST_JOBS env var is used to specify the number of parallel processes - improved test summary - a bit of code cleanup Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-07-24Create a unit-test pluginDave Barach1-1/+2
Move the tcp unit test to the plugin Add a bihash unit test and a "make test" program to call it Adjust framework.py to load the plugin, which is disabled by default Change-Id: Ic229d386a56a9d28dbd54974f231149053ca8f93 Signed-off-by: Dave Barach <dave@barachs.net>
2018-07-24test frame work pump thread exit: set flag then wakeNeale Ranns1-2/+2
Change-Id: Id95b8a7ad1bf550f615c50d77b16a530cc8fc2c4 Signed-off-by: Neale Ranns <nranns@cisco.com>
2018-07-09IGMP improvementsNeale Ranns1-2/+3
- Enable/Disable an interface for IGMP - improve logging - refactor common code - no orphaned timers - IGMP state changes in main thread only - Large groups split over multiple state-change reports - SSM range configuration API. - more tests Change-Id: If5674f1044e7e97274a711f47807c9ba689d7b9a Signed-off-by: Neale Ranns <nranns@cisco.com>
2018-06-25make test: fix RETRIES when setUpClass throwsKlement Sekera1-1/+21
This change adds handling of special case when setUpClass throws. In this case TestResults receives a mock object called _ErrorHolder. By parsing its description, we find test class name and use it to lookup the test class in test suite to be able to add it to the list of failures for re-running. Change-Id: I656f21e38aa450fc567cdcbcf6e586967f947a64 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-24Revert "Revert "make test: fix broken interfaces""Klement Sekera1-6/+4
This reverts commit c8efa29b6f9a91381897b54f1147daf922ed7164. Change-Id: I1d5c5773d5f86a63073e255336bd9de628e26179 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-24Revert "Revert "ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums""Klement Sekera1-5/+14
This reverts commit e0d2bd6bd7fc59c0c6ac48195d7f825dc99bfd91. Change-Id: If491e16f9ea66b2493a6a7c7f3c684ed585f8f51 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-22Revert "ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums"Ole Troan1-14/+5
This reverts commit a98346f664aae148d26a8e158008b773d73db96f. Change-Id: Iee5b3a5ddff0e8fd3a30fe5973cee24de434fe12 Signed-off-by: Ole Troan <ot@cisco.com>
2018-06-22Revert "make test: fix broken interfaces"Ole Troan1-4/+6
This reverts commit d5c60b96a3fd93916fc4af5c8d6d25625c28242e. Change-Id: I3632b9c3f76c615aee897f28f76d094e7031e689 Signed-off-by: Ole Troan <ot@cisco.com>
2018-06-22make test: fix broken interfacesKlement Sekera1-6/+4
Change-Id: I2e092774f81503e04b53cc6c6b5d357fe3fc52ab Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-21ipsec: VPP-1316 calculate IP/TCP/UDP inner checksumsKlement Sekera1-5/+14
Calculate IP/TCP/UDP checksums in software before adding authentication. Change-Id: I3e121cb00aeba667764f39ade8d62170f18f8b6b Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-05-16make test: unify packet checksum verificationsKlement Sekera1-0/+85
Change-Id: If9cc7c5e32ebecff398fd38b39e8f485754a4ad4 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-04-17Add logging supportDamjan Marion1-0/+1
Change-Id: Ieac9cf50156dbbb4962411e900d59256441915ef Signed-off-by: Damjan Marion <damarion@cisco.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-24make test: fix DEBUG=gdb/gdbserver optionsKlement Sekera1-4/+3
Change-Id: I5a7fa3b1c247ad5611907db27835724dcd31f575 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-23make test: code cleanupKlement Sekera1-49/+25
Change-Id: Ic689de569e5b6e6209d16d6acdb13c489daca1f5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-21reassembly: feature/concurrencyKlement Sekera1-0/+4
This change makes ip reassembly an interface feature, while adding concurrency support. Due to this, punt is no longer needed to test reassembly. Change-Id: I467669514ec33283ce935be0f1dd08f07684f0c7 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-03-14make test: early core detection, code cleanupKlement Sekera1-8/+5
Change-Id: I503d69f902f1c8d3107fb3f2b9f31b0083ebb747 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-27make test: refactor vcl test casesDave Wallace1-2/+3
- Reduce replicated code in test cases - Configure separate namespace secrets for thru hoststack test case to validate namespace secret functionality. - Pass per-instance environment variables to Worker class init function. Change-Id: I3cd5d4538f105cbfb09671c4d761541b40714b8f Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2018-02-22make test: tidy worker subprocess outputKlement Sekera1-1/+1
Change-Id: I362765a67762a59775863af12b712abb47445b3a Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-19make test: Add VPP VCL cut-thru test.Dave Wallace1-0/+2
Change-Id: Id3ec196bfeb90b141123adee97f15d9712351680 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2018-02-17make test: add test framework debuggingKlement Sekera1-0/+9
New option TEST_DEBUG=1 turns on test framework debugging, which currently consists of printing difference in allocated objects/memory and also creates reference graphs for any unfreed VppPapiProvider/VPP objects - these take a lot of memory and thus should be freed regularly. Change-Id: I29db0c1341009d4b5c5df9222d14f3095883fd0f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-14make-test: use api custom-dump to reduce log size.Dave Wallace1-1/+1
- Reduce log size by using "api trace custom-dump" instead of "api trace dump". - Fix custom-dump output of cli_inband_t api message to include cli command being executed. New output: DBGvpp# api trace custom-dump /tmp/test vl_api_memclnt_delete_t: index: 2 handle: 0x301d8e10 SCRIPT: memclnt_create name vpp_api_test SCRIPT: sw_interface_dump all SCRIPT: control_ping SCRIPT: exec show run Old output: DBGvpp# api trace dump /tmp/test ---------- trace 0 ----------- vl_api_memclnt_delete_t: index: 33554432 handle: 0x108e1d3000000000 ---------- trace 1 ----------- vl_api_memclnt_create_t: name: vpp_api_test input_queue: 0x808e1d3000000000 context: 0 ctx_quota: 0 ---------- trace 2 ----------- vl_api_sw_interface_dump_t: _vl_msg_id: 61 client_index: 33554432 context: 0 name_filter_valid: 0 ---------- trace 3 ----------- vl_api_control_ping_t: _vl_msg_id: 712 client_index: 33554432 context: 0 ---------- trace 4 ----------- vl_api_cli_inband_t: _vl_msg_id: 715 client_index: 33554432 context: 0 length: 9 Change-Id: If740c861649a3a59b8cc7a777c23c3cf94b8ff87 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2018-02-02make test: use random seedKlement Sekera1-1/+1
This fixes a constant setting of random seed forgotten from testing. Change-Id: Ie3c4db8bb2b4b73ba33de1ffc02cb563391fd31c Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-02-01IPv4/6 reassemblyKlement Sekera1-7/+15
Change-Id: Ic5dcadd13c88b8a5e7896dab82404509c081614a Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-01-09test: consolidate the multiple versions of send_and_*Neale Ranns1-0/+19
Change-Id: I7fa7d0ebf73dab8264a2e5ddbd412600d78ead05 Signed-off-by: Neale Ranns <nranns@cisco.com>
2017-11-10make test: fix logic for CACHE_OUTPUT optionKlement Sekera1-1/+1
Change-Id: I9938c0154c860913e27cecb18ec68f247943a2e6 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-10make test: automatically seed random generatorKlement Sekera1-10/+34
Change-Id: I286b61d41cc2b557de3eb8801ff95c643f680acd Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-06make test: add CACHE_OUTPUT optionKlement Sekera1-0/+12
This option allows one to toggle whether captured VPP stdout/stderr output is printed inline (each line logged immediately after being collected). Default setting is 0/no (behaviour same as before). This allows one to see the output in case of vpp crashing during test run even if test framework gets stuck waiting for shared memory mutex (which crashed vpp can no longer unlock). Complete stdout/stderr output is always printed as a block to the logger after test finishes. Change-Id: If3645a9b85562dba26ed9c87daa27ea5ef34f2e5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-01VPP Object Model (VOM)Neale Ranns1-0/+30
The VOM is a C++ library for use by clients/agents of VPP for programming state. It uses the binary APIs to do so. Various other common client side functions are also provided. Please see om.hpp for a more detailed description. Change-Id: Ib756bfe99817093815a9e26ccf464aa5583fc523 Signed-off-by: Neale Ranns <neale.ranns@cisco.com> Co-authored-by: Mohsin Kazmi <sykazmi@cisco.com>
2017-10-10make test: add RETRIES optionKlement Sekera1-12/+39
Change-Id: Ibe31e932bc997f0101a8947e01df90a90d1f100f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-03make test: Copy api_post_mortem.$$ file tmp test dir for archiving.Dave Wallace1-5/+5
Change-Id: I4baf89ef383dbc2f309081a6b56b13ebcb8fc2df Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-30make test: Create link to failed test dir on timeout.Dave Wallace1-1/+1
- Also change default coredump configuration from "coredump-size unlimited" to "full-coredump" Change-Id: Iefedc2636f2d9696b7575b34e91dd7be49f601fa Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-29make test: fix/disable VAPI tests on centosKlement Sekera1-0/+9
Workaround old `check' library on centos. Disable building/running of C++ VAPI test as centos's gcc can't compile our C++ code (vapi.hpp) due to bug. Change-Id: If9da9c7f1bc076f7cdfb9bd3016dfe60a08afa36 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-17make test: collect symlinks to failed testsKlement Sekera1-1/+21
Compress files in temporary directories of failed tests and symlink the directories under /tmp/vpp-failed-unittests location - preparation for jenkins archivation. Automatically cleanup the directory at start of test run. The compression is performed only when environment variable COMPRESS_FAILED_TEST_LOGS is set to one of "yes", "y", "1". This is set in verify target, but left unset by default, so when invoking make test by hand, files won't be compressed. Change-Id: I84c8f1c6aa79aa9c0b753357022b1f195f17a283 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-12make test: fix broken DEBUG=gdb* optionsKlement Sekera1-2/+6
Change-Id: I5d80982eeab78a629760f567eda3b1539d96e3a8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-10make test: detect hung testsKlement Sekera1-2/+40
Run tests in a forked process with a set of pipes to communicate keep-alives and overall result. This allows us to detect when e.g. vpp dies mid-API call causing the test to hang waiting for response (which will never come since vpp died). Support setting a (per test case) TIMEOUT make test option to set timeout, with a default timeout of 120 seconds. Example - fail the test suite if any test-case fails to finish within 300s: make TIMEOUT=300 test Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-07-18make test: improve console output messagesKlement Sekera1-3/+3
Remove the word derp and replace it with a proper, more gramatically correct message. Change-Id: I04fd44cc67dace1a31ca48fc8ce67b246162ba79 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-06-07make test: improve debugabilityKlement Sekera1-1/+2
Introduce faulthandler to print stack trace to stderr on python crash. Don't disable automatic garbage collection to decrease the chances of running out of memory. Change-Id: I6927a5f6ea9569735d084d4ed3d258950a400d74 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-20make test: improve bfd reliabilityKlement Sekera1-1/+11
Change-Id: Iaf446a2d7d8e595c3379fb9ed61a954351c17b90 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-12make test: don't rely on cPython GC to close fdsKlement Sekera1-8/+9
This code improvement allows running in pypy (and other interpreters) without exhausting file descriptors. Change-Id: Icb692a0fe1343c12cbbb15af6c58753420e74330 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-10make test: automatic "vpp finishes startup" checkKlement Sekera1-13/+28
Add code which checks if vpp doesn't crash/exit immediately after startup to aid debugging stuff like mistyped graph node name or so. Refuse to run tests if the vpp is unable to start, complain loudly and print vpp's stderr at critical log level if this happens to make spotting these problems in jenkins easy. Change-Id: I40d3fbd05c822c0534713bae6bef05ecfb0e0c1d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-06BFD-FIB interactionsNeale Ranns1-1/+1
- single-hop BFD: attach a delegate to the appropriate adjacency - multi-hop BFD [not supported yet]: attach a delegate to the FIB entry. adjacency/fib_entry state tracks the BFD session state. when the state is down the object does not contribute forwarding hence and hence dependent objects will not use it. For example, if a route is ECMP via two adjacencies and one of them is BFD down, then only the other is used to forward (i.e. we don't drop half the traffic). Change-Id: I0ef53e20e73b067001a132cd0a3045408811a822 Signed-off-by: Neale Ranns <nranns@cisco.com>
2017-03-29Sub-net broadcast addresses for IPv4Neale Ranns1-1/+1
Change-Id: Ib2189d01e8bc61de57404159690fb70f89c47277 Signed-off-by: Neale Ranns <nranns@cisco.com>
2017-03-24make test: fix broken plugin pathsKlement Sekera1-0/+2
Change-Id: I25a6882ec503fc5bb3694411fbdc2eb1f1e1fafc Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-22make test: support out-of-tree pluginsKlement Sekera1-2/+10
usage: env EXTERN_PLUGINS=/path/to/plugins make test Change-Id: I8eece726dfafeff1cffd921c1e18cd3eb7eb64ed Signed-off-by: Klement Sekera <ksekera@cisco.com>