aboutsummaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2017-11-10make test: fix logic for CACHE_OUTPUT optionKlement Sekera1-1/+1
Change-Id: I9938c0154c860913e27cecb18ec68f247943a2e6 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-10make test: automatically seed random generatorKlement Sekera1-10/+34
Change-Id: I286b61d41cc2b557de3eb8801ff95c643f680acd Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-06make test: add CACHE_OUTPUT optionKlement Sekera1-0/+12
This option allows one to toggle whether captured VPP stdout/stderr output is printed inline (each line logged immediately after being collected). Default setting is 0/no (behaviour same as before). This allows one to see the output in case of vpp crashing during test run even if test framework gets stuck waiting for shared memory mutex (which crashed vpp can no longer unlock). Complete stdout/stderr output is always printed as a block to the logger after test finishes. Change-Id: If3645a9b85562dba26ed9c87daa27ea5ef34f2e5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-11-01VPP Object Model (VOM)Neale Ranns1-0/+30
The VOM is a C++ library for use by clients/agents of VPP for programming state. It uses the binary APIs to do so. Various other common client side functions are also provided. Please see om.hpp for a more detailed description. Change-Id: Ib756bfe99817093815a9e26ccf464aa5583fc523 Signed-off-by: Neale Ranns <neale.ranns@cisco.com> Co-authored-by: Mohsin Kazmi <sykazmi@cisco.com>
2017-10-10make test: add RETRIES optionKlement Sekera1-12/+39
Change-Id: Ibe31e932bc997f0101a8947e01df90a90d1f100f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-10-03make test: Copy api_post_mortem.$$ file tmp test dir for archiving.Dave Wallace1-5/+5
Change-Id: I4baf89ef383dbc2f309081a6b56b13ebcb8fc2df Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-30make test: Create link to failed test dir on timeout.Dave Wallace1-1/+1
- Also change default coredump configuration from "coredump-size unlimited" to "full-coredump" Change-Id: Iefedc2636f2d9696b7575b34e91dd7be49f601fa Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-09-29make test: fix/disable VAPI tests on centosKlement Sekera1-0/+9
Workaround old `check' library on centos. Disable building/running of C++ VAPI test as centos's gcc can't compile our C++ code (vapi.hpp) due to bug. Change-Id: If9da9c7f1bc076f7cdfb9bd3016dfe60a08afa36 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-17make test: collect symlinks to failed testsKlement Sekera1-1/+21
Compress files in temporary directories of failed tests and symlink the directories under /tmp/vpp-failed-unittests location - preparation for jenkins archivation. Automatically cleanup the directory at start of test run. The compression is performed only when environment variable COMPRESS_FAILED_TEST_LOGS is set to one of "yes", "y", "1". This is set in verify target, but left unset by default, so when invoking make test by hand, files won't be compressed. Change-Id: I84c8f1c6aa79aa9c0b753357022b1f195f17a283 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-12make test: fix broken DEBUG=gdb* optionsKlement Sekera1-2/+6
Change-Id: I5d80982eeab78a629760f567eda3b1539d96e3a8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-08-10make test: detect hung testsKlement Sekera1-2/+40
Run tests in a forked process with a set of pipes to communicate keep-alives and overall result. This allows us to detect when e.g. vpp dies mid-API call causing the test to hang waiting for response (which will never come since vpp died). Support setting a (per test case) TIMEOUT make test option to set timeout, with a default timeout of 120 seconds. Example - fail the test suite if any test-case fails to finish within 300s: make TIMEOUT=300 test Change-Id: I0d04f26a7232968f4bf043adf5d5b508f5018717 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-07-18make test: improve console output messagesKlement Sekera1-3/+3
Remove the word derp and replace it with a proper, more gramatically correct message. Change-Id: I04fd44cc67dace1a31ca48fc8ce67b246162ba79 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-06-07make test: improve debugabilityKlement Sekera1-1/+2
Introduce faulthandler to print stack trace to stderr on python crash. Don't disable automatic garbage collection to decrease the chances of running out of memory. Change-Id: I6927a5f6ea9569735d084d4ed3d258950a400d74 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-20make test: improve bfd reliabilityKlement Sekera1-1/+11
Change-Id: Iaf446a2d7d8e595c3379fb9ed61a954351c17b90 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-12make test: don't rely on cPython GC to close fdsKlement Sekera1-8/+9
This code improvement allows running in pypy (and other interpreters) without exhausting file descriptors. Change-Id: Icb692a0fe1343c12cbbb15af6c58753420e74330 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-10make test: automatic "vpp finishes startup" checkKlement Sekera1-13/+28
Add code which checks if vpp doesn't crash/exit immediately after startup to aid debugging stuff like mistyped graph node name or so. Refuse to run tests if the vpp is unable to start, complain loudly and print vpp's stderr at critical log level if this happens to make spotting these problems in jenkins easy. Change-Id: I40d3fbd05c822c0534713bae6bef05ecfb0e0c1d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-04-06BFD-FIB interactionsNeale Ranns1-1/+1
- single-hop BFD: attach a delegate to the appropriate adjacency - multi-hop BFD [not supported yet]: attach a delegate to the FIB entry. adjacency/fib_entry state tracks the BFD session state. when the state is down the object does not contribute forwarding hence and hence dependent objects will not use it. For example, if a route is ECMP via two adjacencies and one of them is BFD down, then only the other is used to forward (i.e. we don't drop half the traffic). Change-Id: I0ef53e20e73b067001a132cd0a3045408811a822 Signed-off-by: Neale Ranns <nranns@cisco.com>
2017-03-29Sub-net broadcast addresses for IPv4Neale Ranns1-1/+1
Change-Id: Ib2189d01e8bc61de57404159690fb70f89c47277 Signed-off-by: Neale Ranns <nranns@cisco.com>
2017-03-24make test: fix broken plugin pathsKlement Sekera1-0/+2
Change-Id: I25a6882ec503fc5bb3694411fbdc2eb1f1e1fafc Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-22make test: support out-of-tree pluginsKlement Sekera1-2/+10
usage: env EXTERN_PLUGINS=/path/to/plugins make test Change-Id: I8eece726dfafeff1cffd921c1e18cd3eb7eb64ed Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-13ACL plugin rejects ICMP messages (VPP-624)Pavel Kotucek1-1/+8
Change-Id: I95113a277b94cce5ff332fcf9f57ec6f385acec0 Signed-off-by: Pavel Kotucek <pkotucek@cisco.com>
2017-03-10Retire vpp_liteDamjan Marion1-1/+3
vpp_lite platform is not needed anymore as same efect can be achieved with following startup.conf config: plugins { plugin dpdk_plugin.so { disable } } Change-Id: I690ea8ceb1c6e1fe32e01e7da54e9958019a93bf Signed-off-by: Damjan Marion <damarion@cisco.com>
2017-03-08make test: split into basic and extended testsKlement Sekera1-3/+27
Implement plumbing to allow decorating tests as extended, e.g.: @unittest.skipUnless(running_extended_tests(), "part of extended tests") both methods and classes can be decorated this way. Change make test and make test-debug to run only non-extended tests. Introduce make test-all and make test-all-debug to run the full suite. Run full suite as part of make verify. Decorate most BFD tests as extended. Change-Id: I3bc64f59e9fe238f7f767d7e043dc165d03e9dfa Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-06make test: reset object registry if vpp diesKlement Sekera1-0/+2
Change-Id: If0e30837e07a21f3912676f5147cb242d3d2b235 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-03-06make test: tell vpp to set coredump sizeKlement Sekera1-6/+11
Unless overridden by COREDUMP_SIZE env variable, tell VPP to set coredump size to unlimited, otherwise use $COREDUMP_SIZE as the argument. Change-Id: Ia2a6508207c66a171b33d272c820b1deb4a83e82 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-26BFD: echo functionKlement Sekera1-1/+1
Change-Id: Ib1e301d62b687d4e42434239e7cd412065c28da0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-19make test: save + dump VPP api trace log; VPP-640Dave Wallace1-0/+11
Change-Id: I20aacc927f2b04f42b0a7220c4283560b4d2a359 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2017-02-15BFD: loop back echo packetsKlement Sekera1-0/+1
Change-Id: I772b63ac25ebfccaff9ab9d8d0b1445e85f21df7 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-14make test: improve stabilityKlement Sekera1-20/+58
Disable automatic garbage collection and run it manually before running each test case to minimize stalls. Improve vpp subprocess cleanup. Reduce helper thread count to one and properly clean that thread once it's not needed. Change-Id: I3ea78ed9628552b5ef3ff29cc7bcf2d3fc42f2c3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-09make test: work around scapy truncated packetsKlement Sekera1-0/+32
Under stress, it's possible to hit a race condition, when the packet header is fully written to pcap, but not all packet data - yet. Scapy is stupid enough to: 1. not detect and report this error, truncating the packet instead 2. continue munching more data from wrong offset The work around is to scan the file ahead, parse the packet header, figure out how much data we need, wait for the file to be big enough, then restore the file position back to where it was and finally let scapy parse the packet. Change-Id: I9fc71d3ebdc62ecab6c90b90f177d0eaeb09b8bb Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-08BFD: modify session parametersKlement Sekera1-0/+4
Change-Id: I666e5c0cc71a3693640960c93cdd1907f84fbe23 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-06make test: improve test filteringKlement Sekera1-2/+70
Implement fine-grained test filtering by supporting more complicated filters beside the original file name suffix filter. Change-Id: If5a166d08cffe8c58cc6cf174e6df861c34dbaa6 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-02-02BFD: SHA1 authenticationKlement Sekera1-8/+7
Add authentication support to BFD feature. Out of three existing authentication types, implement SHA1 (sole RFC requirement). Simple password is insecure and MD5 is discouraged by the RFC, so ignore those. Add/change APIs to allow configuring BFD authentication keys and their usage with BFD sessions. Change-Id: Ifb0fb5b19c2e72196d84c1cde919bd4c074ea415 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-01-17make test: fix text output when non-interactiveKlement Sekera1-0/+8
Output everything to stdout instead of mixing stdout and stderr. Change-Id: I3146760e068fda7f1ffaaf36683d5153dcda4561 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-01-13make test: improve test results printingKlement Sekera1-2/+2
Accomodate longer test descriptions by increasing the headings size. Change-Id: I51c90b8d91feaa83b78972d0802d3054f8510f36 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-01-11make test: improve documentation and PEP8 complianceKlement Sekera1-11/+12
Change-Id: Ib4f0353aab6112fcc3c3d8f0bcbed5bc4b567b9b Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-01-11BFD: fix bfd_udp_add APIKlement Sekera1-0/+3
Fix reporting of bs_index in the return message. Enhance test suite to cover this case. Change-Id: I37d35b850818bc1a05abe67ca919c22aeac978b6 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2017-01-09make test: Loopback interface CRUD testMatej Klotton1-5/+6
Change-Id: I0581da7a682bfe4dd6520ecf1b2ea6bd8c20b1b3 Signed-off-by: Matej Klotton <mklotton@cisco.com>
2016-12-23make test: improve handling of packet capturesKlement Sekera1-21/+32
Perform accounting of expected packets based on created packet infos. Use this accounting info to automatically expect (and verify) the correct number of packets to be captured. Automatically retry the read of the capture file if scapy raises an exception while doing so to handle rare cases when capture file is read while only partially written during busy wait. Don't fail assert_nothing_captured if only junk packets arrived. Change-Id: I16ec2e9410ef510d313ec16b7e13c57d0b2a63f5 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-23make test: add log.txtKlement Sekera1-0/+4
Always create a log.txt under test's temporary directory containing V=2 output for debugging purposes. Change-Id: If149c8126f42f80724ee17b63c411ddec6645f85 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-16make test: improve robustness and performanceKlement Sekera1-11/+30
Introduce an API which asserts empty capture for interface. Throw exception in old API if the capture does not exist, thus making it clear if the test expects packets to arrive or not. Improve performance by not doing sleeps after starting the packet generator, rather lazily deleting captures when needed. Fix wrong usage of packet.show() in various tests. Change-Id: I456cb23316eef99b3f35f80344fe595c4db9a21c Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-09make test: FIB add/update/delete - ip4 routesMatej Klotton1-0/+7
- JIRA:CSIT-483 Change-Id: Idb4c5bd7a234bc975f3380ece58c0e8d4bfdafd9 Signed-off-by: Matej Klotton <mklotton@cisco.com>
2016-12-09BFD: fix timing in the main loopKlement Sekera1-19/+27
Properly wakeup the main process loop only when necessary to avoid missed events. Improve BFD unit test reliability and speed. Add timestamps to logs and replace Queue with more efficient deque. Change-Id: I01255a02057d3146917b43b669ccdc74f31ef0c8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-08BFD: improve ip header handling, fix assertKlement Sekera1-0/+2
Instead of storing a pointer, store an offset within vlib_buffer marking the start of the ip header (similar to storing start of ethernet header). Handle negative timeout in the bfd process main loop. Change-Id: I05a1ff3ac41da5bdc3b2ac6f9e03b3241994688b Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-07BFD: basic asynchronous session up/downKlement Sekera1-0/+28
This is a work-in-progress basic BFD session handling. Only asynchronous mode is supported at the moment. Setting the session flags doesn't work. Change-Id: Idba27f721b5c35be5a66a6d202a63d23ff7ecf6f Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-05make test: fix missing log/packet messagesKlement Sekera1-1/+1
Change-Id: Idb3119792943664748c4abc3829ad723f4156dfe Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-12-02make test: handle exceptions from VppPapiProvider.__init__Klement Sekera1-6/+8
Change-Id: I3c0372d5d09d554ad4ebe0adfa97c111571f628a Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-11-28make test: detect early vpp crashKlement Sekera1-14/+17
If VPP process dies right at start, do a quick detection instead of being stuck in the connect timeout (60s). Change-Id: I41675181635fb81a6a7d93fbf652480a16bf78a0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2016-11-22GRE tests and fixesNeale Ranns1-1/+1
Change-Id: I234240e9bdd4b69ad64a17b1449ae1e81c0edaca Signed-off-by: Neale Ranns <nranns@cisco.com>
2016-11-15Update test documentation.Matej Klotton1-19/+8
- update IRB, IPv4, ipv6 doc - revert 778c2765c8ea5c6628f6d668847f0b9ae06dbf3d Change-Id: I9af5ed9329ce5fe01392cf28d5bf321cfc647e48 Signed-off-by: Matej Klotton <mklotton@cisco.com>