summaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2020-08-27tests: "force solo" testcase supportAndrew Yourtchenko1-1/+16
Some of the tests are time-sensitive, and at present require a non-trivial modification in order to run at high concurrency. Without these modifications, they intermittently fail, and require the test retries. Rather than setting them to the extended tests and forgetting about them, put them into a "solo" set, which gets run in a single-threaded mode after the rest of the tests are done. Mark a few of the tests that showed errors during TEST_JOBS=48 as forced-solo. Also, give a better diagnostic if the testcase misses a docstring needed to represent it in the diagnostic outputs. Type: fix Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-05-05tests: clean up loggingPaul Vinciguerra1-12/+17
Tests currently expect the logger to be poked from run_tests.py. The tests should run without any magic values. This change sets a default null logger and removes the hasattr checks for the logger. For reference, see: https://docs.python-guide.org/writing/logging/ Type: test Change-Id: I98f953d73d12d00e74b59c94a0fb8c7a625b9c44 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-05-04misc: binary api fuzz test fixesDave Barach1-0/+5
Add a hook to src/vlibapi/api_shared.c to fuzz (screw up) binary API messages, e.g. by xoring random data into them before processing. We specifically exempt client connection messages, and inband debug CLI messages. We step over msg_id, client index, client context, and sw_if_index. Otherwise, "make test" vectors fail too rapidly to learn anything. The goal is to reduce the number of crashes caused to zero. We're fairly close with this patch. Add vl_msg_api_max_length(void *mp), which returns the maximum plausible length for a binary API message. Use it to hardern vl_api_from_api_to_new_vec(...) which takes an additional argument - message pointer - so it can verify that astr->length is sane. If it's not sane, return a u8 *vector of the form "insane astr->length nnnn\0". Verify array lengths in vl_api_dhcp6_send_client_message_t_handler(...) and vl_api_dhcp6_pd_send_client_message_t_handler(...). Add a fairly effective binary API fuzz hook to the unittest plugin, and modify the "make test" framework.py to pass "api-fuzz { on|off }" to enable API fuzzing: "make API_FUZZ=on TEST=xxx test-debug" or similar Type: improvement Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I0157267652a163c01553d5267620f719cc6c3bde
2020-04-28tests: implement ipaddress convenience methodsPaul Vinciguerra1-1/+4
Add vpp specific properties to ip addresses for use in the api. .vapi_af -- returns [ADDRESS_IP4, ADDRESS_IP6] .vapi_af_name -- returns the string ['ip4', 'ip6'] Update tests to demonstrate usage. Type: feature Change-Id: I43447a1522769d99f89debdc714c51700068d771 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-04-28vlib: startup multi-arch variant configurationRay Kinsella1-0/+8
Support for startup node multi-arch variant selection through startup.conf. This is to facilitate unit, functional testing and benchmarking of non-default multi-arch variant node code path. Also added parameters to make test, to specific using multi-arch variants in unit testing. Type: improvement Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
2020-03-12vppinfra: refactor clib_timebase_tDave Barach1-1/+6
Add a clib_time_t * argument to clib_timebase_init(...), to encourage client code to share the vlib_main_t's clib_time_t object. Display the current day / date in GMT via the "show time" debug CLI. Fix the test framework so it processes the new "show time" output format. Type: refactor Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I5e52d57eb164b7cdb6355362d520df6928491711
2020-03-10tests: add running_gcov_tests to framework.pyDave Barach1-0/+7
Mark a few code coverage tests appropriately: @unittest.skipUnless(running_gcov_tests, "part of code coverage tests") def test_my_coverage_unittest(self): <etc> Almost exactly like "make EXTENDED_TESTS=yes test". Type: feature Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Ic6d0a097a608ba93442b3d88252f66f8e6805b97
2020-02-07tests: support python 3.8Ole Troan1-2/+1
Make test framework python3 version independence. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
2020-01-14tests: fix worker thread initializationPaul Vinciguerra1-4/+4
from threading.thread __init__: This constructor should always be called with keyword arguments. If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread. Type: test Change-Id: Ifa89202e97053a4baf19e9a0ca0913430d5087a3 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-01-02tests: configure 32 mb physmemDave Barach1-0/+1
Absolutely nothing good happens when we force the kernel to briefly map and then unmap 16gb as vpp starts. Effect exacerbated when TEST_JOBS = 20...40, and so forth. Type: test Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Id8e3ce1763cad3a0891d5d6c8d2c1e3e610682d7
2019-12-23ipsec: Test and fix IPSec worker hand-offNeale Ranns1-4/+4
Type: fix Change-Id: I5cb9a3845ddbc5f4de4eb4e9c481f606fe5cec9a Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-10/+18
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-19papi: lazily initialize stats clientPaul Vinciguerra1-14/+0
remove wait-loop on stats socket from test framework. Type: refactor Change-Id: I5bb95a7c597707a87f9d9a471215c4b4af1a2280 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-05tests: fix wrong checksum error messageKlement Sekera1-1/+3
This change fixes the error message, which would previously report mismatch on IPv6 layer instead of TCP layer. Type: fix Change-Id: Ie78779685362a420d155f928b3cdb341bc3fb8bc Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-12-05papi: add call statsOle Troan1-0/+1
Type: feature Change-Id: Ic6d44122d3e62e09402e3f1946f7e57e9b5e7c5f Signed-off-by: Ole Troan <ot@cisco.com>
2019-12-05tests: Revert "Python Tests: Make VppTestCase unit testable."Paul Vinciguerra1-5/+0
Type: test This reverts commit 087c811b63aa6b0575149749e188bb37e7fb0513. Change-Id: Ica12597f27d5a75c33c12c2d27a227dc4866adcb Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-05tests: add test run time.Ole Troan1-3/+6
Type: feature Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: Id7b30b5fe63e68ae88f3a42aa77d2614ccda724d
2019-12-04tests: fix exception handling around vapi.connect()/.disconnect()Paul Vinciguerra1-5/+5
vapi.connect() only raises subclasses of vpp_papi.VPPIOError. vapi.disconnect() doesn't raise any exceptions, removed uneeded try block. Type: test Change-Id: Ide6e6db298b205f14382065862e358757619a5ba Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-11-27vlib: add 'wait' cli commandPaul Vinciguerra1-4/+3
When running exec scripts, there can be a need to wait between statements. Type: feature Change-Id: I2a45b390697e09fc222358c9354f28e3368a06ba Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-11-27tests: display stdout and stderr when output caching is disabledBenoît Ganne1-2/+2
Type: fix Change-Id: I4286ba58bb15dd67c1199f76166bedf3b4e0344c Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-11-05misc: Fix python scripts shebang lineRenato Botelho do Couto1-1/+1
Type: fix Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python' as a valid shebang line. It requires scripts to explicitly chose between python2 or python3. Change all to use python3 as suggested by Paul Vinciguerra. Depends-On: https://gerrit.fd.io/r/23170 Signed-off-by: Renato Botelho do Couto <renato@netgate.com> Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
2019-11-05tests: support setting random seedKlement Sekera1-1/+3
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-10-31tests: Revert "tests: add sending SIGQUIT to vpp quit sequence"Andrew Yourtchenko1-7/+6
This reverts commit 4465fe1e30556b40c9ef8a1a155aa1665102dd22. That commit makes *every* test to send SIGQUIT to VPP when it is done, thereby triggering the coredump if there was none. It is most surely not what we want. Type: test Change-Id: I3cc4c96f44c05527d586142f8feccc2f93c3e536 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2019-10-30tests: add sending SIGQUIT to vpp quit sequencePaul Vinciguerra1-6/+7
As suggested in the docs by: https://fd.io/docs/vpp/master/usecases/contiv/core_files#let-vpp-crash Type: test Change-Id: I10caf16220e2507351636c1b26e57e19bcf8a542 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-10-29tests: decode worker stdout/stderr outputDave Wallace1-2/+2
Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I309254e9e99920fbeaa50eea503c1c2a9470abfe
2019-10-28tests: make RA tests run on VPP timeAndrew Yourtchenko1-0/+14
the IPv6 RA tests take timing into the account, but the time inside VPP may go slightly differently compared to the time inside the driving python thread, if the machine running the tests is heavily loaded. Make a sleep function which sleeps "on VPP time" and use it. Change-Id: I3b34b0164f6e0ec7a619b92ee308089a4a8935e3 Type: test Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2019-10-22tests: DEBUG=gdb[server]-all to debug worker(s)Dave Wallace1-13/+70
Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I0264451632c1ce780b38a2c15a7e34350fc6d521
2019-10-21tests: add PID of VPP under test to API dump filename for the testAndrew Yourtchenko1-1/+2
Some testclasses (e.g. IPSec) have methods with the same names, this causes a race when running in parallel to save the API and move it into the test's directory - the name of the file has only the test method name as a discriminator. Saving two traces from two VPP instances "succeeds", in that it silently overwrites the file. But only one mv operation works - the other gives an error "file not found" and fails the test. Solution: add the PID of VPP under test to the API dump filename, this way the filenames become guaranteed unique and the race is avoided. Change-Id: I1bb47ed73369fea0f1bfbf907588a51e2cabfb7f Type: test Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2019-10-17vppinfra: test support python3Ole Troan1-1/+2
Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: Id1794f38fcb776ded9b769141f5f47d7be75f247
2019-10-16tests: cli wrapper should return stringOle Troan1-2/+4
Python3 fixes. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I648b2142d45dfab9146a02eeb1b12de11103ff9f Signed-off-by: Ole Troan <ot@cisco.com>
2019-10-14tests: make pg_start() wait until pg completesAndrew Yourtchenko1-19/+13
A sizable number of tests call pg_start() to get the packets flowing and then immediately expect to have the entirety of the packets gone through. This works on powerful and unstressed hardware, but fails in beautifully random ways under load. This also necessitates the complicated logic of remembering the "zombie captures", then sleeping for some time before cleaning them up.... The solution is simple: in pg_start(), start the generators, wait till they all finish, clean up, done. Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: I930e51b7aae39c9841d22dd905a4d13a465a672b Type: test
2019-10-14tests: log error which happens during connect and wait longer for stats socketAndrew Yourtchenko1-2/+3
Intermittently, a test would start VPP, but no testcases would execute. This would be more probable apparent during the high load or if there is another testcase dumping the core at that moment. Adding the logging to the connection revealed it was the stats socket connection erroring with error -2. Increasing the deadline from 3 seconds to 5 minutes has eliminated this error. Change-Id: I40bd7e642abb9e2aef0238c612e4c34781de5db2 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Type: test
2019-09-02tests: fix centos detectionKlement Sekera1-1/+1
Type: fix Fixes: defde0f870 Change-Id: Ib81e3610bd0fc22901a902dc76692f1951ead2b8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-08-20ip: reassembly: send packet out on correct workerKlement Sekera1-1/+4
Note which worker received fragment with offset zero and use this worker to send out the reassembled packet. Type: fix Change-Id: I1d3cee16788db3b230682525239c0100d51dc380 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-07-31tests: Split IPSec ESP into parameterized tests per engineNeale Ranns1-2/+0
Type: feature Change-Id: Icb1bd3fce768aebf8919c63a104f771ca7fa1d6f Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-07-28pg: add GSO supportMohsin Kazmi1-2/+2
Type: feature Change-Id: I72676495a85fbecc946aa266a75234cce70c3a5e Signed-off-by: Mohsin Kazmi <sykazmi@cisco.com>
2019-07-14tests: fix error in VppDiedError exceptionPaul Vinciguerra1-2/+2
Discovered running test-debug job in CI. - fix missing paren () around format value. Type: test Change-Id: Iebddd3035a435f8ad1cb1d6fa4e8e8c2d4ddaf96 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09tests: remove unused class attribute in VppTestCasePaul Vinciguerra1-1/+0
Identified in post-merge review. Type: test Change-Id: I46e19285479437561a43975ba9b5cb68f478736c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09tests: treat all truthy env vars the same wayPaul Vinciguerra1-8/+27
Introduce a new class, that returns the truthiness of a env var. Since an environment variable is just a string, it would normally be true if not unset. The new class returns true when the env var is set to a string that would be considered true. Type: test Depends-on: https://gerrit.fd.io/r/20484 Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09vppinfra: allocate bihash virtual space on demandDave Barach1-0/+1
Reduces the vpp image virtual size by multiple gigabytes Add a "show bihash" command which displays configured and current virtual space in use by bihash tables. Modify the .py test framework to call "show bihash" on test tear-down Type: refactor Change-Id: Ifc1b7e2c43d29bbef645f6802fa29ff8ef09940c Signed-off-by: Dave Barach <dave@barachs.net>
2019-07-03tests: Have worker return immediately on bad executablePaul Vinciguerra1-1/+10
No reason to waste CI cycles if we know the test will not run. See: 17:55:11 ============================================================================== 17:55:11 VPP Object Model Test 17:55:11 ============================================================================== 17:55:11 Exception in thread Thread-4: 17:55:11 Traceback (most recent call last): 17:55:11 File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner 17:55:11 self.run() 17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/framework.py", line 1475, in run 17:55:11 stdout=subprocess.PIPE, stderr=subprocess.PIPE) 17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 617, in __init__ 17:55:11 restore_signals, start_new_session) 17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 1415, in _execute_child 17:55:11 raise child_exception_type(errno_num, err_msg) 17:55:11 OSError: [Errno 2] No such file or directory: '/w/workspace/vpp-beta-verify-master-ubuntu1804/test/build/vom_test/vom_test' 17:55:11 17:55:11 17:55:11,328 Timeout! Worker did not finish in 120s 17:55:11 run C++ VOM tests ERROR [ temp dir used by test case: /tmp/vpp-unittest-VOMTestCase-vpMcWF ] 17:55:11 Type: fix Change-Id: I3d8252807e98a09a8abd70de8a22517151f9d786 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-28tests: handle TypeErrors in calls to VppDiedErrorPaul Vinciguerra1-5/+4
Type: test Change-Id: I869768ed25efa3350f025fba329466db69805803 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-25tests: if vpp is dead, specify the test name in the outputPaul Vinciguerra1-3/+15
'Exception: VPP is dead when setting up the test' is more helpful with the test info. Type: test Change-Id: I6a262fdcf2ecb33aae17bef3d19745053a0bade8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-24tests: refactor VppDiedError.Paul Vinciguerra1-7/+27
- Move Exception into same module as TestCase. - Move the error reporting logic inside the error. - Allows testing of the returncode and signal_name for tests to consume. - Fix the signal reporting code: VppDiedError: VPP subprocess died unexpectedly with returncode -6 [unknown]. displays as: VppDiedError: VPP subprocess died unexpectedly with return code: -6 [SIGABRT]. Type: test Change-Id: I8488ab318a596c9b737308829cedfb7e96e57302 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-20tests: add sudo to gdb commandsPaul Vinciguerra1-2/+3
Prepend sudo to the gdb command line. Type: test Change-Id: I09013c3ca512e26de0c46c02f643b21064ba499c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-07tests: framework gracefully handle 'VppTransportShmemIOError'Paul Vinciguerra1-12/+17
Catches: ---- Traceback (most recent call last): File "/vpp/test/framework.py", line 593, in tearDown self.logger.info(self.vapi.ppcli("api trace save %s" % api_trace)) File "/vpp/test/vpp_papi_provider.py", line 413, in ppcli return cli + "\n" + str(self.cli(cli)) File "/vpp/test/vpp_papi_provider.py", line 402, in cli r = self.papi.cli_inband(cmd=cli) File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 100, in __call__ return self._func(**kwargs) File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 414, in f return self._call_vpp(i, msg, multipart, **kwargs) File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 634, in _call_vpp msg = self.transport.read() File "/vpp/src/vpp-api/python/vpp_papi/vpp_transport_shmem.py", line 120, in read raise VppTransportShmemIOError(rv, 'vac_read failed') VppTransportShmemIOError: [Errno -1] vac_read failed ---- Change-Id: I767e48c4d03081eb5df6a8aa67da7e192d25e4cc Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-05-29make test: fix packet counter assertKlement Sekera1-15/+11
Change-Id: Ic6e6ac0629bbf07f7dad1803a85aabfdccbc2035 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-22stats: support multiple works for error countersOle Troan1-0/+5
The current code only allowed access to the main thread error counters. That is not so useful for a multi worker instance. No return a vector indexed by thread of counter_t values. Type: fix Change-Id: Ie322c8889c0c8175e1116e71de04a2cf453b9ed7 Signed-off-by: Ole Troan <ot@cisco.com>
2019-05-20reassembly: prevent long chain attackKlement Sekera1-0/+13
limit max # of fragments to 3 per packet by default add API option to configure the limit at runtime Change-Id: Ie4b9507bf5c6095b9a5925972b37fe0032f4f9e8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-09add mactime plugin unit / code coverage testsDave Barach1-0/+4
The unit and code coverage tests are boring. The rest of the patch involves test and packet-generator infra cleanups. Teach the "make test-xxx" family of targets to set the api test plugin path correctly, to make "binary-api <api-message-name> <args>" debug CLI commands work correctly in the "make test" environment. Unfortunately involves both the top-level and test Makefiles. Add a minor pg cli feature, a CLI to manually set s->sw_if_index[VLIB_TX]. Consider the case where one configures an interface with both a device-input and an output feature. To test the output feature using the pg, it's necessary to inject packets into the interface output node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set correctly. For example: packet-generator new { name tx limit 15 size 128-128 interface local0 # rx: device input feature not configured on local0 tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX] node loop0-output data { hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405 } } Fix a longstanding bug in the packet generator stream setup. Remove kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places] instead of using the stream value s->sw_if_index[VLIB_TX], and setting THAT datum correctly. Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed Signed-off-by: Dave Barach <dave@barachs.net>