aboutsummaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2019-11-18tests: support setting random seedKlement Sekera1-1/+3
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com> (cherry picked from commit 45a95dd782b91e9ae5665b5f95be4b6d7f99b879)
2019-11-18tests: decode worker stdout/stderr outputDave Wallace1-2/+2
Type: fix Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I309254e9e99920fbeaa50eea503c1c2a9470abfe (cherry picked from commit 97ea2f4ec0ec337a4d391f5225e89d12ba2ddb38)
2019-11-15tests: DEBUG=gdb[server]-all to debug worker(s)Dave Wallace1-13/+70
Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I0264451632c1ce780b38a2c15a7e34350fc6d521 (cherry picked from commit 2456433df46ae99dfbcec6260d14e592b5823c83)
2019-11-05tests: make RA tests run on VPP timeAndrew Yourtchenko1-0/+14
the IPv6 RA tests take timing into the account, but the time inside VPP may go slightly differently compared to the time inside the driving python thread, if the machine running the tests is heavily loaded. Make a sleep function which sleeps "on VPP time" and use it. Change-Id: I3b34b0164f6e0ec7a619b92ee308089a4a8935e3 Type: test Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> (cherry picked from commit 63cb8827e586f02f53512d23a8e7596fe9c19aa8)
2019-10-29tests: add PID of VPP under test to API dump filename for the testAndrew Yourtchenko1-1/+2
Some testclasses (e.g. IPSec) have methods with the same names, this causes a race when running in parallel to save the API and move it into the test's directory - the name of the file has only the test method name as a discriminator. Saving two traces from two VPP instances "succeeds", in that it silently overwrites the file. But only one mv operation works - the other gives an error "file not found" and fails the test. Solution: add the PID of VPP under test to the API dump filename, this way the filenames become guaranteed unique and the race is avoided. Change-Id: I1bb47ed73369fea0f1bfbf907588a51e2cabfb7f Type: test Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> (cherry picked from commit 586d3edf49f5dd8cd3e4f836afd562e21766914d)
2019-10-27tests: make pg_start() wait until pg completesAndrew Yourtchenko1-19/+13
A sizable number of tests call pg_start() to get the packets flowing and then immediately expect to have the entirety of the packets gone through. This works on powerful and unstressed hardware, but fails in beautifully random ways under load. This also necessitates the complicated logic of remembering the "zombie captures", then sleeping for some time before cleaning them up.... The solution is simple: in pg_start(), start the generators, wait till they all finish, clean up, done. Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: I930e51b7aae39c9841d22dd905a4d13a465a672b Type: test (cherry picked from commit 8d829f6c480cdd6536537fc49356baa1878b9570)
2019-10-27tests: log error which happens during connect and wait longer for stats socketAndrew Yourtchenko1-2/+3
Intermittently, a test would start VPP, but no testcases would execute. This would be more probable apparent during the high load or if there is another testcase dumping the core at that moment. Adding the logging to the connection revealed it was the stats socket connection erroring with error -2. Increasing the deadline from 3 seconds to 5 minutes has eliminated this error. Change-Id: I40bd7e642abb9e2aef0238c612e4c34781de5db2 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Type: test (cherry picked from commit 4f05a8e408cba09057841d97cd5e7da3058836d1)
2019-09-28tests: fix centos detectionKlement Sekera1-1/+1
Type: fix Fixes: defde0f870 Change-Id: Ib81e3610bd0fc22901a902dc76692f1951ead2b8 Signed-off-by: Klement Sekera <ksekera@cisco.com> (cherry picked from commit 3a350702ec24d1bde0483da1e946015c2b81e0c4)
2019-09-25ip: reassembly: send packet out on correct workerKlement Sekera1-1/+4
Note which worker received fragment with offset zero and use this worker to send out the reassembled packet. Type: fix Change-Id: I1d3cee16788db3b230682525239c0100d51dc380 Signed-off-by: Klement Sekera <ksekera@cisco.com> (cherry picked from commit 630ab5846bceddf8d663e9f488a2dc0378949827)
2019-07-31tests: Split IPSec ESP into parameterized tests per engineNeale Ranns1-2/+0
Type: feature Change-Id: Icb1bd3fce768aebf8919c63a104f771ca7fa1d6f Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-07-28pg: add GSO supportMohsin Kazmi1-2/+2
Type: feature Change-Id: I72676495a85fbecc946aa266a75234cce70c3a5e Signed-off-by: Mohsin Kazmi <sykazmi@cisco.com>
2019-07-14tests: fix error in VppDiedError exceptionPaul Vinciguerra1-2/+2
Discovered running test-debug job in CI. - fix missing paren () around format value. Type: test Change-Id: Iebddd3035a435f8ad1cb1d6fa4e8e8c2d4ddaf96 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09tests: remove unused class attribute in VppTestCasePaul Vinciguerra1-1/+0
Identified in post-merge review. Type: test Change-Id: I46e19285479437561a43975ba9b5cb68f478736c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09tests: treat all truthy env vars the same wayPaul Vinciguerra1-8/+27
Introduce a new class, that returns the truthiness of a env var. Since an environment variable is just a string, it would normally be true if not unset. The new class returns true when the env var is set to a string that would be considered true. Type: test Depends-on: https://gerrit.fd.io/r/20484 Change-Id: I90ef010156f6fec246bde5c0e208ced1869b180f Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-07-09vppinfra: allocate bihash virtual space on demandDave Barach1-0/+1
Reduces the vpp image virtual size by multiple gigabytes Add a "show bihash" command which displays configured and current virtual space in use by bihash tables. Modify the .py test framework to call "show bihash" on test tear-down Type: refactor Change-Id: Ifc1b7e2c43d29bbef645f6802fa29ff8ef09940c Signed-off-by: Dave Barach <dave@barachs.net>
2019-07-03tests: Have worker return immediately on bad executablePaul Vinciguerra1-1/+10
No reason to waste CI cycles if we know the test will not run. See: 17:55:11 ============================================================================== 17:55:11 VPP Object Model Test 17:55:11 ============================================================================== 17:55:11 Exception in thread Thread-4: 17:55:11 Traceback (most recent call last): 17:55:11 File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner 17:55:11 self.run() 17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/framework.py", line 1475, in run 17:55:11 stdout=subprocess.PIPE, stderr=subprocess.PIPE) 17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 617, in __init__ 17:55:11 restore_signals, start_new_session) 17:55:11 File "/w/workspace/vpp-beta-verify-master-ubuntu1804/test/run/venv/local/lib/python2.7/site-packages/subprocess32.py", line 1415, in _execute_child 17:55:11 raise child_exception_type(errno_num, err_msg) 17:55:11 OSError: [Errno 2] No such file or directory: '/w/workspace/vpp-beta-verify-master-ubuntu1804/test/build/vom_test/vom_test' 17:55:11 17:55:11 17:55:11,328 Timeout! Worker did not finish in 120s 17:55:11 run C++ VOM tests ERROR [ temp dir used by test case: /tmp/vpp-unittest-VOMTestCase-vpMcWF ] 17:55:11 Type: fix Change-Id: I3d8252807e98a09a8abd70de8a22517151f9d786 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-28tests: handle TypeErrors in calls to VppDiedErrorPaul Vinciguerra1-5/+4
Type: test Change-Id: I869768ed25efa3350f025fba329466db69805803 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-25tests: if vpp is dead, specify the test name in the outputPaul Vinciguerra1-3/+15
'Exception: VPP is dead when setting up the test' is more helpful with the test info. Type: test Change-Id: I6a262fdcf2ecb33aae17bef3d19745053a0bade8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-24tests: refactor VppDiedError.Paul Vinciguerra1-7/+27
- Move Exception into same module as TestCase. - Move the error reporting logic inside the error. - Allows testing of the returncode and signal_name for tests to consume. - Fix the signal reporting code: VppDiedError: VPP subprocess died unexpectedly with returncode -6 [unknown]. displays as: VppDiedError: VPP subprocess died unexpectedly with return code: -6 [SIGABRT]. Type: test Change-Id: I8488ab318a596c9b737308829cedfb7e96e57302 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-20tests: add sudo to gdb commandsPaul Vinciguerra1-2/+3
Prepend sudo to the gdb command line. Type: test Change-Id: I09013c3ca512e26de0c46c02f643b21064ba499c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-06-07tests: framework gracefully handle 'VppTransportShmemIOError'Paul Vinciguerra1-12/+17
Catches: ---- Traceback (most recent call last): File "/vpp/test/framework.py", line 593, in tearDown self.logger.info(self.vapi.ppcli("api trace save %s" % api_trace)) File "/vpp/test/vpp_papi_provider.py", line 413, in ppcli return cli + "\n" + str(self.cli(cli)) File "/vpp/test/vpp_papi_provider.py", line 402, in cli r = self.papi.cli_inband(cmd=cli) File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 100, in __call__ return self._func(**kwargs) File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 414, in f return self._call_vpp(i, msg, multipart, **kwargs) File "/vpp/src/vpp-api/python/vpp_papi/vpp_papi.py", line 634, in _call_vpp msg = self.transport.read() File "/vpp/src/vpp-api/python/vpp_papi/vpp_transport_shmem.py", line 120, in read raise VppTransportShmemIOError(rv, 'vac_read failed') VppTransportShmemIOError: [Errno -1] vac_read failed ---- Change-Id: I767e48c4d03081eb5df6a8aa67da7e192d25e4cc Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-05-29make test: fix packet counter assertKlement Sekera1-15/+11
Change-Id: Ic6e6ac0629bbf07f7dad1803a85aabfdccbc2035 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-22stats: support multiple works for error countersOle Troan1-0/+5
The current code only allowed access to the main thread error counters. That is not so useful for a multi worker instance. No return a vector indexed by thread of counter_t values. Type: fix Change-Id: Ie322c8889c0c8175e1116e71de04a2cf453b9ed7 Signed-off-by: Ole Troan <ot@cisco.com>
2019-05-20reassembly: prevent long chain attackKlement Sekera1-0/+13
limit max # of fragments to 3 per packet by default add API option to configure the limit at runtime Change-Id: Ie4b9507bf5c6095b9a5925972b37fe0032f4f9e8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-09add mactime plugin unit / code coverage testsDave Barach1-0/+4
The unit and code coverage tests are boring. The rest of the patch involves test and packet-generator infra cleanups. Teach the "make test-xxx" family of targets to set the api test plugin path correctly, to make "binary-api <api-message-name> <args>" debug CLI commands work correctly in the "make test" environment. Unfortunately involves both the top-level and test Makefiles. Add a minor pg cli feature, a CLI to manually set s->sw_if_index[VLIB_TX]. Consider the case where one configures an interface with both a device-input and an output feature. To test the output feature using the pg, it's necessary to inject packets into the interface output node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set correctly. For example: packet-generator new { name tx limit 15 size 128-128 interface local0 # rx: device input feature not configured on local0 tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX] node loop0-output data { hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405 } } Fix a longstanding bug in the packet generator stream setup. Remove kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places] instead of using the stream value s->sw_if_index[VLIB_TX], and setting THAT datum correctly. Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed Signed-off-by: Dave Barach <dave@barachs.net>
2019-05-07make test: add python3 compatibility in extend_packetAlexandre Poirrier1-2/+2
Change-Id: I342ff8e44e0d1ebd9f2c8cc386372b2f5a731080 Signed-off-by: Alexandre Poirrier <apoirrie@cisco.com>
2019-05-06make test: Fix truncated core fileDave Wallace1-0/+23
- Wait for coredump to complete before killing VPP. Change-Id: Ib7b96b339f796f9ef7e0eadff81d181464b8a03a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-05-06gcov / test framework: sigterm not sigkillDave Barach1-1/+1
Otherwise, gcov data vanishes without a trace. Add a __gcov_flush() call to the unix signal handler, under #ifdef CLIB_GCOV. Add -DCLIB_GCOV to vpp_gcov_TAG_CFLAGS. Change-Id: I2726e671b26dfbe7fae88f46a8207bb2b5106884 Signed-off-by: Dave Barach <dave@barachs.net>
2019-04-23API: Python and Unix domain socket improvementOle Troan1-2/+5
Handle the case where buffer overflows. Then SOCK_SEQPACKET assumption that multiple API messages are not returned by recv() is broken. Use SOCK_STREAM for API exchanges instead. Add support for running tests over sockets. make test SOCKET=1 Change-Id: Ibe5fd69b1bf617de4c7ba6cce0a7c2b3f97a2821 Signed-off-by: Ole Troan <ot@cisco.com>
2019-04-11Tests: Refactor tearDown show command logging, add lifecycle markers.Paul Vinciguerra1-3/+13
This change adds a consistent interface for adding test-specific show commands to log.txt. It also adds log markers for the execution of setUp[Class], tearDown[Class] in the logs. Change-Id: I7d42e396e594a59e866a7d55dac0af25548e657a Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-04-10API: Fix shared memory only action handlers.Ole Troan1-0/+1
Some API action handlers called vl_msg_ai_send_shmem() directly. That breaks Unix domain socket API transport. A couple (bond / vhost) also tried to send a sw_interface_event directly, but did not send the message to all that had registred interest. That scheme never worked correctly. Refactored and improved the interface event code. Change-Id: Idb90edfd8703c6ae593b36b4eeb4d3ed7da5c808 Signed-off-by: Ole Troan <ot@cisco.com>
2019-03-29IPSEC: tunnel fragmentationNeale Ranns1-2/+4
Change-Id: I63741a22bc82f5f861e1c0f26a93b5569cc52061 Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-21BVI InterfaceNeale Ranns1-0/+15
a new dedicated BVI interface as opposed to [re]using a loopback. benefits: - removes ambiguity over the purpose of a loopback interface - TX node dedicated to BVI only functions. Change-Id: I749d6b38440d450ac5b909a28053c75ec9df946a Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-13Tests: post-merge review comments for https://gerrit.fd.io/r/#/c/18081/Paul Vinciguerra1-2/+3
Change-Id: I2195b3264b5db34d869facf669f3e71a04fe3ff9 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11VPP-1508: Use scapy.compat to manage packet level library differences.Paul Vinciguerra1-6/+14
Change-Id: Icdf6abc9e53d33b26fd1d531c7dda6be0bb9cb55 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11Tests: fix time.sleep(0) # yield. Reduce sleep related log messages.Paul Vinciguerra1-2/+15
Reduce the incidence of: 20:04:23,606 unexpected time.sleep() result - slept for 2.187967e-03s instead of ~6.837845e-04s! Change-Id: Ic576fda7f75e571c9019111588a6a936ee2cf5c2 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07Tests: Refactor payload_to_info()Paul Vinciguerra1-3/+5
All callers of payload_to_info were required to wrap payload with str(). Refactor to call scapy's payload.load for raw payloads or specify the specific fieldname. Change-Id: I1c80599d4df8dc129dbb8274733afaad406d5bcf Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07VPP-1508: Python3 tests. Explicitly specify string formatting.Paul Vinciguerra1-1/+1
.format() doesn't understand what to do with binary values by default. Specify that we expect coersion to string. Change-Id: Ic4b697f70852124dc85ab231f76b7934d0d71f4d Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-26make test: Add exception handling around subprocess.Paul Vinciguerra1-1/+10
This cleans up exception catching to identify oserrors. By raising the specific exception closer to the offending call, we get additional stack history and can add clearer error logging to assist in troubleshooting. Change-Id: I592e4d46844b822a816485bf54910f8daed92088 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-20TESTS: refactor framework send_and_expect.Paul Vinciguerra1-11/+8
Create a common pg_send method for cases when results are just discarded. Change-Id: I786960d2d7bbb96dcb407f6e59aa96951b7b19e7 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-04VTL Cleanup: Fix missing calls to setUpClass/tearDownClass, fix numerous ↵Paul Vinciguerra1-0/+3
TypeErrors. * TypeError: assertIsNotNone() got an unexpected keyword argument 'msg' * Correct missing calls to setUpClass/tearDownClass. If you want the setUpClass and tearDownClass on base classes called then you must call up to them yourself. The implementations in TestCase are empty. https://docs.python.org/2/library/unittest.html#setupclass-and-teardownclass Cleans up issues in parallel test mode: ------------- FAILURES AND ERRORS IN TESTS: Testcase name: VCL Thru Host Stack Bidir Nsock FAILURE: test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock [test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock] Testcase name: Bidirectional Forwarding Detection (BFD) (changing auth) ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate] Testcase name: Punt Socket for IPv4 ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_dump [test_punt.TestIP4PuntSocket.test_punt_socket_dump] ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets] ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket] ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket] Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6) ERROR: test_bfd.BFD6TestCase.test_echo [test_bfd.BFD6TestCase.test_echo] ERROR: test_bfd.BFD6TestCase.test_echo_looped_back [test_bfd.BFD6TestCase.test_echo_looped_back] ERROR: test_bfd.BFD6TestCase.test_intf_deleted [test_bfd.BFD6TestCase.test_intf_deleted] ERROR: test_bfd.BFD6TestCase.test_session_up [test_bfd.BFD6TestCase.test_session_up] ERROR: test_bfd.BFD6TestCase.test_session_up_by_ip [test_bfd.BFD6TestCase.test_session_up_by_ip] Testcase name: Bidirectional Forwarding Detection (BFD) (CLI) ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp] ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6 [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6] ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth] ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth] ERROR: test_bfd.BFDCLITestCase.test_admin_up_down [test_bfd.BFDCLITestCase.test_admin_up_down] ERROR: test_bfd.BFDCLITestCase.test_auth_on_off [test_bfd.BFDCLITestCase.test_auth_on_off] ERROR: test_bfd.BFDCLITestCase.test_auth_on_off_delayed [test_bfd.BFDCLITestCase.test_auth_on_off_delayed] ERROR: test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key [test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key] ERROR: test_bfd.BFDCLITestCase.test_set_del_sha1_key [test_bfd.BFDCLITestCase.test_set_del_sha1_key] ERROR: test_bfd.BFDCLITestCase.test_set_del_udp_echo_source [test_bfd.BFDCLITestCase.test_set_del_udp_echo_source] ERROR: test_bfd.BFDCLITestCase.test_show [test_bfd.BFDCLITestCase.test_show] Testcase name: VAPI test ERROR: test_vapi.VAPITestCase.test_vapi_c [test_vapi.VAPITestCase.test_vapi_c] Testcase name: Container integration extended testcases ERROR: test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test [test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test] ERROR: test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands [test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands] ERROR: test_container.ContainerIntegrationTestCase.test_0410_spoof_test [test_container.ContainerIntegrationTestCase.test_0410_spoof_test] ERROR: test_container.ContainerIntegrationTestCase.test_0411_spoof_test [test_container.ContainerIntegrationTestCase.test_0411_spoof_test] Testcase name: Re-enable IPFIX ERROR: test_flowprobe.ReenableIPFIX.test_0011 [test_flowprobe.ReenableIPFIX.test_0011] Testcase name: VXLAN over IPv6 Test Case ERROR: setUpClass [setUpClass (test_vxlan6.TestVxlan6)] Testcase name: JVPP Core Test Case ERROR: test_jvpp.TestJVpp.test_vpp_acl_callback_api [test_jvpp.TestJVpp.test_vpp_acl_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_acl_future_api [test_jvpp.TestJVpp.test_vpp_acl_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_core_callback_api [test_jvpp.TestJVpp.test_vpp_core_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_core_future_api [test_jvpp.TestJVpp.test_vpp_core_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api [test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_future_api [test_jvpp.TestJVpp.test_vpp_ioamexport_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_callback_api [test_jvpp.TestJVpp.test_vpp_ioampot_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_future_api [test_jvpp.TestJVpp.test_vpp_ioampot_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_snat_callback_api [test_jvpp.TestJVpp.test_vpp_snat_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_snat_future_api [test_jvpp.TestJVpp.test_vpp_snat_future_api] Testcase name: LDP Cut Thru Tests FAILURE: test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3 [test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3] Testcase name: ACL plugin connection-oriented extended testcases ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test] Testcase name: LDP Thru Host Stack Echo FAILURE: test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo [test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo] Testcase name: Bidirectional Forwarding Detection (BFD) - API ERROR: test_bfd.BFDAPITestCase.test_activate_auth [test_bfd.BFDAPITestCase.test_activate_auth] ERROR: test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key [test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key] ERROR: test_bfd.BFDAPITestCase.test_add_bfd [test_bfd.BFDAPITestCase.test_add_bfd] ERROR: test_bfd.BFDAPITestCase.test_add_bfd6 [test_bfd.BFDAPITestCase.test_add_bfd6] ERROR: test_bfd.BFDAPITestCase.test_add_bfd_sha1 [test_bfd.BFDAPITestCase.test_add_bfd_sha1] ERROR: test_bfd.BFDAPITestCase.test_add_sha1_keys [test_bfd.BFDAPITestCase.test_add_sha1_keys] ERROR: test_bfd.BFDAPITestCase.test_change_key [test_bfd.BFDAPITestCase.test_change_key] ERROR: test_bfd.BFDAPITestCase.test_deactivate_auth [test_bfd.BFDAPITestCase.test_deactivate_auth] ERROR: test_bfd.BFDAPITestCase.test_double_add [test_bfd.BFDAPITestCase.test_double_add] ERROR: test_bfd.BFDAPITestCase.test_double_add_sha1 [test_bfd.BFDAPITestCase.test_double_add_sha1] ERROR: test_bfd.BFDAPITestCase.test_mod_bfd [test_bfd.BFDAPITestCase.test_mod_bfd] ERROR: test_bfd.BFDAPITestCase.test_set_del_udp_echo_source [test_bfd.BFDAPITestCase.test_set_del_udp_echo_source] ERROR: test_bfd.BFDAPITestCase.test_shared_sha1_key [test_bfd.BFDAPITestCase.test_shared_sha1_key] Testcase name: LDP Thru Host Stack Iperf FAILURE: test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3 [test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3] Testcase name: Bidirectional Forwarding Detection (BFD) (SHA1 auth) ERROR: test_bfd.BFDSHA1TestCase.test_session_up [test_bfd.BFDSHA1TestCase.test_session_up] Testcase name: Punt Socket for IPv6 ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_dump [test_punt.TestIP6PuntSocket.test_punt_socket_dump] ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets] ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket] ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket] Testcase name: Disable Flowprobe feature ERROR: test_flowprobe.DisableFP.test_0001 [test_flowprobe.DisableFP.test_0001] Testcase name: Disable IPFIX ERROR: test_flowprobe.DisableIPFIX.test_0001 [test_flowprobe.DisableIPFIX.test_0001] Testcase name: VPP Object Model Test ERROR: test_vom.VOMTestCase.test_vom_cpp [test_vom.VOMTestCase.test_vom_cpp] Testcase name: BFD-FIB interactions (IPv6) ERROR: test_bfd.BFDFIBTestCase.test_session_with_fib [test_bfd.BFDFIBTestCase.test_session_with_fib] Testcase name: Bidirectional Forwarding Detection (BFD) ERROR: test_bfd.BFD4TestCase.test_echo [test_bfd.BFD4TestCase.test_echo] ERROR: test_bfd.BFD4TestCase.test_echo_looped_back [test_bfd.BFD4TestCase.test_echo_looped_back] ERROR: test_bfd.BFD4TestCase.test_intf_deleted [test_bfd.BFD4TestCase.test_intf_deleted] ERROR: test_bfd.BFD4TestCase.test_session_down [test_bfd.BFD4TestCase.test_session_down] ERROR: test_bfd.BFD4TestCase.test_session_up [test_bfd.BFD4TestCase.test_session_up] ERROR: test_bfd.BFD4TestCase.test_session_up_by_ip [test_bfd.BFD4TestCase.test_session_up_by_ip] Testcase name: VXLAN-GPE Test Case ERROR: test_vxlan_gpe.TestVxlanGpe.test_decap [test_vxlan_gpe.TestVxlanGpe.test_decap] ERROR: test_vxlan_gpe.TestVxlanGpe.test_encap [test_vxlan_gpe.TestVxlanGpe.test_encap] ERROR: test_vxlan_gpe.TestVxlanGpe.test_ucast_flood [test_vxlan_gpe.TestVxlanGpe.test_ucast_flood] Testcase name: Re-enable Flowprobe feature ERROR: test_flowprobe.ReenableFP.test_0001 [test_flowprobe.ReenableFP.test_0001] Testcase name: LDP IPv6 Cut Thru Tests FAILURE: test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo [test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo] TESTCASES WHERE NO TESTS WERE SUCCESSFULLY EXECUTED: VXLAN over IPv6 Test Case ============================================================================== 0 attempt(s) left. Change-Id: Id202ed6ee7f57670f34ec87380244c568b509416 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-31make test: fix loggingKlement Sekera1-1/+1
Change-Id: I3867d50746f0f424c8e94929171363e5b2e4d470 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-22VTL: VppTestResult: Must implement empty constructor.Paul Vinciguerra1-1/+2
Subclasses of (unittest.TestResult|unittest.TextTestResult) must implement an empty constructor. The standard python library unittests depend on the empty constructor. Change-Id: I6d1bd2e7d6996d85a9da66cb7d40af0ed3c30e2b Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-21VTL: Add **kwargs to VppTestRunner constructor.Paul Vinciguerra1-2/+2
Subclasses should accept **kwargs to ensure compatibility as the interface changes. https://github.com/python/cpython/blob/master/Lib/unittest/runner.py#L133 Change-Id: Ia64ffd88899d207a5422f066700d2f7f9c77622e Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-16make test: raise packet tracing limit to 1000Klement Sekera1-2/+2
Change-Id: I4309cefe13ee861342e7962c7652919a47748656 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-14VTL: Allow running simple unittest.TestCases.Paul Vinciguerra1-11/+13
It came to my attention that Ole added a simple test in: https://gerrit.fd.io/r/#/c/16381/ and the framework forced him to launch an instance of VPP to test the formatting of a mac address. This change allows the test framework to run standard unittest.TestCases without the need to spawn a VPP instance. Change-Id: I56651ab27c4c6bf920081a526f168a743d643201 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-09test framework. Remove unused code.Paul Vinciguerra1-14/+3
Change-Id: I1bd14d5aff4918b58aef18fa0a9264e033989f6e Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21Revert "tests: Rework vpp config generation."Ole Troan1-23/+27
This reverts commit 919efad2671993d4c6d5a0dba8eeb99d5c60edf1. Change-Id: I5ec4fa08a03f508dac684dc1ddbd7e6b85c49b97 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-21Python Tests: Make VppTestCase unit testable.Paul Vinciguerra1-0/+5
* This allows for VppTestCase to instantiated with an empty constructor. Change-Id: Ib71d3bfb4529e113f9b73c39de9b95f12ee9cfb4 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21tests: Rework vpp config generation.Paul Vinciguerra1-27/+23
* Allows test cases to configure the VPP runtime config during fixture setup. * Sample use in a TestCase: @classmethod def setUpConstants(cls): tempdir = cls.tempdir cls.config.add('punt', 'socket', '%s/socket_punt' % cls.tempdir) super(TestPuntSocket, cls).setUpConstants() # enable/disabe a plugin via: #cls.config.add_plugin('dpdk_plugin.so', 'disable') * Supports the following config stanzas: 'unix', 'acl-plugin' 'api-queue' 'api-trace' 'api-segment' 'cj' 'cpu' 'dns' 'dpdk # currently don't support dynamic keys # 'heapsize' 'l2learn' 'l2tp' 'mactime' 'mc' 'nat' 'oam' 'plugins' 'punt' 'session' 'socksvr' 'statseg' 'tapcli' 'tcp' 'tuntap' 'vhost-user' Change-Id: I44f276487267d26eaa46f87e730bdd861003b234 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>