aboutsummaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2019-05-22stats: support multiple works for error countersOle Troan1-0/+5
The current code only allowed access to the main thread error counters. That is not so useful for a multi worker instance. No return a vector indexed by thread of counter_t values. Type: fix Change-Id: Ie322c8889c0c8175e1116e71de04a2cf453b9ed7 Signed-off-by: Ole Troan <ot@cisco.com>
2019-05-20reassembly: prevent long chain attackKlement Sekera1-0/+13
limit max # of fragments to 3 per packet by default add API option to configure the limit at runtime Change-Id: Ie4b9507bf5c6095b9a5925972b37fe0032f4f9e8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-05-09add mactime plugin unit / code coverage testsDave Barach1-0/+4
The unit and code coverage tests are boring. The rest of the patch involves test and packet-generator infra cleanups. Teach the "make test-xxx" family of targets to set the api test plugin path correctly, to make "binary-api <api-message-name> <args>" debug CLI commands work correctly in the "make test" environment. Unfortunately involves both the top-level and test Makefiles. Add a minor pg cli feature, a CLI to manually set s->sw_if_index[VLIB_TX]. Consider the case where one configures an interface with both a device-input and an output feature. To test the output feature using the pg, it's necessary to inject packets into the interface output node with both b->sw_if_index[VLIB_TX] and b->sw_if_index[VLIB_RX] set correctly. For example: packet-generator new { name tx limit 15 size 128-128 interface local0 # rx: device input feature not configured on local0 tx-interface loop0 # tx: output node requires b->sw_if_index[VLIB_TX] node loop0-output data { hex 0x01005e7ffffa000dead0000008000102030405060708090a0b0c0d0e0f0102030405 } } Fix a longstanding bug in the packet generator stream setup. Remove kludges which set b->sw_if_index[VLIB_TX] to ~0 [in multiple places] instead of using the stream value s->sw_if_index[VLIB_TX], and setting THAT datum correctly. Change-Id: I1097a18e8db73661ded6b822c1d718f7e5cf36ed Signed-off-by: Dave Barach <dave@barachs.net>
2019-05-07make test: add python3 compatibility in extend_packetAlexandre Poirrier1-2/+2
Change-Id: I342ff8e44e0d1ebd9f2c8cc386372b2f5a731080 Signed-off-by: Alexandre Poirrier <apoirrie@cisco.com>
2019-05-06make test: Fix truncated core fileDave Wallace1-0/+23
- Wait for coredump to complete before killing VPP. Change-Id: Ib7b96b339f796f9ef7e0eadff81d181464b8a03a Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2019-05-06gcov / test framework: sigterm not sigkillDave Barach1-1/+1
Otherwise, gcov data vanishes without a trace. Add a __gcov_flush() call to the unix signal handler, under #ifdef CLIB_GCOV. Add -DCLIB_GCOV to vpp_gcov_TAG_CFLAGS. Change-Id: I2726e671b26dfbe7fae88f46a8207bb2b5106884 Signed-off-by: Dave Barach <dave@barachs.net>
2019-04-23API: Python and Unix domain socket improvementOle Troan1-2/+5
Handle the case where buffer overflows. Then SOCK_SEQPACKET assumption that multiple API messages are not returned by recv() is broken. Use SOCK_STREAM for API exchanges instead. Add support for running tests over sockets. make test SOCKET=1 Change-Id: Ibe5fd69b1bf617de4c7ba6cce0a7c2b3f97a2821 Signed-off-by: Ole Troan <ot@cisco.com>
2019-04-11Tests: Refactor tearDown show command logging, add lifecycle markers.Paul Vinciguerra1-3/+13
This change adds a consistent interface for adding test-specific show commands to log.txt. It also adds log markers for the execution of setUp[Class], tearDown[Class] in the logs. Change-Id: I7d42e396e594a59e866a7d55dac0af25548e657a Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-04-10API: Fix shared memory only action handlers.Ole Troan1-0/+1
Some API action handlers called vl_msg_ai_send_shmem() directly. That breaks Unix domain socket API transport. A couple (bond / vhost) also tried to send a sw_interface_event directly, but did not send the message to all that had registred interest. That scheme never worked correctly. Refactored and improved the interface event code. Change-Id: Idb90edfd8703c6ae593b36b4eeb4d3ed7da5c808 Signed-off-by: Ole Troan <ot@cisco.com>
2019-03-29IPSEC: tunnel fragmentationNeale Ranns1-2/+4
Change-Id: I63741a22bc82f5f861e1c0f26a93b5569cc52061 Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-21BVI InterfaceNeale Ranns1-0/+15
a new dedicated BVI interface as opposed to [re]using a loopback. benefits: - removes ambiguity over the purpose of a loopback interface - TX node dedicated to BVI only functions. Change-Id: I749d6b38440d450ac5b909a28053c75ec9df946a Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-03-13Tests: post-merge review comments for https://gerrit.fd.io/r/#/c/18081/Paul Vinciguerra1-2/+3
Change-Id: I2195b3264b5db34d869facf669f3e71a04fe3ff9 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11VPP-1508: Use scapy.compat to manage packet level library differences.Paul Vinciguerra1-6/+14
Change-Id: Icdf6abc9e53d33b26fd1d531c7dda6be0bb9cb55 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-11Tests: fix time.sleep(0) # yield. Reduce sleep related log messages.Paul Vinciguerra1-2/+15
Reduce the incidence of: 20:04:23,606 unexpected time.sleep() result - slept for 2.187967e-03s instead of ~6.837845e-04s! Change-Id: Ic576fda7f75e571c9019111588a6a936ee2cf5c2 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07Tests: Refactor payload_to_info()Paul Vinciguerra1-3/+5
All callers of payload_to_info were required to wrap payload with str(). Refactor to call scapy's payload.load for raw payloads or specify the specific fieldname. Change-Id: I1c80599d4df8dc129dbb8274733afaad406d5bcf Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-03-07VPP-1508: Python3 tests. Explicitly specify string formatting.Paul Vinciguerra1-1/+1
.format() doesn't understand what to do with binary values by default. Specify that we expect coersion to string. Change-Id: Ic4b697f70852124dc85ab231f76b7934d0d71f4d Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-26make test: Add exception handling around subprocess.Paul Vinciguerra1-1/+10
This cleans up exception catching to identify oserrors. By raising the specific exception closer to the offending call, we get additional stack history and can add clearer error logging to assist in troubleshooting. Change-Id: I592e4d46844b822a816485bf54910f8daed92088 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-20TESTS: refactor framework send_and_expect.Paul Vinciguerra1-11/+8
Create a common pg_send method for cases when results are just discarded. Change-Id: I786960d2d7bbb96dcb407f6e59aa96951b7b19e7 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-02-04VTL Cleanup: Fix missing calls to setUpClass/tearDownClass, fix numerous ↵Paul Vinciguerra1-0/+3
TypeErrors. * TypeError: assertIsNotNone() got an unexpected keyword argument 'msg' * Correct missing calls to setUpClass/tearDownClass. If you want the setUpClass and tearDownClass on base classes called then you must call up to them yourself. The implementations in TestCase are empty. https://docs.python.org/2/library/unittest.html#setupclass-and-teardownclass Cleans up issues in parallel test mode: ------------- FAILURES AND ERRORS IN TESTS: Testcase name: VCL Thru Host Stack Bidir Nsock FAILURE: test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock [test_vcl.VCLThruHostStackBidirNsock.test_vcl_thru_host_stack_bi_dir_nsock] Testcase name: Bidirectional Forwarding Detection (BFD) (changing auth) ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_delayed] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_change_key_immediate] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_off_delayed] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_off_immediate] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed [test_bfd.BFDAuthOnOffTestCase.test_auth_on_delayed] ERROR: test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate [test_bfd.BFDAuthOnOffTestCase.test_auth_on_immediate] Testcase name: Punt Socket for IPv4 ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_dump [test_punt.TestIP4PuntSocket.test_punt_socket_dump] ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets] ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_multi_ports_single_socket] ERROR: test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP4PuntSocket.test_punt_socket_traffic_single_port_single_socket] Testcase name: Bidirectional Forwarding Detection (BFD) (IPv6) ERROR: test_bfd.BFD6TestCase.test_echo [test_bfd.BFD6TestCase.test_echo] ERROR: test_bfd.BFD6TestCase.test_echo_looped_back [test_bfd.BFD6TestCase.test_echo_looped_back] ERROR: test_bfd.BFD6TestCase.test_intf_deleted [test_bfd.BFD6TestCase.test_intf_deleted] ERROR: test_bfd.BFD6TestCase.test_session_up [test_bfd.BFD6TestCase.test_session_up] ERROR: test_bfd.BFD6TestCase.test_session_up_by_ip [test_bfd.BFD6TestCase.test_session_up_by_ip] Testcase name: Bidirectional Forwarding Detection (BFD) (CLI) ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp] ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6 [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6] ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp6_auth] ERROR: test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth [test_bfd.BFDCLITestCase.test_add_mod_del_bfd_udp_auth] ERROR: test_bfd.BFDCLITestCase.test_admin_up_down [test_bfd.BFDCLITestCase.test_admin_up_down] ERROR: test_bfd.BFDCLITestCase.test_auth_on_off [test_bfd.BFDCLITestCase.test_auth_on_off] ERROR: test_bfd.BFDCLITestCase.test_auth_on_off_delayed [test_bfd.BFDCLITestCase.test_auth_on_off_delayed] ERROR: test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key [test_bfd.BFDCLITestCase.test_set_del_meticulous_sha1_key] ERROR: test_bfd.BFDCLITestCase.test_set_del_sha1_key [test_bfd.BFDCLITestCase.test_set_del_sha1_key] ERROR: test_bfd.BFDCLITestCase.test_set_del_udp_echo_source [test_bfd.BFDCLITestCase.test_set_del_udp_echo_source] ERROR: test_bfd.BFDCLITestCase.test_show [test_bfd.BFDCLITestCase.test_show] Testcase name: VAPI test ERROR: test_vapi.VAPITestCase.test_vapi_c [test_vapi.VAPITestCase.test_vapi_c] Testcase name: Container integration extended testcases ERROR: test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0010_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0011_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test [test_container.ContainerIntegrationTestCase.test_0050_loopback_prepare_test] ERROR: test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0110_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0111_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0200_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0210_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test [test_container.ContainerIntegrationTestCase.test_0211_basic_conn_test] ERROR: test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands [test_container.ContainerIntegrationTestCase.test_0300_unconfigure_commands] ERROR: test_container.ContainerIntegrationTestCase.test_0410_spoof_test [test_container.ContainerIntegrationTestCase.test_0410_spoof_test] ERROR: test_container.ContainerIntegrationTestCase.test_0411_spoof_test [test_container.ContainerIntegrationTestCase.test_0411_spoof_test] Testcase name: Re-enable IPFIX ERROR: test_flowprobe.ReenableIPFIX.test_0011 [test_flowprobe.ReenableIPFIX.test_0011] Testcase name: VXLAN over IPv6 Test Case ERROR: setUpClass [setUpClass (test_vxlan6.TestVxlan6)] Testcase name: JVPP Core Test Case ERROR: test_jvpp.TestJVpp.test_vpp_acl_callback_api [test_jvpp.TestJVpp.test_vpp_acl_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_acl_future_api [test_jvpp.TestJVpp.test_vpp_acl_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_core_callback_api [test_jvpp.TestJVpp.test_vpp_core_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_core_future_api [test_jvpp.TestJVpp.test_vpp_core_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api [test_jvpp.TestJVpp.test_vpp_ioamexport_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamexport_future_api [test_jvpp.TestJVpp.test_vpp_ioamexport_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_callback_api [test_jvpp.TestJVpp.test_vpp_ioampot_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioampot_future_api [test_jvpp.TestJVpp.test_vpp_ioampot_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api [test_jvpp.TestJVpp.test_vpp_ioamtrace_future_api] ERROR: test_jvpp.TestJVpp.test_vpp_snat_callback_api [test_jvpp.TestJVpp.test_vpp_snat_callback_api] ERROR: test_jvpp.TestJVpp.test_vpp_snat_future_api [test_jvpp.TestJVpp.test_vpp_snat_future_api] Testcase name: LDP Cut Thru Tests FAILURE: test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3 [test_vcl.LDPCutThruTestCase.test_ldp_cut_thru_iperf3] Testcase name: ACL plugin connection-oriented extended testcases ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0000_conn_prepare_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0001_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0002_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0005_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0006_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0011_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_0012_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1001_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1002_basic_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1005_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1006_clear_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1011_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_1012_active_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2000_prepare_for_tcp_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2001_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2002_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2003_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2004_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2005_tcp_transient_teardown_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_2006_tcp_transient_teardown_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3001_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3002_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3003_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3004_tcp_transient_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3005_tcp_transient_teardown_conn_test] ERROR: test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test [test_acl_plugin_conns.ACLPluginConnTestCase.test_3006_tcp_transient_teardown_conn_test] Testcase name: LDP Thru Host Stack Echo FAILURE: test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo [test_vcl.LDPThruHostStackEcho.test_ldp_thru_host_stack_echo] Testcase name: Bidirectional Forwarding Detection (BFD) - API ERROR: test_bfd.BFDAPITestCase.test_activate_auth [test_bfd.BFDAPITestCase.test_activate_auth] ERROR: test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key [test_bfd.BFDAPITestCase.test_add_auth_nonexistent_key] ERROR: test_bfd.BFDAPITestCase.test_add_bfd [test_bfd.BFDAPITestCase.test_add_bfd] ERROR: test_bfd.BFDAPITestCase.test_add_bfd6 [test_bfd.BFDAPITestCase.test_add_bfd6] ERROR: test_bfd.BFDAPITestCase.test_add_bfd_sha1 [test_bfd.BFDAPITestCase.test_add_bfd_sha1] ERROR: test_bfd.BFDAPITestCase.test_add_sha1_keys [test_bfd.BFDAPITestCase.test_add_sha1_keys] ERROR: test_bfd.BFDAPITestCase.test_change_key [test_bfd.BFDAPITestCase.test_change_key] ERROR: test_bfd.BFDAPITestCase.test_deactivate_auth [test_bfd.BFDAPITestCase.test_deactivate_auth] ERROR: test_bfd.BFDAPITestCase.test_double_add [test_bfd.BFDAPITestCase.test_double_add] ERROR: test_bfd.BFDAPITestCase.test_double_add_sha1 [test_bfd.BFDAPITestCase.test_double_add_sha1] ERROR: test_bfd.BFDAPITestCase.test_mod_bfd [test_bfd.BFDAPITestCase.test_mod_bfd] ERROR: test_bfd.BFDAPITestCase.test_set_del_udp_echo_source [test_bfd.BFDAPITestCase.test_set_del_udp_echo_source] ERROR: test_bfd.BFDAPITestCase.test_shared_sha1_key [test_bfd.BFDAPITestCase.test_shared_sha1_key] Testcase name: LDP Thru Host Stack Iperf FAILURE: test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3 [test_vcl.LDPThruHostStackIperf.test_ldp_thru_host_stack_iperf3] Testcase name: Bidirectional Forwarding Detection (BFD) (SHA1 auth) ERROR: test_bfd.BFDSHA1TestCase.test_session_up [test_bfd.BFDSHA1TestCase.test_session_up] Testcase name: Punt Socket for IPv6 ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_dump [test_punt.TestIP6PuntSocket.test_punt_socket_dump] ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_port_multi_sockets] ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_multi_ports_single_socket] ERROR: test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket [test_punt.TestIP6PuntSocket.test_punt_socket_traffic_single_port_single_socket] Testcase name: Disable Flowprobe feature ERROR: test_flowprobe.DisableFP.test_0001 [test_flowprobe.DisableFP.test_0001] Testcase name: Disable IPFIX ERROR: test_flowprobe.DisableIPFIX.test_0001 [test_flowprobe.DisableIPFIX.test_0001] Testcase name: VPP Object Model Test ERROR: test_vom.VOMTestCase.test_vom_cpp [test_vom.VOMTestCase.test_vom_cpp] Testcase name: BFD-FIB interactions (IPv6) ERROR: test_bfd.BFDFIBTestCase.test_session_with_fib [test_bfd.BFDFIBTestCase.test_session_with_fib] Testcase name: Bidirectional Forwarding Detection (BFD) ERROR: test_bfd.BFD4TestCase.test_echo [test_bfd.BFD4TestCase.test_echo] ERROR: test_bfd.BFD4TestCase.test_echo_looped_back [test_bfd.BFD4TestCase.test_echo_looped_back] ERROR: test_bfd.BFD4TestCase.test_intf_deleted [test_bfd.BFD4TestCase.test_intf_deleted] ERROR: test_bfd.BFD4TestCase.test_session_down [test_bfd.BFD4TestCase.test_session_down] ERROR: test_bfd.BFD4TestCase.test_session_up [test_bfd.BFD4TestCase.test_session_up] ERROR: test_bfd.BFD4TestCase.test_session_up_by_ip [test_bfd.BFD4TestCase.test_session_up_by_ip] Testcase name: VXLAN-GPE Test Case ERROR: test_vxlan_gpe.TestVxlanGpe.test_decap [test_vxlan_gpe.TestVxlanGpe.test_decap] ERROR: test_vxlan_gpe.TestVxlanGpe.test_encap [test_vxlan_gpe.TestVxlanGpe.test_encap] ERROR: test_vxlan_gpe.TestVxlanGpe.test_ucast_flood [test_vxlan_gpe.TestVxlanGpe.test_ucast_flood] Testcase name: Re-enable Flowprobe feature ERROR: test_flowprobe.ReenableFP.test_0001 [test_flowprobe.ReenableFP.test_0001] Testcase name: LDP IPv6 Cut Thru Tests FAILURE: test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo [test_vcl.LDPIpv6CutThruTestCase.test_ldp_ipv6_cut_thru_echo] TESTCASES WHERE NO TESTS WERE SUCCESSFULLY EXECUTED: VXLAN over IPv6 Test Case ============================================================================== 0 attempt(s) left. Change-Id: Id202ed6ee7f57670f34ec87380244c568b509416 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-31make test: fix loggingKlement Sekera1-1/+1
Change-Id: I3867d50746f0f424c8e94929171363e5b2e4d470 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-22VTL: VppTestResult: Must implement empty constructor.Paul Vinciguerra1-1/+2
Subclasses of (unittest.TestResult|unittest.TextTestResult) must implement an empty constructor. The standard python library unittests depend on the empty constructor. Change-Id: I6d1bd2e7d6996d85a9da66cb7d40af0ed3c30e2b Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-21VTL: Add **kwargs to VppTestRunner constructor.Paul Vinciguerra1-2/+2
Subclasses should accept **kwargs to ensure compatibility as the interface changes. https://github.com/python/cpython/blob/master/Lib/unittest/runner.py#L133 Change-Id: Ia64ffd88899d207a5422f066700d2f7f9c77622e Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-16make test: raise packet tracing limit to 1000Klement Sekera1-2/+2
Change-Id: I4309cefe13ee861342e7962c7652919a47748656 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-14VTL: Allow running simple unittest.TestCases.Paul Vinciguerra1-11/+13
It came to my attention that Ole added a simple test in: https://gerrit.fd.io/r/#/c/16381/ and the framework forced him to launch an instance of VPP to test the formatting of a mac address. This change allows the test framework to run standard unittest.TestCases without the need to spawn a VPP instance. Change-Id: I56651ab27c4c6bf920081a526f168a743d643201 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-09test framework. Remove unused code.Paul Vinciguerra1-14/+3
Change-Id: I1bd14d5aff4918b58aef18fa0a9264e033989f6e Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21Revert "tests: Rework vpp config generation."Ole Troan1-23/+27
This reverts commit 919efad2671993d4c6d5a0dba8eeb99d5c60edf1. Change-Id: I5ec4fa08a03f508dac684dc1ddbd7e6b85c49b97 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-21Python Tests: Make VppTestCase unit testable.Paul Vinciguerra1-0/+5
* This allows for VppTestCase to instantiated with an empty constructor. Change-Id: Ib71d3bfb4529e113f9b73c39de9b95f12ee9cfb4 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21tests: Rework vpp config generation.Paul Vinciguerra1-27/+23
* Allows test cases to configure the VPP runtime config during fixture setup. * Sample use in a TestCase: @classmethod def setUpConstants(cls): tempdir = cls.tempdir cls.config.add('punt', 'socket', '%s/socket_punt' % cls.tempdir) super(TestPuntSocket, cls).setUpConstants() # enable/disabe a plugin via: #cls.config.add_plugin('dpdk_plugin.so', 'disable') * Supports the following config stanzas: 'unix', 'acl-plugin' 'api-queue' 'api-trace' 'api-segment' 'cj' 'cpu' 'dns' 'dpdk # currently don't support dynamic keys # 'heapsize' 'l2learn' 'l2tp' 'mactime' 'mc' 'nat' 'oam' 'plugins' 'punt' 'session' 'socksvr' 'statseg' 'tapcli' 'tcp' 'tuntap' 'vhost-user' Change-Id: I44f276487267d26eaa46f87e730bdd861003b234 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20VPP-1508 python3 tests: raw_inputPaul Vinciguerra1-3/+9
'raw_input' does not exist in python3. Change-Id: I95d916db2834fab83e5e791f35103a4513e610b8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20Tests: Cleanup @skip decorator.Paul Vinciguerra1-4/+12
The runnning environment is static as of module load time, so only evalute the conditions once at module load time. Track-by: VPP-1518 Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-17make test: add option for adding extra vpp configKlement Sekera1-1/+2
Change-Id: Ief316756635794dadf2f0f4190aa20745df5e53c Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-12-13VPP-1522: harden reassembly codeKlement Sekera1-9/+12
Change-Id: Ib5a20bff7d8a340ecf50bcd4a023d6bf36382ba3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-12-13API: Use string type instead of u8.Ole Troan1-1/+1
The new string type is modelled after string in proto3. It is always variable length. Change-Id: I64884067e28a80072c8dac31b7c7c82d6e306051 Signed-off-by: Ole Troan <ot@cisco.com> Signed-off-by: Michal Cmarada <mcmarada@cisco.com> Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-10test/framework.py: change logger to return 'e' format.Paul Vinciguerra1-15/+13
* When the framework logs an unexpected sleep time, display the units in the same format. Typically, it has been returning a float and an 'e'. ex. unexpected time.sleep() result - slept for 0.000107049942017s instead of ~4.88758087158e-05s! * Defer logger interpolation. Change-Id: I543ad6d41c7f263d61615341437f80973c0017de Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-10Test framework: StringIO fixes for Python3Ole Troan1-3/+3
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-07VPP-1506: dump local punts and registered punt socketsPavel Kotucek1-0/+4
Change-Id: If7835e9b80ec9402404bfc8d271eb11a10ef992b Signed-off-by: Pavel Kotucek <pavel.kotucek@pantheon.tech>
2018-12-06Skip failing ARM testcases for CIjuraj.linkes1-0/+9
There are a few tests that fail on ARM and thus we can't enable testing in CI. Skip the failing tests until they're fixed so that we catch new failures in CI (when we enable testing for the remaining tests). Change-Id: Ie896ef5c449ef965029633e38d317a8d5ac26352 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-29make test: create virtualenv under /test/Klement Sekera1-3/+3
instead of using build-root, use /test/venv directory for virtualenv similarly, don't pollute build-root with test-built binaries Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-26Split non-parallel testsuitejuraj.linkes1-6/+26
Split one big suite into smaller suites when not running tests in parallel. This results in all tests being executed in one iteration. Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-26Revert "VPP-1508 python3 tests: raw_input"Ole Trøan1-24/+19
This reverts commit 72f0004ac5f6e201cbe042593d76df6f2491d743. Reason for revert: Traceback (most recent call last): File "/home/ksekera/vpp/test/test_ipsec_ah.py", line 36, in setUpClass super(TemplateIpsecAh, cls).setUpClass() File "/home/ksekera/vpp/test/template_ipsec.py", line 105, in setUpClass super(TemplateIpsec, cls).setUpClass() File "/home/ksekera/vpp/test/framework.py", line 459, in setUpClass cls.quit() File "/home/ksekera/vpp/test/framework.py", line 475, in quit six.input("When done debugging, press ENTER to kill the " AttributeError: 'module' object has no attribute 'input' Change-Id: Idf0bbfea231730b37bae5dcb4557a0f82ab1b810 Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-26Fix swapped decorators in VppTestCase.Paul Vinciguerra1-5/+5
Change-Id: I39b5a8e368f0e0b5c83203141d01f22d909b6f9d Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-26VPP-1508 python3 tests: raw_inputPaul Vinciguerra1-19/+24
Raw input does not exist in python3 use six.input. Change-Id: Ie461696ff869057bcc3969a7571602dde570f5b8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-25make test: Fix too wide subprocess exceptionsPaul Vinciguerra1-1/+1
When a command fails, CalledProcessError is raised. testing with except: masks other failures. Change-Id: I7e3a6739411cb6a4c13e96dd123aff9159213fea Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-23Fix test framework keepalivejuraj.linkes1-2/+1
The pipe used for sending keepalive messages was being added to an instance of KeepAliveReporter and then used by the class itself. This worked in the past but doesn't anymore. Fix the issue by adding the pipe to the class instead of an instance. Change-Id: If7cdca2de23ca78448e80569b155e9e29e81ff94 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-23Fix test logging propagationjuraj.linkes1-0/+1
When running tests with child processes the logs from child processes would propagate to root logger, potentially resulting in some logs being emitted twice. Fix this by disabling log propagation to parent loggers in child processes. Change-Id: I31eb265c2b7f7bceff627043956a67d6def3da2b Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-22Revert "Fix instance method call in test/framework.py"Ole Trøan1-6/+9
This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968. Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71 Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-22Fix instance method call in test/framework.pyPaul Vinciguerra1-9/+6
Instance method breaking encapsulation by calling a global fn. Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-19VPP-1498: test/framework.py: AttributeErrorPaul Vinciguerra1-1/+2
Traceback if .send_keep_alive is called before pipe is configured. Traceback (most recent call last): File "/vpp/test/test_span.py", line 27, in setUpClass super(TestSpan, cls).setUpClass() File "/vpp/test/framework.py", line 411, in setUpClass cls.reporter.send_keep_alive(cls, 'setUpClass') File "/vpp/test/framework.py", line 172, in send_keep_alive if self.pipe is None: File "/vpp/test/framework.py", line 160, in pipe return self._pipe AttributeError: 'KeepAliveReporter' object has no attribute '_pipe' Change-Id: I561d2748441702478a84fbb4580a4d2667d70ffd Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-16VPP-1495: Fix intermittent stat sock test failures.Paul Vinciguerra1-1/+7
Loosen polling loop while waiting for stats socket. ============================================================================== Classifier IP6 UDP proto Test Case ============================================================================== connect: No such file or directory ============================================================================== ERROR: setUpClass (test_ip4.TestIPLoadBalance) ------------------------------------------------------------------------------ Traceback (most recent call last): File "/vpp/test/framework.py", line 424, in setUpClass cls.statistics = VPPStats(socketname=cls.stats_sock) File "build/bdist.linux-x86_64/egg/vpp_papi/vpp_stats.py", line 120, in _init_ raise IOError() IOError Change-Id: I80523122f96eafc41b3ebbf27bee73f6c637d781 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-09Fix logging into multiple filesjuraj.linkes1-5/+4
When running tests in one process, only one logger was used and each testcase added its own file handler, which resulted in logs appearing in multiple files. Fix this by restoring the creation of new loggers for each testcase and only reuse the stream handler from parent process. Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>