summaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2021-11-22vlib: add virtual time supportBenoît Ganne1-0/+4
Type: feature Change-Id: Iabd76558e9c72ed8286cfeeb1fbaa4fde4832a90 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2021-11-02build: remove unnecessary executable bitsRay Kinsella1-0/+0
Run 'find src -executable -type f', remove unnecessary executable bits from the source tree. Type: refactor Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I070e22c8fb2ef8712bc3ea620727ee5ab3d9a9fb
2021-10-14nat: static mappings in flow hashKlement Sekera1-0/+2
Put static mappings in flow hash, drop existing hash tables used for static mappings. Drop refcount variables and use hash table as a single point of truth. Allow creating a static mapping conflicting with dynamic mapping, which will take precedence after dynamic mapping is freed, so that the existing flow can finish transferring data. Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: Idfde8efabc09971be38921d4b0ca5ccf4e9fe412
2021-10-12Revert "nat: static mappings in flow hash"Ole Troan1-2/+0
This reverts commit 69b7599e4b061a8996205f0304232ede84cb70d4. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: If531b122ae5a9f91c2fe6eaa0da69922a91f16d3
2021-10-12nat: static mappings in flow hashKlement Sekera1-0/+2
Put static mappings in flow hash, drop existing hash tables used for static mappings. Drop refcount variables and use hash table as a single point of truth. Allow creating a static mapping conflicting with dynamic mapping, which will take precedence after dynamic mapping is freed, so that the existing flow can finish transferring data. Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: Ieeba691d83a83887d0a0baccd5f3832f66126096
2021-10-05build: don't hardcode triplet, allow specifying custom lib dirDamjan Marion1-16/+4
Type: fix Change-Id: I33f364fda88914f88f9b976cb83e6d3ff466f0bb Signed-off-by: Damjan Marion <damarion@cisco.com> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
2021-09-27classify: Large and nested classifer unit testsRay Kinsella1-8/+16
Type: test Large and nested unit tests to test AVX-512 optimized versions of the classify hash and match algorithims. Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: Ie423fee5e0fd1cb4bdf3bec8e0230a5f7cfc75fc
2021-06-16api: remove custom dumpFilip Tehlar1-2/+0
Type: improvement Change-Id: I4b9b2be8817be10e46accc19219deb2b544f266b Signed-off-by: Filip Tehlar <ftehlar@cisco.com>
2021-06-14pg: A Tunnel mode variant of a pg interfaceNeale Ranns1-2/+28
Type: feature this allows VPP to simulate linux tun devices. Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: I3adf38b49a254804370f78edd5d275d192fd00a6
2021-05-21tests: add role to VCLAppWorkerDave Wallace1-2/+3
- Improve readability of log output with respect to worker role Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I1a5787421b31483fa1a2558951568417dfc111d9
2021-05-20tests: don't buffer worker stdin/stdoutDave Wallace1-2/+3
- Required to get VCLAppWorker stdin / stdout data into log.txt when a test fails. Type: test Change-Id: I4db467986940f701750aec6d842a4d9aee22b4e6 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-05-04stats: adding symlinks for nodes and interfaces in the stat segmentArthur de Kerhor1-1/+3
A given interface counter (e.g rx) can be accessed via /interfaces/<interface_name>/<counter_name>. Same goes with nodes: /nodes/<node_name>/<counter_name> As interfaces may contain '/' in their names, these are replaced by '_' in symlinks Also added 2 tests for the stat segment Type: feature Signed-off-by: Arthur de Kerhor <arthurdekerhor@gmail.com> Change-Id: I27da252e7b3dc177815616ca46b5c200a456bf0f Signed-off-by: Ole Troan <ot@cisco.com>
2021-04-28tests: add worker count to class descriptionKlement Sekera1-1/+9
While running test: ============================================================================== NAT44ED Test Case ============================================================================== ==> ============================================================================== NAT44ED Test Case [main thread only] ============================================================================== ============================================================================== NAT44ED Test Case [1 worker thread] ============================================================================== ============================================================================== NAT44ED Test Case [4 worker threads] ============================================================================== Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I61904b411ff1b58700564698079dc4c07cea5c5e
2021-04-16tests: cpus awarenessKlement Sekera1-60/+75
Introduce MAX_CPUS parameters to control maximum number of CPUs used by VPP(s) during testing, with default value 'auto' corresponding to all CPUs available. Calculate test CPU requirements by taking into account the number of workers, so a test requires 1 (main thread) + # of worker CPUs. When running tests, keep track of both running test jobs (controlled by TEST_JOBS parameter) and free CPUs. This then causes two limits in the system - to not exceed number of jobs in parallel but also to not exceed number of CPUs available. Skip tests which require more CPUs than are available in system (or more than MAX_CPUS) and print a warning message. Type: improvement Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-12tests: support attaching to existing vppKlement Sekera1-29/+70
Introduce a new option DEBUG=attach to run a test against existing already running vpp. A new target 'make test-start-gdb' will spawn VPP in gdb for this purpose. Customization options explained in test-help. Type: improvement Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-09tests: improve namingKlement Sekera1-12/+12
Change capture to pcap to remove confusion in framework.py Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: Idf8bd61f174b8b2df3bbd13908e54d597da4a83b
2021-04-01tests: support injecting multiple worker pcaps on one PGKlement Sekera1-4/+11
This change allows one to inject multiple streams for different workers on the same PG interface at the same time. Type: improvement Change-Id: I29d80369aabada261eda466e5a5d8d3518bb8bc8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-25stats: python vpp_stats rewrite to access stat segment directlyOle Troan1-1/+1
This module implement Python access to the VPP statistics segment. It accesses the data structures directly in shared memory. VPP uses optimistic locking, so data structures may change underneath us while we are reading. Data is copied out and it's important to spend as little time as possible "holding the lock". Counters are stored in VPP as a two dimensional array. Index by thread and index (typically sw_if_index). Simple counters count only packets, Combined counters count packets and octets. Counters can be accessed in either dimension. stat['/if/rx'] - returns 2D lists stat['/if/rx'][0] - returns counters for all interfaces for thread 0 stat['/if/rx'][0][1] - returns counter for interface 1 on thread 0 stat['/if/rx'][0][1]['packets'] - returns the packet counter for interface 1 on thread 0 stat['/if/rx'][:, 1] - returns the counters for interface 1 on all threads stat['/if/rx'][:, 1].packets() - returns the packet counters for interface 1 on all threads stat['/if/rx'][:, 1].sum_packets() - returns the sum of packet counters for interface 1 on all threads stat['/if/rx-miss'][:, 1].sum() - returns the sum of packet counters for interface 1 on all threads for simple counters Type: refactor Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1fe7f7c7d11378d06be8276db5e1900ecdb8f515 Signed-off-by: Ole Troan <ot@cisco.com>
2021-03-20tests: add support for worker awarenessKlement Sekera1-25/+33
VppTestCase now has vpp_worker_count property set to number of workers. This can be overriden by child classes. Also overriden by VPP_WORKER_CONFIG variable for legacy reasons. Type: improvement Change-Id: Ic328bacb9003ddf9e92815767653bd362aa7f086 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-15tests: use socket transport instead of shared memoryOle Troan1-12/+16
Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I9e65c94a5a05047a5104e9361ea36eac77b40442 Signed-off-by: Ole Troan <ot@cisco.com>
2021-03-15tests: fix NoneType printing if VPP died earlyKlement Sekera1-12/+15
Make error message more meaningful. Type: fix Change-Id: I3c49cb179c3ee7a59657b3ae9a06311f57dc52ac Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-02classify: fix multiple filters supportBenoît Ganne1-7/+9
This fix the classify filter if we attach several different filters. This also fix some issues with l3 and l4 parsing. Type: fix Change-Id: I9dc6c55049a3bbc0110d1097b40d9da27633626b Signed-off-by: Benoît Ganne <bganne@cisco.com>
2021-02-15tests: explicitly close subprocess filesPaul Vinciguerra1-0/+2
Resolve the following error: /vpp/test/framework.py:657: ResourceWarning: unclosed file <_io.BufferedReader name=6> del cls.vpp Object allocated at (most recent call last): File "/usr/lib/python3.8/subprocess.py", lineno 844 self.stdout = io.open(c2pread, 'rb', bufsize) Type: test Change-Id: Ia2974da594f0582dcff1f4bdf40d25475769c46c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2021-02-08tests: allow for externally supplied VPP workers config for testsAndrew Yourtchenko1-1/+17
Allow to supply the external VPP worker config for tests which do not specify the workers config explicitly, and use the tags infra to flag those that need attention in this configuration. This commit shows one example use of such a tag, there will be a separate commit with the rest of the places needing it, since that change is rather mechanical. Thus, the assumption is that the test should by default be agnostic of the VPP configuration, unless it explicitly specifies so. Type: test Change-Id: I3c0077e4e22a75cb9561fb98d3b783b93486b2be Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-01-22tests: add generalized tags for tests, use them for run-solo testsAndrew Yourtchenko1-3/+29
We have accumulated several scenarios in prod or wishlists where it would be useful to have a general infra to say yes/no about a certain test, and potentially make decisions based on that, for example: - runs solo (aka 'time-dependent') - (wishlist) part of quick smoke-test set - (wishlist) intermittent failure unrelated to timing - (wishlist) test broken with a multi-worker config in vpp Refactor the current "run-solo" code to allow for this extension. Type: test Change-Id: Ia5b3810e57c0543753c8e0dc4dc0cfb4a30b36ac Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2020-12-08tests: restore vapi C++ test under Centos-8Paul Vinciguerra1-8/+0
The Centos-7 compiler had problems with vapi_cpp_test.cpp. The Centos-8 compiler does not. This change adds back the test. ============================================================================== VAPI test ============================================================================== run C VAPI tests 1.55 OK run C++ VAPI tests 0.44 OK ============================================================================== TEST RESULTS: Scheduled tests: 2 Executed tests: 2 Passed tests: 2 ============================================================================== Type: test Change-Id: Ic6a655ae99eb3ba0e73b7feef4436a0ca5fd92be Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-12-06tests: remove the dependency on subprocess32Paul Vinciguerra1-13/+1
Type: test Change-Id: I39afe56a085648d881a18c7eafcf7caf2f375f31 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-09-22lisp: Move to pluginNeale Ranns1-0/+2
Type: refactor Change-Id: I54df533a8f863c4e49742903cf2457f18b4fc506 Signed-off-by: Neale Ranns <nranns@cisco.com>
2020-08-27tests: "force solo" testcase supportAndrew Yourtchenko1-1/+16
Some of the tests are time-sensitive, and at present require a non-trivial modification in order to run at high concurrency. Without these modifications, they intermittently fail, and require the test retries. Rather than setting them to the extended tests and forgetting about them, put them into a "solo" set, which gets run in a single-threaded mode after the rest of the tests are done. Mark a few of the tests that showed errors during TEST_JOBS=48 as forced-solo. Also, give a better diagnostic if the testcase misses a docstring needed to represent it in the diagnostic outputs. Type: fix Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-05-05tests: clean up loggingPaul Vinciguerra1-12/+17
Tests currently expect the logger to be poked from run_tests.py. The tests should run without any magic values. This change sets a default null logger and removes the hasattr checks for the logger. For reference, see: https://docs.python-guide.org/writing/logging/ Type: test Change-Id: I98f953d73d12d00e74b59c94a0fb8c7a625b9c44 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-05-04misc: binary api fuzz test fixesDave Barach1-0/+5
Add a hook to src/vlibapi/api_shared.c to fuzz (screw up) binary API messages, e.g. by xoring random data into them before processing. We specifically exempt client connection messages, and inband debug CLI messages. We step over msg_id, client index, client context, and sw_if_index. Otherwise, "make test" vectors fail too rapidly to learn anything. The goal is to reduce the number of crashes caused to zero. We're fairly close with this patch. Add vl_msg_api_max_length(void *mp), which returns the maximum plausible length for a binary API message. Use it to hardern vl_api_from_api_to_new_vec(...) which takes an additional argument - message pointer - so it can verify that astr->length is sane. If it's not sane, return a u8 *vector of the form "insane astr->length nnnn\0". Verify array lengths in vl_api_dhcp6_send_client_message_t_handler(...) and vl_api_dhcp6_pd_send_client_message_t_handler(...). Add a fairly effective binary API fuzz hook to the unittest plugin, and modify the "make test" framework.py to pass "api-fuzz { on|off }" to enable API fuzzing: "make API_FUZZ=on TEST=xxx test-debug" or similar Type: improvement Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I0157267652a163c01553d5267620f719cc6c3bde
2020-04-28tests: implement ipaddress convenience methodsPaul Vinciguerra1-1/+4
Add vpp specific properties to ip addresses for use in the api. .vapi_af -- returns [ADDRESS_IP4, ADDRESS_IP6] .vapi_af_name -- returns the string ['ip4', 'ip6'] Update tests to demonstrate usage. Type: feature Change-Id: I43447a1522769d99f89debdc714c51700068d771 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-04-28vlib: startup multi-arch variant configurationRay Kinsella1-0/+8
Support for startup node multi-arch variant selection through startup.conf. This is to facilitate unit, functional testing and benchmarking of non-default multi-arch variant node code path. Also added parameters to make test, to specific using multi-arch variants in unit testing. Type: improvement Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
2020-03-12vppinfra: refactor clib_timebase_tDave Barach1-1/+6
Add a clib_time_t * argument to clib_timebase_init(...), to encourage client code to share the vlib_main_t's clib_time_t object. Display the current day / date in GMT via the "show time" debug CLI. Fix the test framework so it processes the new "show time" output format. Type: refactor Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I5e52d57eb164b7cdb6355362d520df6928491711
2020-03-10tests: add running_gcov_tests to framework.pyDave Barach1-0/+7
Mark a few code coverage tests appropriately: @unittest.skipUnless(running_gcov_tests, "part of code coverage tests") def test_my_coverage_unittest(self): <etc> Almost exactly like "make EXTENDED_TESTS=yes test". Type: feature Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Ic6d0a097a608ba93442b3d88252f66f8e6805b97
2020-02-07tests: support python 3.8Ole Troan1-2/+1
Make test framework python3 version independence. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
2020-01-14tests: fix worker thread initializationPaul Vinciguerra1-4/+4
from threading.thread __init__: This constructor should always be called with keyword arguments. If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread. Type: test Change-Id: Ifa89202e97053a4baf19e9a0ca0913430d5087a3 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-01-02tests: configure 32 mb physmemDave Barach1-0/+1
Absolutely nothing good happens when we force the kernel to briefly map and then unmap 16gb as vpp starts. Effect exacerbated when TEST_JOBS = 20...40, and so forth. Type: test Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Id8e3ce1763cad3a0891d5d6c8d2c1e3e610682d7
2019-12-23ipsec: Test and fix IPSec worker hand-offNeale Ranns1-4/+4
Type: fix Change-Id: I5cb9a3845ddbc5f4de4eb4e9c481f606fe5cec9a Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-10/+18
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-19papi: lazily initialize stats clientPaul Vinciguerra1-14/+0
remove wait-loop on stats socket from test framework. Type: refactor Change-Id: I5bb95a7c597707a87f9d9a471215c4b4af1a2280 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-05tests: fix wrong checksum error messageKlement Sekera1-1/+3
This change fixes the error message, which would previously report mismatch on IPv6 layer instead of TCP layer. Type: fix Change-Id: Ie78779685362a420d155f928b3cdb341bc3fb8bc Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-12-05papi: add call statsOle Troan1-0/+1
Type: feature Change-Id: Ic6d44122d3e62e09402e3f1946f7e57e9b5e7c5f Signed-off-by: Ole Troan <ot@cisco.com>
2019-12-05tests: Revert "Python Tests: Make VppTestCase unit testable."Paul Vinciguerra1-5/+0
Type: test This reverts commit 087c811b63aa6b0575149749e188bb37e7fb0513. Change-Id: Ica12597f27d5a75c33c12c2d27a227dc4866adcb Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-05tests: add test run time.Ole Troan1-3/+6
Type: feature Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: Id7b30b5fe63e68ae88f3a42aa77d2614ccda724d
2019-12-04tests: fix exception handling around vapi.connect()/.disconnect()Paul Vinciguerra1-5/+5
vapi.connect() only raises subclasses of vpp_papi.VPPIOError. vapi.disconnect() doesn't raise any exceptions, removed uneeded try block. Type: test Change-Id: Ide6e6db298b205f14382065862e358757619a5ba Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-11-27vlib: add 'wait' cli commandPaul Vinciguerra1-4/+3
When running exec scripts, there can be a need to wait between statements. Type: feature Change-Id: I2a45b390697e09fc222358c9354f28e3368a06ba Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-11-27tests: display stdout and stderr when output caching is disabledBenoît Ganne1-2/+2
Type: fix Change-Id: I4286ba58bb15dd67c1199f76166bedf3b4e0344c Signed-off-by: Benoît Ganne <bganne@cisco.com>
2019-11-05misc: Fix python scripts shebang lineRenato Botelho do Couto1-1/+1
Type: fix Since CentOS 8, RPM build script doesn't accept '#!/usr/bin/env python' as a valid shebang line. It requires scripts to explicitly chose between python2 or python3. Change all to use python3 as suggested by Paul Vinciguerra. Depends-On: https://gerrit.fd.io/r/23170 Signed-off-by: Renato Botelho do Couto <renato@netgate.com> Change-Id: Ie72af9f60fd0609e07f05b70f8d96e738b2754d1
2019-11-05tests: support setting random seedKlement Sekera1-1/+3
Log the random seed used when running tests and provide means to re-use it in a later run. Type: feature Change-Id: I18d2a36ee802b901d4cca5577df41cec07f09cc0 Signed-off-by: Klement Sekera <ksekera@cisco.com>