summaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2019-01-16make test: raise packet tracing limit to 1000Klement Sekera1-2/+2
Change-Id: I4309cefe13ee861342e7962c7652919a47748656 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2019-01-14VTL: Allow running simple unittest.TestCases.Paul Vinciguerra1-11/+13
It came to my attention that Ole added a simple test in: https://gerrit.fd.io/r/#/c/16381/ and the framework forced him to launch an instance of VPP to test the formatting of a mac address. This change allows the test framework to run standard unittest.TestCases without the need to spawn a VPP instance. Change-Id: I56651ab27c4c6bf920081a526f168a743d643201 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-01-09test framework. Remove unused code.Paul Vinciguerra1-14/+3
Change-Id: I1bd14d5aff4918b58aef18fa0a9264e033989f6e Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21Revert "tests: Rework vpp config generation."Ole Troan1-23/+27
This reverts commit 919efad2671993d4c6d5a0dba8eeb99d5c60edf1. Change-Id: I5ec4fa08a03f508dac684dc1ddbd7e6b85c49b97 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-21Python Tests: Make VppTestCase unit testable.Paul Vinciguerra1-0/+5
* This allows for VppTestCase to instantiated with an empty constructor. Change-Id: Ib71d3bfb4529e113f9b73c39de9b95f12ee9cfb4 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-21tests: Rework vpp config generation.Paul Vinciguerra1-27/+23
* Allows test cases to configure the VPP runtime config during fixture setup. * Sample use in a TestCase: @classmethod def setUpConstants(cls): tempdir = cls.tempdir cls.config.add('punt', 'socket', '%s/socket_punt' % cls.tempdir) super(TestPuntSocket, cls).setUpConstants() # enable/disabe a plugin via: #cls.config.add_plugin('dpdk_plugin.so', 'disable') * Supports the following config stanzas: 'unix', 'acl-plugin' 'api-queue' 'api-trace' 'api-segment' 'cj' 'cpu' 'dns' 'dpdk # currently don't support dynamic keys # 'heapsize' 'l2learn' 'l2tp' 'mactime' 'mc' 'nat' 'oam' 'plugins' 'punt' 'session' 'socksvr' 'statseg' 'tapcli' 'tcp' 'tuntap' 'vhost-user' Change-Id: I44f276487267d26eaa46f87e730bdd861003b234 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20VPP-1508 python3 tests: raw_inputPaul Vinciguerra1-3/+9
'raw_input' does not exist in python3. Change-Id: I95d916db2834fab83e5e791f35103a4513e610b8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-20Tests: Cleanup @skip decorator.Paul Vinciguerra1-4/+12
The runnning environment is static as of module load time, so only evalute the conditions once at module load time. Track-by: VPP-1518 Change-Id: I73b0d17ae1ff90789e70307f168d43921829aec8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-17make test: add option for adding extra vpp configKlement Sekera1-1/+2
Change-Id: Ief316756635794dadf2f0f4190aa20745df5e53c Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-12-13VPP-1522: harden reassembly codeKlement Sekera1-9/+12
Change-Id: Ib5a20bff7d8a340ecf50bcd4a023d6bf36382ba3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-12-13API: Use string type instead of u8.Ole Troan1-1/+1
The new string type is modelled after string in proto3. It is always variable length. Change-Id: I64884067e28a80072c8dac31b7c7c82d6e306051 Signed-off-by: Ole Troan <ot@cisco.com> Signed-off-by: Michal Cmarada <mcmarada@cisco.com> Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-10test/framework.py: change logger to return 'e' format.Paul Vinciguerra1-15/+13
* When the framework logs an unexpected sleep time, display the units in the same format. Typically, it has been returning a float and an 'e'. ex. unexpected time.sleep() result - slept for 0.000107049942017s instead of ~4.88758087158e-05s! * Defer logger interpolation. Change-Id: I543ad6d41c7f263d61615341437f80973c0017de Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-12-10Test framework: StringIO fixes for Python3Ole Troan1-3/+3
Add 2/3 support to binarytomac and mactobinary and move to vpp_mac.py Change-Id: I3dc7e4a24486aee22140c781aae7e44e58935877 Signed-off-by: Ole Troan <ot@cisco.com>
2018-12-07VPP-1506: dump local punts and registered punt socketsPavel Kotucek1-0/+4
Change-Id: If7835e9b80ec9402404bfc8d271eb11a10ef992b Signed-off-by: Pavel Kotucek <pavel.kotucek@pantheon.tech>
2018-12-06Skip failing ARM testcases for CIjuraj.linkes1-0/+9
There are a few tests that fail on ARM and thus we can't enable testing in CI. Skip the failing tests until they're fixed so that we catch new failures in CI (when we enable testing for the remaining tests). Change-Id: Ie896ef5c449ef965029633e38d317a8d5ac26352 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-29make test: create virtualenv under /test/Klement Sekera1-3/+3
instead of using build-root, use /test/venv directory for virtualenv similarly, don't pollute build-root with test-built binaries Change-Id: I1e63c04037eaee718b27b34ef16c9eb0252afa53 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-26Split non-parallel testsuitejuraj.linkes1-6/+26
Split one big suite into smaller suites when not running tests in parallel. This results in all tests being executed in one iteration. Change-Id: I0d3d357a95d9cc596b606d5911a5819e8ffdeee5 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-26Revert "VPP-1508 python3 tests: raw_input"Ole Trøan1-24/+19
This reverts commit 72f0004ac5f6e201cbe042593d76df6f2491d743. Reason for revert: Traceback (most recent call last): File "/home/ksekera/vpp/test/test_ipsec_ah.py", line 36, in setUpClass super(TemplateIpsecAh, cls).setUpClass() File "/home/ksekera/vpp/test/template_ipsec.py", line 105, in setUpClass super(TemplateIpsec, cls).setUpClass() File "/home/ksekera/vpp/test/framework.py", line 459, in setUpClass cls.quit() File "/home/ksekera/vpp/test/framework.py", line 475, in quit six.input("When done debugging, press ENTER to kill the " AttributeError: 'module' object has no attribute 'input' Change-Id: Idf0bbfea231730b37bae5dcb4557a0f82ab1b810 Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-26Fix swapped decorators in VppTestCase.Paul Vinciguerra1-5/+5
Change-Id: I39b5a8e368f0e0b5c83203141d01f22d909b6f9d Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-26VPP-1508 python3 tests: raw_inputPaul Vinciguerra1-19/+24
Raw input does not exist in python3 use six.input. Change-Id: Ie461696ff869057bcc3969a7571602dde570f5b8 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-25make test: Fix too wide subprocess exceptionsPaul Vinciguerra1-1/+1
When a command fails, CalledProcessError is raised. testing with except: masks other failures. Change-Id: I7e3a6739411cb6a4c13e96dd123aff9159213fea Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-23Fix test framework keepalivejuraj.linkes1-2/+1
The pipe used for sending keepalive messages was being added to an instance of KeepAliveReporter and then used by the class itself. This worked in the past but doesn't anymore. Fix the issue by adding the pipe to the class instead of an instance. Change-Id: If7cdca2de23ca78448e80569b155e9e29e81ff94 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-23Fix test logging propagationjuraj.linkes1-0/+1
When running tests with child processes the logs from child processes would propagate to root logger, potentially resulting in some logs being emitted twice. Fix this by disabling log propagation to parent loggers in child processes. Change-Id: I31eb265c2b7f7bceff627043956a67d6def3da2b Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-22Revert "Fix instance method call in test/framework.py"Ole Trøan1-6/+9
This reverts commit c32023b9fd6970ed1cac1b3c7f36233b536d9968. Change-Id: Ic934d223b10028093b0262e28515bde3ae1ccb71 Signed-off-by: Ole Troan <ot@cisco.com>
2018-11-22Fix instance method call in test/framework.pyPaul Vinciguerra1-9/+6
Instance method breaking encapsulation by calling a global fn. Change-Id: Ifde2a207951143764aed75f1b191aed0bac83704 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-19VPP-1498: test/framework.py: AttributeErrorPaul Vinciguerra1-1/+2
Traceback if .send_keep_alive is called before pipe is configured. Traceback (most recent call last): File "/vpp/test/test_span.py", line 27, in setUpClass super(TestSpan, cls).setUpClass() File "/vpp/test/framework.py", line 411, in setUpClass cls.reporter.send_keep_alive(cls, 'setUpClass') File "/vpp/test/framework.py", line 172, in send_keep_alive if self.pipe is None: File "/vpp/test/framework.py", line 160, in pipe return self._pipe AttributeError: 'KeepAliveReporter' object has no attribute '_pipe' Change-Id: I561d2748441702478a84fbb4580a4d2667d70ffd Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-16VPP-1495: Fix intermittent stat sock test failures.Paul Vinciguerra1-1/+7
Loosen polling loop while waiting for stats socket. ============================================================================== Classifier IP6 UDP proto Test Case ============================================================================== connect: No such file or directory ============================================================================== ERROR: setUpClass (test_ip4.TestIPLoadBalance) ------------------------------------------------------------------------------ Traceback (most recent call last): File "/vpp/test/framework.py", line 424, in setUpClass cls.statistics = VPPStats(socketname=cls.stats_sock) File "build/bdist.linux-x86_64/egg/vpp_papi/vpp_stats.py", line 120, in _init_ raise IOError() IOError Change-Id: I80523122f96eafc41b3ebbf27bee73f6c637d781 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2018-11-09Fix logging into multiple filesjuraj.linkes1-5/+4
When running tests in one process, only one logger was used and each testcase added its own file handler, which resulted in logs appearing in multiple files. Fix this by restoring the creation of new loggers for each testcase and only reuse the stream handler from parent process. Change-Id: I5b8471e041dc769128fddb433d33812bfcb5ecf6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-11-08make test: p2p ethernet refactorKlement Sekera1-1/+13
Change-Id: I48b4b3eaa84e4174e4d0305594675bf983184fa0 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-11-07GBP Endpoint LearningNeale Ranns1-1/+29
Learning GBP endpoints over vxlan-gbp tunnels Change-Id: I1db9fda5a16802d9ad8b4efd4e475614f3b21502 Signed-off-by: Neale Ranns <neale.ranns@cisco.com>
2018-10-10Setup, teardown, DEBUG=core, FAILFAST fixesjuraj.linkes1-83/+109
- Fixed a bug when an error occuring in tearDownClass would not result in test being marked as failed - Improved test results reporting in cases when an error occurs in setUpClass and tearDownClass - Fixed DEBUG=core when the core is produced in setUpClass or tearDownClass - Reworked DEBUG=core to always be handled after all tests have been executed - Fixed FAILFAST=1 for parallel test runs Change-Id: I3e9cd3b97ba6fa802fa0aa2dd7678ff82eee09ec Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-10-10Test FW: Use unittest temp dir as unix runtime dirJakub Grajciar1-7/+8
Change-Id: I5273d5f3f59cc3c43da0a15bb0c4a4056098adcf Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
2018-10-03ipsec: add missing ipv6 ah code & ipv6 testsKlement Sekera1-6/+18
Change-Id: I89e90193ded1beb6cb0950c15737f9467efac1c3 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-09-28stats: Split stat_segment and stats code in preparation for deprecation.Ole Troan1-1/+1
Split the stat_segment.c code from stats.c. Rename stats.[ch] to prepare for removing (19.01?) In addition stats.api can be removed. Since the stats aggregation for the stat segment does not use the API, that part is now done on the main thread. (Old stats aggregator is also left in place). Change-Id: I9867429f4fc547b1a7ab7f88bc4f3625428d681b Signed-off-by: Ole Troan <ot@cisco.com>
2018-09-20VPP-1421: Reworked results gatheringjuraj.linkes1-9/+23
The previous version sent the whole VPPTestResult object through pipe, which uses pickle to transfer objects. Pickle does not support sending any arbitrary objects and was causing issues. Now just a tuple of (test_id, result) is sent. Change-Id: I3a3a9e6f1b9ac9b05889babfc1f7560c7ac4471c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-06Fix test summary and retriesjuraj.linkes1-2/+2
There was an issue when tests crashed and weren't properly retried. Change-Id: Id5ef828ecc9a8dc0f08c50183721db06e162e6c3 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-09-05STATS: Update Prometheus portOle Troan1-1/+1
Update temporary port allocation with fixed allocated from: https://github.com/prometheus/prometheus/wiki/Default-port-allocations Change-Id: I99a7e069fb95d00884458aeacaba06e4713fbb76 Signed-off-by: Ole Troan <ot@cisco.com>
2018-09-02STATS: Python binding to access VPP statistics and counters.Ole Troan1-1/+5
from vpp_papi.vpp_stats import VPPStats s = VPPStats(socketname='/var/run/stats.sock') c = s.ls('/if/rx') counters = s.dump(c) print(s.set_error_str()) Change-Id: I203ebe60b0c9ee5742aadc737c0f29051757959d Signed-off-by: Ole Troan <ot@cisco.com>
2018-08-30Fix hanging test runner when child process diesjuraj.linkes1-2/+12
When fixing the test summary after a test run doesn't finish properly I introduced a bug where child process which died would leave the whole run hanging. This patch fixed the bug while still having the correct test summary. Change-Id: I206b1a7dab4032d24cbc50667b8dd0bdcebb67a6 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-25Fix logging to file and summary for crashed testsjuraj.linkes1-0/+2
Logging was not being logged to log.txt Crashed tests were not reported properly when running tests in one process Change-Id: Ica2d703d88351cc8a94bad86764ae4376dc79590 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-08-23CSIT-1139: Implement parallel test executionjuraj.linkes1-132/+72
The implementation of parallel test execution in VPP Test Framework. - VPPTestCase test methods are grouped together - tests are running in separate processes - VPP instances spawned by tests are assigned to different cores - output from these processes is redirected through pipes and printed out testcase by testcase - TEST_JOBS env var is used to specify the number of parallel processes - improved test summary - a bit of code cleanup Change-Id: I9ca93904d9fe2c3daf980500c64a8611838ae28c Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2018-07-24Create a unit-test pluginDave Barach1-1/+2
Move the tcp unit test to the plugin Add a bihash unit test and a "make test" program to call it Adjust framework.py to load the plugin, which is disabled by default Change-Id: Ic229d386a56a9d28dbd54974f231149053ca8f93 Signed-off-by: Dave Barach <dave@barachs.net>
2018-07-24test frame work pump thread exit: set flag then wakeNeale Ranns1-2/+2
Change-Id: Id95b8a7ad1bf550f615c50d77b16a530cc8fc2c4 Signed-off-by: Neale Ranns <nranns@cisco.com>
2018-07-09IGMP improvementsNeale Ranns1-2/+3
- Enable/Disable an interface for IGMP - improve logging - refactor common code - no orphaned timers - IGMP state changes in main thread only - Large groups split over multiple state-change reports - SSM range configuration API. - more tests Change-Id: If5674f1044e7e97274a711f47807c9ba689d7b9a Signed-off-by: Neale Ranns <nranns@cisco.com>
2018-06-25make test: fix RETRIES when setUpClass throwsKlement Sekera1-1/+21
This change adds handling of special case when setUpClass throws. In this case TestResults receives a mock object called _ErrorHolder. By parsing its description, we find test class name and use it to lookup the test class in test suite to be able to add it to the list of failures for re-running. Change-Id: I656f21e38aa450fc567cdcbcf6e586967f947a64 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-24Revert "Revert "make test: fix broken interfaces""Klement Sekera1-6/+4
This reverts commit c8efa29b6f9a91381897b54f1147daf922ed7164. Change-Id: I1d5c5773d5f86a63073e255336bd9de628e26179 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-24Revert "Revert "ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums""Klement Sekera1-5/+14
This reverts commit e0d2bd6bd7fc59c0c6ac48195d7f825dc99bfd91. Change-Id: If491e16f9ea66b2493a6a7c7f3c684ed585f8f51 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2018-06-22Revert "ipsec: VPP-1316 calculate IP/TCP/UDP inner checksums"Ole Troan1-14/+5
This reverts commit a98346f664aae148d26a8e158008b773d73db96f. Change-Id: Iee5b3a5ddff0e8fd3a30fe5973cee24de434fe12 Signed-off-by: Ole Troan <ot@cisco.com>
2018-06-22Revert "make test: fix broken interfaces"Ole Troan1-4/+6
This reverts commit d5c60b96a3fd93916fc4af5c8d6d25625c28242e. Change-Id: I3632b9c3f76c615aee897f28f76d094e7031e689 Signed-off-by: Ole Troan <ot@cisco.com>
2018-06-22make test: fix broken interfacesKlement Sekera1-6/+4
Change-Id: I2e092774f81503e04b53cc6c6b5d357fe3fc52ab Signed-off-by: Klement Sekera <ksekera@cisco.com>