diff options
Diffstat (limited to 'test/doc/index.rst')
| -rw-r--r-- | test/doc/index.rst | 157 |
1 files changed, 150 insertions, 7 deletions
diff --git a/test/doc/index.rst b/test/doc/index.rst index c18357a4860..323b608fdfb 100644 --- a/test/doc/index.rst +++ b/test/doc/index.rst @@ -1,15 +1,158 @@ -.. VPP test framework documentation master file, created by - sphinx-quickstart on Thu Oct 13 08:45:03 2016. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. _unittest: https://docs.python.org/2/library/unittest.html +.. _TestCase: https://docs.python.org/2/library/unittest.html#unittest.TestCase +.. _AssertionError: https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError +.. _SkipTest: https://docs.python.org/2/library/unittest.html#unittest.SkipTest +.. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/ +.. _scapy: http://www.secdev.org/projects/scapy/ -Welcome to VPP test framework's documentation! -============================================== +.. |vtf| replace:: VPP Test Framework -Contents: +|vtf| +===== +Overview +######## + +The goal of the |vtf| is to ease writing, running and debugging +unit tests for the VPP. For this, python was chosen as a high level language +allowing rapid development with scapy_ providing the necessary tool for creating +and dissecting packets. + +Anatomy of a test case +###################### + +Python's unittest_ is used as the base framework upon which the VPP test +framework is built. A test suite in the |vtf| constists of multiple classes +derived from `VppTestCase`, which is itself derived from TestCase_. +The test class defines one or more test functions, which act as test cases. + +Function flow when running a test case is: + +1. `setUpClass <VppTestCase.setUpClass>`: + This function is called once for each test class, allowing a one-time test + setup to be executed. If this functions throws an exception, + none of the test functions are executed. +2. `setUp <VppTestCase.setUp>`: + The setUp function runs before each of the test functions. If this function + throws an exception other than AssertionError_ or SkipTest_, then this is + considered an error, not a test failure. +3. *test_<name>*: + This is the guts of the test case. It should execute the test scenario + and use the various assert functions from the unittest framework to check + necessary. +4. `tearDown <VppTestCase.tearDown>`: + The tearDown function is called after each test function with the purpose + of doing partial cleanup. +5. `tearDownClass <VppTestCase.tearDownClass>`: + Method called once after running all of the test funnctions to perform + the final cleanup. + +Test temporary directory and VPP life cycle +################################################ + +Test separation is achieved by separating the test files and vpp instances. +Each test creates a temporary directory and it's name is used to create +a shared memory prefix which is used to run a VPP instance. +This way, there is no conflict between any other VPP instances running +on the box and the test VPP. Any temporary files created by the test case +are stored in this temporary test directory. + +Virtual environment +################### + +Virtualenv_ is a python module which provides a means to crate an environment +containing the dependencies required by the |vtf|, allowing a separation +from any existing system-wide packages. |vtf|'s Makefile automatically +creates a virtualenv_ inside build-root and installs the required packages +in that environment. The environment is entered whenever executing a test +via one of the make test targets. + +Naming conventions +################## + +Most unit tests do some kind of packet manipulation - sending and receiving +packets between VPP and virtual hosts connected to the VPP. Referring +to the sides, addresses, etc.. is always done as if looking from the VPP side, +thus: + +* *local_* prefix is used for the VPP side. + So e.g. `local_ip4 <VppInterface.local_ip4>` address is the IPv4 address + assigned to the VPP interface. +* *remote_* prefix is used for the virtual host side. + So e.g. `remote_mac <VppInterface.remote_mac>` address is the MAC address + assigned to the virtual host connected to the VPP. + +Packet flow in the |vtf| +######################## + +Test framework -> VPP +~~~~~~~~~~~~~~~~~~~~~ + +|vtf| doesn't send any packets to VPP directly. Traffic is instead injected +using packet-generator interfaces, represented by the `VppPGInterface` class. +Packets are written into a temporary .pcap file, which is then read by the VPP +and the packets are injected into the VPP world. + +VPP -> test framework +~~~~~~~~~~~~~~~~~~~~~ + +Similarly, VPP doesn't send any packets to |vtf| directly. Instead, packet +capture feature is used to capture and write traffic to a temporary .pcap file, +which is then read and analyzed by the |vtf|. + +Test framework objects +###################### + +The following objects provide VPP abstraction and provide a means to do +common tasks easily in the test cases. + +* `VppInterface`: abstract class representing generic VPP interface + and contains some common functionality, which is then used by derived classes +* `VppPGInterface`: class representing VPP packet-generator interface. + The interface is created/destroyed when the object is created/destroyed. +* `VppSubInterface`: VPP sub-interface abstract class, containing common + functionality for e.g. `VppDot1QSubint` and `VppDot1ADSubint` classes + +How VPP API/CLI is called +######################### + +Vpp provides python bindings in a python module called vpp-papi, which the test +framework installs in the virtual environment. A shim layer represented by +the `VppPapiProvider` class is built on top of the vpp-papi, serving these +purposes: + +1. Automatic return value checks: + After each API is called, the return value is checked against the expected + return value (by default 0, but can be overridden) and an exception + is raised if the check fails. +2. Automatic call of hooks: + + a. `before_cli <Hook.before_cli>` and `before_api <Hook.before_api>` hooks + are used for debug logging and stepping through the test + b. `after_cli <Hook.after_cli>` and `after_api <Hook.after_api>` hooks + are used for monitoring the vpp process for crashes +3. Simplification of API calls: + Many of the VPP APIs take a lot of parameters and by providing sane defaults + for these, the API is much easier to use in the common case and the code is + more readable. E.g. ip_add_del_route API takes ~25 parameters, of which + in the common case, only 3 are needed. + +Example: how to add a new test +############################## + +In this example, we will describe how to add a new test case which tests VPP... + +1. Add a new file called... +2. Add a setUpClass function containing... +3. Add the test code in test... +4. Run the test... + +|vtf| module documentation +########################## + .. toctree:: :maxdepth: 2 + :glob: modules.rst |