.. _unittest: https://docs.python.org/2/library/unittest.html .. _TestCase: https://docs.python.org/2/library/unittest.html#unittest.TestCase .. _AssertionError: https://docs.python.org/2/library/exceptions.html#exceptions.AssertionError .. _SkipTest: https://docs.python.org/2/library/unittest.html#unittest.SkipTest .. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/ .. _scapy: http://www.secdev.org/projects/scapy/ .. _logging: https://docs.python.org/2/library/logging.html .. |vtf| replace:: VPP Test Framework |vtf| ===== .. contents:: :local: :depth: 1 Overview ######## The goal of the |vtf| is to ease writing, running and debugging unit tests for the VPP. For this, python was chosen as a high level language allowing rapid development with scapy_ providing the necessary tool for creating and dissecting packets. Anatomy of a test case ###################### Python's unittest_ is used as the base framework upon which the VPP test framework is built. A test suite in the |vtf| consists of multiple classes derived from `VppTestCase`, which is itself derived from TestCase_. The test class defines one or more test functions, which act as test cases. Function flow when running a test case is: 1. `setUpClass `: This function is called once for each test class, allowing a one-time test setup to be executed. If this functions throws an exception, none of the test functions are executed. 2. `setUp `: The setUp function runs before each of the test functions. If this function throws an exception other than AssertionError_ or SkipTest_, then this is considered an error, not a test failure. 3. *test_*: This is the guts of the test case. It should execute the test scenario and use the various assert functions from the unittest framework to check necessary. Multiple test_ methods can exist in a test case. 4. `tearDown `: The tearDown function is called after each test function with the purpose of doing partial cleanup. 5. `tearDownClass `: Method called once after running all of the test functions to perform the final cleanup. Logging ####### Each test case has a logger automatically created for it, stored in 'logger' property, based on logging_. Use the logger's standard methods debug(), info(), error(), ... to emit log messages to the logger. All the log messages go always into a log file in temporary directory (see below). To control the messages printed to console, specify the V= parameter. .. code-block:: shell make test # minimum verbosity make test V=1 # moderate verbosity make test V=2 # maximum verbosity Test temporary directory and VPP life cycle ########################################### Test separation is achieved by separating the test files and vpp instances. Each test creates a temporary directory and it's name is used to create a shared memory prefix which is used to run a VPP instance. The temporary directory name contains the testcase class name for easy reference, so for testcase named 'TestVxlan' the directory could be named e.g. vpp-unittest-TestVxlan-UNUP3j. This way, there is no conflict between any other VPP instances running on the box and the test VPP. Any temporary files created by the test case are stored in this temporary test directory. The test temporary directory holds the following interesting files: * log.txt - this contains the logger output on max verbosity * pg*_in.pcap - last injected packet stream into VPP, named after the interface, so for pg0, the file will be named pg0_in.pcap * pg*_out.pcap - last capture file created by VPP for interface, similarly, named after the interface, so for e.g. pg1, the file will be named pg1_out.pcap * history files - whenever the capture is restarted or a new stream is added, the existing files are rotated and renamed, soo all the pcap files are always saved for later debugging if needed * core - if vpp dumps a core, it'll be stored in the temporary directory * vpp_stdout.txt - file containing output which vpp printed to stdout * vpp_stderr.txt - file containing output which vpp printed to stderr *NOTE*: existing temporary directories named vpp-unittest-* are automatically removed when invoking 'make test*' or 'make retest*' to keep the temporary directory clean. Virtual environment ################### Virtualenv_ is a python module which provides a means to create an environment containing the dependencies required by the |vtf|, allowing a separation from any existing system-wide packages. |vtf|'s Makefile automatically creates a virtualenv_ inside build-root and installs the required packages in that environment. The environment is entered whenever executing a test via one of the make test targets. Naming conventions ################## Most unit tests do some kind of packet manipulation - sending and receiving packets between VPP and virtual hosts connected to the VPP. Referring to the sides, addresses, etc. is always done as if looking from the VPP side, thus: * *local_* prefix is used for the VPP side. So e.g. `local_ip4 ` address is the IPv4 address assigned to the VPP interface. * *remote_* prefix is used for the virtual host side. So e.g. `remote_mac ` address is the MAC address assigned to the virtual host connected to the VPP. Automatically generated addresses ################################# To send packets, one needs to typically provide some addresses, otherwise the packets will be dropped. The interface objects in |vtf| automatically provide addresses based on (typically) their indexes, which ensures there are no conflicts and eases debugging by making the addressing scheme consistent. The developer of a test case typically doesn't need to work with the actual numbers, rather using the properties of the objects. The addresses typically come in two flavors: '

' and '

n' - note the 'n' suffix. The former address is a Python string, while the latter is translated using socket.inet_pton to raw format in network byte order - this format is suitable for passing as an argument to VPP APIs. e.g. for the IPv4 address assigned to the VPP interface: * local_ip4 - Local IPv4 address on VPP interface (string) * local_ip4n - Local IPv4 address - raw, suitable as API parameter. These addresses need to be configured in VPP to be usable using e.g. `config_ip4` API. Please see the documentation to `VppInterface` for more details. By default, there is one remote address of each kind created for L3: remote_ip4 and remote_ip6. If the test needs more addresses, because it's simulating more remote hosts, they can be generated using `generate_remote_hosts` API and the entries for them inserted into the ARP table using `configure_ipv4_neighbors` API. Packet flow in the |vtf| ######################## Test framework -> VPP ~~~~~~~~~~~~~~~~~~~~~ |vtf| doesn't send any packets to VPP directly. Traffic is instead injected using packet-generator interfaces, represented by the `VppPGInterface` class. Packets are written into a temporary .pcap file, which is then read by the VPP and the packets are injected into the VPP world. To add a list of packets to an interface, call the `add_stream` method on that interface. Once everything is prepared, call `pg_start` method to start the packet generator on the VPP side. VPP -> test framework ~~~~~~~~~~~~~~~~~~~~~ Similarly, VPP doesn't send any packets to |vtf| directly. Instead, packet capture feature is used to capture and write traffic to a temporary .pcap file, which is then read and analyzed by the |vtf|. The following APIs are available to the test case for reading pcap files. * `get_capture`: this API is suitable for bulk & batch style of test, where a list of packets is prepared & sent, then the received packets are read and verified. The API needs the number of packets which are expected to be captured (ignoring filtered packets - see below) to know when the pcap file is completely written by the VPP. If using packet infos for verifying packets, then the counts of the packet infos can be automatically used by `get_capture` to get the proper count (in this case the default value None can be supplied as expected_count or ommitted altogether). * `wait_for_packet`: this API is suitable for interactive style of test, e.g. when doing session management, three-way handsakes, etc. This API waits for and returns a single packet, keeping the capture file in place and remembering context. Repeated invocations return following packets (or raise Exception if timeout is reached) from the same capture file (= packets arriving on the same interface). *NOTE*: it is not recommended to mix these APIs unless you understand how they work internally. None of these APIs rotate the pcap capture file, so calling e.g. `get_capture` after `wait_for_packet` will return already read packets. It is safe to switch from one API to another after calling `enable_capture` as that API rotates the capture file. Automatic filtering of packets: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Both APIs (`get_capture` and `wa