csit rls1701 report edits:

- simplified section structure for clarity and readability,
- updated overview sections,
- moved not rls related content from rls_notes to overview sections,
- removed section title suffixes: HW, VIRL.
- completed vpp_unit_tests and vpp_unittest_results sections.
- updated all documentation sections.
- updated rls_notes sections for vpp performance and vpp functional.

Change-Id: Id2c2abbf9d3531ec9f63ecd353f385a0b55ae1ba
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Signed-off-by: pmikus <pmikus@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Signed-off-by: pmikus <pmikus@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>

5 files changed, 252 insertions, 0 deletions

diff --git a/docs/report/honeycomb_functional_tests/csit_release_notes.rst b/docs/report/honeycomb_functional_tests/csit_release_notes.rst
new file mode 100644
index 0000000000..68b5eb6fb5
--- /dev/null
+++ b/docs/report/honeycomb_functional_tests/csit_release_notes.rst
@@ -0,0 +1,35 @@
+CSIT Release Notes

+==================

+

+Changes in CSIT |release|

+-------------------------

+

+#. Naming change for all Honeycomb functional test suites in VIRL

+

+    - Honeycomb functional test case names stayed unchanged

+

+#. Added Honeycomb functional tests

+

+    - NSH_SFC

+    - LISP

+    - NAT

+    - SPAN

+

+Known Issues

+------------

+

+Here is the list of known issues in CSIT |release| for Honeycomb functional

+tests in VIRL:

+

++---+-------------------------------------------------+-----------------------------------------------------------------+

+| # | Issue                                           | Description                                                     |

++---+-------------------------------------------------+-----------------------------------------------------------------+

+| 1 | Operational data for Vhost-user interfaces      | Honeycomb Operational data reports Vhost-user interfaces        |

+|   | "server" flag                                   | as client, even if they are server.                             |

++---+-------------------------------------------------+-----------------------------------------------------------------+

+| 2 | Persistence of VxLAN tunnels                    | Configuration persistence often fails to restore                |

+|   |                                                 | Honeycomb's internal naming context for VxLAN interfaces.       |

+|   |                                                 | The interface is renamed to "vxlan_tunnel0" but is otherwise    |

+|   |                                                 | configured correctly.                                           |

++---+-------------------------------------------------+-----------------------------------------------------------------+

+

diff --git a/docs/report/honeycomb_functional_tests/documentation.rst b/docs/report/honeycomb_functional_tests/documentation.rst
new file mode 100644
index 0000000000..e53052ddb1
--- /dev/null
+++ b/docs/report/honeycomb_functional_tests/documentation.rst
@@ -0,0 +1,6 @@
+Documentation

+=============

+

+`CSIT Honeycomb Functional Tests Documentation

+<https://docs.fd.io/csit/master/doc/tests.func.html>`_ contains detailed

+functional description and input parameters for each test case.

diff --git a/docs/report/honeycomb_functional_tests/index.rst b/docs/report/honeycomb_functional_tests/index.rst
new file mode 100644
index 0000000000..d5d1b6180b
--- /dev/null
+++ b/docs/report/honeycomb_functional_tests/index.rst
@@ -0,0 +1,10 @@
+HoneyComb Functional Tests
+==========================
+
+.. toctree::
+
+    overview
+    csit_release_notes
+    test_environment
+    documentation
+
diff --git a/docs/report/honeycomb_functional_tests/overview.rst b/docs/report/honeycomb_functional_tests/overview.rst
new file mode 100644
index 0000000000..d93de132a9
--- /dev/null
+++ b/docs/report/honeycomb_functional_tests/overview.rst
@@ -0,0 +1,168 @@
+Overview

+========

+

+Tested Virtual Topologies

+-------------------------

+

+CSIT Honeycomb functional tests are executed on virtualized topologies created

+using Virtual Internet Routing Lab (VIRL) simulation platform contributed by

+Cisco. VIRL runs on physical baremetal servers hosted by LF FD.io project. All

+tests are executed in two node logical test topology - Traffic Generator (TG)

+node and Systems Under Test (SUT1) node connected in a loop. Logical test

+topology is shown in the figure below.

+

+::

+

+                     +------------------------+

+                     |                        |

+                     |  +------------------+  |

+        +--------------->                  <--------------+

+        |            |  |                  |  |           |

+        |  |------------>       DUT1       <-----------+  |

+        |  |         |  +------------------+  |        |  |

+        |  |         |                        |        |  |

+        |  |         |                  SUT1  |        |  |

+        |  |         +------------------------+        |  |

+        |  |                                           |  |

+        |  |                                           |  |

+        |  |               +-----------+               |  |

+        |  +--------------->           <---------------+  |

+        |                  |    TG     |                  |

+        +------------------>           <------------------+

+                           +-----------+

+

+SUT1 is a VM (Ubuntu or Centos, depending on the test suite), TG is a Traffic

+Generator (TG, another Ubuntu VM). SUTs run Honeycomb and VPP SW applications

+in Linux user-mode as a Device Under Test (DUT) within the VM. TG runs Scapy

+SW application as a packet Traffic Generator. Logical connectivity between

+SUTs and to TG is provided using virtual NICs using VMs' virtio driver.

+

+Virtual testbeds are created on-demand whenever a verification job is started

+(e.g. triggered by the gerrit patch submission) and destroyed upon completion

+of all functional tests. Each node is a Virtual Machine and each connection

+that is drawn on the diagram is available for use in any test case. During the

+test execution, all nodes are reachable thru the Management network connected

+to every node via dedicated virtual NICs and virtual links (not shown above

+for clarity).

+

+Functional Tests Coverage

+-------------------------

+

+The following Honeycomb functional test areas are included in the CSIT |release|

+with results listed in this report:

+

+- **Basic interface management** - CRUD for interface state,

+  - ipv4/ipv6 address, ipv4 neighbor, MTU value.

+  - Test case count: 7

+- **L2BD** - CRUD for L2 Bridge-Domain, interface assignment.

+  - Create up to two bridge domains with all implemented functions turned on.

+  - (flooding, unknown-unicast flooding, forwarding, learning, arp-termination)

+  - Assign up to two physical interfaces to a single bridge domain.

+  - Remove interface assignments, remove bridge domains.

+  - Test case count: 5

+- **L2FIB** - CRD for L2-FIB entries.

+  - Create 4 FIB entries

+  - (one of each for filter/forward, static/dynamic combinations).

+  - Remove FIB entries.

+  - Test case count: 7

+- **VxLAN** - CRD for VxLAN tunnels.

+  - Create VxLAN interface.

+  - Disable VxLAN interface.

+  - Re-create a disabled VxLAN interface.

+  - Test case count: 6

+- **VxLAN-GPE** - CRD for VxLAN GPE tunnels.

+  - Create VxLAN GPE interface.

+  - Disable VxLAN interface.

+  - Re-create a disabled VxLAN interface.

+  - Test case count: 7

+- **Vhost-user** - CRUD for Vhost-user interfaces.

+  - Create, modify and delete Vhost-user interface, as client and server.

+  - Test case count: 8

+- **TAP** - CRUD for Tap interface management.

+  - Create, modify and delete TAP interface.

+  - Test case count: 3

+- **VLAN** - CRUD for VLAN sub-interface management.

+  - Create VLAN sub-interface over a physical interface.

+  - Toggle interface state separately for super-interface and sub-interface.

+  - Configure IP address and bridge domain assignment on sub-interface.

+  - Configure VLAN tag rewrite on sub-interface.

+  - Test case count: 17

+- **ACL** - CRD for low-level classifiers: table and session management,

+  - interface assignment.

+  - Configure up to 2 classify tables.

+  - Configure up to 2 classify sessions on one table.

+  - Assign classify session to a physical interface.

+  - Remove tables, sessions, interface assignments.

+  - Test case count: 9

+- **PBB** - CRD for provider backbone bridge sub-interface.

+  - Configure, modify and remove a PBB sub-interface over a physical interface.

+  - Test case count: 9

+- **NSH_SFC** - CRD for NSH maps and entries, using NSH_SFC plugin.

+  - Configure up to 2 NSH entries.

+  - Configure up to 2 NSH maps.

+  - Modify and delete NSH maps and entries.

+  - Test case count: 8

+- **LISP** - CRD for Lisp: mapping, locator set, adjacency, map resolver.

+  - Toggle Lisp feature status.

+  - Configure and delete Lisp mapping as local and remote.

+  - Configure and delete Lisp adjacency mapping

+  - Configure and delete Lisp map resolver, proxy ITR.

+  - Test case count: 11

+- **NAT** - CRD for NAT entries, interface assignment.

+  - Configure and delete up to two NAT entries.

+  - Assign NAT entries to a physical interface.

+  - Test case count: 6

+- **Port mirroring** - CRD for SPAN port mirroring, interface assignment.

+  - Configure SPAN port mirroring on a physical interface, mirroring

+  - up to 2 interfaces.

+  - Remove SPAN configuration from interfaces.

+  - Test case count: 3

+- **Honeycomb Infractructure** - configuration persistence,

+  - Netconf notifications for interface events,

+  - Netconf negative tests aimed at specific issues

+

+Total 111 Honeycomb tests in the CSIT |release|.

+

+Operational data in Honeycomb should mirror configuration data at all times.

+Because of this, test cases follow this general pattern:

+

+#. read operational data of the feature using restconf.

+#. read status of the feature using VPP API dump.

+#. modify configuration of the feature using restconf.

+#. verify changes to operational data using restconf.

+#. verify changes using VPP API dump.

+

+Test cases involving network interfaces utilize the first two interfaces on

+the DUT node.

+

+Functional Tests Naming

+-----------------------

+

+CSIT |release| introduced a common structured naming convention for all

+performance and functional tests. This change was driven by substantially

+growing number and type of CSIT test cases. Firstly, the original practice did

+not always follow any strict naming convention. Secondly test names did not

+always clearly capture tested packet encapsulations, and the actual type or

+content of the tests. Thirdly HW configurations in terms of NICs, ports and

+their locality were not captured either. These were but few reasons that drove

+the decision to change and define a new more complete and stricter test naming

+convention, and to apply this to all existing and new test cases.

+

+The new naming should be intuitive for majority of the tests. The complete

+description of CSIT test naming convention is provided on `CSIT test naming

+page <https://wiki.fd.io/view/CSIT/csit-test-naming>`_.

+

+Here few illustrative examples of the new naming usage for functional test

+suites:

+

+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**

+

+    - *eth2p-ethip4-ip4base-func.robot* => 2 ports of Ethernet, IPv4 baseline

+      routed forwarding, functional tests.

+

+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,

+   P2V2P, NIC2VMchain2NIC, P2V2V2P**

+

+    - *eth2p-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-func.robot* => 2 ports of

+      Ethernet, IPv4 VXLAN Ethernet, L2 bridge-domain switching to/from two vhost

+      interfaces and one VM, functional tests.

diff --git a/docs/report/honeycomb_functional_tests/test_environment.rst b/docs/report/honeycomb_functional_tests/test_environment.rst
new file mode 100644
index 0000000000..e706aa5573
--- /dev/null
+++ b/docs/report/honeycomb_functional_tests/test_environment.rst
@@ -0,0 +1,33 @@
+Test Environment

+================

+

+CSIT Honeycomb tests are currently executed in VIRL, as mentioned above. The

+physical VIRL testbed infrastructure consists of three identical VIRL hosts,

+each host being a Cisco UCS C240-M4 (2x Intel(R) Xeon(R) CPU E5-2699 v3 @

+2.30GHz, 18c, 512GB RAM) running Ubuntu 14.04.3 and the following VIRL software

+versions:

+

+  STD server version 0.10.24.7

+  UWM server version 0.10.24.7

+

+Current VPP tests have been executed on a single VM operating system and

+version only, as described in the following paragraphs.

+

+In CSIT terminology, the VM operating system for both SUTs and TG that VPP 17.01

+has been tested with, is the following:

+

+  ubuntu-14.04.4_2016-10-07_1.3

+

+which implies Ubuntu 14.04.3 LTS, current as of 2016/10/07 (that is, package

+versions are those that would have been installed by a "apt-get update",

+"apt-get upgrade" on October 7), produced by CSIT disk image build scripts

+version 1.6.

+

+The exact list of installed packages and their versions (including the Linux

+kernel package version) are included in CSIT source repository:

+

+  resources/tools/disk-image-builder/ubuntu/lists/ubuntu-14.04.4_2016-10-07_1.3

+

+A replica of this VM image can be built by running the "build.sh" script in CSIT

+repository resources/tools/disk-image-builder/.

+