hs-test: transition to ginkgo test framework

Type: test

Change-Id: Ia38bf5549d20b22876f6082085b69a52a03d0142
Signed-off-by: Adrian Villin <avillin@cisco.com>
Signed-off-by: Matus Fabian <matfabia@cisco.com>

1 files changed, 132 insertions, 43 deletions

diff --git a/extras/hs-test/README.rst b/extras/hs-test/README.rst
index 6db832b7fbe..1dc1039b33f 100644
--- a/extras/hs-test/README.rst
+++ b/extras/hs-test/README.rst
@@ -9,7 +9,10 @@ End-to-end tests often want multiple VPP instances, network namespaces, differen
 and to execute external tools or commands. With such requirements the existing VPP test framework is not sufficient.
 For this, ``Go`` was chosen as a high level language, allowing rapid development, with ``Docker`` and ``ip`` being the tools for creating required topology.
 
-Go's package `testing`_ together with `go test`_ command form the base framework upon which the *hs-test* is built and run.
+`Ginkgo`_ forms the base framework upon which the *hs-test* is built and run.
+All tests are technically in a single suite because we are only using ``package main``. We simulate suite behavior by grouping tests by the topology they require.
+This allows us to run those mentioned groups in parallel, but not individual tests in parallel.
+
 
 Anatomy of a test case
 ----------------------
@@ -24,15 +27,16 @@ Anatomy of a test case
 **Action flow when running a test case**:
 
 #. It starts with running ``make test``. Optional arguments are VERBOSE, PERSIST (topology configuration isn't cleaned up after test run),
-   and TEST=<test-name> to run specific test.
-#. ``make list-tests`` (or ``make help``) shows all test names.
-#. ``go test`` compiles package ``main`` along with any files with names matching the file pattern ``*_test.go``
-   and then runs the resulting test binaries
-#. The go test framework runs each function matching :ref:`naming convention<test-convention>`. Each of these corresponds to a `test suite`_
-#. Testify toolkit's ``suite.Run(t *testing.T, suite TestingSuite)`` function runs the suite and does the following:
-
+   TEST=<test-name> to run a specific test and PARALLEL=[n-cpus].
+#. ``make list-tests`` (or ``make help``) shows all tests. The current `list of tests`_ is at the bottom of this document.
+#. ``Ginkgo`` looks for a spec suite in the current directory and then compiles it to a .test binary
+#. The Ginkgo test framework runs each function that was registered manually using ``registerMySuiteTest(s *MySuite)``. Each of these functions correspond to a suite
+#. Ginkgo's ``RunSpecs(t, "Suite description")`` function is the entry point and does the following:
+
+  #. Ginkgo compiles the spec, builds a spec tree
+  #. ``Describe`` container nodes in suite\_\*_test.go files are run (in series by default, or in parallel with the argument PARALLEL=[n-cpus])
   #. Suite is initialized. The topology is loaded and configured in this step
-  #. Test suite runs all the tests attached to it
+  #. Registered tests are run in generated ``It`` subject nodes
   #. Execute tear-down functions, which currently consists of stopping running containers
      and clean-up of test topology
 
@@ -43,23 +47,25 @@ This describes adding a new test case to an existing suite.
 For adding a new suite, please see `Modifying the framework`_ below.
 
 #. To write a new test case, create a file whose name ends with ``_test.go`` or pick one that already exists
-#. Declare method whose name starts with ``Test`` and specifies its receiver as a pointer to the suite's struct (defined in ``framework_test.go``)
+#. Declare method whose name ends with ``Test`` and specifies its parameter as a pointer to the suite's struct (defined in ``suite_*_test.go``)
 #. Implement test behaviour inside the test method. This typically includes the following:
 
-  #. Retrieve a running container in which to run some action. Method ``getContainerByName``
-     from ``HstSuite`` struct serves this purpose
-  #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI
-  #. Run arbitrary commands inside the containers with ``exec`` method
-  #. Run other external tool with one of the preexisting functions in the ``utils.go`` file.
-     For example, use ``wget`` with ``startWget`` function
-  #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
-  #. Verify results of your tests using ``assert`` methods provided by the test suite,
-     implemented by HstSuite struct
+   #. Retrieve a running container in which to run some action. Method ``getContainerByName``
+      from ``HstSuite`` struct serves this purpose
+   #. Interact with VPP through the ``VppInstance`` struct embedded in container. It provides ``vppctl`` method to access debug CLI
+   #. Run arbitrary commands inside the containers with ``exec`` method
+   #. Run other external tool with one of the preexisting functions in the ``utils.go`` file.
+      For example, use ``wget`` with ``startWget`` function
+   #. Use ``exechelper`` or just plain ``exec`` packages to run whatever else
+   #. Verify results of your tests using ``assert`` methods provided by the test suite, implemented by HstSuite struct or use ``Gomega`` assert functions.
+
+#. Create an ``init()`` function and register the test using ``register*SuiteTests(testCaseFunction)``
+
 
 **Example test case**
 
 Assumed are two docker containers, each with its own VPP instance running. One VPP then pings the other.
-This can be put in file ``extras/hs-test/my_test.go`` and run with command ``./test -run TestMySuite/TestMyCase``.
+This can be put in file ``extras/hs-test/my_test.go`` and run with command ``make test TEST=MyTest`` or ``ginkgo -v --trace --focus MyTest``.
 
 ::
 
@@ -69,7 +75,11 @@ This can be put in file ``extras/hs-test/my_test.go`` and run with command ``./t
                 "fmt"
         )
 
-        func (s *MySuite) TestMyCase() {
+        func init(){
+                registerMySuiteTest(MyTest)
+        }
+
+        func MyTest(s *MySuite) {
                 clientVpp := s.getContainerByName("client-vpp").vppInstance
 
                 serverVethAddress := s.netInterfaces["server-iface"].AddressString()
@@ -86,8 +96,7 @@ Modifying the framework
 
 .. _test-convention:
 
-#. Adding a new suite takes place in ``framework_test.go`` and by creating a new file for the suite.
-   Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced
+#. To add a new suite, create a new file. Naming convention for the suite files is ``suite_name_test.go`` where *name* will be replaced
    by the actual name
 
 #. Make a ``struct``, in the suite file, with at least ``HstSuite`` struct as its member.
@@ -99,7 +108,17 @@ Modifying the framework
                         HstSuite
                 }
 
-#. In suite file, implement ``SetupSuite`` method which testify runs once before starting any of the tests.
+#. Create a new slice that will contain test functions with a pointer to the suite's struct: ``var myTests = []func(s *MySuite){}``
+
+#. Then create a new function that will append test functions to that slice:
+
+        ::
+
+                func registerMySuiteTests(tests ...func(s *MySuite)) {
+	                nginxTests = append(myTests, tests...)
+                }
+
+#. In suite file, implement ``SetupSuite`` method which Ginkgo runs once before starting any of the tests.
    It's important here to call ``configureNetworkTopology`` method,
    pass the topology name to the function in a form of file name of one of the *yaml* files in ``topo-network`` folder.
    Without the extension. In this example, *myTopology* corresponds to file ``extras/hs-test/topo-network/myTopology.yaml``
@@ -111,6 +130,8 @@ Modifying the framework
         ::
 
                 func (s *MySuite) SetupSuite() {
+                        s.HstSuite.SetupSuite()
+
                         // Add custom setup code here
 
                         s.configureNetworkTopology("myTopology")
@@ -123,19 +144,62 @@ Modifying the framework
         ::
 
                 func (s *MySuite) SetupTest() {
+                        s.HstSuite.setupTest()
                         s.SetupVolumes()
                         s.SetupContainers()
                 }
 
-#. In order for ``go test`` to run this suite, we need to create a normal test function and pass our suite to ``suite.Run``.
-   These functions are placed at the end of ``framework_test.go``
+#. In order for ``Ginkgo`` to run this suite, we need to create a ``Describe`` container node with setup nodes and an ``It`` subject node.
+   Place them at the end of the suite file
+
+   * Declare a suite struct variable before anything else
+   * To use ``BeforeAll()`` and ``AfterAll()``, the container has to be marked as ``Ordered``
+   * Because the container is now marked as Ordered, if a test fails, all the subsequent tests are skipped.
+     To override this behavior, decorate the container node with ``ContinueOnFailure``
 
         ::
 
-                func TestMySuite(t *testing.T) {
-                        var m MySuite
-                        suite.Run(t, &m)
-                }
+                var _ = Describe("MySuite", Ordered, ContinueOnFailure, func() {
+	        var s MySuite
+	        BeforeAll(func() {
+		        s.SetupSuite()
+	        })
+	        BeforeEach(func() {
+		        s.SetupTest()
+	        })
+	        AfterAll(func() {
+		        s.TearDownSuite()
+	        })
+	        AfterEach(func() {
+		        s.TearDownTest()
+        	})
+	        for _, test := range mySuiteTests {
+		        test := test
+		        pc := reflect.ValueOf(test).Pointer()
+		        funcValue := runtime.FuncForPC(pc)
+		        It(strings.Split(funcValue.Name(), ".")[2], func(ctx SpecContext) {
+			        test(&s)
+		        }, SpecTimeout(time.Minute*5))
+	        }
+                })
+
+#. Notice the loop - it will generate multiple ``It`` nodes, each running a different test.
+   ``test := test`` is necessary, otherwise only the last test in a suite will run.
+   For a more detailed description, check Ginkgo's documentation: https://onsi.github.io/ginkgo/#dynamically-generating-specs\.
+
+#. ``funcValue.Name()`` returns the full name of a function (e.g. ``fd.io/hs-test.MyTest``), however, we only need the test name (``MyTest``).
+
+#. To run certain tests solo, create a new slice that will only contain tests that have to run solo and a new register function.
+   Add a ``Serial`` decorator to the container node and ``Label("SOLO")`` to the ``It`` subject node:
+
+        ::
+
+                var _ = Describe("MySuiteSolo", Ordered, ContinueOnFailure, Serial, func() {
+                        ...
+                        It(strings.Split(funcValue.Name(), ".")[2], Label("SOLO"), func(ctx SpecContext) {
+			test(&s)
+		        }, SpecTimeout(time.Minute*5))
+                })
 
 #. Next step is to add test cases to the suite. For that, see section `Adding a test case`_ above
 
@@ -186,14 +250,9 @@ Alternatively copy the executable from host system to the Docker image, similarl
 **Skipping tests**
 
 ``HstSuite`` provides several methods that can be called in tests for skipping it conditionally or unconditionally such as:
-``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``.
+``skip()``, ``SkipIfMultiWorker()``, ``SkipUnlessExtendedTestsBuilt()``. You can also use Ginkgo's ``Skip()``.
 However the tests currently run under test suites which set up topology and containers before actual test is run. For the reason of saving
-test run time it is not advisable to use aforementioned skip methods and instead prefix test name with ``Skip``:
-
-::
-
-    func (s *MySuite) SkipTest(){
-
+test run time it is not advisable to use aforementioned skip methods and instead, just don't register the test.
 
 **Debugging a test**
 
@@ -201,11 +260,11 @@ It is possible to debug VPP by attaching ``gdb`` before test execution by adding
 
 ::
 
-    $ make test TEST=TestVeths/TestLDPreloadIperfVpp DEBUG=true
+    $ make test TEST=LDPreloadIperfVppTest DEBUG=true
     ...
     run following command in different terminal:
-    docker exec -it server-vpp gdb -ex "attach $(docker exec server-vpp pidof vpp)"
-    Afterwards press CTRL+C to continue
+    docker exec -it server-vpp2456109 gdb -ex "attach $(docker exec server-vpp2456109 pidof vpp)"
+    Afterwards press CTRL+\ to continue
 
 If a test consists of more VPP instances then this is done for each of them.
 
@@ -223,8 +282,38 @@ Generally, these will be updated on a per-need basis, for example when a bug is
 or a new version incompatibility issue occurs.
 
 
-.. _testing: https://pkg.go.dev/testing
-.. _go test: https://pkg.go.dev/cmd/go#hdr-Test_packages
-.. _test suite: https://github.com/stretchr/testify#suite-package
+.. _ginkgo: https://onsi.github.io/ginkgo/
 .. _volumes: https://docs.docker.com/storage/volumes/
 
+**List of tests**
+
+.. _list of tests:
+
+Please update this list whenever you add a new test by pasting the output below.
+
+* NsSuite/HttpTpsTest
+* NsSuite/VppProxyHttpTcpTest
+* NsSuite/VppProxyHttpTlsTest
+* NsSuite/EnvoyProxyHttpTcpTest
+* NginxSuite/MirroringTest
+* VethsSuiteSolo TcpWithLossTest [SOLO]
+* NoTopoSuiteSolo HttpStaticPromTest [SOLO]
+* TapSuite/LinuxIperfTest
+* NoTopoSuite/NginxHttp3Test
+* NoTopoSuite/NginxAsServerTest
+* NoTopoSuite/NginxPerfCpsTest
+* NoTopoSuite/NginxPerfRpsTest
+* NoTopoSuite/NginxPerfWrkTest
+* VethsSuite/EchoBuiltinTest
+* VethsSuite/HttpCliTest
+* VethsSuite/LDPreloadIperfVppTest
+* VethsSuite/VppEchoQuicTest
+* VethsSuite/VppEchoTcpTest
+* VethsSuite/VppEchoUdpTest
+* VethsSuite/XEchoVclClientUdpTest
+* VethsSuite/XEchoVclClientTcpTest
+* VethsSuite/XEchoVclServerUdpTest
+* VethsSuite/XEchoVclServerTcpTest
+* VethsSuite/VclEchoTcpTest
+* VethsSuite/VclEchoUdpTest
+* VethsSuite/VclRetryAttachTest