Capture VPP stats for Long and Short tests

JIRA: CSIT-71
- remove warmup for short runs
- clear and collect stats for short run
- run short test with no failure for long test with NDR/PDR rate

Change-Id: I1b8879c3b891165072822207d0781687d8159336
Signed-off-by: Miroslav Miklus <mmiklus@cisco.com>

6 files changed, 79 insertions, 23 deletions

diff --git a/resources/libraries/python/VppCounters.py b/resources/libraries/python/VppCounters.py
index 4bb7d63135..af3ea47560 100644
--- a/resources/libraries/python/VppCounters.py
+++ b/resources/libraries/python/VppCounters.py
@@ -108,6 +108,28 @@ class VppCounters(object):
         vat.execute_script('clear_interface.vat', node)
         vat.script_should_have_passed()
 
+    @staticmethod
+    def vpp_clear_hardware_counters(node):
+        """Clear interface hardware counters on VPP node.
+
+        :param node: Node to clear hardware counters on.
+        :type node: dict
+        """
+        vat = VatExecutor()
+        vat.execute_script('clear_hardware.vat', node)
+        vat.script_should_have_passed()
+
+    @staticmethod
+    def vpp_clear_errors_counters(node):
+        """Clear errors counters on VPP node.
+
+        :param node: Node to clear errors counters on.
+        :type node: dict
+        """
+        vat = VatExecutor()
+        vat.execute_script('clear_errors.vat', node)
+        vat.script_should_have_passed()
+
     def vpp_dump_stats_table(self, node):
         """Dump stats table on VPP node.
 
diff --git a/resources/libraries/robot/counters.robot b/resources/libraries/robot/counters.robot
index 0cb18c15db..1fa4ca491c 100644
--- a/resources/libraries/robot/counters.robot
+++ b/resources/libraries/robot/counters.robot
@@ -38,6 +38,13 @@
 | | ${ipv4_counter}= | Vpp get ipv4 interface counter | ${node} | ${interface}
 | | Should Be Equal | ${ipv4_counter} | ${value}
 
+| Show statistics on all DUTs
+| | [Documentation] | Show VPP statistics on all DUTs after the test failed
+| | Sleep | 10 | Waiting for statistics to be collected
+| | ${duts}= | Get Matches | ${nodes} | DUT*
+| | :FOR | ${dut} | IN | @{duts}
+| | | Vpp show stats | ${nodes['${dut}']}
+
 | Vpp show stats
 | | [Documentation] | Show [error, hardware, interface] stats
 | | [Arguments] | ${node}
@@ -46,14 +53,40 @@
 | | Vpp Show Hardware Detail | ${node}
 | | Vpp Show Runtime | ${node}
 
-| Clear runtime statistics on all DUTs
-| | [Documentation] | Clear VPP runtime statistics on all DUTs
+| Clear all counters on all DUTs
+| | [Documentation] | Clear runtime, interface, hardware and error counters
+| | ... | on all DUTs with VPP instance
+| | Clear runtime counters on all DUTs
+| | Clear interface counters on all DUTs
+| | Clear hardware counters on all DUTs
+| | Clear errors counters on all DUTs
+
+| Clear runtime counters on all DUTs
+| | [Documentation] | Clear VPP runtime counters on all DUTs
 | | ${duts}= | Get Matches | ${nodes} | DUT*
 | | :FOR | ${dut} | IN | @{duts}
 | | | Vpp clear runtime | ${nodes['${dut}']}
 
-| Show runtime statistics on all DUTs
-| | [Documentation] | Show VPP runtime statistics on all DUTs
+| Clear interface counters on all DUTs
+| | [Documentation] | Clear VPP interface counters on all DUTs
+| | ${duts}= | Get Matches | ${nodes} | DUT*
+| | :FOR | ${dut} | IN | @{duts}
+| | | Vpp clear interface counters | ${nodes['${dut}']}
+
+| Clear hardware counters on all DUTs
+| | [Documentation] | Clear VPP hardware counters on all DUTs
+| | ${duts}= | Get Matches | ${nodes} | DUT*
+| | :FOR | ${dut} | IN | @{duts}
+| | | Vpp clear hardware counters | ${nodes['${dut}']}
+
+| Clear errors counters on all DUTs
+| | [Documentation] | Clear VPP errors counters on all DUTs
+| | ${duts}= | Get Matches | ${nodes} | DUT*
+| | :FOR | ${dut} | IN | @{duts}
+| | | Vpp clear errors counters | ${nodes['${dut}']}
+
+| Show runtime counters on all DUTs
+| | [Documentation] | Show VPP runtime counters on all DUTs
 | | ${duts}= | Get Matches | ${nodes} | DUT*
 | | :FOR | ${dut} | IN | @{duts}
 | | | Vpp show runtime | ${nodes['${dut}']}
diff --git a/resources/libraries/robot/default.robot b/resources/libraries/robot/default.robot
index 51cafdff96..8e03e62983 100644
--- a/resources/libraries/robot/default.robot
+++ b/resources/libraries/robot/default.robot
@@ -28,13 +28,6 @@
 | | [Documentation] | Prepare all TGs before traffic scripts execution
 | | All TGs Set Interface Default Driver | ${nodes}
 
-| Show statistics on all DUTs
-| | [Documentation] | Show VPP statistics on all DUTs after the test failed
-| | Sleep | 10 | Waiting for statistics to be collected
-| | ${duts}= | Get Matches | ${nodes} | DUT*
-| | :FOR | ${dut} | IN | @{duts}
-| | | Vpp show stats | ${nodes['${dut}']}
-
 | Setup '${m}' worker threads and rss '${n}' without HTT on all DUTs
 | | [Documentation] |  Setup M worker threads without HTT and rss N in startup
 | | ...             |  configuration of VPP on all DUTs
diff --git a/resources/libraries/robot/performance.robot b/resources/libraries/robot/performance.robot
index 8201ae405f..6c06288f53 100644
--- a/resources/libraries/robot/performance.robot
+++ b/resources/libraries/robot/performance.robot
@@ -181,8 +181,9 @@
 | | Linear Search | ${start_rate} | ${topology_type}
 | | ${rate_per_stream}= | Verify Search Result
 | | Display result of NDR search | ${rate_per_stream} | ${framesize} | 2
-| | Clear and show runtime stats with running traffic | ${duration}
-| | ...  | ${rate_per_stream}pps | ${framesize} | ${topology_type}
+| | Traffic should pass with no loss | ${duration} | ${rate_per_stream}pps
+| | ...                              | ${framesize} | ${topology_type}
+| | ...                              | fail_on_loss=${False}
 
 | Find NDR using binary search and pps
 | | [Documentation] | Find throughput by using RFC2544 binary search
@@ -197,8 +198,9 @@
 | | Binary Search | ${binary_min} | ${binary_max} | ${topology_type}
 | | ${rate_per_stream}= | Verify Search Result
 | | Display result of NDR search | ${rate_per_stream} | ${framesize} | 2
-| | Clear and show runtime stats with running traffic | ${duration}
-| | ...  | ${rate_per_stream}pps | ${framesize} | ${topology_type}
+| | Traffic should pass with no loss | ${duration} | ${rate_per_stream}pps
+| | ...                              | ${framesize} | ${topology_type}
+| | ...                              | fail_on_loss=${False}
 
 | Find NDR using combined search and pps
 | | [Documentation] | Find throughput by using RFC2544 combined search
@@ -215,8 +217,9 @@
 | | Combined Search | ${start_rate} | ${topology_type}
 | | ${rate_per_stream}= | Verify Search Result
 | | Display result of NDR search | ${rate_per_stream} | ${framesize} | 2
-| | Clear and show runtime stats with running traffic | ${duration}
-| | ...  | ${rate_per_stream}pps | ${framesize} | ${topology_type}
+| | Traffic should pass with no loss | ${duration} | ${rate_per_stream}pps
+| | ...                              | ${framesize} | ${topology_type}
+| | ...                              | fail_on_loss=${False}
 
 | Display result of NDR search
 | | [Documentation] | Display result of NDR search in packet per seconds (total
@@ -230,17 +233,20 @@
 
 | Traffic should pass with no loss
 | | [Arguments] | ${duration} | ${rate} | ${framesize} | ${topology_type}
-| | Clear and show runtime stats with running traffic | ${duration}
+| | ...         | ${fail_on_loss}=${True}
+| | Clear and show runtime counters with running traffic | ${duration}
 | | ...  | ${rate} | ${framesize} | ${topology_type}
+| | Clear all counters on all DUTs
 | | Send traffic on tg | ${duration} | ${rate} | ${framesize}
-| | ...                | ${topology_type}
-| | No traffic loss occurred
+| | ...                | ${topology_type} | warmup_time=0
+| | Show statistics on all DUTs
+| | Run Keyword If | ${fail_on_loss} | No traffic loss occurred
 
-| Clear and show runtime stats with running traffic
+| Clear and show runtime counters with running traffic
 | | [Arguments] | ${duration} | ${rate} | ${framesize} | ${topology_type}
 | | Send traffic on tg | -1 | ${rate} | ${framesize}
 | | ...                | ${topology_type} | warmup_time=0 | async_call=True
-| | Clear runtime statistics on all DUTs
+| | Clear runtime counters on all DUTs
 | | Sleep | ${duration}
-| | Show runtime statistics on all DUTs
+| | Show runtime counters on all DUTs
 | | Stop traffic on tg
diff --git a/resources/templates/vat/clear_errors.vat b/resources/templates/vat/clear_errors.vat
new file mode 100644
index 0000000000..b6dd4c2650
--- /dev/null
+++ b/resources/templates/vat/clear_errors.vat
@@ -0,0 +1 @@
+exec clear errors
diff --git a/resources/templates/vat/clear_hardware.vat b/resources/templates/vat/clear_hardware.vat
new file mode 100644
index 0000000000..235115155e
--- /dev/null
+++ b/resources/templates/vat/clear_hardware.vat
@@ -0,0 +1 @@
+exec clear hardware