CSIT-530: Add common setup and teardown to perf tests

Change-Id: I2e76de710b4b758eac1d24d447a9c5938f937385
Signed-off-by: Tibor Frank <tifrank@cisco.com>

2 files changed, 109 insertions, 42 deletions

diff --git a/resources/libraries/python/QemuUtils.py b/resources/libraries/python/QemuUtils.py
index acc00de455..2123e42a4b 100644
--- a/resources/libraries/python/QemuUtils.py
+++ b/resources/libraries/python/QemuUtils.py
@@ -296,6 +296,7 @@ class QemuUtils(object):
             if time() - start > timeout:
                 raise RuntimeError('timeout, VM {0} not booted on {1}'.format(
                     self._qemu_opt['disk_image'], self._node['host']))
+            out = None
             try:
                 self._qemu_qga_flush()
                 out = self._qemu_qga_exec('guest-ping')
diff --git a/resources/libraries/robot/performance.robot b/resources/libraries/robot/performance.robot
index 47fde73007..a43972a7b1 100644
--- a/resources/libraries/robot/performance.robot
+++ b/resources/libraries/robot/performance.robot
@@ -1,4 +1,4 @@
-# Copyright (c) 2016 Cisco and/or its affiliates.
+# Copyright (c) 2017 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -46,8 +46,10 @@
 | | ...
 | | ... | *Example:*
 | | ...
-| | ... | \| Calculate pps \| 10000000000 | 64
+| | ... | \| Calculate pps \| 10000000000 \| 64 \|
+| | ...
 | | [Arguments] | ${bps} | ${framesize}
+| | ...
 | | ${framesize}= | Get Frame Size | ${framesize}
 | | ${ret}= | Evaluate | (${bps}/((${framesize}+20)*8)).__trunc__()
 | | Return From Keyword | ${ret}
@@ -63,25 +65,14 @@
 | | ...
 | | ... | *Example:*
 | | ...
-| | ... | \| Get Frame Size \| IMIX_v4_1
+| | ... | \| Get Frame Size \| IMIX_v4_1 \|
+| | ...
 | | [Arguments] | ${framesize}
+| | ...
 | | Run Keyword If | '${framesize}' == 'IMIX_v4_1'
 | | ... | Return From Keyword | 353.83333
 | | Return From Keyword | ${framesize}
 
-| Setup performance global Variables
-| | [Documentation]
-| | ... | Setup suite Variables. Variables are used across performance testing.
-| | ...
-| | ... | _NOTE:_ This KW sets following suite variables:
-| | ... | - glob_loss_acceptance - Loss acceptance treshold
-| | ... | - glob_loss_acceptance_type - Loss acceptance treshold type
-| | ... | - glob_vm_image - Guest VM disk image
-| | ...
-| | Set Suite Variable | ${glob_loss_acceptance} | 0.5
-| | Set Suite Variable | ${glob_loss_acceptance_type} | percentage
-| | Set Suite Variable | ${glob_vm_image} | /var/lib/vm/csit-nested-1.6.img
-
 | 2-node circular Topology Variables Setup
 | | [Documentation]
 | | ... | Compute path for testing on two given nodes in circular
@@ -164,7 +155,9 @@
 | | ...
 | | ... | \| 2-node circular Topology Variables Setup with DUT interface model \
 | | ... | \| Intel-X520-DA2 \|
+| | ...
 | | [Arguments] | ${iface_model}
+| | ...
 | | ${iface_model_list}= | Create list | ${iface_model}
 | | Append Node | ${nodes['TG']}
 | | Append Node | ${nodes['DUT1']} | filter_list=${iface_model_list}
@@ -205,7 +198,9 @@
 | | ...
 | | ... | \| 3-node circular Topology Variables Setup with DUT interface model \
 | | ... | \| Intel-X520-DA2 \|
+| | ...
 | | [Arguments] | ${iface_model}
+| | ...
 | | ${iface_model_list}= | Create list | ${iface_model}
 | | Append Node | ${nodes['TG']}
 | | Append Node | ${nodes['DUT1']} | filter_list=${iface_model_list}
@@ -310,7 +305,9 @@
 | | ...
 | | ... | \| Scale IPv4 forwarding initialized in a 3-node circular topology \
 | | ... | \| 100000 \|
+| | ...
 | | [Arguments] | ${count}
+| | ...
 | | Set Interface State | ${dut1} | ${dut1_if1} | up
 | | Set Interface State | ${dut1} | ${dut1_if2} | up
 | | Set Interface State | ${dut2} | ${dut2_if1} | up
@@ -359,7 +356,9 @@
 | | ...
 | | ... | \| IPv4 forwarding with vhost initialized in a 3-node circular \
 | | ... | topology \| /tmp/sock1 \| /tmp/sock2 \|
+| | ...
 | | [Arguments] | ${sock1} | ${sock2}
+| | ...
 | | VPP interfaces in path are up in a 3-node circular topology
 | | VPP Vhost interfaces for L2BD forwarding are setup | ${dut1}
 | | ... | ${sock1} | ${sock2}
@@ -430,21 +429,13 @@
 | | Set Test Variable | ${dut2_vif1_mac}
 | | Set Test Variable | ${dut2_vif2_mac}
 | | Add arp on dut | ${dut1} | ${dut1_if1} | 1.1.1.1 | ${tg1_if1_mac}
-| | ... | vrf=${fib_table_1}
 | | Add arp on dut | ${dut1} | ${dut1_if2} | 2.2.2.2 | ${dut2_if1_mac}
-| | ... | vrf=${fib_table_2}
 | | Add arp on dut | ${dut1} | ${dut1_vif1} | 4.4.4.2 | 52:54:00:00:04:01
-| | ... | vrf=${fib_table_1}
 | | Add arp on dut | ${dut1} | ${dut1_vif2} | 5.5.5.2 | 52:54:00:00:04:02
-| | ... | vrf=${fib_table_2}
 | | Add arp on dut | ${dut2} | ${dut2_if1} | 2.2.2.1 | ${dut1_if2_mac}
-| | ... | vrf=${fib_table_1}
 | | Add arp on dut | ${dut2} | ${dut2_if2} | 3.3.3.2 | ${tg1_if2_mac}
-| | ... | vrf=${fib_table_2}
 | | Add arp on dut | ${dut2} | ${dut2_vif1} | 4.4.4.2 | 52:54:00:00:04:01
-| | ... | vrf=${fib_table_1}
 | | Add arp on dut | ${dut2} | ${dut2_vif2} | 5.5.5.2 | 52:54:00:00:04:02
-| | ... | vrf=${fib_table_2}
 | | Vpp Route Add | ${dut1} | 20.20.20.0 | 24 | 4.4.4.2 | ${dut1_vif1}
 | | ... | vrf=${fib_table_1}
 | | Vpp Route Add | ${dut1} | 10.10.10.0 | 24 | 1.1.1.1 | ${dut1_if1}
@@ -532,7 +523,9 @@
 | | ...
 | | ... | \| Scale IPv6 forwarding initialized in a 3-node circular topology \
 | | ... | \| 100000 \|
+| | ...
 | | [Arguments] | ${count}
+| | ...
 | | ${subn_prefix}= | Set Variable | 64
 | | ${host_prefix}= | Set Variable | 128
 | | VPP Set If IPv6 Addr | ${dut1} | ${dut1_if1} | 2001:3::1 | ${subn_prefix}
@@ -621,7 +614,9 @@
 | | ...
 | | ... | \| L2 xconnect with Vhost-User initialized in a 3-node \
 | | ... | circular topology \| /tmp/sock1 \| /tmp/sock2 \|
+| | ...
 | | [Arguments] | ${sock1} | ${sock2}
+| | ...
 | | VPP Vhost interfaces for L2BD forwarding are setup | ${dut1}
 | | ... | ${sock1} | ${sock2}
 | | L2 setup xconnect on DUT | ${dut1} | ${dut1_if1} | ${vhost_if1}
@@ -672,7 +667,7 @@
 | | ... | Setup L2 DB topology by adding two interfaces on each DUT into BD
 | | ... | that is created automatically with index 1. Learning is enabled.
 | | ... | Interfaces are brought up.
-| | ... |
+| | ...
 | | Vpp l2bd forwarding setup | ${dut1} | ${dut1_if1} | ${dut1_if2}
 | | Vpp l2bd forwarding setup | ${dut2} | ${dut2_if1} | ${dut2_if2}
 | | All Vpp Interfaces Ready Wait | ${nodes}
@@ -693,7 +688,9 @@
 | | ...
 | | ... | \| L2 bridge domains with Vhost-User initialized in a 3-node \
 | | ... | circular topology \| 1 \| 2 \| /tmp/sock1 \| /tmp/sock2 \|
+| | ...
 | | [Arguments] | ${bd_id1} | ${bd_id2} | ${sock1} | ${sock2}
+| | ...
 | | VPP Vhost interfaces for L2BD forwarding are setup | ${dut1}
 | | ... | ${sock1} | ${sock2}
 | | Interface is added to bridge domain | ${dut1} | ${dut1_if1} | ${bd_id1}
@@ -864,7 +861,6 @@
 | | [Arguments] | ${topology_type} | ${nic_model}
 | | ...
 | | Show vpp version on all DUTs
-| | Setup performance global Variables
 | | 2-node circular Topology Variables Setup with DUT interface model
 | | ... | ${nic_model}
 | | Setup 2-node startup configuration of VPP on all DUTs
@@ -893,7 +889,6 @@
 | | ... | ${tg_if2_dest_mac}
 | | ...
 | | Show vpp version on all DUTs
-| | Setup performance global Variables
 | | 2-node circular Topology Variables Setup with DUT interface model
 | | ... | ${nic_model}
 | | Setup 2-node startup configuration of VPP on all DUTs
@@ -915,9 +910,10 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| 3-node Performance Suite Setup \| L2 \| Intel-X520-DA2 \|
+| | ...
 | | [Arguments] | ${topology_type} | ${nic_model}
+| | ...
 | | Show vpp version on all DUTs
-| | Setup performance global Variables
 | | 3-node circular Topology Variables Setup with DUT interface model
 | | ... | ${nic_model}
 | | Setup default startup configuration of VPP on all DUTs
@@ -959,7 +955,7 @@
 | | [Arguments] | ${framesize} | ${start_rate} | ${step_rate}
 | | ... | ${topology_type} | ${min_rate} | ${max_rate}
 | | ...
-| | ${duration}= | Set Variable | 10
+| | ${duration}= | Set Variable | ${perf_trial_duration}
 | | Set Duration | ${duration}
 | | Set Search Rate Boundaries | ${max_rate} | ${min_rate}
 | | Set Search Linear Step | ${step_rate}
@@ -1008,7 +1004,7 @@
 | | ... | ${topology_type} | ${min_rate} | ${max_rate}
 | | ... | ${loss_acceptance}=0 | ${loss_acceptance_type}='frames'
 | | ...
-| | ${duration}= | Set Variable | 10
+| | ${duration}= | Set Variable | ${perf_trial_duration}
 | | Set Duration | ${duration}
 | | Set Search Rate Boundaries | ${max_rate} | ${min_rate}
 | | Set Search Linear Step | ${step_rate}
@@ -1048,7 +1044,7 @@
 | | [Arguments] | ${framesize} | ${binary_min} | ${binary_max}
 | | ... | ${topology_type} | ${min_rate} | ${max_rate} | ${threshold}
 | | ...
-| | ${duration}= | Set Variable | 10
+| | ${duration}= | Set Variable | ${perf_trial_duration}
 | | Set Duration | ${duration}
 | | Set Search Rate Boundaries | ${max_rate} | ${min_rate}
 | | Set Search Frame Size | ${framesize}
@@ -1099,7 +1095,7 @@
 | | ... | ${topology_type} | ${min_rate} | ${max_rate} | ${threshold}
 | | ... | ${loss_acceptance}=0 | ${loss_acceptance_type}='frames'
 | | ...
-| | ${duration}= | Set Variable | 10
+| | ${duration}= | Set Variable | ${perf_trial_duration}
 | | Set Duration | ${duration}
 | | Set Search Rate Boundaries | ${max_rate} | ${min_rate}
 | | Set Search Frame Size | ${framesize}
@@ -1140,7 +1136,7 @@
 | | [Arguments] | ${framesize} | ${start_rate} | ${step_rate}
 | | ... | ${topology_type} | ${min_rate} | ${max_rate} | ${threshold}
 | | ...
-| | ${duration}= | Set Variable | 10
+| | ${duration}= | Set Variable | ${perf_trial_duration}
 | | Set Duration | ${duration}
 | | Set Search Rate Boundaries | ${max_rate} | ${min_rate}
 | | Set Search Linear Step | ${step_rate}
@@ -1193,7 +1189,7 @@
 | | ... | ${topology_type} | ${min_rate} | ${max_rate} | ${threshold}
 | | ... | ${loss_acceptance}=0 | ${loss_acceptance_type}='frames'
 | | ...
-| | ${duration}= | Set Variable | 10
+| | ${duration}= | Set Variable | ${perf_trial_duration}
 | | Set Duration | ${duration}
 | | Set Search Rate Boundaries | ${max_rate} | ${min_rate}
 | | Set Search Linear Step | ${step_rate}
@@ -1319,6 +1315,8 @@
 | | ... | - rate - Rate for sending packets. Type: string
 | | ... | - framesize - L2 Frame Size [B]. Type: integer
 | | ... | - topology_type - Topology type. Type: string
+| | ... | - fail_on_loss - If True, the keyword fails if loss occurred.
+| | ... | Type: boolean
 | | ...
 | | ... | *Example:*
 | | ...
@@ -1445,7 +1443,7 @@
 | | Run keyword | ${vm_name}.Qemu Set Node | ${dut_node}
 | | Run keyword | ${vm_name}.Qemu Set Smp | ${count} | ${count} | 1 | 1
 | | Run keyword | ${vm_name}.Qemu Set Mem Size | 2048
-| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${glob_vm_image}
+| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${perf_vm_image}
 | | ${vm}= | Run keyword | ${vm_name}.Qemu Start
 | | Run keyword | ${vm_name}.Qemu Set Affinity | @{qemu_cpus}
 | | Run keyword | ${vm_name}.Qemu Set Scheduler Policy
@@ -1489,7 +1487,7 @@
 | | Run keyword | ${vm_name}.Qemu Set Node | ${dut_node}
 | | Run keyword | ${vm_name}.Qemu Set Smp | ${count} | ${count} | 1 | 1
 | | Run keyword | ${vm_name}.Qemu Set Mem Size | 2048
-| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${glob_vm_image}
+| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${perf_vm_image}
 | | ${vm}= | Run keyword | ${vm_name}.Qemu Start
 | | Run keyword | ${vm_name}.Qemu Set Affinity | @{qemu_cpus}
 | | Run keyword | ${vm_name}.Qemu Set Scheduler Policy
@@ -1535,7 +1533,7 @@
 | | Run keyword | ${vm_name}.Qemu Set Node | ${dut_node}
 | | Run keyword | ${vm_name}.Qemu Set Smp | ${count} | ${count} | 1 | 1
 | | Run keyword | ${vm_name}.Qemu Set Mem Size | 2048
-| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${glob_vm_image}
+| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${perf_vm_image}
 | | ${vm}= | Run keyword | ${vm_name}.Qemu Start
 | | Run keyword | ${vm_name}.Qemu Set Affinity | @{qemu_cpus}
 | | Run keyword | ${vm_name}.Qemu Set Scheduler Policy
@@ -1582,7 +1580,7 @@
 | | Run keyword | ${vm_name}.Qemu Set Node | ${dut_node}
 | | Run keyword | ${vm_name}.Qemu Set Smp | ${count} | ${count} | 1 | 1
 | | Run keyword | ${vm_name}.Qemu Set Mem Size | 2048
-| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${glob_vm_image}
+| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${perf_vm_image}
 | | ${vm}= | Run keyword | ${vm_name}.Qemu Start
 | | Run keyword | ${vm_name}.Qemu Set Affinity | @{qemu_cpus}
 | | Run keyword | ${vm_name}.Qemu Set Scheduler Policy
@@ -1624,7 +1622,7 @@
 | | Run keyword | ${vm_name}.Qemu Set Node | ${dut_node}
 | | Run keyword | ${vm_name}.Qemu Set Smp | ${count} | ${count} | 1 | 1
 | | Run keyword | ${vm_name}.Qemu Set Mem Size | 2048
-| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${glob_vm_image}
+| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${perf_vm_image}
 | | ${vm}= | Run keyword | ${vm_name}.Qemu Start
 | | Run keyword | ${vm_name}.Qemu Set Affinity | @{qemu_cpus}
 | | Run keyword | ${vm_name}.Qemu Set Scheduler Policy
@@ -1670,7 +1668,7 @@
 | | Run keyword | ${vm_name}.Qemu Set Node | ${dut_node}
 | | Run keyword | ${vm_name}.Qemu Set Smp | ${count} | ${count} | 1 | 1
 | | Run keyword | ${vm_name}.Qemu Set Mem Size | 2048
-| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${glob_vm_image}
+| | Run keyword | ${vm_name}.Qemu Set Disk Image | ${perf_vm_image}
 | | ${vm}= | Run keyword | ${vm_name}.Qemu Start
 | | Run keyword | ${vm_name}.Qemu Set Affinity | @{qemu_cpus}
 | | Run keyword | ${vm_name}.Qemu Set Scheduler Policy
@@ -1923,7 +1921,6 @@
 | | ...
 | | [Arguments] | ${topology_type} | ${nic_model}
 | | ...
-| | Setup performance global Variables
 | | 2-node circular Topology Variables Setup with DUT interface model
 | | ... | ${nic_model}
 | | Initialize traffic generator | ${tg} | ${tg_if1} | ${tg_if2}
@@ -1947,7 +1944,6 @@
 | | ...
 | | [Arguments] | ${topology_type} | ${nic_model}
 | | ...
-| | Setup performance global Variables
 | | 3-node circular Topology Variables Setup with DUT interface model
 | | ... | ${nic_model}
 | | Initialize traffic generator | ${tg} | ${tg_if1} | ${tg_if2}
@@ -1979,3 +1975,73 @@
 | | ... | Variable Should Exist | ${DPDK_TEST}
 | | Return From Keyword If | "${ret}" == "PASS" | ${TRUE}
 | | Return From Keyword | ${FALSE}
+
+| Performance test setup
+| | [Documentation] | Common test setup for performance tests.
+| | ...
+| | Setup all DUTs before test
+| | Reset VAT History On All DUTs | ${nodes}
+
+| Performance test teardown
+| | [Documentation] | Common test teardown for ndrdisc and pdrdisc performance \
+| | ... | tests.
+| | ...
+| | ... | *Arguments:*
+| | ... | - rate - Rate for sending packets. Type: string
+| | ... | - framesize - L2 Frame Size [B]. Type: integer
+| | ... | - topology_type - Topology type. Type: string
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Performance test teardown \| 4.0mpps \| 64 \| 3-node-IPv4 \|
+| | ...
+| | [Arguments] | ${rate} | ${framesize} | ${topology_type}
+| | ...
+| | Show VAT History On All DUTs | ${nodes}
+| | Show statistics on all DUTs
+| | Run Keyword If Test Failed
+| | ... | Traffic should pass with no loss | ${perf_trial_duration} | ${rate}
+| | ... | ${framesize} | ${topology_type} | fail_on_loss=${False}
+| | Remove startup configuration of VPP from all DUTs
+
+| Performance ndrchk test teardown
+| | [Documentation] | Common test teardown for ndrchk performance tests.
+| | ...
+| | Show VAT History On All DUTs | ${nodes}
+| | Show statistics on all DUTs
+| | Remove startup configuration of VPP from all DUTs
+
+| Performance test with vhost and VM with dpdk-testpmd teardown
+| | [Documentation] | Common test teardown for performance tests which use
+| | ... | vhost(s) and VM(s) with dpdk-testpmd.
+| | ...
+| | ... | *Arguments:*
+| | ... | - rate - Rate for sending packets. Type: string
+| | ... | - framesize - L2 Frame Size [B]. Type: integer
+| | ... | - topology_type - Topology type. Type: string
+| | ... | - dut1_node - Node where to clean qemu. Type: dictionary
+| | ... | - dut1_vm_refs - VM references on node. Type: dictionary
+| | ... | - dut2_node - Node where to clean qemu. Type: dictionary
+| | ... | - dut2_vm_refs - VM references on node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Performance test with vhost and VM with dpdk-testpmd teardown \
+| | ... | \| 4.0mpps \| 64 \| 3-node-IPv4 \| ${node['DUT1']} \| ${dut_vm_refs} \
+| | ... | \| ${node['DUT2']} \| ${dut_vm_refs} \|
+| | ...
+| | [Arguments] | ${rate} | ${framesize} | ${topology_type}
+| | ... | ${dut1_node}=${None} | ${dut1_vm_refs}=${None}
+| | ... | ${dut2_node}=${None} | ${dut2_vm_refs}=${None}
+| | ...
+| | Show VAT History On All DUTs | ${nodes}
+| | Show Vpp Vhost On All DUTs
+| | Show statistics on all DUTs
+| | Run Keyword If Test Failed
+| | ... | Traffic should pass with no loss | ${perf_trial_duration} | ${rate}
+| | ... | ${framesize} | ${topology_type} | fail_on_loss=${False}
+| | Remove startup configuration of VPP from all DUTs
+| | Run keyword unless | ${dut1_node}==${None}
+| | ... | Guest VM with dpdk-testpmd Teardown | ${dut1} | ${dut1_vm_refs}
+| | Run keyword unless | ${dut2_node}==${None}
+| | ... | Guest VM with dpdk-testpmd Teardown | ${dut2} | ${dut2_vm_refs}