CSIT-457: Address Honeycomb test failures

 - tag all not-implemented features with EXPECTED_FAILING
 - add workarounds for known issues with VAT dump commands
 - replace hardcoded test data with parametrized
 - separate IP neighbor and IP address into two test cases
 - support unordered lists with compare_data_structures keyword

Change-Id: I2c683162325cee720ba496489787fa50e9c72b8c
Signed-off-by: selias <samelias@cisco.com>

4 files changed, 65 insertions, 76 deletions

diff --git a/tests/func/honeycomb/010_interface_management.robot b/tests/func/honeycomb/010_interface_management.robot
index 2c639c5c6a..b68b082772 100644
--- a/tests/func/honeycomb/010_interface_management.robot
+++ b/tests/func/honeycomb/010_interface_management.robot
@@ -20,7 +20,7 @@
 | ${ipv4_mask}= | 255.255.255.0
 | ${ipv4_prefix}= | ${24}
 | @{ipv4_neighbor}= | 192.168.0.4 | 08:00:27:c0:5d:37
-| &{ipv4_settings}= | enabled=${True} | forwarding=${True} | mtu=${9000}
+| &{ipv4_settings}= | mtu=${9000}
 | @{ipv6_address}= | 10::10 | ${64}
 | @{ipv6_neighbor}= | 10::11 | 08:00:27:c0:5d:37
 | &{ipv6_settings}= | enabled=${True} | forwarding=${True} | mtu=${9000}
@@ -42,11 +42,12 @@
 | ... | Test suite uses the first interface of the first DUT node.
 
 *** Test Cases ***
-# TODO: Remove "continue on failure" once VPP bugs (VPP-132, VPP-333) are fixed.
 | Honeycomb configures and reads interface state
 | | [Documentation] | Check if Honeycomb API can modify the admin state of\
 | | ... | VPP interfaces.
-| | Given Interface State Is | ${node} | ${interface} | down
+| | Given Interface state from Honeycomb should be
+| | ... | ${node} | ${interface} | down
+| | And Interface state from VAT should be | ${node} | ${interface} | down
 | | When Honeycomb sets interface state | ${node} | ${interface} | up
 | | Then Interface state from Honeycomb should be
 | | ... | ${node} | ${interface} | up
@@ -56,70 +57,71 @@
 | | ... | ${node} | ${interface} | down
 | | And Interface state from VAT should be | ${node} | ${interface} | down
 
-| Honeycomb modifies interface configuration - ipv4 (netmask)
+| Honeycomb modifies interface IPv4 address with netmask
 | | [Documentation] | Check if Honeycomb API can configure interfaces for ipv4\
 | | ... | with address and netmask provided.
+| | Given IPv4 address from Honeycomb should be empty | ${node} | ${interface}
+| | And ipv4 address from VAT should be empty | ${node} | ${interface}
 | | When Honeycomb sets interface ipv4 address | ${node} | ${interface}
-| | ... | ${ipv4_address} | ${ipv4_mask} | ${ipv4_settings}
-| | And Honeycomb adds interface ipv4 neighbor
-| | ... | ${node} | ${interface} | @{ipv4_neighbor}
-| | Run Keyword And Continue On Failure
-| | ... | Then IPv4 config from Honeycomb should be
-| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
-| | ... | @{ipv4_neighbor} | ${ipv4_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv4 config from VAT should be
+| | ... | ${ipv4_address} | ${ipv4_mask}
+| | Then IPv4 address from Honeycomb should be
 | | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
+| | And IPv4 address from VAT should be
+| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_mask}
 
-| Honeycomb removes ipv4 address from interface
+| Honeycomb removes IPv4 address from interface
 | | [Documentation] | Check if Honeycomb API can remove configured ipv4\
 | | ... | addresses from interface.
-| | Run Keyword And Continue On Failure
-| | ... | Given IPv4 config from Honeycomb should be
-| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
-| | ... | @{ipv4_neighbor} | ${ipv4_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv4 config from VAT should be
+| | Given IPv4 address from Honeycomb should be
 | | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
+| | And IPv4 address from VAT should be
+| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_mask}
 | | When Honeycomb removes interface ipv4 addresses | ${node} | ${interface}
 | | Then IPv4 address from Honeycomb should be empty | ${node} | ${interface}
 | | And ipv4 address from VAT should be empty | ${node} | ${interface}
 
-| Honeycomb modifies interface configuration - ipv4 (prefix)
+| Honeycomb modifies interface IPv4 address with prefix
 | | [Documentation] | Check if Honeycomb API can configure interfaces for ipv4\
 | | ... | with address and prefix provided.
 | | [Teardown] | Honeycomb removes interface ipv4 addresses | ${node}
 | | ... | ${interface}
+| | Given IPv4 address from Honeycomb should be empty | ${node} | ${interface}
+| | And ipv4 address from VAT should be empty | ${node} | ${interface}
 | | When Honeycomb sets interface ipv4 address with prefix
 | | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_prefix}
-| | ... | ${ipv4_settings}
-| | And Honeycomb adds interface ipv4 neighbor
-| | ... | ${node} | ${interface} | @{ipv4_neighbor}
-| | Run Keyword And Continue On Failure
-| | ... | Then IPv4 config from Honeycomb should be
-| | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_prefix}
-| | ... | @{ipv4_neighbor}
-| | ... | ${ipv4_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv4 config from VAT should be
+| | Then IPv4 address from Honeycomb should be
 | | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_prefix}
+| | And IPv4 address from VAT should be
+| | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_mask}
 
-| Honeycomb modifies interface configuration - ipv6
+| Honeycomb modifies IPv4 neighbor table
+| | [Documentation] | Check if Honeycomb API can add and remove ARP entries.
+# Feature not implemented
+| | [Tags] | EXPECTED_FAILING
+| | [Teardown] | Honeycomb clears all interface ipv4 neighbors
+| | ... | ${node} | ${interface}
+| | When Honeycomb adds interface ipv4 neighbor
+| | ... | ${node} | ${interface} | @{ipv4_neighbor}
+| | Then IPv4 neighbor from Honeycomb should be
+| | ... | ${node} | ${interface} | @{ipv4_neighbor}
+# VAT dump not available
+
+| Honeycomb modifies interface configuration - IPv6
 | | [Documentation] | Check if Honeycomb API can configure interfaces for ipv6.
-| | When Honeycomb sets interface ipv6 configuration
-| | ... | ${node} | ${interface} | @{ipv6_address} | @{ipv6_neighbor}
-| | ... | ${ipv6_settings}
-| | Run Keyword And Continue On Failure
-| | ... | Then IPv6 config from Honeycomb should be
-| | ... | ${node} | ${interface} | @{ipv6_address} | @{ipv6_neighbor}
-| | ... | ${ipv6_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv6 config from VAT should be
+# Feature not implemented
+| | [Tags] | EXPECTED_FAILING
+| | When Honeycomb sets interface ipv6 address
+| | ... | ${node} | ${interface} | @{ipv6_address}
+| | Then IPv6 address from Honeycomb should be
+| | ... | ${node} | ${interface} | @{ipv6_address}
+| | And IPv6 address from VAT should be
 | | ... | ${node} | ${interface} | @{ipv6_address}
 
 | Honeycomb modifies interface configuration - ethernet,routing
 | | [Documentation] | Check if Honeycomb API can configure interface ethernet\
 | | ... | and routing settings.
+# Feature not implemented
+| | [Tags] | EXPECTED_FAILING
 | | When Honeycomb sets interface ethernet and routing configuration
 | | ... | ${node} | ${interface} | ${ethernet} | ${routing}
 | | Then Interface ethernet and routing configuration from Honeycomb should be
diff --git a/tests/func/honeycomb/031_vxlan_gpe.robot b/tests/func/honeycomb/031_vxlan_gpe.robot
index 54347e5725..9c6134b3cb 100644
--- a/tests/func/honeycomb/031_vxlan_gpe.robot
+++ b/tests/func/honeycomb/031_vxlan_gpe.robot
@@ -44,7 +44,6 @@
 | ... | Restart Honeycomb And VPP And Clear Persisted Configuration | ${node}
 
 *** Test Cases ***
-# TODO: Remove "continue on failure" once VPP bugs (VPP-217, VPP-156) are fixed.
 | Honeycomb creates VxLAN GPE tunnel
 | | [Documentation] | Check if Honeycomb API can configure a VxLAN GPE tunnel.
 | | ...
@@ -58,8 +57,7 @@
 | | Then VxLAN GPE configuration from Honeycomb should be
 | | ... | ${node} | ${vxlan_gpe_if1}
 | | ... | ${vxlan_gpe_base_settings} | ${vxlan_gpe_settings}
-| | And run keyword and continue on failure
-| | ... | VxLAN GPE configuration from VAT should be
+| | And VxLAN GPE configuration from VAT should be
 | | ... | ${node} | ${vxlan_gpe_if1} | ${vxlan_gpe_settings}
 | | And VxLAN GPE Interface indices from Honeycomb and VAT should correspond
 | | ... | ${node} | ${vxlan_gpe_if1}
@@ -67,12 +65,10 @@
 | Honeycomb removes VxLAN GPE tunnel
 | | [Documentation] | Check if Honeycomb API can remove VxLAN GPE tunnel.
 | | ...
-| | Run Keyword And Continue On Failure
-| | ... | Given VxLAN GPE configuration from Honeycomb should be
+| | Given VxLAN GPE configuration from Honeycomb should be
 | | ... | ${node} | ${vxlan_gpe_if1}
 | | ... | ${vxlan_gpe_base_settings} | ${vxlan_gpe_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And VxLAN GPE configuration from VAT should be
+| | VxLAN GPE configuration from VAT should be
 | | ... | ${node} | ${vxlan_gpe_if1} | ${vxlan_gpe_settings}
 | | When Honeycomb removes VxLAN GPE interface
 | | ... | ${node} | ${vxlan_gpe_if1}
@@ -161,8 +157,7 @@
 | | Then VxLAN GPE configuration from Honeycomb should be
 | | ... | ${node} | ${vxlan_gpe_if6}
 | | ... | ${vxlan_gpe_base_ipv6_settings2} | ${vxlan_gpe_ipv6_settings2}
-| | And run keyword and continue on failure
-| | ... | VxLAN GPE configuration from VAT should be
+| | And VxLAN GPE configuration from VAT should be
 | | ... | ${node} | ${vxlan_gpe_if6} | ${vxlan_gpe_ipv6_settings2}
 | | And VxLAN GPE Interface indices from Honeycomb and VAT should correspond
 | | ... | ${node} | ${vxlan_gpe_if6}
diff --git a/tests/func/honeycomb/060_sub_interface.robot b/tests/func/honeycomb/060_sub_interface.robot
index 8f12dd65dc..5d0d7a37a5 100644
--- a/tests/func/honeycomb/060_sub_interface.robot
+++ b/tests/func/honeycomb/060_sub_interface.robot
@@ -45,39 +45,34 @@
 | | ... | ${node} | ${sub_if_name}
 | | When Honeycomb creates sub-interface | ${node} | ${super_if}
 | | ... | ${sub_if_1_match} | ${sub_if_1_tags} | ${sub_if_1_settings}
-| | Then run keyword and continue on failure
-| | ... | Sub-interface configuration from Honeycomb should be
+| | Then Sub-interface configuration from Honeycomb should be
 | | ... | ${node} | ${super_if} | ${sub_if_id} | ${sub_if_1_oper}
-| | And run keyword and continue on failure
-| | ... | Sub-interface configuration from VAT should be
+| | And Sub-interface configuration from VAT should be
 | | ... | ${node} | ${sub_if_name} | ${sub_if_1_oper}
 | | And sub-interface indices from Honeycomb and VAT should correspond
 | | ... | ${node} | ${super_if} | ${sub_if_id}
 
 | Honeycomb sets interface and sub-interface up
-| | [Documentation] | Honeycomb changes the state of interface up and then \
-| | ... | changes the state of its sub-interface up, in this order.
+| | [Documentation] | Honeycomb changes the state of interface\
+| | ... | and of its sub-interface to up.
 | | ...
 | | Given interface state from Honeycomb should be
 | | ... | ${node} | ${super_if} | down
 | | And interface state from VAT should be
 | | ... | ${node} | ${super_if} | down
+| | Sub-interface state from Honeycomb should be
+| | ... | ${node} | ${super_if} | ${sub_if_id} | down | down
+| | Sub-interface state from VAT should be
+| | ... | ${node} | ${sub_if_name} | down | down
 | | When Honeycomb sets interface state
 | | ... | ${node} | ${super_if} | up
 | | Then interface state from Honeycomb should be
 | | ... | ${node} | ${super_if} | up
 | | And interface state from VAT should be
 | | ... | ${node} | ${super_if} | up
-| | Given run keyword and continue on failure
-| | ... | Sub-interface state from Honeycomb should be
-| | ... | ${node} | ${super_if} | ${sub_if_id} | down | down
-| | And run keyword and continue on failure
-| | ... | Sub-interface state from VAT should be
-| | ... | ${node} | ${sub_if_name} | down | down
 | | When Honeycomb sets the sub-interface up
 | | ... | ${node} | ${super_if} | ${sub_if_id}
-| | Then run keyword and continue on failure
-| | ... | Sub-interface state from Honeycomb should be
+| | Then Sub-interface state from Honeycomb should be
 | | ... | ${node} | ${super_if} | ${sub_if_id} | up | up
 | | And sub-interface state from VAT should be
 | | ... | ${node} | ${sub_if_name} | up | up
@@ -99,14 +94,14 @@
 | | ... | ${node} | ${super_if} | up
 | | When Honeycomb sets the sub-interface down
 | | ... | ${node} | ${super_if} | ${sub_if_id}
-| | Then sub-interface state from Honeycomb should be
-| | ... | ${node} | ${super_if} | ${sub_if_id} | down | down
-| | And sub-interface state from VAT should be
-| | ... | ${node} | ${sub_if_name} | down | down
-| | And interface state from Honeycomb should be
+| | Then interface state from Honeycomb should be
 | | ... | ${node} | ${super_if} | up
 | | And interface state from VAT should be
 | | ... | ${node} | ${super_if} | up
+| | And sub-interface state from Honeycomb should be
+| | ... | ${node} | ${super_if} | ${sub_if_id} | down | up
+| | And sub-interface state from VAT should be
+| | ... | ${node} | ${sub_if_name} | down | up
 
 | Honeycomb sets interface and sub-interface down
 | | [Documentation] | Honeycomb changes the state of interface down and then \
@@ -186,7 +181,7 @@
 | | ... | ${node} | ${super_if} | ${sub_if_id} | ${sub_if_1_oper}
 | | And sub-interface configuration from VAT should be
 | | ... | ${node} | ${sub_if_name} | ${sub_if_1_oper}
-| | When Honeycomb creates L2 bridge domain
+| | When Honeycomb creates first L2 bridge domain
 | | ... | ${node} | ${bd_name} | ${bd_settings}
 | | Then bridge domain configuration from Honeycomb should be
 | | ... | ${node} | ${bd_name} | ${bd_settings}
@@ -351,7 +346,6 @@
 | | ... | ${node} | ${sub_if_name}
 | | ... | ${ipv4['address']} | ${ipv4['prefix-length']}
 
-#TODO: Remove "continue on failure" once VPP bug VPP-132 is fixed.
 | Honeycomb removes sub-interface ipv4 address
 | | [Documentation] | Check if Honeycomb can remove configured ipv4 addresses\
 | | ... | from the sub-interface.
diff --git a/tests/func/honeycomb/081_ietf_acl_traffic.robot b/tests/func/honeycomb/081_ietf_acl_traffic.robot
index 93a0e70330..01dbe5750c 100644
--- a/tests/func/honeycomb/081_ietf_acl_traffic.robot
+++ b/tests/func/honeycomb/081_ietf_acl_traffic.robot
@@ -345,11 +345,9 @@
 | | Honeycomb sets interface state | ${dut_node} | ${dut_to_tg_if1} | up
 | | Honeycomb sets interface state | ${dut_node} | ${dut_to_tg_if2} | up
 | | Honeycomb sets interface ipv4 address with prefix | ${dut_node}
-| | ... | ${dut_to_tg_if1} | ${dut_to_tg_if1_ip}
-| | ... | ${prefix_length} | ${if_settings}
+| | ... | ${dut_to_tg_if1} | ${dut_to_tg_if1_ip} | ${prefix_length}
 | | Honeycomb sets interface ipv4 address with prefix | ${dut_node}
-| | ... | ${dut_to_tg_if2} | ${dut_to_tg_if2_ip}
-| | ... | ${prefix_length} | ${if_settings}
+| | ... | ${dut_to_tg_if2} | ${dut_to_tg_if2_ip} | ${prefix_length}
 # TODO: Configure routes through Honeycomb when implemented.(Honeycomb-58)
 | | Add ARP on DUT
 | | ... | ${node} | ${dut_to_tg_if2} | ${gateway} | ${tg_to_dut_if2_mac}
@@ -368,7 +366,7 @@
 | | ... | ${test_data_id} | ${acl_name}
 | | Honeycomb Sets Interface State | ${dut_node} | ${dut_to_tg_if1} | up
 | | Honeycomb Sets Interface State | ${dut_node} | ${dut_to_tg_if2} | up
-| | Honeycomb Creates L2 Bridge Domain
+| | Honeycomb Creates first L2 Bridge Domain
 | | ... | ${dut_node} | ${bd_name} | ${bd_settings}
 | | Honeycomb Adds Interfaces To Bridge Domain
 | | ... | ${dut_node} | ${dut_to_tg_if1} | ${dut_to_tg_if2}