aboutsummaryrefslogtreecommitdiffstats
path: root/tests/func
diff options
context:
space:
mode:
authorselias <samelias@cisco.com>2016-10-24 18:15:52 +0200
committerPeter Mikus <pmikus@cisco.com>2016-11-04 08:50:25 +0000
commit6bfae6634aa26c2e88dddf49d91245d880b0222b (patch)
treefb554021ff68a8d862b5252583f296e681525863 /tests/func
parentb8fe6e5a6c0dc1f9972b7f3919041dedb1427cfd (diff)
CSIT-457: Address Honeycomb test failures
- tag all not-implemented features with EXPECTED_FAILING - add workarounds for known issues with VAT dump commands - replace hardcoded test data with parametrized - separate IP neighbor and IP address into two test cases - support unordered lists with compare_data_structures keyword Change-Id: I2c683162325cee720ba496489787fa50e9c72b8c Signed-off-by: selias <samelias@cisco.com>
Diffstat (limited to 'tests/func')
-rw-r--r--tests/func/honeycomb/010_interface_management.robot84
-rw-r--r--tests/func/honeycomb/031_vxlan_gpe.robot13
-rw-r--r--tests/func/honeycomb/060_sub_interface.robot36
-rw-r--r--tests/func/honeycomb/081_ietf_acl_traffic.robot8
4 files changed, 65 insertions, 76 deletions
diff --git a/tests/func/honeycomb/010_interface_management.robot b/tests/func/honeycomb/010_interface_management.robot
index 2c639c5c6a..b68b082772 100644
--- a/tests/func/honeycomb/010_interface_management.robot
+++ b/tests/func/honeycomb/010_interface_management.robot
@@ -20,7 +20,7 @@
| ${ipv4_mask}= | 255.255.255.0
| ${ipv4_prefix}= | ${24}
| @{ipv4_neighbor}= | 192.168.0.4 | 08:00:27:c0:5d:37
-| &{ipv4_settings}= | enabled=${True} | forwarding=${True} | mtu=${9000}
+| &{ipv4_settings}= | mtu=${9000}
| @{ipv6_address}= | 10::10 | ${64}
| @{ipv6_neighbor}= | 10::11 | 08:00:27:c0:5d:37
| &{ipv6_settings}= | enabled=${True} | forwarding=${True} | mtu=${9000}
@@ -42,11 +42,12 @@
| ... | Test suite uses the first interface of the first DUT node.
*** Test Cases ***
-# TODO: Remove "continue on failure" once VPP bugs (VPP-132, VPP-333) are fixed.
| Honeycomb configures and reads interface state
| | [Documentation] | Check if Honeycomb API can modify the admin state of\
| | ... | VPP interfaces.
-| | Given Interface State Is | ${node} | ${interface} | down
+| | Given Interface state from Honeycomb should be
+| | ... | ${node} | ${interface} | down
+| | And Interface state from VAT should be | ${node} | ${interface} | down
| | When Honeycomb sets interface state | ${node} | ${interface} | up
| | Then Interface state from Honeycomb should be
| | ... | ${node} | ${interface} | up
@@ -56,70 +57,71 @@
| | ... | ${node} | ${interface} | down
| | And Interface state from VAT should be | ${node} | ${interface} | down
-| Honeycomb modifies interface configuration - ipv4 (netmask)
+| Honeycomb modifies interface IPv4 address with netmask
| | [Documentation] | Check if Honeycomb API can configure interfaces for ipv4\
| | ... | with address and netmask provided.
+| | Given IPv4 address from Honeycomb should be empty | ${node} | ${interface}
+| | And ipv4 address from VAT should be empty | ${node} | ${interface}
| | When Honeycomb sets interface ipv4 address | ${node} | ${interface}
-| | ... | ${ipv4_address} | ${ipv4_mask} | ${ipv4_settings}
-| | And Honeycomb adds interface ipv4 neighbor
-| | ... | ${node} | ${interface} | @{ipv4_neighbor}
-| | Run Keyword And Continue On Failure
-| | ... | Then IPv4 config from Honeycomb should be
-| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
-| | ... | @{ipv4_neighbor} | ${ipv4_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv4 config from VAT should be
+| | ... | ${ipv4_address} | ${ipv4_mask}
+| | Then IPv4 address from Honeycomb should be
| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
+| | And IPv4 address from VAT should be
+| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_mask}
-| Honeycomb removes ipv4 address from interface
+| Honeycomb removes IPv4 address from interface
| | [Documentation] | Check if Honeycomb API can remove configured ipv4\
| | ... | addresses from interface.
-| | Run Keyword And Continue On Failure
-| | ... | Given IPv4 config from Honeycomb should be
-| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
-| | ... | @{ipv4_neighbor} | ${ipv4_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv4 config from VAT should be
+| | Given IPv4 address from Honeycomb should be
| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_prefix}
+| | And IPv4 address from VAT should be
+| | ... | ${node} | ${interface} | ${ipv4_address} | ${ipv4_mask}
| | When Honeycomb removes interface ipv4 addresses | ${node} | ${interface}
| | Then IPv4 address from Honeycomb should be empty | ${node} | ${interface}
| | And ipv4 address from VAT should be empty | ${node} | ${interface}
-| Honeycomb modifies interface configuration - ipv4 (prefix)
+| Honeycomb modifies interface IPv4 address with prefix
| | [Documentation] | Check if Honeycomb API can configure interfaces for ipv4\
| | ... | with address and prefix provided.
| | [Teardown] | Honeycomb removes interface ipv4 addresses | ${node}
| | ... | ${interface}
+| | Given IPv4 address from Honeycomb should be empty | ${node} | ${interface}
+| | And ipv4 address from VAT should be empty | ${node} | ${interface}
| | When Honeycomb sets interface ipv4 address with prefix
| | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_prefix}
-| | ... | ${ipv4_settings}
-| | And Honeycomb adds interface ipv4 neighbor
-| | ... | ${node} | ${interface} | @{ipv4_neighbor}
-| | Run Keyword And Continue On Failure
-| | ... | Then IPv4 config from Honeycomb should be
-| | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_prefix}
-| | ... | @{ipv4_neighbor}
-| | ... | ${ipv4_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv4 config from VAT should be
+| | Then IPv4 address from Honeycomb should be
| | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_prefix}
+| | And IPv4 address from VAT should be
+| | ... | ${node} | ${interface} | ${ipv4_address2} | ${ipv4_mask}
-| Honeycomb modifies interface configuration - ipv6
+| Honeycomb modifies IPv4 neighbor table
+| | [Documentation] | Check if Honeycomb API can add and remove ARP entries.
+# Feature not implemented
+| | [Tags] | EXPECTED_FAILING
+| | [Teardown] | Honeycomb clears all interface ipv4 neighbors
+| | ... | ${node} | ${interface}
+| | When Honeycomb adds interface ipv4 neighbor
+| | ... | ${node} | ${interface} | @{ipv4_neighbor}
+| | Then IPv4 neighbor from Honeycomb should be
+| | ... | ${node} | ${interface} | @{ipv4_neighbor}
+# VAT dump not available
+
+| Honeycomb modifies interface configuration - IPv6
| | [Documentation] | Check if Honeycomb API can configure interfaces for ipv6.
-| | When Honeycomb sets interface ipv6 configuration
-| | ... | ${node} | ${interface} | @{ipv6_address} | @{ipv6_neighbor}
-| | ... | ${ipv6_settings}
-| | Run Keyword And Continue On Failure
-| | ... | Then IPv6 config from Honeycomb should be
-| | ... | ${node} | ${interface} | @{ipv6_address} | @{ipv6_neighbor}
-| | ... | ${ipv6_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And IPv6 config from VAT should be
+# Feature not implemented
+| | [Tags] | EXPECTED_FAILING
+| | When Honeycomb sets interface ipv6 address
+| | ... | ${node} | ${interface} | @{ipv6_address}
+| | Then IPv6 address from Honeycomb should be
+| | ... | ${node} | ${interface} | @{ipv6_address}
+| | And IPv6 address from VAT should be
| | ... | ${node} | ${interface} | @{ipv6_address}
| Honeycomb modifies interface configuration - ethernet,routing
| | [Documentation] | Check if Honeycomb API can configure interface ethernet\
| | ... | and routing settings.
+# Feature not implemented
+| | [Tags] | EXPECTED_FAILING
| | When Honeycomb sets interface ethernet and routing configuration
| | ... | ${node} | ${interface} | ${ethernet} | ${routing}
| | Then Interface ethernet and routing configuration from Honeycomb should be
diff --git a/tests/func/honeycomb/031_vxlan_gpe.robot b/tests/func/honeycomb/031_vxlan_gpe.robot
index 54347e5725..9c6134b3cb 100644
--- a/tests/func/honeycomb/031_vxlan_gpe.robot
+++ b/tests/func/honeycomb/031_vxlan_gpe.robot
@@ -44,7 +44,6 @@
| ... | Restart Honeycomb And VPP And Clear Persisted Configuration | ${node}
*** Test Cases ***
-# TODO: Remove "continue on failure" once VPP bugs (VPP-217, VPP-156) are fixed.
| Honeycomb creates VxLAN GPE tunnel
| | [Documentation] | Check if Honeycomb API can configure a VxLAN GPE tunnel.
| | ...
@@ -58,8 +57,7 @@
| | Then VxLAN GPE configuration from Honeycomb should be
| | ... | ${node} | ${vxlan_gpe_if1}
| | ... | ${vxlan_gpe_base_settings} | ${vxlan_gpe_settings}
-| | And run keyword and continue on failure
-| | ... | VxLAN GPE configuration from VAT should be
+| | And VxLAN GPE configuration from VAT should be
| | ... | ${node} | ${vxlan_gpe_if1} | ${vxlan_gpe_settings}
| | And VxLAN GPE Interface indices from Honeycomb and VAT should correspond
| | ... | ${node} | ${vxlan_gpe_if1}
@@ -67,12 +65,10 @@
| Honeycomb removes VxLAN GPE tunnel
| | [Documentation] | Check if Honeycomb API can remove VxLAN GPE tunnel.
| | ...
-| | Run Keyword And Continue On Failure
-| | ... | Given VxLAN GPE configuration from Honeycomb should be
+| | Given VxLAN GPE configuration from Honeycomb should be
| | ... | ${node} | ${vxlan_gpe_if1}
| | ... | ${vxlan_gpe_base_settings} | ${vxlan_gpe_settings}
-| | Run Keyword And Continue On Failure
-| | ... | And VxLAN GPE configuration from VAT should be
+| | VxLAN GPE configuration from VAT should be
| | ... | ${node} | ${vxlan_gpe_if1} | ${vxlan_gpe_settings}
| | When Honeycomb removes VxLAN GPE interface
| | ... | ${node} | ${vxlan_gpe_if1}
@@ -161,8 +157,7 @@
| | Then VxLAN GPE configuration from Honeycomb should be
| | ... | ${node} | ${vxlan_gpe_if6}
| | ... | ${vxlan_gpe_base_ipv6_settings2} | ${vxlan_gpe_ipv6_settings2}
-| | And run keyword and continue on failure
-| | ... | VxLAN GPE configuration from VAT should be
+| | And VxLAN GPE configuration from VAT should be
| | ... | ${node} | ${vxlan_gpe_if6} | ${vxlan_gpe_ipv6_settings2}
| | And VxLAN GPE Interface indices from Honeycomb and VAT should correspond
| | ... | ${node} | ${vxlan_gpe_if6}
diff --git a/tests/func/honeycomb/060_sub_interface.robot b/tests/func/honeycomb/060_sub_interface.robot
index 8f12dd65dc..5d0d7a37a5 100644
--- a/tests/func/honeycomb/060_sub_interface.robot
+++ b/tests/func/honeycomb/060_sub_interface.robot
@@ -45,39 +45,34 @@
| | ... | ${node} | ${sub_if_name}
| | When Honeycomb creates sub-interface | ${node} | ${super_if}
| | ... | ${sub_if_1_match} | ${sub_if_1_tags} | ${sub_if_1_settings}
-| | Then run keyword and continue on failure
-| | ... | Sub-interface configuration from Honeycomb should be
+| | Then Sub-interface configuration from Honeycomb should be
| | ... | ${node} | ${super_if} | ${sub_if_id} | ${sub_if_1_oper}
-| | And run keyword and continue on failure
-| | ... | Sub-interface configuration from VAT should be
+| | And Sub-interface configuration from VAT should be
| | ... | ${node} | ${sub_if_name} | ${sub_if_1_oper}
| | And sub-interface indices from Honeycomb and VAT should correspond
| | ... | ${node} | ${super_if} | ${sub_if_id}
| Honeycomb sets interface and sub-interface up
-| | [Documentation] | Honeycomb changes the state of interface up and then \
-| | ... | changes the state of its sub-interface up, in this order.
+| | [Documentation] | Honeycomb changes the state of interface\
+| | ... | and of its sub-interface to up.
| | ...
| | Given interface state from Honeycomb should be
| | ... | ${node} | ${super_if} | down
| | And interface state from VAT should be
| | ... | ${node} | ${super_if} | down
+| | Sub-interface state from Honeycomb should be
+| | ... | ${node} | ${super_if} | ${sub_if_id} | down | down
+| | Sub-interface state from VAT should be
+| | ... | ${node} | ${sub_if_name} | down | down
| | When Honeycomb sets interface state
| | ... | ${node} | ${super_if} | up
| | Then interface state from Honeycomb should be
| | ... | ${node} | ${super_if} | up
| | And interface state from VAT should be
| | ... | ${node} | ${super_if} | up
-| | Given run keyword and continue on failure
-| | ... | Sub-interface state from Honeycomb should be
-| | ... | ${node} | ${super_if} | ${sub_if_id} | down | down
-| | And run keyword and continue on failure
-| | ... | Sub-interface state from VAT should be
-| | ... | ${node} | ${sub_if_name} | down | down
| | When Honeycomb sets the sub-interface up
| | ... | ${node} | ${super_if} | ${sub_if_id}
-| | Then run keyword and continue on failure
-| | ... | Sub-interface state from Honeycomb should be
+| | Then Sub-interface state from Honeycomb should be
| | ... | ${node} | ${super_if} | ${sub_if_id} | up | up
| | And sub-interface state from VAT should be
| | ... | ${node} | ${sub_if_name} | up | up
@@ -99,14 +94,14 @@
| | ... | ${node} | ${super_if} | up
| | When Honeycomb sets the sub-interface down
| | ... | ${node} | ${super_if} | ${sub_if_id}
-| | Then sub-interface state from Honeycomb should be
-| | ... | ${node} | ${super_if} | ${sub_if_id} | down | down
-| | And sub-interface state from VAT should be
-| | ... | ${node} | ${sub_if_name} | down | down
-| | And interface state from Honeycomb should be
+| | Then interface state from Honeycomb should be
| | ... | ${node} | ${super_if} | up
| | And interface state from VAT should be
| | ... | ${node} | ${super_if} | up
+| | And sub-interface state from Honeycomb should be
+| | ... | ${node} | ${super_if} | ${sub_if_id} | down | up
+| | And sub-interface state from VAT should be
+| | ... | ${node} | ${sub_if_name} | down | up
| Honeycomb sets interface and sub-interface down
| | [Documentation] | Honeycomb changes the state of interface down and then \
@@ -186,7 +181,7 @@
| | ... | ${node} | ${super_if} | ${sub_if_id} | ${sub_if_1_oper}
| | And sub-interface configuration from VAT should be
| | ... | ${node} | ${sub_if_name} | ${sub_if_1_oper}
-| | When Honeycomb creates L2 bridge domain
+| | When Honeycomb creates first L2 bridge domain
| | ... | ${node} | ${bd_name} | ${bd_settings}
| | Then bridge domain configuration from Honeycomb should be
| | ... | ${node} | ${bd_name} | ${bd_settings}
@@ -351,7 +346,6 @@
| | ... | ${node} | ${sub_if_name}
| | ... | ${ipv4['address']} | ${ipv4['prefix-length']}
-#TODO: Remove "continue on failure" once VPP bug VPP-132 is fixed.
| Honeycomb removes sub-interface ipv4 address
| | [Documentation] | Check if Honeycomb can remove configured ipv4 addresses\
| | ... | from the sub-interface.
diff --git a/tests/func/honeycomb/081_ietf_acl_traffic.robot b/tests/func/honeycomb/081_ietf_acl_traffic.robot
index 93a0e70330..01dbe5750c 100644
--- a/tests/func/honeycomb/081_ietf_acl_traffic.robot
+++ b/tests/func/honeycomb/081_ietf_acl_traffic.robot
@@ -345,11 +345,9 @@
| | Honeycomb sets interface state | ${dut_node} | ${dut_to_tg_if1} | up
| | Honeycomb sets interface state | ${dut_node} | ${dut_to_tg_if2} | up
| | Honeycomb sets interface ipv4 address with prefix | ${dut_node}
-| | ... | ${dut_to_tg_if1} | ${dut_to_tg_if1_ip}
-| | ... | ${prefix_length} | ${if_settings}
+| | ... | ${dut_to_tg_if1} | ${dut_to_tg_if1_ip} | ${prefix_length}
| | Honeycomb sets interface ipv4 address with prefix | ${dut_node}
-| | ... | ${dut_to_tg_if2} | ${dut_to_tg_if2_ip}
-| | ... | ${prefix_length} | ${if_settings}
+| | ... | ${dut_to_tg_if2} | ${dut_to_tg_if2_ip} | ${prefix_length}
# TODO: Configure routes through Honeycomb when implemented.(Honeycomb-58)
| | Add ARP on DUT
| | ... | ${node} | ${dut_to_tg_if2} | ${gateway} | ${tg_to_dut_if2_mac}
@@ -368,7 +366,7 @@
| | ... | ${test_data_id} | ${acl_name}
| | Honeycomb Sets Interface State | ${dut_node} | ${dut_to_tg_if1} | up
| | Honeycomb Sets Interface State | ${dut_node} | ${dut_to_tg_if2} | up
-| | Honeycomb Creates L2 Bridge Domain
+| | Honeycomb Creates first L2 Bridge Domain
| | ... | ${dut_node} | ${bd_name} | ${bd_settings}
| | Honeycomb Adds Interfaces To Bridge Domain
| | ... | ${dut_node} | ${dut_to_tg_if1} | ${dut_to_tg_if2}