CSIT-619 HC Test: Honeycomb performance testing - initial commit

 - keywords and scripts for HC performance testing setup
 - basic performance suite: operational data read
 - traffic script and keywords used in tests

Change-Id: Ic0290be73a7c925ea2561f8cd2524c5cb83fcda2
Signed-off-by: selias <samelias@cisco.com>

4 files changed, 332 insertions, 0 deletions

diff --git a/tests/vpp/perf/honeycomb/__init__.robot b/tests/vpp/perf/honeycomb/__init__.robot
new file mode 100644
index 0000000000..b984993139
--- /dev/null
+++ b/tests/vpp/perf/honeycomb/__init__.robot
@@ -0,0 +1,61 @@
+# Copyright (c) 2017 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Variables***
+# Honeycomb node to run tests on.
+| ${node}= | ${nodes['DUT1']}
+
+*** Settings ***
+| Resource | resources/libraries/robot/shared/default.robot
+| Resource | resources/libraries/robot/shared/interfaces.robot
+| Resource | resources/libraries/robot/honeycomb/honeycomb.robot
+| Resource | resources/libraries/robot/honeycomb/performance.robot
+| Library | resources.libraries.python.SetupFramework
+| Library | resources.libraries.python.CpuUtils
+| Library | resources.libraries.python.honeycomb.Performance
+| Suite Setup | Setup suite for Honeycomb performance tests
+| Suite Teardown | Run Keywords
+| ... | Stop VPP Service on DUT | ${node}
+| ... | AND | Archive Honeycomb Log File | ${node} | perf=${True}
+| ... | AND | Stop honeycomb service on DUTs | ${node}
+
+*** Keywords ***
+| Setup suite for Honeycomb performance tests
+| | [Documentation] | Setup variables and configure VPP and Honeycomb
+| | ... | for performance testing.
+| | ...
+| | Set Global Variable | ${node}
+| | ${cores}= | Get Length | ${node['cpuinfo']}
+| | Set Global Variable | ${cores}
+| | Stop VPP Service on DUT | ${node}
+| | Stop Honeycomb service on DUTs | ${node}
+| | Generate VPP Startup Configuration for Honeycomb Test on DUT | ${node}
+| | Configure Restconf binding address | ${node}
+| | Configure Log Level | ${node} | INFO
+| | Configure Persistence | ${node} | disable
+| | Configure jVPP timeout | ${node} | ${14}
+| | Generate Honeycomb startup configuration for performance test
+| | ... | ${node} | ${cores}
+| | Clear Persisted Honeycomb Configuration | ${node}
+
+| Configure ODL Client for performance tests
+| | [Documentation] | Setup ODL client for performance testing.
+| | ...
+| | ${use_odl_client}= | Get Variable Value | ${HC_ODL}
+| | Run Keyword If | '${use_odl_client}' != '${NONE}'
+| | ... | Run Keywords
+| | ... | Set Global Variable | ${use_odl_client}
+| | ... | AND | Copy ODL client | ${node} | ${HC_ODL} | ~ | ${install_dir}
+| | ... | AND | Configure ODL Client Service On DUT | ${node} | ${install_dir}
+| | ... | ELSE | Log | Variable HC_ODL is not present. Not using ODL.
+| | ... | level=INFO
diff --git a/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf16t-crud-read.robot b/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf16t-crud-read.robot
new file mode 100644
index 0000000000..c13851cc62
--- /dev/null
+++ b/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf16t-crud-read.robot
@@ -0,0 +1,87 @@
+# Copyright (c) 2017 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Resource | resources/libraries/robot/honeycomb/performance.robot
+| ...
+| Suite Setup | Run Keywords
+| ... | Configure Honeycomb Netconf threads | ${node} | ${16}
+| ... | AND | Honeycomb Performance Suite Setup Generic | ${node}
+| ...
+| Suite Teardown | Honeycomb Performance Suite Teardown Generic | ${node}
+| ...
+| Force Tags | HC_PERF
+| ...
+| Documentation | *Base Netconf operational read test suite with sixteen Netconf
+| ... | threads.*
+| ...
+| ... | *[Top] Network Topologies:* DUT1 single-node topology.
+| ... | *[Enc] Packet Encapsulations:* Eth-IPv4-TCP.
+| ... | *[Cfg] DUT configuration:* Default
+| ... | *[Ver] verification:* DUT verifies Honeycomb's response times with zero
+| ... | error or loss tolerance. Test packets are generated by DUT1 on localhost
+| ... | loopback interface.
+| ... | Traffic profile contains flow-groups containing REST requests
+| ... | for operational reads.
+| ... | *[Ref] Applicable standard specifications:* RFC6241.
+
+*** Test Cases ***
+| TC01: Base operational read with 1 TG thread
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in a single thread.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${1}
+| | ${entries}= | Set Variable | ${100000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC02: Base operational read with 2 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in two threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${2}
+| | ${entries}= | Set Variable | ${50000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC03: Base operational read with 4 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in four threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${4}
+| | ${entries}= | Set Variable | ${25000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC04: Base operational read with 8 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in eigth threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${8}
+| | ${entries}= | Set Variable | ${12500}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC05: Base operational read with 16 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in sixteen threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${16}
+| | ${entries}= | Set Variable | ${6250}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
diff --git a/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf1t-crud-read.robot b/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf1t-crud-read.robot
new file mode 100644
index 0000000000..d46fdf795b
--- /dev/null
+++ b/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf1t-crud-read.robot
@@ -0,0 +1,87 @@
+# Copyright (c) 2017 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Resource | resources/libraries/robot/honeycomb/performance.robot
+| ...
+| Suite Setup | Run Keywords
+| ... | Configure Honeycomb Netconf threads | ${node} | ${1}
+| ... | AND | Honeycomb Performance Suite Setup Generic | ${node}
+| ...
+| Suite Teardown | Honeycomb Performance Suite Teardown Generic | ${node}
+| ...
+| Force Tags | HC_PERF
+| ...
+| Documentation | *Base Netconf operational read test suite with one Netconf
+| ... | thread.*
+| ...
+| ... | *[Top] Network Topologies:* DUT1 single-node topology.
+| ... | *[Enc] Packet Encapsulations:* Eth-IPv4-TCP.
+| ... | *[Cfg] DUT configuration:* Default
+| ... | *[Ver] verification:* DUT verifies Honeycomb's response times with zero
+| ... | error or loss tolerance. Test packets are generated by DUT1 on localhost
+| ... | loopback interface.
+| ... | Traffic profile contains flow-groups containing REST requests
+| ... | for operational reads.
+| ... | *[Ref] Applicable standard specifications:* RFC6241.
+
+*** Test Cases ***
+| TC01: Base operational read with 1 TG thread
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in a single thread.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${1}
+| | ${entries}= | Set Variable | ${100000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC02: Base operational read with 2 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in two threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${2}
+| | ${entries}= | Set Variable | ${50000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC03: Base operational read with 4 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in four threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${4}
+| | ${entries}= | Set Variable | ${25000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC04: Base operational read with 8 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in eigth threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${8}
+| | ${entries}= | Set Variable | ${12500}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC05: Base operational read with 16 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in sixteen threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${16}
+| | ${entries}= | Set Variable | ${6250}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
diff --git a/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf8t-crud-read.robot b/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf8t-crud-read.robot
new file mode 100644
index 0000000000..9928f15f1d
--- /dev/null
+++ b/tests/vpp/perf/honeycomb/localhostp1-tcp-netconf8t-crud-read.robot
@@ -0,0 +1,97 @@
+# Copyright (c) 2017 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Resource | resources/libraries/robot/honeycomb/performance.robot
+| ...
+| Suite Setup | Run Keywords
+| ... | Configure Honeycomb Netconf threads | ${node} | ${8}
+| ... | AND | Honeycomb Performance Suite Setup Generic | ${node}
+| ...
+| Suite Teardown | Honeycomb Performance Suite Teardown Generic | ${node}
+| ...
+| Force Tags | HC_PERF
+| ...
+| Documentation | *Base Netconf operational read test suite with eight Netconf
+| ... | threads.*
+| ...
+| ... | *[Top] Network Topologies:* DUT1 single-node topology.
+| ... | *[Enc] Packet Encapsulations:* Eth-IPv4-TCP.
+| ... | *[Cfg] DUT configuration:* Default
+| ... | *[Ver] verification:* DUT verifies Honeycomb's response times with zero
+| ... | error or loss tolerance. Test packets are generated by DUT1 on localhost
+| ... | loopback interface.
+| ... | Traffic profile contains flow-groups containing REST requests
+| ... | for operational reads.
+| ... | *[Ref] Applicable standard specifications:* RFC6241.
+
+*** Test Cases ***
+| TC01: Base operational read with 1 TG thread
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in a single thread.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${1}
+| | ${entries}= | Set Variable | ${100000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC02: Base operational read with 2 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in two threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${2}
+| | ${entries}= | Set Variable | ${50000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC03: Base operational read with 4 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in four threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${4}
+| | ${entries}= | Set Variable | ${25000}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC04: Base operational read with 8 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in eigth threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${8}
+| | ${entries}= | Set Variable | ${12500}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC05: Base operational read with 16 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in sixteen threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${16}
+| | ${entries}= | Set Variable | ${6250}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}
+
+| TC05: Base operational read with 24 TG threads
+| | [Documentation]
+| | ... | [Ver] Measure response time using single trial throughput test
+| | ... | with 100K requests generated in sixteen threads.
+| | ${cycles}= | Set Variable | ${1}
+| | ${threads}= | Set Variable | ${24}
+| | ${entries}= | Set Variable | ${6250}
+| | Run base operational read performance trial
+| | ... | ${node} | ${cores} | ${cycles} | ${threads} | ${entries}