CSIT-619 HC Test: Honeycomb performance testing - initial commit

 - keywords and scripts for HC performance testing setup
 - basic performance suite: operational data read
 - traffic script and keywords used in tests

Change-Id: Ic0290be73a7c925ea2561f8cd2524c5cb83fcda2
Signed-off-by: selias <samelias@cisco.com>

2 files changed, 202 insertions, 4 deletions

diff --git a/resources/libraries/robot/honeycomb/honeycomb.robot b/resources/libraries/robot/honeycomb/honeycomb.robot
index 9017584f5a..d0f9f0fbf6 100644
--- a/resources/libraries/robot/honeycomb/honeycomb.robot
+++ b/resources/libraries/robot/honeycomb/honeycomb.robot
@@ -73,6 +73,7 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Clear persisted Honeycomb configuration \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | @{duts}
 | | Clear persisted Honeycomb config | @{duts}
 
@@ -86,14 +87,34 @@
 | | ...
 | | ... | \| Restart Honeycomb and VPP and clear persisted configuration \
 | | ... | \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | ${node}
 | | Stop Honeycomb service on DUTs | ${node}
 | | Clear persisted Honeycomb configuration | ${node}
 | | Setup DUT | ${node}
+| | Sleep | 10s | Wait 10sec so VPP is up for sure.
 | | Configure Honeycomb service on DUTs | ${node}
 
 | Restart Honeycomb and VPP
-| | [Documentation] | Restarts Honeycomb service and wait until it starts up.
+| | [Documentation] | Stops the Honeycomb service and verifies it is stopped.
+| | ... | Then restarts VPP, starts Honeycomb again and verifies it is running.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - information about a DUT node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Restart Honeycomb and VPP \| ${nodes['DUT1']} \|
+| | ...
+| | [Arguments] | ${node}
+| | Stop Honeycomb service on DUTs | ${node}
+| | Setup DUT | ${node}
+| | Sleep | 10s | Wait 10sec so VPP is up for sure.
+| | Configure Honeycomb service on DUTs | ${node}
+
+| Restart Honeycomb and VPP in performance test
+| | [Documentation] | Stops Honeycomb and VPP and verifies HC is stopped.
+| | ... | Then restarts VPP, starts Honeycomb again and verifies it is running.
 | | ...
 | | ... | *Arguments:*
 | | ... | - node - information about a DUT node. Type: dictionary
@@ -101,10 +122,15 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Restart Honeycomb and VPP \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | ${node}
 | | Stop Honeycomb service on DUTs | ${node}
+| | Stop VPP service on DUT | ${node}
 | | Setup DUT | ${node}
+| | Sleep | 10s | Wait 10sec so VPP is up for sure.
 | | Configure Honeycomb service on DUTs | ${node}
+| | Wait until keyword succeeds | 2min | 16sec
+| | ... | Check honeycomb startup state | ${node}
 
 | Archive Honeycomb log file
 | | [Documentation] | Copy honeycomb.log file from Honeycomb node\
@@ -112,12 +138,14 @@
 | | ...
 | | ... | *Arguments:*
 | | ... | - node - information about a DUT node. Type: dictionary
+| | ... | - perf - Running on performance testbed? Yes/no Type: boolean
 | | ...
 | | ... | *Example:*
 | | ...
 | | ... | \| Archive Honeycomb log file \| ${nudes['DUT1']} \|
-| | [Arguments] | ${node}
-| | Archive Honeycomb log | ${node}
+| | ...
+| | [Arguments] | ${node} | ${perf}=${False}
+| | Archive Honeycomb log | ${node} | ${perf}
 
 | Configure ODL Client Service On DUT
 | | [Documentation] | Configure and start ODL client, then repeatedly check if
@@ -131,6 +159,7 @@
 | | ...
 | | ... | \| Configure ODL Client Service on DUT \| ${nodes['DUT1']} \
 | | ... | \| carbon-SR1 \|
+| | ...
 | | [Arguments] | ${node} | ${odl_name}
 | | Copy ODL Client | ${node} | ${odl_name} | /mnt/common | /tmp
 | | Setup ODL Client | ${node} | /tmp
@@ -153,6 +182,7 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Configure Honeycomb for functional testing \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | ${node}
 | | Configure Restconf binding address | ${node}
 | | Configure Log Level | ${node} | TRACE
@@ -172,6 +202,7 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Configure ODL Client for functional testing \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | ${node}
 | | ${use_odl_client}= | Get Variable Value | ${HC_ODL}
 | | Run Keyword If | '${use_odl_client}' != '${NONE}'
@@ -192,6 +223,7 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Set Up Honeycomb Functional Test Suite \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | ${node}
 | | Setup DUT | ${node}
 | | Configure all TGs for traffic script
@@ -209,6 +241,7 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Tear Down Honeycomb Functional Test Suite \| ${nodes['DUT1']} \|
+| | ...
 | | [Arguments] | ${node}
 | | ${use_odl_client}= | Get Variable Value | ${HC_ODL}
 | | Run Keyword If | '${use_odl_client}' != '${NONE}'
@@ -227,6 +260,7 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Enable Honeycomb Feature \| ${nodes['DUT1']} \| NSH \|
+| | ...
 | | [arguments] | ${node} | ${feature}
 | | Manage Honeycomb Features | ${node} | ${feature}
 
@@ -240,5 +274,49 @@
 | | ... | *Example:*
 | | ...
 | | ... | \| Disable Honeycomb Feature \| ${nodes['DUT1']} \| NSH \|
+| | ...
 | | [arguments] | ${node} | ${feature}
-| | Manage Honeycomb Features | ${node} | ${feature} | disable=${True}
\ No newline at end of file
+| | Manage Honeycomb Features | ${node} | ${feature} | disable=${True}
+
+| Stop VPP Service on DUT
+| | [Documentation] | Stop the VPP service on the specified node.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - information about a DUT node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Stop VPP Service on DUT \| ${nodes['DUT1']} \|
+| | ...
+| | [Arguments] | ${node}
+| | Stop VPP Service | ${node}
+
+| Honeycomb Performance Suite Setup Generic
+| | [Documentation] | Generic test suite setup for Honeycomb performance tests.
+| | ... | Performs multiple attempts to start Honeycomb+VPP stack.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - information about a DUT node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Honeycomb Performance Suite Setup Generic \| ${nodes['DUT1']} \|
+| | ...
+| | [Arguments] | ${node}
+| | Wait until keyword succeeds | 8min | 2min
+| | ... | Restart Honeycomb and VPP in Performance test | ${node}
+
+| Honeycomb Performance Suite Teardown Generic
+| | [Documentation] | Generic test suite teardown for Honeycomb performance
+| | ... | tests. Logs CPU usage before stopping Honeycomb.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - information about a DUT node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Honeycomb Performance Suite Teardown Generic \| ${nodes['DUT1']} \|
+| | ...
+| | [Arguments] | ${node}
+| | Log Honeycomb and VPP process distribution on cores | ${node}
+| | Stop Honeycomb service on DUTs | ${node}
diff --git a/resources/libraries/robot/honeycomb/performance.robot b/resources/libraries/robot/honeycomb/performance.robot
new file mode 100644
index 0000000000..bd906e1742
--- /dev/null
+++ b/resources/libraries/robot/honeycomb/performance.robot
@@ -0,0 +1,120 @@
+# Copyright (c) 2017 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+| Library | resources.libraries.python.honeycomb.Performance
+| Library | resources.libraries.python.InterfaceUtil
+| Resource | resources/libraries/robot/honeycomb/honeycomb.robot
+| Documentation | Keywords used in Honeycomb performance testing.
+
+*** Keywords ***
+| Configure Honeycomb Netconf threads
+| | [Documentation] | Modify thread configuration of Honeycomb's Netconf server,
+| | ... | Requires a restart of Honeycomb to take effect.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - Node to change configuration on. Type: dictionary
+| | ... | - threads - Number of threads to configure. Type: integer
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Configure Honeycomb Netconf Threads \| ${nodes[DUT1]} \| ${2} \|
+| | ...
+| | [Arguments] | ${node} | ${threads}
+| | Configure Netconf Threads | ${node} | ${threads}
+
+| Run base operational read performance trial
+| | [Documentation] | Send Netconf requests over plain TCP to obtain VPP version
+| | ... | from Honeycomb operational data.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - Node to run test on. Type: dictionary
+| | ... | - cores - Number of available processor cores. Type: integer
+| | ... | - cycles - Number of test cycles to run. Final results will\
+| | ... | be averaged across all runs. Type: integer
+| | ... | - threads - Number of threads to use for generating traffic.\
+| | ... | Type: integer
+| | ... | - requests - Number of requests to send in each thread and cycle.\
+| | ... | Type: integer
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Run base operational read performance trial \| ${nodes[DUT1]} \
+| | ... | \| ${36} \| ${1} \| ${4} \| ${10000} \|
+| | ...
+| | [Arguments] | ${node} | ${cores} | ${cycles} | ${threads} | ${requests}
+| | ${result}= | Run traffic script on DUT | ${node} | read_vpp_version.py
+| | ... | ${cores} | cycles=${cycles} | threads=${threads}
+| | ... | requests=${requests}
+| | Set Test Message | ${result}
+
+| Generate VPP Startup Configuration for Honeycomb Test on DUT
+| | [Arguments] | ${node}
+| | [Documentation] | Create VPP base startup configuration on DUT, then restart
+| | ... | VPP to apply the configuration.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - VPP node to configure. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Generate VPP Startup Configuration for Honeycomb Test on DUT \
+| | ... | \| ${nodes[DUT1]} \|
+| | ...
+| | Import Library | resources.libraries.python.VppConfigGenerator
+| | ... | WITH NAME | VPP_config
+| | Run keyword | VPP_config.Set Node | ${node}
+| | Run keyword | VPP_config.Add Unix Log
+| | Run keyword | VPP_config.Add Unix CLI Listen
+| | Run keyword | VPP_config.Add Unix Nodaemon
+| | Run keyword | VPP_config.Add CPU Main Core | ${1}
+| | Run keyword | VPP_config.Apply Config
+
+| Log Honeycomb and VPP process distribution on cores
+| | [Documentation] | Log the distribution of VPP and Honeycomb child processes
+| | ... | over the CPU cores.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - Honeycomb node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Log Honeycomb and VPP process distribution on cores \
+| | ... | \| ${nodes[DUT1]} \|
+| | ...
+| | [Arguments] | ${node}
+| | Log Core Schedule | ${node} | vpp
+| | Log Core Schedule | ${node} | java
+
+| Generate Honeycomb startup configuration for performance test
+| | [Documentation] | Create HC startup configuration and apply to config
+| | ... | file on DUT. Requires Honeycomb restart to take effect.
+| | ...
+| | ... | *Arguments:*
+| | ... | - node - Honeycomb node. Type: dictionary
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Generate Honeycomb startup configuration for performance test \
+| | ... | \| ${nodes[DUT1]} \|
+| | ...
+| | [Arguments] | ${node} | ${cores}
+| | Import Library | resources.libraries.python.honeycomb.HoneycombSetup.HoneycombStartupConfig
+| | ... | WITH NAME | HC_config
+| | Run Keyword | HC_config.Set CPU Scheduler | FIFO
+| | Run Keyword | HC_config.Set CPU Core Affinity | ${2} | ${cores}
+| | Run Keyword | HC_config.Set JIT Compiler Mode | server
+| | Run Keyword | HC_config.Set Memory Size | ${512} | ${2048}
+| | Run Keyword | HC_config.Set Metaspace Size | ${128} | ${512}
+| | Run Keyword | HC_config.Set NUMA Optimization
+| | Run Keyword | HC_config.apply config | ${node}