aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot/honeycomb/honeycomb.robot
blob: 165dbc036bb33063d0315b55ebc81f8138ddc503 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
# Copyright (c) 2017 Cisco and/or its affiliates.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at:
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

*** Settings ***
| Library | resources.libraries.python.honeycomb.HoneycombSetup
| Library | resources.libraries.python.honeycomb.HoneycombUtil
| Library | resources.libraries.python.honeycomb.HcPersistence
| Library | resources.libraries.python.DUTSetup
| Library | resources.libraries.python.InterfaceUtil
| Resource | resources/libraries/robot/shared/default.robot

*** Keywords ***
| Configure Honeycomb service on DUTs
| | [Documentation] | *Setup environment for honeycomb testing.*
| | ...
| | ... | _Setup steps:_
| | ... | - 1. Login to each honeycomb node using ssh
| | ... | - 2. Startup honeycomb service
| | ... | - 3. Monitor service startup using HTTP GET request loop
| | ... | Expected sequence of HTTP replies:
| | ... | connection refused -> 404 -> 401 -> 503 or 500 -> 200 (pass)
| | ... | - 4. Configure honeycomb nodes using HTTP PUT request
| | ...
| | ... | _Arguments:_
| | ... | - duts - list of nodes to setup Honeycomb on
| | ...
| | ... | _Used global constants and variables:_
| | ... | - RESOURCES_TPL_HC - path to honeycomb templates directory
| | ... | - HTTPCodes - HTTP protocol status codes
| | ...
| | [Arguments] | @{duts}
| | Start honeycomb on DUTs | @{duts}
| | :FOR | ${dut} | IN | @{duts}
| | | Check honeycomb startup state | @{duts}
| | | Sleep | 5s | Make sure all modules are loaded and ready.

| Stop Honeycomb service on DUTs
| | [Documentation] | *Cleanup environment after honeycomb testing.*
| | ...
| | ... | _Teardown steps:_
| | ... | - 1. Login to each honeycomb node using ssh
| | ... | - 2. Stop honeycomb service
| | ... | - 3. Monitor service shutdown using HTTP GET request loop
| | ... | Expected sequence of HTTP replies:
| | ... | 200 -> 404 -> connection refused (pass)
| | ...
| | ... | _Arguments:_
| | ... | - duts - list of nodes to stop Honeycomb on
| | ...
| | ... | _Used global constants and variables:_
| | ... | - RESOURCES_TPL_HC - path to honeycomb templates directory
| | ... | - HTTPCodes - HTTP protocol status codes
| | ...
| | [Arguments] | @{duts}
| | Stop honeycomb on DUTs | @{duts}
| | :FOR | ${dut} | IN | @{duts}
| | | Wait until keyword succeeds | 60sec | 15sec
| | | ... | Check honeycomb shutdown state | @{duts}

| Clear persisted Honeycomb configuration
| | [Documentation] | *Delete saved configuration.*
| | ...
| | ... | *Arguments:*
| | ... | - duts - one or more nodes to clear persistence on. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Clear persisted Honeycomb configuration \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | @{duts}
| | Clear persisted Honeycomb config | @{duts}

| Restart Honeycomb and VPP and clear persisted configuration
| | [Documentation] | Restarts Honeycomb and VPP with default configuration.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Restart Honeycomb and VPP and clear persisted configuration \
| | ... | \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Stop Honeycomb service on DUTs | ${node}
| | Clear persisted Honeycomb configuration | ${node}
| | Setup DUT | ${node}
| | Sleep | 10s | Wait 10sec so VPP is up for sure.
| | Configure Honeycomb service on DUTs | ${node}

| Restart Honeycomb and VPP
| | [Documentation] | Stops the Honeycomb service and verifies it is stopped.
| | ... | Then restarts VPP, starts Honeycomb again and verifies it is running.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Restart Honeycomb and VPP \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Stop Honeycomb service on DUTs | ${node}
| | Setup DUT | ${node}
| | Sleep | 10s | Wait 10sec so VPP is up for sure.
| | Configure Honeycomb service on DUTs | ${node}

| Restart Honeycomb and VPP in performance test
| | [Documentation] | Stops Honeycomb and VPP and verifies HC is stopped.
| | ... | Then restarts VPP, starts Honeycomb again and verifies it is running.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Restart Honeycomb and VPP \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Stop Honeycomb service on DUTs | ${node}
| | Stop VPP service on DUT | ${node}
| | Setup DUT | ${node}
| | Sleep | 10s | Wait 10sec so VPP is up for sure.
| | Configure Honeycomb service on DUTs | ${node}
| | Check honeycomb startup state | ${node} | timeout=120

| Archive Honeycomb log file
| | [Documentation] | Copy honeycomb.log file from Honeycomb node\
| | ... | to test executor.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ... | - perf - Running on performance testbed? Yes/no Type: boolean
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Archive Honeycomb log file \| ${nudes['DUT1']} \|
| | ...
| | [Arguments] | ${node} | ${perf}=${False}
| | Archive Honeycomb log | ${node} | ${perf}

| Configure ODL Client Service On DUT
| | [Documentation] | Configure and start ODL client, then repeatedly check if
| | ... | it is running.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ... | - odl_name - Name of ODL client version. Type: string
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Configure ODL Client Service on DUT \| ${nodes['DUT1']} \
| | ... | \| carbon-SR1 \|
| | ...
| | [Arguments] | ${node} | ${odl_name}
| | Copy ODL Client | ${node} | ${odl_name} | /mnt/common | /tmp
| | Setup ODL Client | ${node} | /tmp
| | Wait until keyword succeeds | 2min | 30sec
| | ... | Install ODL Features | ${node} | /tmp
| | Wait until keyword succeeds | 4min | 15sec
| | ... | Mount Honeycomb on ODL | ${node}
| | Wait until keyword succeeds | 2min | 15sec
| | ... | Check ODL startup state | ${node}
| | Check honeycomb startup state | ${node} | timeout=120

| Configure Honeycomb for functional testing
| | [Documentation] | Configure Honeycomb with parameters for functional
| | ... | testing, then start Honeycomb and repeatedly check startup status.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Configure Honeycomb for functional testing \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Configure Restconf binding address | ${node}
| | Configure Log Level | ${node} | TRACE
| | Configure Persistence | ${node} | disable
| | Configure jVPP timeout | ${node} | ${10}
| | Generate Honeycomb startup configuration for ODL test | ${node}
| | Clear Persisted Honeycomb Configuration | ${node}
| | Configure Honeycomb service on DUTs | ${node}

| Configure ODL Client for functional testing
| | [Documentation] | Read external variable HC_ODL. Depending on its
| | ... | value either: do nothing, or setup ODL client for testing and
| | ... | create a global variable that modifies Restconf requests to use ODL.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Configure ODL Client for functional testing \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | ${use_odl_client}= | Get Variable Value | ${HC_ODL}
| | Run Keyword If | '${use_odl_client}' != '${NONE}'
| | ... | Run Keywords
| | ... | Set Global Variable | ${use_odl_client} | AND
| | ... | Configure ODL Client Service On DUT | ${node} | ${use_odl_client}
| | ... | ELSE | Log | Variable HC_ODL is not present. Not using ODL.
| | ... | level=INFO

| Set Up Honeycomb Functional Test Suite
| | [Documentation] | Generic test suite setup for Honeycomb functional tests.
| | ... | Restarts VPP, then enables Honeycomb and optionally ODL, based
| | ... | on external variable.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Set Up Honeycomb Functional Test Suite \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Setup DUT | ${node}
| | Configure all TGs for traffic script
| | Configure Honeycomb for functional testing | ${node}
| | Configure ODL Client for functional testing | ${node}

| Tear Down Honeycomb Functional Test Suite
| | [Documentation] | Generic test suite teardown for Honeycomb functional
| | ... | tests. Stops ODL client (if used), then stops Honeycomb and verifies
| | ... | that they are both stopped.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Tear Down Honeycomb Functional Test Suite \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Append suite to Honeycomb log file | ${node}
| | ${use_odl_client}= | Get Variable Value | ${HC_ODL}
| | Run Keyword If | '${use_odl_client}' != '${NONE}'
| | ... | Run Keywords
| | ... | Stop ODL Client | ${node} | /tmp | AND
| | ... | Wait until keyword succeeds | 3min | 15sec
| | ... | Check ODL shutdown state | ${node} | AND
| | ... | Set Global Variable | ${use_odl_client} | ${NONE}
| | Stop Honeycomb service on DUTs | ${node}
| | Clear Honeycomb Log | ${node}
| | Stop VPP Service on DUT | ${node}

| Enable Honeycomb Feature
| | [Documentation] | Enable the specified feature in Honeycomb configuration.
| | ... | Requires a restart of Honeycomb to take effect.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Enable Honeycomb Feature \| ${nodes['DUT1']} \| NSH \|
| | ...
| | [arguments] | ${node} | ${feature}
| | Manage Honeycomb Features | ${node} | ${feature}

| Disable Honeycomb Feature
| | [Documentation] | Disable the specified feature in Honeycomb configuration.
| | ... | Requires a restart of Honeycomb to take effect.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Disable Honeycomb Feature \| ${nodes['DUT1']} \| NSH \|
| | ...
| | [arguments] | ${node} | ${feature}
| | Manage Honeycomb Features | ${node} | ${feature} | disable=${True}

| Honeycomb Performance Suite Setup Generic
| | [Documentation] | Generic test suite setup for Honeycomb performance tests.
| | ... | Performs multiple attempts to start Honeycomb+VPP stack.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Honeycomb Performance Suite Setup Generic \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Wait until keyword succeeds | 8min | 2min
| | ... | Restart Honeycomb and VPP in Performance test | ${node}

| Honeycomb Performance Suite Teardown Generic
| | [Documentation] | Generic test suite teardown for Honeycomb performance
| | ... | tests. Logs CPU usage before stopping Honeycomb.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Honeycomb Performance Suite Teardown Generic \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Log Honeycomb and VPP process distribution on cores | ${node}
| | Append suite to Honeycomb log file | ${node}
| | Stop Honeycomb service on DUTs | ${node}
| | Clear Honeycomb Log | ${node}
| | Stop VPP Service on DUT | ${node}

| Append suite to Honeycomb log file
| | [Documentation] | Add the contents of honeycomb.log for the current suite\
| | ... | to the full log which will be archived.
| | ...
| | ... | *Arguments:*
| | ... | - node - information about a DUT node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Append suite to Honeycomb log file \| ${nodes['DUT1']} \|
| | ...
| | [Arguments] | ${node}
| | Append Honeycomb log | ${node} | ${SUITE_NAME}

| Generate Honeycomb startup configuration for ODL test
| | [Documentation] | Create HC startup configuration and apply to config
| | ... | file on DUT. Requires Honeycomb restart to take effect.
| | ...
| | ... | *Arguments:*
| | ... | - node - Honeycomb node. Type: dictionary
| | ...
| | ... | *Example:*
| | ...
| | ... | \| Generate Honeycomb startup configuration for ODL test \
| | ... | \| ${nodes[DUT1]} \|
| | ...
| | [Arguments] | ${node}
| | Import Library | resources.libraries.python.honeycomb.HoneycombSetup.HoneycombStartupConfig
| | ... | WITH NAME | HC_config
| | Run Keyword | HC_config.Set SSH Security provider
| | Run Keyword | HC_config.Set Memory Size | ${32}
| | Run Keyword | HC_config.Set Metaspace Size | ${32}
| | Run Keyword | HC_config.Apply config | ${node}