1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
|
# Copyright (c) 2021 Cisco and/or its affiliates.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at:
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
*** Settings ***
| Documentation | Performance suite keywords - Handling of various values
| ... | to allow autodetection, computation and overriding by suite variables.
| Library | Collections
| Variables | ${CURDIR}/../../python/Constants.py
*** Variables ***
| ${extended_debug}= | ${EXTENDED_DEBUG}
*** Keywords ***
| Get Disable Latency
| | [Documentation]
| | ... | If Get Use Latency returns true, return false.
| | ... | Otherwise return value of \${disable_latency} variable,
| | ... | or \${False} if not defined.
| |
| | ... | The return value controls whether latency trials in NDRPDR tests
| | ... | are executed. For example, ASTF tests do not support latency
| | ... | measurements yet, so executing the trials just wastes time.
| | ... | Return type: bool.
| |
| | ... | *Example:*
| |
| | ... | \| \${disable_latency} = \| Get Disable Latency \|
| |
| | ${use_latency} = | Get Use Latency
| | Return From Keyword If | ${use_latency} | ${False}
| | ${disable_latency} = | Get Variable Value | \${disable_latency} | ${False}
| | Return From Keyword | ${disable_latency}
| Get Max Rate
| | [Documentation]
| | ... | Return value of \${max_rate} variable,
| | ... | fail if it is not defined.
| | ... | Call this just before calling a Python keyword,
| | ... | as those have restricted access to Robot variables.
| |
| | ... | The return value controls the maximal unidirectional packet rate.
| | ... | The value is also usable for minimal TPS value for ASTF tests.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${max_rate} = \| Get Max Rate \|
| |
| | ${max_rate} = | Get Variable Value | \${max_rate} | ${0.0}
| | Return From Keyword If | ${max_rate} | ${max_rate}
| | Fail | \${max_rate} is not defined. Call Set Max Rate And Jumbo keyword.
| Get Min Rate Hard
| | [Documentation]
| | ... | Return a hardcoded value.
| | ... | The return value controls the minimal unidirectional packet rate,
| | ... | to be used anywhere, including latency measurements at 0% load.
| | ... | The current value is the smallest one permitted
| | ... | by STL profiles with latency streams.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${min_rate_hard} = \| Get Min Rate Hard \|
| |
| | Return From Keyword | ${9001.0}
| Get Min Rate Soft
| | [Documentation]
| | ... | If ramp up rate is not defined, return the hard min value.
| | ... | If ramp up rate is defined (and larger than hard min), return that.
| | ... | The reason is, ramp up rate should already guarantee no loss.
| |
| | ... | The return value controls the minimal unidirectional packet rate,
| | ... | to be used in various search algorithms.
| | ... | Latency measurements may want even lower loads, use hard min for that.
| |
| | ... | The value is also usable for minimal TPS value for ASTF tests.
| | ... | Return type: float.
| |
| | ... | Currently, undefined ramp up rate is reported as zero,
| | ... | so we return the maximum of ramp up rate and the hard min rate.
| |
| | ... | *Example:*
| |
| | ... | \| \${min_rate_soft} = \| Get Min Rate Soft \|
| |
| | ${min_rate_hard} = | Get Min Rate Hard
| | ${ramp_up_rate} = | Get Ramp Up Rate
| | ${min_rate_soft} = | Evaluate | max(${ramp_up_rate}, ${min_rate_hard})
| | Return From Keyword | ${min_rate_soft}
| Get Mrr Trial Duration
| | [Documentation]
| | ... | Return value from Constants. This is an abstraction, useful in case
| | ... | we start allowing various other overrides or computations.
| | ... | Call this just before calling a Python keyword,
| | ... | as those have restricted access to Robot variables.
| |
| | ... | The return value controls the duration of main trial measurement
| | ... | for MRR type tests.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${mrr_trial_duration} = \| Get Mrr Trial Duration \|
| |
| | Return From Keyword | ${PERF_TRIAL_DURATION}
| Get Mrr Trial Multiplicity
| | [Documentation]
| | ... | Return value from Constants. This is an abstraction, useful in case
| | ... | we start allowing various other overrides or computations.
| | ... | Call this just before calling a Python keyword,
| | ... | as those have restricted access to Robot variables.
| |
| | ... | The return value controls the number of main trial measurement
| | ... | for (B)MRR type tests.
| | ... | Return type: integer.
| |
| | ... | *Example:*
| |
| | ... | \| \${mrr_trial_multiplicity} = \| Get Mrr Trial Multiplicity \|
| |
| | Return From Keyword | ${PERF_TRIAL_MULTIPLICITY}
| Get Packet Loss Ratio
| | [Documentation]
| | ... | Return a hardcoded value. This is an abstraction, useful in case
| | ... | we start allowing various other overrides or computations.
| | ... | Call this just before calling a Python keyword,
| | ... | as those have restricted access to Robot variables.
| |
| | ... | The return value controls the default packet loss ration for PDR
| | ... | in NDRPDR tests. Some other usages of MLRsearch (e.g. reconf tests)
| | ... | may use a different value.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${packet_loss_ratio} = \| Get Packet Loss Ratio \|
| |
| | Return From Keyword | ${0.005}
| Get Packets Per Transaction Aggregated
| | [Documentation]
| | ... | Return value of \${packets_per_transaction_aggregated};
| | ... | if not defined, assume traffic is symmetric and compute
| | ... | from unidirectional values.
| |
| | ... | The return value is used when reporting PPS values from TPS found
| | ... | by some search (e.g. NDRPDR).
| | ... | Return type: integer.
| |
| | ... | *Example:*
| |
| | ... | \| \${ppta} = \| Get Packets Per Transaction Aggregated \|
| |
| | ${ppta} = | Get Variable Value | \${packets_per_transaction_aggregated}
| | ... | ${0}
| | Return From Keyword If | "${ppta}" != "0" | ${ppta}
| | # TODO: Insert TCP computation from packet size here.
| | ${pptad} = | Get Packets Per Transaction And Direction
| | ${traffic_directions} = | Get Traffic Directions
| | # We do not support ASTF profiles with multiple transactions, yet.
| | ${ppta} = | Evaluate | ${pptad} * ${traffic_directions}
| | Return From Keyword | ${ppta}
| Get Packets Per Transaction And Direction
| | [Documentation]
| | ... | Return value of \${packets_per_transaction_and_direction},
| | ... | or ${1} if not defined.
| |
| | ... | The return value is used when computing max rate (TPS),
| | ... | so for asymmetric transaction use the more numerous direction.
| | ... | Return type: integer.
| |
| | ... | *Example:*
| |
| | ... | \| \${pptad} = \| Get Packets Per Transaction And Direction \|
| |
| | ${pptad} = | Get Variable Value | \${packets_per_transaction_and_direction}
| | ... | ${1}
| | Return From Keyword | ${pptad}
| Get Ramp Up Duration
| | [Documentation]
| | ... | Return value of \${ramp_up_duration},
| | ... | or ${0.0} if not defined.
| |
| | ... | The return value determines the required duration of ramp-up phase.
| | ... | Typically used to prepare a specific state on DUT.
| | ... | If the value is zero, ramp-up phase is either skipped or size-limited.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${ramp_up_duration} = \| Get Ramp Up Duration \|
| |
| | ${ramp_up_duration} = | Get Variable Value | \${ramp_up_duration} | ${0.0}
| | Return From Keyword | ${ramp_up_duration}
| Get Ramp Up Rate
| | [Documentation]
| | ... | Return value of \${ramp_up_rate},
| | ... | if not defined, return zero.
| |
| | ... | The return value determines the rate for ramp-up phase.
| | ... | Typically used to limit the rate when max rate
| | ... | would lose packets in the ramp up phase, thus not setting
| | ... | the DUT state correctly.
| | ... | If the value is zero, ramp-up phase should be skipped.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${ramp_up_rate} = \| Get Ramp Up Rate \|
| |
| | ${ramp_up_rate} = | Get Variable Value | \${ramp_up_rate} | ${0.0}
| | Return From Keyword | ${ramp_up_rate}
| Get Rate For Teardown
| | [Documentation]
| | ... | Return value of \${rate_for_teardown},
| | ... | if not defined (or zero) return the soft min rate.
| |
| | ... | The return value determines the rate for teardown trial,
| | ... | that is executed if a perf test fails.
| | ... | The \${rate_for_teardown} is usually not defined in suite,
| | ... | but search keywords set it in places where failure can occur,
| | ... | so the trial is done at the rate interesting for the failure.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${rate_for_teardown} = \| Get Rate For Teardown \|
| |
| | ${rate_for_teardown} = | Get Variable Value | \${rate_for_teardown} | ${0.0}
| | Return From Keyword If | ${rate_for_teardown} | ${rate_for_teardown}
| | Run Keyword And Return | Get Min Rate Soft
| Get Resetter
| | [Documentation]
| | ... | Return value of \${resetter} variable,
| | ... | or \${None} if not defined.
| |
| | ... | If not \${None}, the returned value is callable.
| | ... | Its use is to reset DUT to initial conditions,
| | ... | for example to remove NAT sessions created in the previous trial.
| |
| | ... | *Example:*
| |
| | ... | \| \${resetter} = \| Get Resetter \|
| |
| | ${resetter} = | Get Variable Value | \${resetter} | ${None}
| | Return From Keyword | ${resetter}
| Get Runtime Duration
| | [Documentation]
| | ... | Return value of \${runtime_duration} variable,
| | ... | if not defined return ${1.0}.
| |
| | ... | The return value controls the duration of runtime trial,
| | ... | which also acts as a warmup. Usually one second is enough,
| | ... | but some suites need longer time to set up state on DUT.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${runtime_duration} = \| Get Runtime Duration \|
| |
| | ${runtime_duration} = | Get Variable Value | \${runtime_duration} | ${1.0}
| | Return From Keyword | ${runtime_duration}
| Get Runtime Rate
| | [Documentation]
| | ... | Return value of \${runtime_rate} variable,
| | ... | if not defined return the max rate.
| |
| | ... | The return value controls the rate (TPS unidir) of runtime trial,
| | ... | which also acts as a warmup. No plans to ever use a different rate,
| | ... | but keywords look better if access to such values is uniform.
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${runtime_rate} = \| Get Runtime Rate \|
| |
| | ${runtime_rate} = | Get Variable Value | \${runtime_rate} | ${0.0}
| | Return From Keyword If | ${runtime_rate} | ${runtime_rate}
| | Run Keyword And Return | Get Max Rate
| Get Traffic Directions
| | [Documentation]
| | ... | Return value of \${traffic_directions},
| | ... | or ${2} if not defined.
| |
| | ... | The return value used when parsing for measurement results.
| | ... | This needs to be known already in profile driver,
| | ... | as bidirectional parsing may fail on unidirectional traffic.
| | ... | Return type: integer.
| |
| | ... | *Example:*
| |
| | ... | \| \${traffic_directions} = \| Get Traffic Directions \|
| |
| | ${traffic_directions} = | Get Variable Value | \${traffic_directions} | ${2}
| | Return From Keyword | ${traffic_directions}
| Get Transaction Duration
| | [Documentation]
| | ... | Return value of \${transaction_duration} variable,
| | ... | or \${0.0} if not defined.
| |
| | ... | The return value is the expected duration of single (ASTF) transaction
| | ... | if it is not negligible for overall trial duration computation.
| | ... | Most tests use very short transactions (without explicit delays),
| | ... | so the zero default works (and suite saves one line
| | ... | of Variables table).
| | ... | Return type: float.
| |
| | ... | *Example:*
| |
| | ... | \| \${transaction_duration} = \| Get Transaction Duration \|
| |
| | ${transaction_duration} = | Get Variable Value | \${transaction_duration}
| | ... | ${0.0}
| | Return From Keyword | ${transaction_duration}
| Get Transaction Scale
| | [Documentation]
| | ... | Return value of \${transaction_scale} variable,
| | ... | or \${0} if not defined.
| |
| | ... | Zero return value means the number of transactions is not limited,
| | ... | which is true for most STL TRex profiles (transaction is a packet).
| | ... | Nonzero return value means the number of transactions is fixed,
| | ... | for example in stateful NAT scale tests.
| | ... | Return type: integer.
| |
| | ... | *Example:*
| |
| | ... | \| \${transaction_scale} = \| Get Transaction Scale \|
| |
| | ${transaction_scale} = | Get Variable Value | \${transaction_scale} | ${0}
| | Return From Keyword | ${transaction_scale}
| Get Transaction Type
| | [Documentation]
| | ... | Return value of \${transaction_type} variable,
| | ... | or "packet" if not defined.
| |
| | ... | The return value describes the type of transaction
| | ... | the test is executed. For example "packet" means a transaction
| | ... | is just a single packet. For more sophisticated transactions,
| | ... | the logic to determine the number of passed transactions
| | ... | is different from merely counting the packets received from DUT.
| | ... | Return type: string.
| |
| | ... | *Example:*
| |
| | ... | \| \${transaction_type} = \| Get Transaction Type \|
| |
| | ${transaction_type} = | Get Variable Value | \${transaction_type} | packet
| | Return From Keyword | ${transaction_type}
| Get Use Latency
| | [Documentation]
| | ... | Return value of \${use_latency} variable,
| | ... | if not defined return the value from Constants.
| |
| | ... | The return value controls whether latency streams are active
| | ... | during the main search.
| | ... | Return type: bool.
| |
| | ... | *Example:*
| |
| | ... | \| \${use_latency} = \| Get Use Latency \|
| |
| | ${use_latency} = | Get Variable Value | ${use_latency} | ${PERF_USE_LATENCY}
| | Return From Keyword | ${use_latency}
| Set Jumbo
| | [Documentation]
| | ... | For jumbo frames detection, the maximal packet size is relevant,
| | ... | encapsulation overhead (if any) has effect.
| |
| | ... | This keyword computes jumbo boolean (some suites need that for
| | ... | configuration decisions).
| | ... | To streamline suite autogeneration, both input and output values
| | ... | are communicated as test (or broader scope) variables,
| | ... | instead of explicit arguments and return values.
| |
| | ... | *Test (or broader scope) variables read:*
| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
| | ... | string
| |
| | ... | *Test variables set:*
| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
| | ... | enabled. Type: boolean
| |
| | ... | *Example:*
| |
| | ... | \| Set Jumbo \|
| |
| | # Already called by Set Max Rate And Jumbo, but some suites (e.g. device)
| | # are calling this directly.
| | Set Numeric Frame Sizes
| | ${jumbo} = | Set Variable If | ${max_frame_size} < 1522
| | ... | ${False} | ${True}
| | Set Test Variable | \${jumbo}
| Set Max Rate And Jumbo
| | [Documentation]
| | ... | Input framesize can be either integer in case of a single packet
| | ... | in stream, or IMIX string defining mix of packets.
| | ... | For jumbo frames detection, the maximal packet size is relevant.
| | ... | For maximal transmit rate, the average packet size is relevant.
| | ... | In both cases, encapsulation overhead (if any) has effect.
| | ... | The maximal rate is computed from NIC name.
| | ... | The implementation works by mapping from exact
| | ... | whitelisted NIC names.
| | ... | The mapping is hardcoded in nic_limits.yaml
| | ... | TODO: Make the mapping from NIC names case insensistive.
| |
| | ... | This keyword computes maximal unidirectional transmit rate
| | ... | and jumbo boolean (some suites need that for configuration decisions).
| | ... | To streamline suite autogeneration, both input and output values
| | ... | are communicated as test (or broader scope) variables,
| | ... | instead of explicit arguments and return values.
| |
| | ... | If this keyword detects the test is interested in (unidirectional)
| | ... | transactons per second maximal rate (tps), that is returned (not pps).
| |
| | ... | *Test (or broader scope) variables read:*
| | ... | - nic_name - Name of bottleneck NIC. Type: string
| | ... | - overhead - Overhead in bytes; default value: 0. Type: integer
| | ... | - frame_size - L2 Frame Size [B] or IMIX string. Type: integer or
| | ... | string
| | ... | - packets_per_transaction_and_direction - Pps-tps conversion.
| | ... | Optional, default 1.
| |
| | ... | *Test variables set:*
| | ... | - max_rate - Calculated unidirectional maximal transmit rate [pps].
| | ... | This never exceeds bandwidth on TG-DUT nor DUT-DUT links.
| | ... | Type: float
| | ... | - jumbo - Jumbo boolean, true if jumbo packet support has to be
| | ... | enabled. Type: boolean
| | ... | avg_frame_size - Average frame size including overhead. Type: float
| | ... | max_frame_size - Maximal frame size including overhead. Type: float
| |
| | ... | *Example:*
| |
| | ... | \| Set Max Rate And Jumbo \|
| |
| | # TODO: Re-check overhead values in suites with both traffics encapsulated.
| | # TODO: Improve layered setup to detect encap/decap and update overhead.
| | ${pps_limit} = | Get From Dictionary
| | ... | ${NIC_NAME_TO_PPS_LIMIT} | ${nic_name}
| | ${bps_limit} = | Get From Dictionary
| | ... | ${NIC_NAME_TO_BPS_LIMIT} | ${nic_name}
| | Set Numeric Frame Sizes
| | # We need to add 20B (Ethernet preamble and inter-frame gap)
| | # to avg_frame_size
| | ${rate} = | Evaluate | ${bps_limit} / ((${avg_frame_size} + 20.0) * 8)
| | ${max_rate} = | Set Variable If | ${rate} > ${pps_limit}
| | ... | ${pps_limit} | ${rate}
| | ${pptad} = | Get Packets Per Transaction And Direction
| | ${max_rate} = | Evaluate | ${max_rate} / ${pptad}
| | Set Test Variable | \${max_rate}
| | Set Jumbo
| Set Numeric Frame Sizes
| | [Documentation]
| | ... | Framesize can be either integer in case of a single packet
| | ... | in stream, or set of packets in case of IMIX type or simmilar.
| | ... | For jumbo decisions, we need a numeric size of the biggest packet.
| | ... | For max rate decisions, we need a numeric average packet size.
| | ... | This keyword computes both and sets them as test variables.
| |
| | ... | Each suite sets a value named \${overhead},
| | ... | which describes by how many bytes the frames on DUT-DUT link
| | ... | are larger (due to encapsulation) than those
| | ... | on the primary TG-DUT link. But for some suites that value
| | ... | can be negaive (if TG-DUT is encapsulated more heavily).
| | ... | For calculations in this keyword, we need largest sizes
| | ... | across links, so zero is used if \${overhead} is negative.
| |
| | ... | *Test variables read:*
| | ... | - frame_size - Framesize. Type: integer or string
| | ... | - overhead - Overhead in bytes; default value: ${0}. Type: integer
| |
| | ... | *Test variables set*
| | ... | avg_frame_size - Average frame size including overhead. Type: float
| | ... | max_frame_size - Maximal frame size including overhead. Type: float
| |
| | ... | *Example:*
| |
| | ... | \| Set Numeric Frame Sizes \|
| |
| | ${max_overhead} = | Set Variable If | ${overhead} >= 0 | ${overhead} | ${0}
| | ${bare_avg_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
| | ... | Set Variable | ${353.83333}
| | ... | ELSE
| | ... | Convert To Number | ${frame_size}
| | # Do not use $max_overhead (without braces), that does not tolerate string.
| | ${avg_frame_size} = | Evaluate | ${bare_avg_frame_size} + ${max_overhead}
| | Set Test Variable | \${avg_frame_size}
| | ${bare_max_frame_size} = | Run Keyword If | '${frame_size}' == 'IMIX_v4_1'
| | ... | Set Variable | ${1518}
| | ... | ELSE
| | ... | Convert To Number | ${frame_size}
| | ${max_frame_size} = | Evaluate | $bare_max_frame_size + $max_overhead
| | Set Test Variable | ${max_frame_size}
| Set Rates For Policer
| | [Documentation]
| | ... | Policer tests need these values,
| | ... | currently computed from \${avg_frame_size}.
| | ... | TODO: Verify the units match and computation is correct.
| |
| | ... | *Test (or broader scope) variables read:*
| | ... | - avg_frame_size - Average L2 Frame Size [B]. Type: float
| | ... | Set by Set Max Rate And Jumbo keyword.
| |
| | ... | *Test variables set:*
| | ... | - eb - Excess burst rate for policer. Type: float
| | ... | - cb - Committed burst rate for policer. Type: float
| |
| | ... | *Example:*
| |
| | ... | \| Set Rates For Policer \|
| |
| | Set Test Variable | \${eb} | ${avg_frame_size}
| | Set Test Variable | \${cb} | ${avg_frame_size}
|