From 01d8f262afc567c3d49a23c3cb2cdeaced8a6887 Mon Sep 17 00:00:00 2001 From: Vratko Polak Date: Wed, 15 Dec 2021 17:14:36 +0100 Subject: UTI: Export results + Model version 1.0.0. - Only some result types are exported. + MRR, NDRPDR and SOAK. - Other result types to be added later. + In contrast, all test types are detected. + Convert custom classes to JSON-serializable equivalents. + Sort dict keys before converting to JSON. + Override the order for some known keys. + Export sets as sorted arrays. + Convert to info content from serialized raw content. + Also export outputs for suite setups and teardowns. + Info files for setup/teardown exist only temporarily. + The data is merged into suite.info.json file. + This simplifies presentation of total suite duration. + Define model via JSON schema: - Just test case, suite setup/teardown/suite to be added later. - Just info, raw to be added later. + Proper descriptions. + Json is generated from yaml. + This is a convenience for maintainers. + The officially used schema is the .json one. + TODOs written into a separate .txt file. + Validate exported instance against the schema. + Include format checking. + Update CSIT requirements for validation dependencies. + This needs python-dateutil==2.8.2, only a patch bump. + Compute bandwidth also for soak tests. + This unifies with NDRPDR to simplify schema definition. - PAL may need an update for parsing soak test message. + Include SSH log items, raw output only. + Generate all outputs in a single filesystem tree. + Move raw outputs into test_output_raw.tar.xz. + Rename existing tar with suites to generated_robot_files.tar.xz. Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c Signed-off-by: Vratko Polak --- .../robot/performance/performance_display.robot | 32 ++++++++++++++++++++-- .../robot/performance/performance_utils.robot | 19 ++++++++++--- 2 files changed, 44 insertions(+), 7 deletions(-) (limited to 'resources/libraries/robot/performance') diff --git a/resources/libraries/robot/performance/performance_display.robot b/resources/libraries/robot/performance/performance_display.robot index de515412a5..db2b522091 100644 --- a/resources/libraries/robot/performance/performance_display.robot +++ b/resources/libraries/robot/performance/performance_display.robot @@ -46,6 +46,31 @@ | | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other} | | Fail | ${message} +| Compute bandwidth +| | [Documentation] +| | ... | Compute (bidir) bandwidth from given (unidir) transaction rate. +| | ... +| | ... | This keyword reads "ppta" and "avg_frame_size" set elsewhere. +| | ... | The implementation should work for both pps and cps rates. +| | ... | +| | ... | *Arguments:* +| | ... | - tps - Transaction rate (unidirectional) [tps]. Type: float +| | ... +| | ... | *Returns:* +| | ... | - Computed bandwidth in Gbps. +| | ... | - Computed aggregate packet rate in pps. +| | +| | ... | *Example:* +| | +| | ... | |\ \${bandwidth} \| \${pps} = \| Compute Bandwidth \| \${12345.67} \| +| | +| | [Arguments] | ${tps} +| | +| | ${ppta} = | Get Packets Per Transaction Aggregated +| | ${pps} = | Evaluate | ${tps} * ${ppta} +| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9 +| | Return From Keyword | ${bandwidth} | ${pps} + | Display Reconfig Test Message | | [Documentation] | | ... | Display the number of packets lost (bidirectionally) @@ -198,6 +223,8 @@ | | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY} | | | | Set Test Message | ${\n}${text}: ${tps} CPS | append=yes +| | ${bandwidth} | ${pps} = | Compute Bandwidth | ${tps} +| | Export Search Bound | ${text} | ${tps} | cps | ${bandwidth} | | Return From Keyword If | not """${latency}""" | | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency} | | ... | append=yes @@ -230,11 +257,10 @@ | | | | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY} | | -| | ${ppta} = | Get Packets Per Transaction Aggregated -| | ${pps} = | Evaluate | ${tps} * ${ppta} -| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9 +| | ${bandwidth} | ${pps} = | Compute Bandwidth | ${tps} | | Set Test Message | ${\n}${text}: ${pps} pps, | append=yes | | Set Test Message | ${bandwidth} Gbps (initial) | append=yes +| | Export Search Bound | ${text} | ${pps} | pps | ${bandwidth * 1e9} | | Return From Keyword If | not """${latency}""" | | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency} | | ... | append=yes diff --git a/resources/libraries/robot/performance/performance_utils.robot b/resources/libraries/robot/performance/performance_utils.robot index 8de74c5707..f2cb873d1a 100644 --- a/resources/libraries/robot/performance/performance_utils.robot +++ b/resources/libraries/robot/performance/performance_utils.robot @@ -13,6 +13,7 @@ *** Settings *** | Library | Collections +| Library | resources.libraries.python.model.ExportResult | Library | resources.libraries.python.topology.Topology | Library | resources.libraries.python.NodePath | Library | resources.libraries.python.InterfaceUtil @@ -301,6 +302,7 @@ | | ... | ramp_up_rate=${ramp_up_rate} | | ${latency} = | Get Latency Int | | Set Test Message | ${\n}${message_prefix} ${latency} | append=${True} +| | Export Ndrpdr Latency | ${message_prefix} | ${latency} | Send ramp-up traffic | | [Documentation] @@ -371,14 +373,17 @@ | | ... | Type: boolean | | ... | - duration_limit - Hard limit for trial duration, overriding duration | | ... | computed from transaction_scale. Default 0.0 means no limit. +| | ... | - export_mrr_unit - Use this unit when exporting MRR values, +| | ... | or empty string for no export. | | | | ... | *Example:* | | | | ... | \| Send traffic at specified rate \| \${1.0} \| ${4000000.0} \ -| | ... | \| \${10} \| ${False} \| ${1.0} \| +| | ... | \| \${10} \| ${False} \| ${1.0} \| pps \| | | | | [Arguments] | ${trial_duration} | ${rate} | ${trial_multiplicity} | | ... | ${use_latency}=${False} | ${duration_limit}=${0.0} +| | ... | ${export_mrr_unit}=${Empty} | | | | ${ppta} = | Get Packets Per Transaction Aggregated | | ${ramp_up_duration} = | Get Ramp Up Duration @@ -415,7 +420,10 @@ | | | ... | ramp_up_rate=${ramp_up_rate} | | | # Out of several quantities for aborted traffic (duration stretching), | | | # the approximated receive rate is the best estimate we have. -| | | Append To List | ${results} | ${result.approximated_receive_rate} +| | | ${value} = | Set Variable | ${result.approximated_receive_rate} +| | | # TODO: Add correct bandwidth computation. +| | | Append Mrr Value | ${value} | ${export_mrr_unit} +| | | Append To List | ${results} | ${value} | | END | | FOR | ${action} | IN | @{stat_post_trial} | | | Run Keyword | Additional Statistics Action For ${action} @@ -637,6 +645,8 @@ | | ${trial_duration} = | Get Mrr Trial Duration | | ${trial_multiplicity} = | Get Mrr Trial Multiplicity | | ${use_latency} = | Get Use Latency +| | ${unit} = | Set Variable If | """_cps""" in """${transaction_type}""" +| | ... | cps | pps | | # The following also sets \${rate_for_teardown} | | ${results} = | Send traffic at specified rate | | ... | rate=${max_rate} @@ -644,9 +654,10 @@ | | ... | trial_multiplicity=${trial_multiplicity} | | ... | use_latency=${use_latency} | | ... | duration_limit=${0.0} -| | ${unit} = | Set Variable If | """_cps""" in """${transaction_type}""" +| | ... | export_mrr_unit=${unit} +| | ${unit_text} = | Set Variable If | """_cps""" in """${transaction_type}""" | | ... | estimated connections per second | packets per second | | Set Test Message | ${\n}Maximum Receive Rate trial results -| | Set Test Message | in ${unit}: ${results} +| | Set Test Message | in ${unit_text}: ${results} | | ... | append=yes | | Fail if no traffic forwarded -- cgit 1.2.3-korg