UTI: Export results

+ Model version 1.0.0.
- Only some result types are exported.
 + MRR, NDRPDR and SOAK.
 - Other result types to be added later.
 + In contrast, all test types are detected.
+ Convert custom classes to JSON-serializable equivalents.
 + Sort dict keys before converting to JSON.
  + Override the order for some known keys.
 + Export sets as sorted arrays.
+ Convert to info content from serialized raw content.
+ Also export outputs for suite setups and teardowns.
 + Info files for setup/teardown exist only temporarily.
 + The data is merged into suite.info.json file.
  + This simplifies presentation of total suite duration.
+ Define model via JSON schema:
 - Just test case, suite setup/teardown/suite to be added later.
 - Just info, raw to be added later.
 + Proper descriptions.
 + Json is generated from yaml.
  + This is a convenience for maintainers.
  + The officially used schema is the .json one.
 + TODOs written into a separate .txt file.
+ Validate exported instance against the schema.
 + Include format checking.
 + Update CSIT requirements for validation dependencies.
  + This needs python-dateutil==2.8.2, only a patch bump.
+ Compute bandwidth also for soak tests.
 + This unifies with NDRPDR to simplify schema definition.
 - PAL may need an update for parsing soak test message.
+ Include SSH log items, raw output only.
+ Generate all outputs in a single filesystem tree.
 + Move raw outputs into test_output_raw.tar.xz.
 + Rename existing tar with suites to generated_robot_files.tar.xz.

Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

7 files changed, 72 insertions, 9 deletions

diff --git a/resources/libraries/robot/performance/performance_display.robot b/resources/libraries/robot/performance/performance_display.robot
index de515412a5..db2b522091 100644
--- a/resources/libraries/robot/performance/performance_display.robot
+++ b/resources/libraries/robot/performance/performance_display.robot
@@ -46,6 +46,31 @@
 | | ... | ${message}${\n}${message_zero} | ${message}${\n}${message_other}
 | | Fail | ${message}
 
+| Compute bandwidth
+| | [Documentation]
+| | ... | Compute (bidir) bandwidth from given (unidir) transaction rate.
+| | ...
+| | ... | This keyword reads "ppta" and "avg_frame_size" set elsewhere.
+| | ... | The implementation should work for both pps and cps rates.
+| | ... |
+| | ... | *Arguments:*
+| | ... | - tps - Transaction rate (unidirectional) [tps]. Type: float
+| | ...
+| | ... | *Returns:*
+| | ... | - Computed bandwidth in Gbps.
+| | ... | - Computed aggregate packet rate in pps.
+| |
+| | ... | *Example:*
+| |
+| | ... | |\ \${bandwidth} \| \${pps} = \| Compute Bandwidth \| \${12345.67} \|
+| |
+| | [Arguments] | ${tps}
+| |
+| | ${ppta} = | Get Packets Per Transaction Aggregated
+| | ${pps} = | Evaluate | ${tps} * ${ppta}
+| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9
+| | Return From Keyword | ${bandwidth} | ${pps}
+
 | Display Reconfig Test Message
 | | [Documentation]
 | | ... | Display the number of packets lost (bidirectionally)
@@ -198,6 +223,8 @@
 | | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY}
 | |
 | | Set Test Message | ${\n}${text}: ${tps} CPS | append=yes
+| | ${bandwidth} | ${pps} = | Compute Bandwidth | ${tps}
+| | Export Search Bound | ${text} | ${tps} | cps | ${bandwidth}
 | | Return From Keyword If | not """${latency}"""
 | | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency}
 | | ... | append=yes
@@ -230,11 +257,10 @@
 | |
 | | [Arguments] | ${text} | ${tps} | ${latency}=${EMPTY}
 | |
-| | ${ppta} = | Get Packets Per Transaction Aggregated
-| | ${pps} = | Evaluate | ${tps} * ${ppta}
-| | ${bandwidth} = | Evaluate | ${pps} * (${avg_frame_size}+20)*8 / 1e9
+| | ${bandwidth} | ${pps} = | Compute Bandwidth | ${tps}
 | | Set Test Message | ${\n}${text}: ${pps} pps, | append=yes
 | | Set Test Message | ${bandwidth} Gbps (initial) | append=yes
+| | Export Search Bound | ${text} | ${pps} | pps | ${bandwidth * 1e9}
 | | Return From Keyword If | not """${latency}"""
 | | Set Test Message | ${\n}LATENCY [min/avg/max/hdrh] per stream: ${latency}
 | | ... | append=yes
diff --git a/resources/libraries/robot/performance/performance_utils.robot b/resources/libraries/robot/performance/performance_utils.robot
index 8de74c5707..f2cb873d1a 100644
--- a/resources/libraries/robot/performance/performance_utils.robot
+++ b/resources/libraries/robot/performance/performance_utils.robot
@@ -13,6 +13,7 @@
 
 *** Settings ***
 | Library | Collections
+| Library | resources.libraries.python.model.ExportResult
 | Library | resources.libraries.python.topology.Topology
 | Library | resources.libraries.python.NodePath
 | Library | resources.libraries.python.InterfaceUtil
@@ -301,6 +302,7 @@
 | | ... | ramp_up_rate=${ramp_up_rate}
 | | ${latency} = | Get Latency Int
 | | Set Test Message | ${\n}${message_prefix} ${latency} | append=${True}
+| | Export Ndrpdr Latency | ${message_prefix} | ${latency}
 
 | Send ramp-up traffic
 | | [Documentation]
@@ -371,14 +373,17 @@
 | | ... | Type: boolean
 | | ... | - duration_limit - Hard limit for trial duration, overriding duration
 | | ... | computed from transaction_scale. Default 0.0 means no limit.
+| | ... | - export_mrr_unit - Use this unit when exporting MRR values,
+| | ... | or empty string for no export.
 | |
 | | ... | *Example:*
 | |
 | | ... | \| Send traffic at specified rate \| \${1.0} \| ${4000000.0} \
-| | ... | \| \${10} \| ${False} \| ${1.0} \|
+| | ... | \| \${10} \| ${False} \| ${1.0} \| pps \|
 | |
 | | [Arguments] | ${trial_duration} | ${rate} | ${trial_multiplicity}
 | | ... | ${use_latency}=${False} | ${duration_limit}=${0.0}
+| | ... | ${export_mrr_unit}=${Empty}
 | |
 | | ${ppta} = | Get Packets Per Transaction Aggregated
 | | ${ramp_up_duration} = | Get Ramp Up Duration
@@ -415,7 +420,10 @@
 | | | ... | ramp_up_rate=${ramp_up_rate}
 | | | # Out of several quantities for aborted traffic (duration stretching),
 | | | # the approximated receive rate is the best estimate we have.
-| | | Append To List | ${results} | ${result.approximated_receive_rate}
+| | | ${value} = | Set Variable | ${result.approximated_receive_rate}
+| | | # TODO: Add correct bandwidth computation.
+| | | Append Mrr Value | ${value} | ${export_mrr_unit}
+| | | Append To List | ${results} | ${value}
 | | END
 | | FOR | ${action} | IN | @{stat_post_trial}
 | | | Run Keyword | Additional Statistics Action For ${action}
@@ -637,6 +645,8 @@
 | | ${trial_duration} = | Get Mrr Trial Duration
 | | ${trial_multiplicity} = | Get Mrr Trial Multiplicity
 | | ${use_latency} = | Get Use Latency
+| | ${unit} = | Set Variable If | """_cps""" in """${transaction_type}"""
+| | ... | cps | pps
 | | # The following also sets \${rate_for_teardown}
 | | ${results} = | Send traffic at specified rate
 | | ... | rate=${max_rate}
@@ -644,9 +654,10 @@
 | | ... | trial_multiplicity=${trial_multiplicity}
 | | ... | use_latency=${use_latency}
 | | ... | duration_limit=${0.0}
-| | ${unit} = | Set Variable If | """_cps""" in """${transaction_type}"""
+| | ... | export_mrr_unit=${unit}
+| | ${unit_text} = | Set Variable If | """_cps""" in """${transaction_type}"""
 | | ... | estimated connections per second | packets per second
 | | Set Test Message | ${\n}Maximum Receive Rate trial results
-| | Set Test Message | in ${unit}: ${results}
+| | Set Test Message | in ${unit_text}: ${results}
 | | ... | append=yes
 | | Fail if no traffic forwarded
diff --git a/resources/libraries/robot/shared/default.robot b/resources/libraries/robot/shared/default.robot
index 32d8863a32..7fa0222f78 100644
--- a/resources/libraries/robot/shared/default.robot
+++ b/resources/libraries/robot/shared/default.robot
@@ -31,6 +31,7 @@
 | Library | resources.libraries.python.IPUtil
 | Library | resources.libraries.python.IPv6Util
 | Library | resources.libraries.python.IrqUtil
+| Library | resources.libraries.python.model.export_json
 | Library | resources.libraries.python.NodePath
 | Library | resources.libraries.python.Namespaces
 | Library | resources.libraries.python.PapiHistory
@@ -334,6 +335,16 @@
 | | Run Keyword If | ${with_trace} | VPP Enable Traces On Dut
 | | ... | ${nodes['${dut}']}
 
+| Get And Export DPDK Version
+| | [Documentation] | Add version to test export as detected on DUT1.
+| |
+| | ... | *Example:*
+| |
+| | ... | \| Get And Export DPDK Version \|
+| |
+| | ${version} = | Get Dpdk Version | ${nodes}[DUT1]
+| | Export Dut Type And Version | DPDK | ${version}
+
 | Save VPP PIDs
 | | [Documentation] | Get PIDs of VPP processes from all DUTs in topology and\
 | | ... | set it as a test variable. The PIDs are stored as dictionary items\
diff --git a/resources/libraries/robot/shared/suite_setup.robot b/resources/libraries/robot/shared/suite_setup.robot
index 26d7f52205..232158a6e0 100644
--- a/resources/libraries/robot/shared/suite_setup.robot
+++ b/resources/libraries/robot/shared/suite_setup.robot
@@ -24,6 +24,7 @@
 | Variables | resources/libraries/python/Constants.py
 |
 | Documentation | Suite setup keywords.
+
 *** Keywords ***
 | Create suite topology variables
 | | [Documentation]
@@ -83,12 +84,14 @@
 | |
 | | [Arguments] | @{actions}
 | |
+| | Start Suite Setup Export
 | | ${nic_model_list}= | Create list | ${nic_name}
 | | &{info}= | Compute Circular Topology
 | | ... | ${nodes} | filter_list=${nic_model_list} | nic_pfs=${nic_pfs}
 | | ... | always_same_link=${False} | topo_has_tg=${True}
 | | Set suite variable | &{topology_info} | &{info}
 | | Create suite topology variables | @{actions}
+| | Finalize Suite Setup Export
 
 | Setup suite topology interfaces with no TG
 | | [Documentation]
@@ -104,12 +107,14 @@
 | |
 | | [Arguments] | @{actions}
 | |
+| | Start Suite Setup Export
 | | ${nic_model_list}= | Create list | ${nic_name}
 | | &{info}= | Compute Circular Topology
 | | ... | ${nodes} | filter_list=${nic_model_list} | nic_pfs=${nic_pfs}
 | | ... | always_same_link=${True} | topo_has_tg=${False}
 | | Set suite variable | &{topology_info} | &{info}
 | | Create suite topology variables | @{actions}
+| | Finalize Suite Setup Export
 
 | Setup suite topology interfaces with no DUT
 | | [Documentation]
@@ -125,12 +130,14 @@
 | |
 | | [Arguments] | @{actions}
 | |
+| | Start Suite Setup Export
 | | ${nic_model_list}= | Create list | ${nic_name}
 | | &{info}= | Compute Circular Topology
 | | ... | ${nodes} | filter_list=${nic_model_list} | nic_pfs=${nic_pfs}
 | | ... | always_same_link=${True} | topo_has_tg=${True} | topo_has_dut=${False}
 | | Set suite variable | &{topology_info} | &{info}
 | | Create suite topology variables | @{actions}
+| | Finalize Suite Setup Export
 
 | Additional Suite Setup Action For scapy
 | | [Documentation]
@@ -151,6 +158,7 @@
 | | | Initialize DPDK Framework | ${nodes['${dut}']}
 | | | ... | ${${dut}_${int}1}[0] | ${${dut}_${int}2}[0] | ${nic_driver}
 | | END
+| | Get And Export DPDK Version
 
 | Additional Suite Setup Action For performance vf
 | | [Documentation]
@@ -238,6 +246,8 @@
 | | ... | Additional Setup for suites which uses performance measurement
 | | ... | for L1 cross connect tests
 | |
+| | # TRex suites have only TG (and a loopback cable), no SUT nor DUT.
+| | Export Dut Type And Version | none | ${EMPTY}
 | | Initialize traffic generator
 | | ... | ${tg} | ${TG_pf1}[0] | ${TG_pf2}[0]
 | | ... | ${tg} | ${TG_pf2}[0]
diff --git a/resources/libraries/robot/shared/suite_teardown.robot b/resources/libraries/robot/shared/suite_teardown.robot
index 20b2776eec..ab6d3b6f8c 100644
--- a/resources/libraries/robot/shared/suite_teardown.robot
+++ b/resources/libraries/robot/shared/suite_teardown.robot
@@ -30,10 +30,12 @@
 | |
 | | [Arguments] | @{actions}
 | |
+| | Start Suite Teardown Export
 | | FOR | ${action} | IN | @{actions}
 | | | Run Keyword | Additional Suite Tear Down Action For ${action}
 | | END
 | | Remove All Added VIF Ports On All DUTs From Topology | ${nodes}
+| | Finalize Suite Teardown Export
 
 | Additional Suite Tear Down Action For ab
 | | [Documentation]
diff --git a/resources/libraries/robot/shared/test_setup.robot b/resources/libraries/robot/shared/test_setup.robot
index e2fcfc5f87..3bc88858c0 100644
--- a/resources/libraries/robot/shared/test_setup.robot
+++ b/resources/libraries/robot/shared/test_setup.robot
@@ -21,13 +21,14 @@
 *** Keywords ***
 | Setup test
 | | [Documentation]
-| | ... | Common test setup for tests.
+| | ... | Common test setup for VPP tests.
 | |
 | | ... | *Arguments:*
 | | ... | - ${actions} - Additional setup action. Type: list
 | |
 | | [Arguments] | @{actions}
 | |
+| | Start Test Export
 | | Reset PAPI History On All DUTs | ${nodes}
 | | ${int} = | Set Variable If | ${nic_vfs} > 0 | prevf | pf
 | | Create base startup configuration of VPP on all DUTs
diff --git a/resources/libraries/robot/shared/test_teardown.robot b/resources/libraries/robot/shared/test_teardown.robot
index 3fe5e6a8ee..474a6e9385 100644
--- a/resources/libraries/robot/shared/test_teardown.robot
+++ b/resources/libraries/robot/shared/test_teardown.robot
@@ -14,7 +14,7 @@
 """Keywords used in test teardowns."""
 
 *** Settings ***
-| Resource | resources/libraries/robot/shared/container.robot
+| Resource | resources/libraries/robot/shared/default.robot
 | Library | resources.libraries.python.PapiHistory
 | Library | resources.libraries.python.topology.Topology
 | Variables | resources/libraries/python/Constants.py
@@ -45,6 +45,7 @@
 | | | Run Keyword | Additional Test Tear Down Action For ${action}
 | | END
 | | Clean Sockets On All Nodes | ${nodes}
+| | Finalize Test Export
 
 | Tear down test raw
 | | [Documentation]
@@ -60,6 +61,7 @@
 | | | Run Keyword | Additional Test Tear Down Action For ${action}
 | | END
 | | Clean Sockets On All Nodes | ${nodes}
+| | Finalize Test Export
 
 # Additional Test Tear Down Actions in alphabetical order
 | Additional Test Tear Down Action For acl