UTI: Export results

+ Model version 1.0.0.
- Only some result types are exported.
 + MRR, NDRPDR and SOAK.
 - Other result types to be added later.
 + In contrast, all test types are detected.
+ Convert custom classes to JSON-serializable equivalents.
 + Sort dict keys before converting to JSON.
  + Override the order for some known keys.
 + Export sets as sorted arrays.
+ Convert to info content from serialized raw content.
+ Also export outputs for suite setups and teardowns.
 + Info files for setup/teardown exist only temporarily.
 + The data is merged into suite.info.json file.
  + This simplifies presentation of total suite duration.
+ Define model via JSON schema:
 - Just test case, suite setup/teardown/suite to be added later.
 - Just info, raw to be added later.
 + Proper descriptions.
 + Json is generated from yaml.
  + This is a convenience for maintainers.
  + The officially used schema is the .json one.
 + TODOs written into a separate .txt file.
+ Validate exported instance against the schema.
 + Include format checking.
 + Update CSIT requirements for validation dependencies.
  + This needs python-dateutil==2.8.2, only a patch bump.
+ Compute bandwidth also for soak tests.
 + This unifies with NDRPDR to simplify schema definition.
 - PAL may need an update for parsing soak test message.
+ Include SSH log items, raw output only.
+ Generate all outputs in a single filesystem tree.
 + Move raw outputs into test_output_raw.tar.xz.
 + Rename existing tar with suites to generated_robot_files.tar.xz.

Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

14 files changed, 20 insertions, 3 deletions

diff --git a/tests/trex/perf/__init__.robot b/tests/trex/perf/__init__.robot
index 7d7531d309..e6fc3fae17 100644
--- a/tests/trex/perf/__init__.robot
+++ b/tests/trex/perf/__init__.robot
@@ -19,13 +19,17 @@
 | Library | resources.libraries.python.SetupFramework.CleanupFramework
 | Library | resources.libraries.python.CpuUtils
 |
-| Suite Setup | Run Keywords | Setup Global Variables
+| Suite Setup | Run Keywords | Start Suite Setup Export
+| ... | AND | Setup Global Variables
 | ... | AND | Setup Framework | ${nodes}
 | ... | AND | Get CPU Info from All Nodes | ${nodes}
 | ... | AND | Update All Interface Data on All Nodes | ${nodes}
 | ... | skip_tg=${True} | skip_vpp=${True}
+| ... | AND | Finalize Suite Setup Export
 |
-| Suite Teardown | Cleanup Framework | ${nodes}
+| Suite Teardown | Run Keywords | Start Suite Teardown Export
+| ... | AND | Cleanup Framework | ${nodes}
+| ... | AND | Finalize Suite Teardown Export
 
 *** Keywords ***
 | Setup Global Variables
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4base-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4base-tg-ndrpdr.robot
index 76d483940b..756e37a3cf 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4base-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4base-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4scale2m-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4scale2m-tg-ndrpdr.robot
index 0d7ae91a08..7dbed4c28f 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4scale2m-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4-ip4scale2m-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
index 5de14432d1..e9fc43ff15 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
@@ -22,6 +22,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
index 6fa25a50ca..3bb788bf2f 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
index 66ef7e8fe5..f15eaacc43 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
index afed63333f..a83e87d902 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4tcp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
index d73bd9b044..7c4ddc2115 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-cps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
index bbf66e06b6..94456e7437 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h1024-p63-s64512-pps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
index 41d49315be..23690c343c 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-cps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
@@ -73,7 +74,7 @@
 | | [Arguments] | ${frame_size}
 | |
 | | Set Test Variable | \${frame_size}
-
+| |
 | | Given Set Max Rate And Jumbo
 | | Then Find NDR and PDR intervals using optimized search
 
diff --git a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
index ecb48413cc..61d29f1eda 100644
--- a/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip4/1n1l-10ge2p1x710-ethip4udp-ip4base-h262144-p63-s16515072-pps-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6base-tg-ndrpdr.robot b/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6base-tg-ndrpdr.robot
index 9788baadac..c13b6b663f 100644
--- a/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6base-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6base-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6scale2m-tg-ndrpdr.robot b/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6scale2m-tg-ndrpdr.robot
index fcdcfead7f..ba3449d898 100644
--- a/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6scale2m-tg-ndrpdr.robot
+++ b/tests/trex/perf/ip6/1n1l-10ge2p1x710-ethip6-ip6scale2m-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template
diff --git a/tests/trex/perf/l2/1n1l-10ge2p1x710-eth-l2bdscale1mmaclrn-tg-ndrpdr.robot b/tests/trex/perf/l2/1n1l-10ge2p1x710-eth-l2bdscale1mmaclrn-tg-ndrpdr.robot
index ed89f95fea..3acc63deb3 100644
--- a/tests/trex/perf/l2/1n1l-10ge2p1x710-eth-l2bdscale1mmaclrn-tg-ndrpdr.robot
+++ b/tests/trex/perf/l2/1n1l-10ge2p1x710-eth-l2bdscale1mmaclrn-tg-ndrpdr.robot
@@ -21,6 +21,7 @@
 |
 | Suite Setup | Setup suite topology interfaces with no DUT | performance_tg_nic
 | Suite Teardown | Tear down suite | performance
+| Test Setup | Start Test Export
 | Test Teardown | Tear down test raw | performance
 |
 | Test Template | Local Template