From 01d8f262afc567c3d49a23c3cb2cdeaced8a6887 Mon Sep 17 00:00:00 2001 From: Vratko Polak Date: Wed, 15 Dec 2021 17:14:36 +0100 Subject: UTI: Export results + Model version 1.0.0. - Only some result types are exported. + MRR, NDRPDR and SOAK. - Other result types to be added later. + In contrast, all test types are detected. + Convert custom classes to JSON-serializable equivalents. + Sort dict keys before converting to JSON. + Override the order for some known keys. + Export sets as sorted arrays. + Convert to info content from serialized raw content. + Also export outputs for suite setups and teardowns. + Info files for setup/teardown exist only temporarily. + The data is merged into suite.info.json file. + This simplifies presentation of total suite duration. + Define model via JSON schema: - Just test case, suite setup/teardown/suite to be added later. - Just info, raw to be added later. + Proper descriptions. + Json is generated from yaml. + This is a convenience for maintainers. + The officially used schema is the .json one. + TODOs written into a separate .txt file. + Validate exported instance against the schema. + Include format checking. + Update CSIT requirements for validation dependencies. + This needs python-dateutil==2.8.2, only a patch bump. + Compute bandwidth also for soak tests. + This unifies with NDRPDR to simplify schema definition. - PAL may need an update for parsing soak test message. + Include SSH log items, raw output only. + Generate all outputs in a single filesystem tree. + Move raw outputs into test_output_raw.tar.xz. + Rename existing tar with suites to generated_robot_files.tar.xz. Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c Signed-off-by: Vratko Polak --- docs/model/current/top.rst | 43 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 39 insertions(+), 4 deletions(-) (limited to 'docs/model/current/top.rst') diff --git a/docs/model/current/top.rst b/docs/model/current/top.rst index 640abe2343..d86e3fde4c 100644 --- a/docs/model/current/top.rst +++ b/docs/model/current/top.rst @@ -22,7 +22,7 @@ especially the export side (UTI), not import side (PAL). Version ~~~~~~~ -This document is valid for CSIT model version 0.1.0. +This document is valid for CSIT model version 1.0.0. It is recommended to use semantic versioning: https://semver.org/ That means, if the new model misses a field present in the old model, @@ -42,7 +42,42 @@ UTI stands for Unified Test Interface. It mainly focuses on exporting information gathered during test run into JSON output files. -Files ------ +Output Structure +----------------- -No files are exported yet in this version. +UTI outputs come in filesystem tree structure (single tree), where directories +correspond to suite levels and files correspond to suite setup, suite teardown +or any test case at this level of suite. +The directory name comes from SUITE_NAME Robot variable (the last part +as the previous parts are higher level suites), converted to lowercase. +If the suite name contains spaces (Robot converts underscores to spaces), +they are replaced with underscores. + +The filesystem tree is rooted under tests/ (as suites in git are there), +and for each component (test case, suite setup, suite teardown) +two files are generated. +The "raw" variant is suitable for debugging (can contain lower level logging), +while the "info" variant is suitable for processing by PAL +(can contain derivative values so PAL does not need to compute them +on every download). +Their structure and content is mostly identical, model definition mentions +if a particular subschema is not identical in the two variants. +It is possible to convert from raw to info, but not the other way. + +Although we expect only ASCII text in the exported files, +we manipulate files using UTF-8 encoding, +so if Robot Framework uses a non-ascii character, it will be handled. + +JSON schemas +------------ + +CSIT model is formally defined as a collection of JSON schema documents, +one for each output file type. + +The current version specifies only one output file type: +Info output for test case. + +The authoritative JSON schema documents are in JSON format. +Git repository also contains YAML formatted document and conversion utility, +which simplifies maintaining of the JSON document +(no need to track brackets and commas), but are not authoritative. -- cgit 1.2.3-korg