aboutsummaryrefslogtreecommitdiffstats
path: root/docs/content/introduction/test_code_guidelines.md
diff options
context:
space:
mode:
authorpmikus <peter.mikus@protonmail.ch>2023-03-09 13:32:33 +0000
committerPeter Mikus <peter.mikus@protonmail.ch>2023-03-15 10:06:55 +0000
commitb928e56347c682fdb9762ccbe2f368329d4037e4 (patch)
tree9c0f6a02efc1704821229b8e18a23468fdf8674a /docs/content/introduction/test_code_guidelines.md
parentddcdf45806d0efa9e89dd4446b4c7da39cfb27a8 (diff)
feat(docs): Hugo
Signed-off-by: pmikus <peter.mikus@protonmail.ch> Change-Id: Id8b43ef1f31f39b19a0629c52581514fda278f3b
Diffstat (limited to 'docs/content/introduction/test_code_guidelines.md')
-rw-r--r--docs/content/introduction/test_code_guidelines.md294
1 files changed, 294 insertions, 0 deletions
diff --git a/docs/content/introduction/test_code_guidelines.md b/docs/content/introduction/test_code_guidelines.md
new file mode 100644
index 0000000000..9707d63ea6
--- /dev/null
+++ b/docs/content/introduction/test_code_guidelines.md
@@ -0,0 +1,294 @@
+---
+bookHidden: true
+title: "CSIT Test Code Guidelines"
+---
+
+# CSIT Test Code Guidelines
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
+"MAY", and "OPTIONAL" in this document are to be interpreted as
+described in [BCP 14](https://tools.ietf.org/html/bcp14),
+[RFC2119](https://tools.ietf.org/html/rfc2119),
+[RFC8174](https://tools.ietf.org/html/rfc8174)
+when, and only when, they appear in all capitals, as shown here.
+
+This document SHALL describe guidelines for writing reliable, maintainable,
+reusable and readable code for CSIT.
+
+# RobotFramework test case files and resource files
+
++ General
+
+ + Contributors SHOULD look at requirements.txt in root CSIT directory
+ for the currently used Robot Framework version.
+ Contributors SHOULD read
+ [Robot Framework User Guide](http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html)
+ for more details.
+
+ + RobotFramework test case files and resource files
+ SHALL use special extension .robot
+
+ + Pipe and space separated file format (without trailing pipe
+ and without pipe aligning) SHALL be used.
+ Tabs are invisible characters, which are error prone.
+ 4-spaces separation is prone to accidental double space
+ acting as a separator.
+
+ + Files SHALL be encoded in UTF-8 (the default Robot source file encoding).
+ Usage of non-ASCII characters SHOULD be avoided if possible.
+ It is RECOMMENDED to
+ [escape](http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#escaping)
+ non-ASCII characters.
+
+ + Line length SHALL be limited to 80 characters.
+
+ + There SHALL be licence text present at the beginning of each file.
+
+ + Copy-pasting of the code NOT RECOMMENDED practice, any code that could be
+ re-used SHOULD be put into a library (Robot resource, Python library, ...).
+
++ Test cases
+
+ + It is RECOMMENDED to use data-driven test case definitions
+ anytime suite contains test cases similar in structure.
+ Typically, a suite SHOULD define a Template keyword, and test cases
+ SHOULD only specify tags and argument values
+
+ *** Settings ***
+ | Test Template | Local Template
+ ...
+
+ *** Test Cases ***
+ | tc01-64B-1c-eth-l2patch-mrr
+ | | [Tags] | 64B | 1C
+ | | framesize=${64} | phy_cores=${1}
+
+ + Test case templates (or testcases) SHALL be written in Behavior-driven style
+ i.e. in readable English, so that even non-technical project stakeholders
+ can understand it
+
+ *** Keywords ***
+ | Local Template
+ | | [Documentation]
+ | | ... | [Cfg] DUT runs L2 patch config with ${phy_cores} phy core(s).
+ | | ... | [Ver] Measure NDR and PDR values using MLRsearch algorithm.\
+ | | ...
+ | | ... | *Arguments:*
+ | | ... | - frame_size - Framesize in Bytes in integer
+ | | ... | or string (IMIX_v4_1). Type: integer, string
+ | | ... | - phy_cores - Number of physical cores. Type: integer
+ | | ... | - rxq - Number of RX queues, default value: ${None}.
+ | | ... | Type: integer
+ | | ...
+ | | [Arguments] | ${frame_size} | ${phy_cores} | ${rxq}=${None}
+ | | ...
+ | | Set Test Variable | \${frame_size}
+ | | ...
+ | | Given Add worker threads and rxqueues to all DUTs
+ | | ... | ${phy_cores} | ${rxq}
+ | | And Add PCI devices to all DUTs
+ | | Set Max Rate And Jumbo And Handle Multi Seg
+ | | And Apply startup configuration on all VPP DUTs
+ | | When Initialize L2 patch
+ | | Then Find NDR and PDR intervals using optimized search
+
+ + Every suite and test case template (or testcase)
+ SHALL contain short documentation.
+ Generated CSIT web pages display the documentation.
+
+ + You SHOULD NOT use hard-coded constants.
+ It is RECOMMENDED to use the variable table
+ (\*\*\*Variables\*\*\*) to define test case specific values.
+ You SHALL use the assignment sign = after the variable name
+ to make assigning variables slightly more explicit
+
+ *** Variables ***
+ | ${traffic_profile}= | trex-stl-2n-ethip4-ip4src254
+
+ + Common test case specific settings of the test environment SHALL be done
+ in Test Setup keyword defined in the Setting table.
+
+ + Run Keywords construction is RECOMMENDED if it is more readable
+ than a keyword.
+
+ + Separate keyword is RECOMMENDED if the construction is less readable.
+
+ + Post-test cleaning and processing actions SHALL be done in Test Teardown
+ part of the Setting table (e.g. download statistics from VPP nodes).
+ This part is executed even if the test case has failed. On the other hand
+ it is possible to disable the tear-down from command line, thus leaving
+ the system in “broken” state for investigation.
+
+ + Every testcase SHALL be correctly tagged. List of defined tags is in
+ csit/docs/introduction/test_tag_documentation.rst
+
+ + Whenever possible, common tags SHALL be set using Force Tags
+ in Settings table.
+
+ + User high-level keywords specific for the particular test suite
+ SHOULD be implemented in the Keywords table of suitable Robot resource file
+ to enable readability and code-reuse.
+
+ + Such keywords MAY be implemented in Keywords table of the suite instead,
+ if the contributor believes no other test will use such keywords.
+ But this is NOT RECOMMENDED in general, as keywords in Resources
+ are easier to maintain.
+
+ + All test case names (and suite names) SHALL conform
+ to current naming convention.
+ https://wiki.fd.io/view/CSIT/csit-test-naming
+
+ + Frequently, different suites use the same test case layout.
+ It is RECOMMENDED to use autogeneration scripts available,
+ possibly extending them if their current functionality is not sufficient.
+
++ Resource files
+
+ + SHALL be used to implement higher-level keywords that are used in test cases
+ or other higher-level (or medium-level) keywords.
+
+ + Every keyword SHALL contain Documentation where the purpose and arguments
+ of the keyword are described. Also document types, return values,
+ and any specific assumptions the particular keyword relies on.
+
+ + A keyword usage example SHALL be the part of the Documentation.
+ The example SHALL use pipe and space separated format
+ (with escaped pipes and) with a trailing pipe.
+
+ + The reason was possbile usage of Robot's libdoc tool
+ to generate tests and resources documentation. In that case
+ example keyword usage would be rendered in table.
+
+ + Keyword name SHALL describe what the keyword does,
+ specifically and in a reasonable length (“short sentence”).
+
+ + Keyword names SHALL be short enough for call sites
+ to fit within line length limit.
+
+ + If a keyword argument has a most commonly used value, it is RECOMMENDED
+ to set it as default. This makes keyword code longer,
+ but suite code shorter, and readability (and maintainability)
+ of suites SHALL always more important.
+
+ + If there is intermediate data (created by one keyword, to be used
+ by another keyword) of singleton semantics (it is clear that the test case
+ can have at most one instance of such data, even if the instance
+ is complex, for example ${nodes}), it is RECOMMENDED to store it
+ in test variables. You SHALL document test variables read or written
+ by a keyword. This makes the test template code less verbose.
+ As soon as the data instance is not unique, you SHALL pass it around
+ via arguments and return values explicitly (this makes lower level keywords
+ more reusable and less bug prone).
+
+ + It is RECOMMENDED to pass arguments explicitly via [Arguments] line.
+ Setting test variables takes more space and is less explicit.
+ Using arguments embedded in keyword name makes them less visible,
+ and it makes it harder for the line containing the resulting long name
+ to fit into the maximum character limit, so you SHOULD NOT use them.
+
+# Python library files
+
++ General
+
+ + SHALL be used to implement low-level keywords that are called from
+ resource files (of higher-level keywords) or from test cases.
+
+ + Higher-level keywords MAY be implemented in python library file too.
+ it is RECOMMENDED especially in the case that their implementation
+ in resource file would be too difficult or impossible,
+ e.g. complex data structures or functional programming.
+
+ + Every keyword, Python module, class, method, enum SHALL contain
+ docstring with the short description and used input parameters
+ and possible return value(s) or raised exceptions.
+
+ + The docstrings SHOULD conform to
+ [PEP 257](https://www.python.org/dev/peps/pep-0257/)
+ and other quality standards.
+
+ + CSIT contributions SHALL use a specific formatting for documenting
+ arguments, return values and similar.
+
+ + Keyword usage examples MAY be grouped and used
+ in the class/module documentation string, to provide better overview
+ of the usage and relationships between keywords.
+
+ + Keyword name SHALL describe what the keyword does,
+ specifically and in a reasonable length (“short sentence”).
+ See https://wiki.fd.io/view/CSIT/csit-test-naming
+
+ + Python implementation of a keyword is a function,
+ so its name in the python library should be lowercase_with_underscores.
+ Robot call sites should usename with first letter capitalized, and spaces.
+
++ Coding
+
+ + It is RECOMMENDED to use some standard development tool
+ (e.g. PyCharm Community Edition) and follow
+ [PEP-8](https://www.python.org/dev/peps/pep-0008/) recommendations.
+
+ + All python code (not only Robot libraries) SHALL adhere to PEP-8 standard.
+ This is reported by CSIT Jenkins verify job.
+
+ + Indentation: You SHALL NOT use tab for indents!
+ Indent is defined as four spaces.
+
+ + Line length: SHALL be limited to 80 characters.
+
+ + CSIT Python code assumes PYTHONPATH is set
+ to the root of cloned CSIT git repository, creating a tree of sub-packages.
+ You SHALL use that tree for importing, for example
+
+ from resources.libraries.python.ssh import exec_cmd_no_error
+
+ + Imports SHALL be grouped in the following order:
+
+ 1. standard library imports,
+ 2. related third party imports,
+ 3. local application/library specific imports.
+
+ You SHALL put a blank line between each group of imports.
+
+ + You SHALL use two blank lines between top-level definitions,
+ one blank line between method definitions.
+
+ + You SHALL NOT execute any active code on library import.
+
+ + You SHALL NOT use global variables inside library files.
+
+ + You MAY define constants inside library files.
+
+ + It is NOT RECOMMENDED to use hard-coded constants (e.g. numbers,
+ paths without any description). It is RECOMMENDED to use
+ configuration file(s), like /csit/resources/libraries/python/Constants.py,
+ with appropriate comments.
+
+ + The code SHALL log at the lowest possible level of implementation,
+ for debugging purposes. You SHALL use same style for similar events.
+ You SHALL keep logging as verbose as necessary.
+
+ + You SHALL use the most appropriate exception not general one (Exception)
+ if possible. You SHOULD create your own exception
+ if necessary and implement there logging, level debug.
+
+ + You MAY use RuntimeException for generally unexpected failures.
+
+ + It is RECOMMENDED to use RuntimeError also for
+ infrastructure failures, e.g. losing SSH connection to SUT.
+
+ + You MAY use EnvironmentError and its cublasses instead,
+ if the distinction is informative for callers.
+
+ + It is RECOMMENDED to use AssertionError when SUT is at fault.
+
+ + For each class (e.g. exception) it is RECOMMENDED to implement __repr__()
+ which SHALL return a string usable as a constructor call
+ (including repr()ed arguments).
+ When logging, you SHOULD log the repr form, unless the internal structure
+ of the object in question would likely result in too long output.
+ This is helpful for debugging.
+
+ + For composing and formatting strings, you SHOULD use .format()
+ with named arguments.
+ Example: "repr() of name: {name!r}".format(name=name)