Add tox.ini and few checker scripts

The plan is to change csit-validate-pylint-master job
to fail if (and only if) tox fails.

This will allow us to easily add checks,
with or without the voting power.

Each check produces log (ignored in .gitignore)
the voting job can archive.

+ Made autogen quiet by default, to avoid spam in autogen checker.
+ Unified the way direct csit subdirectories are git-ignored.

Change-Id: I6477b1ef7da6d3e30f68c5850d04900cc52f431e
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

10 files changed, 260 insertions, 58 deletions

diff --git a/resources/libraries/bash/entry/bootstrap_verify_perf.sh b/resources/libraries/bash/entry/bootstrap_verify_perf.sh
index 1a0d638da1..aad2969ecc 100644
--- a/resources/libraries/bash/entry/bootstrap_verify_perf.sh
+++ b/resources/libraries/bash/entry/bootstrap_verify_perf.sh
@@ -1,6 +1,4 @@
-#!/usr/bin/env bash
-
-# Copyright (c) 2018 Cisco and/or its affiliates.
+# Copyright (c) 2019 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -39,7 +37,7 @@ get_test_tag_string || die
 select_topology || die
 gather_build || die
 check_download_dir || die
-activate_virtualenv "${CSIT_DIR}" || die
+activate_virtualenv || die
 reserve_testbed || die
 select_tags || die
 compose_pybot_arguments || die
diff --git a/resources/libraries/bash/entry/bootstrap_vpp_device.sh b/resources/libraries/bash/entry/bootstrap_vpp_device.sh
index 6d389d6baf..139e1e99ab 100755
--- a/resources/libraries/bash/entry/bootstrap_vpp_device.sh
+++ b/resources/libraries/bash/entry/bootstrap_vpp_device.sh
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-# Copyright (c) 2018 Cisco and/or its affiliates.
+# Copyright (c) 2019 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -37,7 +37,7 @@ get_test_tag_string || die
 select_topology || die
 gather_build || die
 check_download_dir || die
-activate_virtualenv "${CSIT_DIR}" || die
+activate_virtualenv || die
 activate_docker_topology || die
 select_vpp_device_tags || die
 compose_pybot_arguments || die
diff --git a/resources/libraries/bash/entry/check/README.txt b/resources/libraries/bash/entry/check/README.txt
new file mode 100644
index 0000000000..a72274464c
--- /dev/null
+++ b/resources/libraries/bash/entry/check/README.txt
@@ -0,0 +1,27 @@
+# Copyright (c) 2019 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+This directory contains checker scripts and other files they need.
+Each checker script is assumed to be run from tox,
+when working directory is set to ${CSIT_DIR}.
+Each script should:
++ Return nonzero exit code when it fails.
+++ The tox might ignore the code when the check is not blocking.
++ Write less verbose output to stderr.
++ Write (to stderr) PASSED or FAILED to help with debugging.
++ Direct more verbose output to appropriately named .log file.
++ Only the output suitable for automated processing by an external caller
+  should be written to stdout.
+++ The level of "less verbose" depends on check and state of codebase.
++ TODO: Should we carefully document which files are
+  whitelisted/blacklisted for a particulat check?
diff --git a/resources/libraries/bash/entry/check/autogen.sh b/resources/libraries/bash/entry/check/autogen.sh
new file mode 100644
index 0000000000..35ecc281ee
--- /dev/null
+++ b/resources/libraries/bash/entry/check/autogen.sh
@@ -0,0 +1,61 @@
+# Copyright (c) 2019 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -exuo pipefail
+
+# This file should be executed from tox, as the assumend working directory
+# is different from where this file is located.
+# This file does not have executable flag nor shebang,
+# to dissuade non-tox callers.
+
+# This script run every executable *.py script anywhere within tests/ dir,
+# the working directory temporarily changed to where the *.py file is.
+# Proper virtualenv is assumed to be active.
+# If "git diff" sees any change, this script fails.
+# The diff output stored to autogen.log (overwriting).
+# The *.py files are assumed to be robot suite generators,
+# any change means the contribution does not match the generated code.
+
+# "set -eu" handles failures from the following two lines.
+BASH_CHECKS_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))"
+BASH_FUNCTION_DIR="$(readlink -e "${BASH_CHECKS_DIR}/../../function")"
+source "${BASH_FUNCTION_DIR}/common.sh" || {
+    echo "Source failed." >&2
+    exit 1
+}
+
+work_dir="$(pwd)" || die
+trap "cd '${work_dir}'" EXIT || die
+file_list="$(find ./tests -type f -executable -name '*.py')" || die
+
+for gen in ${file_list}; do
+    directory="$(dirname "${gen}")" || die
+    filename="$(basename "${gen}")" || die
+    pushd "${directory}" || die
+    ./"${filename}" || die
+    popd || die
+done
+
+lines="$(git diff | tee "autogen.log" | wc -l)" || die
+if [ "${lines}" != "0" ]; then
+    # TODO: Decide which text goes to stdout and which to stderr.
+    warn "Autogen conflict diff nonzero lines: ${lines}"
+    # TODO: Disable if output size does more harm than good.
+    cat "autogen.log" >&2
+    warn
+    warn "Autogen checker: FAIL"
+    exit 1
+fi
+
+warn
+warn "Autogen checker: PASS"
diff --git a/resources/libraries/bash/entry/check/line.sh b/resources/libraries/bash/entry/check/line.sh
new file mode 100644
index 0000000000..c58c7d0126
--- /dev/null
+++ b/resources/libraries/bash/entry/check/line.sh
@@ -0,0 +1,49 @@
+# Copyright (c) 2019 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -exuo pipefail
+
+# This file should be executed from tox, as the assumend working directory
+# is different from where this file is located.
+# This file does not have executable flag nor shebang,
+# to dissuade non-tox callers.
+
+# This script runs a grep-based command and fails if it detects any lines
+# longer than 80 characters.
+# The grep output stored to lines.log (overwriting).
+
+# "set -eu" handles failures from the following two lines.
+BASH_CHECKS_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))"
+BASH_FUNCTION_DIR="$(readlink -e "${BASH_CHECKS_DIR}/../../function")"
+source "${BASH_FUNCTION_DIR}/common.sh" || {
+    echo "Source failed." >&2
+    exit 1
+}
+
+# docs contains too many wide formatted tables.
+# .txt contains lines with wide URLs.
+piped_command='set -exuo pipefail && grep -rn ".\{81\}" "resources/" "tests/"'
+piped_command+=' | fgrep -v .svg | fgrep -v .txt | tee "lines.log" | wc -l'
+lines="$(bash -c "${piped_command}")" || die
+if [ "${lines}" != "0" ]; then
+    # TODO: Decide which text goes to stdout and which to stderr.
+    warn "Long lines detected: ${lines}"
+    ## TODO: Enable when output size does more good than harm.
+    # cat "lines.log" >&2
+    warn
+    warn "Line length checker: FAIL"
+    exit 1
+fi
+
+warn
+warn "Line length checker: PASS"
diff --git a/resources/libraries/bash/entry/check/pylint.sh b/resources/libraries/bash/entry/check/pylint.sh
new file mode 100644
index 0000000000..a3cad9a0d7
--- /dev/null
+++ b/resources/libraries/bash/entry/check/pylint.sh
@@ -0,0 +1,44 @@
+# Copyright (c) 2019 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -exuo pipefail
+
+# This file should be executed from tox, as the assumend working directory
+# is different from where this file is located.
+# This file does not have executable flag nor shebang,
+# to dissuade non-tox callers.
+
+# This script runs pylint and propagates its exit code.
+# Config is taken from pylint.cfg, and proper virtualenv is assumed to be active.
+# The pylint output stored to pylint.log (overwriting).
+
+# "set -eu" handles failures from the following two lines.
+BASH_CHECKS_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))"
+BASH_FUNCTION_DIR="$(readlink -e "${BASH_CHECKS_DIR}/../../function")"
+source "${BASH_FUNCTION_DIR}/common.sh" || {
+    echo "Source failed." >&2
+    exit 1
+}
+pylint_args=("--rcfile=pylint.cfg" "resources/" "tests/")
+if pylint "${pylint_args[@]}" > "pylint.log"; then
+    warn
+    warn "Pylint checker: PASS"
+else
+    # TODO: Decide which text goes to stdout and which to stderr.
+    warn "Pylint exited with nonzero status."
+    ## TODO: Enable when output size does more good than harm.
+    # cat "pylint.log" >&2
+    warn
+    warn "Pylint checker: FAIL"
+    exit 1
+fi
diff --git a/resources/libraries/bash/entry/tox.sh b/resources/libraries/bash/entry/tox.sh
new file mode 100755
index 0000000000..82b7ef5307
--- /dev/null
+++ b/resources/libraries/bash/entry/tox.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+
+# Copyright (c) 2019 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -exuo pipefail
+
+# This is a launcher script, to be called from Jenkins.
+# It runs tox at $CSIT_DIR (after activating virtualenv with its requirements).
+# Exit code of tox is propagated.
+
+# "set -eu" handles failures from the following two lines.
+BASH_ENTRY_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))"
+BASH_FUNCTION_DIR="$(readlink -e "${BASH_ENTRY_DIR}/../function")"
+source "${BASH_FUNCTION_DIR}/common.sh" || {
+    echo "Source failed." >&2
+    exit 1
+}
+common_dirs || die
+cd "${CSIT_DIR}" || die
+activate_virtualenv "${CSIT_DIR}" "${CSIT_DIR}/tox-requirements.txt" || die
+tox  # Return code is turned into Jenkins job vote.
diff --git a/resources/libraries/bash/function/common.sh b/resources/libraries/bash/function/common.sh
index 7f640678ae..71cf9d3fa6 100644
--- a/resources/libraries/bash/function/common.sh
+++ b/resources/libraries/bash/function/common.sh
@@ -1,4 +1,4 @@
-# Copyright (c) 2018 Cisco and/or its affiliates.
+# Copyright (c) 2019 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -90,40 +90,43 @@ function activate_virtualenv () {
 
     set -exuo pipefail
 
+    # Update virtualenv pip package, delete and create virtualenv directory,
+    # activate the virtualenv, install requirements, set PYTHONPATH.
+
     # Arguments:
-    # - ${1} - Non-empty path to existing directory for creating virtualenv in.
+    # - ${1} - Path to existing directory for creating virtualenv in.
+    #          If missing or empty, ${CSIT_DIR} is used.
+    # - ${2} - Path to requirements file, ${CSIT_DIR}/requirements.txt if empty.
     # Variables read:
     # - CSIT_DIR - Path to existing root of local CSIT git repository.
-    # Variables set:
-    # - ENV_DIR - Path to the created virtualenv subdirectory.
     # Variables exported:
     # - PYTHONPATH - CSIT_DIR, as CSIT Python scripts usually need this.
     # Functions called:
     # - die - Print to stderr and exit.
 
-    # TODO: Do we really need to have ENV_DIR available as a global variable?
-
-    if [[ "${1-}" == "" ]]; then
-        die "Root location of virtualenv to create is not specified."
-    fi
-    ENV_DIR="${1}/env"
-    rm -rf "${ENV_DIR}" || die "Failed to clean previous virtualenv."
+    # TODO: Do we want the callers to be able to set the env dir name?
+    # TODO: + In that case, do we want to support env switching?
+    # TODO:   + In that case we want to make env_dir global.
+    # TODO: Do we want the callers to override PYTHONPATH loaction?
 
+    root_path="${1-$CSIT_DIR}"
+    env_dir="${root_path}/env"
+    req_path=${2-$CSIT_DIR/requirements.txt}
+    rm -rf "${env_dir}" || die "Failed to clean previous virtualenv."
     pip install --upgrade virtualenv || {
         die "Virtualenv package install failed."
     }
-    virtualenv "${ENV_DIR}" || {
+    virtualenv "${env_dir}" || {
         die "Virtualenv creation failed."
     }
     set +u
-    source "${ENV_DIR}/bin/activate" || die "Virtualenv activation failed."
+    source "${env_dir}/bin/activate" || die "Virtualenv activation failed."
     set -u
-    pip install -r "${CSIT_DIR}/requirements.txt" || {
-        die "CSIT requirements installation failed."
+    pip install --upgrade -r "${req_path}" || {
+        die "Requirements installation failed."
     }
-
     # Most CSIT Python scripts assume PYTHONPATH is set and exported.
-    export PYTHONPATH="${CSIT_DIR}" || die "Export failed."
+    export PYTHONPATH="${root_path}" || die "Export failed."
 }
 
 
diff --git a/resources/libraries/python/autogen/Copyright.py b/resources/libraries/python/autogen/Copyright.py
deleted file mode 100644
index e8c72f0624..0000000000
--- a/resources/libraries/python/autogen/Copyright.py
+++ /dev/null
@@ -1,29 +0,0 @@
-# Copyright (c) 2018 Cisco and/or its affiliates.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at:
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Module defining sole constant holding copyright text for current year."""
-
-
-COPYRIGHT = '''# Copyright (c) 2018 Cisco and/or its affiliates.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at:
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-'''
\ No newline at end of file
diff --git a/resources/libraries/python/autogen/Regenerator.py b/resources/libraries/python/autogen/Regenerator.py
index 64bcea917d..bae0e4f9fe 100644
--- a/resources/libraries/python/autogen/Regenerator.py
+++ b/resources/libraries/python/autogen/Regenerator.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2018 Cisco and/or its affiliates.
+# Copyright (c) 2019 Cisco and/or its affiliates.
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at:
@@ -13,16 +13,25 @@
 
 """Module defining utilities for test directory regeneration."""
 
+from __future__ import print_function
+
 from glob import glob
 from os import getcwd
+import sys
 
 from .DefaultTestcase import DefaultTestcase
 
 
+# Copied from https://stackoverflow.com/a/14981125
+def eprint(*args, **kwargs):
+    """Print to stderr."""
+    print(*args, file=sys.stderr, **kwargs)
+
+
 class Regenerator(object):
     """Class containing file generating methods."""
 
-    def __init__(self, testcase_class=DefaultTestcase):
+    def __init__(self, testcase_class=DefaultTestcase, quiet=True):
         """Initialize Testcase class to use.
 
         TODO: See the type doc for testcase_class?
@@ -31,9 +40,12 @@ class Regenerator(object):
 
         :param testcase_class: Subclass of DefaultTestcase for generation.
             Default: DefaultTestcase
+        :param quiet: Reduce log prints (to stderr) when True (default).
         :type testcase_class: subclass of DefaultTestcase accepting suite_id
+        :type quiet: boolean
         """
         self.testcase_class = testcase_class
+        self.quiet = quiet
 
     def regenerate_glob(self, pattern, protocol="ip4", tc_kwargs_list=None):
         """Regenerate files matching glob pattern based on arguments.
@@ -43,6 +55,8 @@ class Regenerator(object):
         test cases, autonumbering them, taking arguments from list.
         If the list is None, use default list, which depends on ip6 usage.
 
+        Log-like prints are emited to sys.stderr.
+
         :param pattern: Glob pattern to select files. Example: *-ndrpdr.robot
         :param is_ip6: Flag determining minimal frame size. Default: False
         :param tc_kwargs_list: Arguments defining the testcases. Default: None
@@ -140,7 +154,8 @@ class Regenerator(object):
                 num = add_testcase(testcase, iface, suite_id, file_out, num,
                                    **tc_kwargs)
 
-        print "Regenerator starts at {cwd}".format(cwd=getcwd())
+        if not self.quiet:
+            eprint("Regenerator starts at {cwd}".format(cwd=getcwd()))
         min_framesize = protocol_to_min_framesize[protocol]
         kwargs_list = tc_kwargs_list if tc_kwargs_list else [
             {"framesize": min_framesize, "phy_cores": 1},
@@ -157,7 +172,8 @@ class Regenerator(object):
             {"framesize": "IMIX_v4_1", "phy_cores": 4}
         ]
         for filename in glob(pattern):
-            print "Regenerating filename:", filename
+            if not self.quiet:
+                eprint("Regenerating filename:", filename)
             with open(filename, "r") as file_in:
                 text = file_in.read()
                 text_prolog = "".join(text.partition("*** Test Cases ***")[:-1])
@@ -166,5 +182,6 @@ class Regenerator(object):
             with open(filename, "w") as file_out:
                 file_out.write(text_prolog)
                 add_testcases(testcase, iface, suite_id, file_out, kwargs_list)
-        print "Regenerator ends."
-        print  # To make autogen check output more readable.
+        if not self.quiet:
+            eprint("Regenerator ends.")
+        eprint()  # To make autogen check output more readable.