Doc: CSIT Python2 to Python3 migration plan proposal

Change-Id: I0046ef57744d8d09117e850b60f9d65c7495ed07
Signed-off-by: Jan Gelety <jgelety@cisco.com>

1 files changed, 182 insertions, 0 deletions

diff --git a/docs/csit_python2_to_python3_migration.md b/docs/csit_python2_to_python3_migration.md
new file mode 100644
index 0000000000..8263f7aabe
--- /dev/null
+++ b/docs/csit_python2_to_python3_migration.md
@@ -0,0 +1,182 @@
+# FD.io CSIT migration Python 2.7 to Python 3
+
+## Python 3 version
+
+There is a pre-agreement to migrate to Python 3 version used by
+Ubuntu 18.04-LTS - currently it is version [3.6.8](https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-8-final).
+
+CentOS7 version 1810 that is used in [FD.io](https://fd.io/) also contains
+Python 3.6.
+
+## Dependency libs
+
+There was used *[caniusepython3](https://pypi.org/project/caniusepython3/)*
+tool to check readiness of current version of csit external libraries for
+Python 3. It identified one external library that needs to be updated to
+support Python 3:
+  ```
+ (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r requirements.txt
+ Finding and checking dependencies ...
+ You need 1 project to transition to Python 3.
+ Of that 1 project, 1 has no direct dependencies blocking its transition:
+   pypcap
+ (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r tox-requirements.txt 
+ Finding and checking dependencies ...
+ You have 0 projects blocking you from using Python 3!
+ (env) vpp@vpp-VirtualBox:~/Documents/csit$
+ ```
+
+The latest released version of *[pypcap](https://pypi.org/project/pypcap/)* is
+version 1.2.3 (Python 3 support implemented in version 1.2.0).
+
+Packages were checked for Python 3.6.8 support too and here are proposed
+package versions:
+
+- directly needed packages
+  - ecdsa==0.13.3
+  - paramiko==2.6.0
+  - pycrypto==2.6.1
+  - pypcap==1.2.3    # min. v1.2.0 for Python 3.6 support
+  - PyYAML==5.1
+  - requests==2.22.0 # min. v2.14.0 for Python 3.6 support
+  - robotframework==3.1.2
+  - scapy==2.4.3     # min. v2.4.0 for Python 3.6 support
+  - scp==0.13.2
+
+- directly needed packages for PLRSearch
+  - dill==0.3.1.1
+  - numpy==1.17.3    # v1.14.5 - compatibility with Python 3.6.2, possible
+    incompatibility with Python 3.6.8; v1.14.6 should be compatible with
+    Python 3.6.8
+  - scipy==1.3.1
+
+- directly needed packages for PAL
+  - hdrhistogram==0.6.1
+  - pandas==0.25.3
+  - plotly==4.1.1
+  - PTable==0.9.2
+  - Sphinx==2.2.1
+  - sphinx-rtd-theme==0.4.0
+  - sphinxcontrib-programoutput==0.15
+
+- packages needed by paramiko package
+  - bcrypt==3.1.7
+  - cffi==1.13.1
+  - cryptography==2.8
+  - pycparser==2.19
+  - PyNaCl==1.3.0
+  - six==1.12.0
+
+- packages needed by request package
+  - certifi==2019.9.11
+  - chardet==3.0.4
+  - idna==2.8
+  - urllib3==1.25.6
+
+- not needed anymore
+  - aenum - enum module in Python 3.6 already contains needed enum types
+  - ipaddress - module already included in Python 3.6
+  - pexpect - can be removed when corresponding unused code is removed from
+    ssh.py
+  - pykwalify + docpot + python-dateutil - can be removed if virl not used
+    anymore
+
+After discussion there is an agreement to use pip freeze for indirect
+dependencies when all direct dependency versions are resolved - see example of
+*[requirements.txt](https://gerrit.fd.io/r/c/csit/+/23207/17/requirements.txt)*
+file in CSIT gerrit commit
+[Python3: PIP requirement](https://gerrit.fd.io/r/c/csit/+/23207).
+
+## Required CSIT code changes
+
+There were identified following code changes that need to be addressed during
+Python 2.7 to Python 3 migration in CSIT:
+- imports relative to package
+  - `import submodul1` => `from . import submodule1`
+  - `from csv import my_csv` => `from .csv import my_csv`
+- StringIO
+  - `import StringIO` => `from io import StringIO`
+- `StandardError` -=> `Exception`
+- raising  exceptions - should be ready
+  - `raise ValueError, "wrong value"` => `raise ValueError("wrong value")`
+- catching exceptions - should be ready
+  - `except ValueError, e:` => `except ValueError as e:`
+- integers
+  - `long` => `int`
+- strings and bytes
+  - `unicode` => `str`
+  - `basestring` => `str`
+  - `str` => `bytes` - not generally, only if bytes type really required
+  - use following string style conventions:
+    - `u"a unicode string literal"`
+    - `b"a bytes string literal"`
+    - `f"a formatted unicode string literal"` - `f"He said his name is {name}"`
+       instead of `"He said his name is {n}".format(n=name)`
+- integer division with rounding down
+  - `2 / 3` =>  `2 // 3`
+- metaclasses - use only new style
+  - `class Form(BaseForm, metaclass=FormType):`
+- for-loop variables and the global namespace leak
+  - for-loop variables don't leak into the global namespace anymore
+- returning iterable objects instead of lists
+  - `xrange` => `range`
+  - `range` => `list(range())`
+  - `map` => `list(map())`
+  - `zip` => `list(zip())`
+  - `filter` => `list(filter())`
+  - dictionaries
+    - `.iteritems()` => `.items()`
+    - `.iterkeys()` => `.keys()`
+    - `.itervalues()` => `.values()`
+    - `.viewitems()` => `.items()`
+    - `.viewkeys()` => `.keys()`
+    - `.viewvalues()` => `.values()`
+    - `.items()`=> `list(.items())`
+    - `.keys()` => `list(.keys())`
+    - `.values()` => `list(.values())`
+    - `dict.has_key(key)` => `key in dict`
+  - lists
+    - `L = list(some_iterable); L.sort()` => `L = sorted(some_iterable)`
+    - parenthesis in list comprehensions
+      - `[... for var in item1, item2, ...]` => `[... for var in (item1, item2, ...)]`
+- file IO with `open`
+  - `f = open('myfile.txt')  # f.read() returns byte string` =>
+  `from io import open` plus
+    - `f = open('myfile.txt', 'rb')  # f.read() should return bytes`
+    - `f = open('myfile.txt', 'rt')  # f.read() should return unicode text`
+- reduce()
+  - `reduce()` => `from functools import reduce; reduce()`
+
+- python files in following directories:
+  - resources/libraries/python
+  - resources/tools
+  - resources/traffic_profiles/trex
+  - resources/traffic_scripts
+
+- check python calls in bash files:
+  - resources/libraries/bash/
+  - csit root directory
+
+## Migration steps
+
+1. Update all external libraries - week(s) before the week W
+1. Install agreed Python 3 version to all servers used by CSIT for test
+   execution - week(s) before the week W
+   1. vpp device servers - already done
+   1. performance testbeds - already done
+   1. jenkins executors - already done
+1. Freeze the CSIT master branch for one week for commits other then Python 2 to
+   Python 3 migration - week W
+   1. Create back up branch of actual master
+   1. Migrate libraries - work split between all available CSIT developers. Each
+      one will submit separate commit for review - csit-vpp-xxx verify jobs will
+      be failing at this phase so committers will need to overwrite verify
+      voting to be able to merged these commits.
+
+      TODO: provide separate spread sheet with listed libraries to be migrated
+      with the name of CSIT developer responsible for the migration of this
+      library.
+   1. Run jobs and tests of all of types when all libraries migrated to confirm
+      functionality or to catch bugs that needs to be fixed - iterate until
+      successful execution of all tests.
+1. Unfreeze the CSIT master branch.