# FD.io CSIT migration Python 2.7 to Python 3

## Python 3 version

There is a pre-agreement to migrate to Python 3 version used by
Ubuntu 18.04-LTS - currently it is version [3.6.8](https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-8-final).

CentOS7 version 1810 that is used in [FD.io](https://fd.io/) also contains
Python 3.6.

## Dependency libs

There was used *[caniusepython3](https://pypi.org/project/caniusepython3/)*
tool to check readiness of current version of csit external libraries for
Python 3. It identified one external library that needs to be updated to
support Python 3:
  ```
 (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r requirements.txt
 Finding and checking dependencies ...
 You need 1 project to transition to Python 3.
 Of that 1 project, 1 has no direct dependencies blocking its transition:
   pypcap
 (env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r tox-requirements.txt 
 Finding and checking dependencies ...
 You have 0 projects blocking you from using Python 3!
 (env) vpp@vpp-VirtualBox:~/Documents/csit$
 ```

The latest released version of *[pypcap](https://pypi.org/project/pypcap/)* is
version 1.2.3 (Python 3 support implemented in version 1.2.0).

Packages were checked for Python 3.6.8 support too and here are proposed
package versions:

- directly needed packages
  - ecdsa==0.13.3
  - paramiko==2.6.0
  - pycrypto==2.6.1
  - pypcap==1.2.3    # min. v1.2.0 for Python 3.6 support
  - PyYAML==5.1
  - requests==2.22.0 # min. v2.14.0 for Python 3.6 support
  - robotframework==3.1.2
  - scapy==2.4.3     # min. v2.4.0 for Python 3.6 support
  - scp==0.13.2

- directly needed packages for PLRSearch
  - dill==0.3.1.1
  - numpy==1.17.3    # v1.14.5 - compatibility with Python 3.6.2, possible
    incompatibility with Python 3.6.8; v1.14.6 should be compatible with
    Python 3.6.8
  - scipy==1.3.1

- directly needed packages for PAL
  - hdrhistogram==0.6.1
  - pandas==0.25.3
  - plotly==4.1.1
  - PTable==0.9.2
  - Sphinx==2.2.1
  - sphinx-rtd-theme==0.4.0
  - sphinxcontrib-programoutput==0.15

- packages needed by paramiko package
  - bcrypt==3.1.7
  - cffi==1.13.1
  - cryptography==2.8
  - pycparser==2.19
  - PyNaCl==1.3.0
  - six==1.12.0

- packages needed by request package
  - certifi==2019.9.11
  - chardet==3.0.4
  - idna==2.8
  - urllib3==1.25.6

- not needed anymore
  - aenum - enum module in Python 3.6 already contains needed enum types
  - ipaddress - module already included in Python 3.6
  - pexpect - can be removed when corresponding unused code is removed from
    ssh.py
  - pykwalify + docpot + python-dateutil - can be removed if virl not used
    anymore

After discussion there is an agreement to use pip freeze for indirect
dependencies when all direct dependency versions are resolved - see example of
*[requirements.txt](https://gerrit.fd.io/r/c/csit/+/23207/17/requirements.txt)*
file in CSIT gerrit commit
[Python3: PIP requirement](https://gerrit.fd.io/r/c/csit/+/23207).

## Required CSIT code changes

There were identified following code changes that need to be addressed during
Python 2.7 to Python 3 migration in CSIT:
- imports relative to package
  - `import submodul1` => `from . import submodule1`
  - `from csv import my_csv` => `from .csv import my_csv`
- StringIO
  - `import StringIO` => `from io import StringIO`
- `StandardError` -=> `Exception`
- raising  exceptions - should be ready
  - `raise ValueError, "wrong value"` => `raise ValueError("wrong value")`
- catching exceptions - should be ready
  - `except ValueError, e:` => `except ValueError as e:`
- integers
  - `long` => `int`
- strings and bytes
  - `unicode` => `str`
  - `basestring` => `str`
  - `str` => `bytes` - not generally, only if bytes type really required
  - use following string style conventions:
    - `u"a unicode string literal"`
    - `b"a bytes string literal"`
    - `f"a formatted unicode string literal"` - `f"He said his name is {name}"`
       instead of `"He said his name is {n}".format(n=name)`
- integer division with rounding down
  - `2 / 3` =>  `2 // 3`
- metaclasses - use only new style
  - `class Form(BaseForm, metaclass=FormType):`
- for-loop variables and the global namespace leak
  - for-loop variables don't leak into the global namespace anymore
- returning iterable objects instead of lists
  - `xrange` => `range`
  - `range` => `list(range())`
  - `map` => `list(map())`
  - `zip` => `list(zip())`
  - `filter` => `list(filter())`
  - dictionaries
    - `.iteritems()` => `.items()`
    - `.iterkeys()` => `.keys()`
    - `.itervalues()` => `.values()`
    - `.viewitems()` => `.items()`
    - `.viewkeys()` => `.keys()`
    - `.viewvalues()` => `.values()`
    - `.items()`=> `list(.items())`
    - `.keys()` => `list(.keys())`
    - `.values()` => `list(.values())`
    - `dict.has_key(key)` => `key in dict`
  - lists
    - `L = list(some_iterable); L.sort()` => `L = sorted(some_iterable)`
    - parenthesis in list comprehensions
      - `[... for var in item1, item2, ...]` => `[... for var in (item1, item2, ...)]`
- file IO with `open`
  - `f = open('myfile.txt')  # f.read() returns byte string` =>
  `from io import open` plus
    - `f = open('myfile.txt', 'rb')  # f.read() should return bytes`
    - `f = open('myfile.txt', 'rt')  # f.read() should return unicode text`
- reduce()
  - `reduce()` => `from functools import reduce; reduce()`

- python files in following directories:
  - resources/libraries/python
  - resources/tools
  - resources/traffic_profiles/trex
  - resources/traffic_scripts

- check python calls in bash files:
  - resources/libraries/bash/
  - csit root directory

## Migration steps

1. Update all external libraries - week(s) before the week W
1. Install agreed Python 3 version to all servers used by CSIT for test
   execution - week(s) before the week W
   1. vpp device servers - already done
   1. performance testbeds - already done
   1. jenkins executors - already done
1. Freeze the CSIT master branch for one week for commits other then Python 2 to
   Python 3 migration - week W
   1. Create back up branch of actual master
   1. Migrate libraries - work split between all available CSIT developers. Each
      one will submit separate commit for review - csit-vpp-xxx verify jobs will
      be failing at this phase so committers will need to overwrite verify
      voting to be able to merged these commits.

      TODO: provide separate spread sheet with listed libraries to be migrated
      with the name of CSIT developer responsible for the migration of this
      library.
   1. Run jobs and tests of all of types when all libraries migrated to confirm
      functionality or to catch bugs that needs to be fixed - iterate until
      successful execution of all tests.
1. Unfreeze the CSIT master branch.