1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
|
# FD.io CSIT migration Python 2.7 to Python 3
## Python 3 version
There is a pre-agreement to migrate to Python 3 version used by
Ubuntu 18.04-LTS - currently it is version [3.6.8](https://docs.python.org/3.6/whatsnew/changelog.html#python-3-6-8-final).
CentOS7 version 1810 that is used in [FD.io](https://fd.io/) also contains
Python 3.6.
## Dependency libs
There was used *[caniusepython3](https://pypi.org/project/caniusepython3/)*
tool to check readiness of current version of csit external libraries for
Python 3. It identified one external library that needs to be updated to
support Python 3:
```
(env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r requirements.txt
Finding and checking dependencies ...
You need 1 project to transition to Python 3.
Of that 1 project, 1 has no direct dependencies blocking its transition:
pypcap
(env) vpp@vpp-VirtualBox:~/Documents/csit$ caniusepython3 -r tox-requirements.txt
Finding and checking dependencies ...
You have 0 projects blocking you from using Python 3!
(env) vpp@vpp-VirtualBox:~/Documents/csit$
```
The latest released version of *[pypcap](https://pypi.org/project/pypcap/)* is
version 1.2.3 (Python 3 support implemented in version 1.2.0).
Packages were checked for Python 3.6.8 support too and here are proposed
package versions:
- directly needed packages
- ecdsa==0.13.3
- paramiko==2.6.0
- pycrypto==2.6.1
- pypcap==1.2.3 # min. v1.2.0 for Python 3.6 support
- PyYAML==5.1
- requests==2.22.0 # min. v2.14.0 for Python 3.6 support
- robotframework==3.1.2
- scapy==2.4.3 # min. v2.4.0 for Python 3.6 support
- scp==0.13.2
- directly needed packages for PLRSearch
- dill==0.3.1.1
- numpy==1.17.3 # v1.14.5 - compatibility with Python 3.6.2, possible
incompatibility with Python 3.6.8; v1.14.6 should be compatible with
Python 3.6.8
- scipy==1.3.1
- directly needed packages for PAL
- hdrhistogram==0.6.1
- pandas==0.25.3
- plotly==4.1.1
- PTable==0.9.2
- Sphinx==2.2.1
- sphinx-rtd-theme==0.4.0
- sphinxcontrib-programoutput==0.15
- packages needed by paramiko package
- bcrypt==3.1.7
- cffi==1.13.1
- cryptography==2.8
- pycparser==2.19
- PyNaCl==1.3.0
- six==1.12.0
- packages needed by request package
- certifi==2019.9.11
- chardet==3.0.4
- idna==2.8
- urllib3==1.25.6
- not needed anymore
- aenum - enum module in Python 3.6 already contains needed enum types
- ipaddress - module already included in Python 3.6
- pexpect - can be removed when corresponding unused code is removed from
ssh.py
- pykwalify + docpot + python-dateutil - can be removed if virl not used
anymore
After discussion there is an agreement to use pip freeze for indirect
dependencies when all direct dependency versions are resolved - see example of
*[requirements.txt](https://gerrit.fd.io/r/c/csit/+/23207/17/requirements.txt)*
file in CSIT gerrit commit
[Python3: PIP requirement](https://gerrit.fd.io/r/c/csit/+/23207).
## Required CSIT code changes
There were identified following code changes that need to be addressed during
Python 2.7 to Python 3 migration in CSIT:
- imports relative to package
- `import submodul1` => `from . import submodule1`
- `from csv import my_csv` => `from .csv import my_csv`
- StringIO
- `import StringIO` => `from io import StringIO`
- `StandardError` -=> `Exception`
- raising exceptions - should be ready
- `raise ValueError, "wrong value"` => `raise ValueError("wrong value")`
- catching exceptions - should be ready
- `except ValueError, e:` => `except ValueError as e:`
- integers
- `long` => `int`
- strings and bytes
- `unicode` => `str`
- `basestring` => `str`
- `str` => `bytes` - not generally, only if bytes type really required
- use following string style conventions:
- `u"a unicode string literal"`
- `b"a bytes string literal"`
- `f"a formatted unicode string literal"` - `f"He said his name is {name}"`
instead of `"He said his name is {n}".format(n=name)`
- integer division with rounding down
- `2 / 3` => `2 // 3`
- metaclasses - use only new style
- `class Form(BaseForm, metaclass=FormType):`
- for-loop variables and the global namespace leak
- for-loop variables don't leak into the global namespace anymore
- returning iterable objects instead of lists
- `xrange` => `range`
- `range` => `list(range())`
- `map` => `list(map())`
- `zip` => `list(zip())`
- `filter` => `list(filter())`
- dictionaries
- `.iteritems()` => `.items()`
- `.iterkeys()` => `.keys()`
- `.itervalues()` => `.values()`
- `.viewitems()` => `.items()`
- `.viewkeys()` => `.keys()`
- `.viewvalues()` => `.values()`
- `.items()`=> `list(.items())`
- `.keys()` => `list(.keys())`
- `.values()` => `list(.values())`
- `dict.has_key(key)` => `key in dict`
- lists
- `L = list(some_iterable); L.sort()` => `L = sorted(some_iterable)`
- parenthesis in list comprehensions
- `[... for var in item1, item2, ...]` => `[... for var in (item1, item2, ...)]`
- file IO with `open`
- `f = open('myfile.txt') # f.read() returns byte string` =>
`from io import open` plus
- `f = open('myfile.txt', 'rb') # f.read() should return bytes`
- `f = open('myfile.txt', 'rt') # f.read() should return unicode text`
- reduce()
- `reduce()` => `from functools import reduce; reduce()`
- python files in following directories:
- resources/libraries/python
- resources/tools
- resources/traffic_profiles/trex
- resources/traffic_scripts
- check python calls in bash files:
- resources/libraries/bash/
- csit root directory
## Migration steps
1. Update all external libraries - week(s) before the week W
1. Install agreed Python 3 version to all servers used by CSIT for test
execution - week(s) before the week W
1. vpp device servers - already done
1. performance testbeds - already done
1. jenkins executors - already done
1. Freeze the CSIT master branch for one week for commits other then Python 2 to
Python 3 migration - week W
1. Create back up branch of actual master
1. Migrate libraries - work split between all available CSIT developers. Each
one will submit separate commit for review - csit-vpp-xxx verify jobs will
be failing at this phase so committers will need to overwrite verify
voting to be able to merged these commits.
TODO: provide separate spread sheet with listed libraries to be migrated
with the name of CSIT developer responsible for the migration of this
library.
1. Run jobs and tests of all of types when all libraries migrated to confirm
functionality or to catch bugs that needs to be fixed - iterate until
successful execution of all tests.
1. Unfreeze the CSIT master branch.
|
