1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
|
RDMA (ibverb) device driver
===========================
This driver relies on Linux rdma-core (libibverb) userspace poll-mode
driver to rx/tx Ethernet packets. Despite using the RDMA APIs, this is
**not** about RDMA (no Infiniband, no RoCE, no iWARP), just pure
traditional Ethernet packets.
Maturity level
--------------
Under development: it should work, but has not been thoroughly tested.
Supported Hardware
------------------
- Mellanox ConnectX-4
- Mellanox ConnectX-5
Features
--------
- bifurcation: MAC based flow steering for transparent sharing of a
single physical port between multiple virtual interfaces including
Linux netdev
- multiqueue
Security considerations
-----------------------
When creating a rdma interface, it will receive all packets to the MAC
address attributed to the interface plus a copy of all broadcast and
multicast traffic. The MAC address is under the control of VPP: **the
user controlling VPP can divert all traffic of any MAC address to the
VPP process, including the Linux netdev MAC address as long as it can
create a rdma interface**. The rights to create a rdma interface are
controlled by the access rights of the
``/dev/infiniband/uverbs[0-9]+``\ device nodes.
Quickstart
----------
1. Make sure the ``ib_uverbs`` module is loaded:
::
~# modprobe ib_uverbs
2. In VPP, create a new rdma virtual interface tied to the Linux netdev
of the physical port you want to use (``enp94s0f0`` in this example):
::
vpp# create int rdma host-if enp94s0f0 name rdma-0
3. Use the interface as usual, e.g.:
::
vpp# set int ip addr rdma-0 1.1.1.1/24
vpp# set int st rdma-0 up
vpp# ping 1.1.1.100`
Containers support
------------------
It should work in containers as long as: - the ``ib_uverbs`` module is
loaded - the device nodes ``/dev/infiniband/uverbs[0-9]+`` are usable
from the container (but see `security
considerations <#Security%20considerations>`__)
SR-IOV VFs support
------------------
It should work on SR-IOV VFs the same way it does with PFs. Because of
VFs security containment features, make sure the MAC address of the rdma
VPP interface matches the MAC address assigned to the underlying VF. For
example:
::
host# echo 1 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs
host# ip l set dev enp94s0f0 vf 0 mac 92:5d:f5:df:b1:6f spoof on trust off
host# ip l set dev enp94s0f2 up
vpp# create int rdma host-if enp94s0f2 name rdma-0
vpp# set int mac address rdma-0 92:5d:f5:df:b1:6f
If you plan to use L2 features such as switching, make sure the
underlying VF is configured in trusted mode and spoof-checking is
disabled (of course, be aware of the `security
considerations <#Security%20considerations>`__):
::
host# ip l set dev enp94s0f0 vf 0 spoof off trust on
Direct Verb mode
----------------
Direct Verb allows the driver to access the NIC HW RX/TX rings directly
instead of having to go through libibverb and suffering associated
overhead. It will be automatically selected if the adapter supports it.
|
