aboutsummaryrefslogtreecommitdiffstats
path: root/README
blob: 792bdefd463b3be37dca8436ec8212027424bbda (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
1. OVERVIEW

   1) To implement a set of libraries for L4 protocol processing (UDP, TCP etc.)
   for both IPv4 and IPv6.

   The goal is to provide lightweight, high performance and highly adaptable
   implementation for L4(UDP, TCP etc.) protocol processing. The provided API
   are not planned to be compatible with BSD socket API. These libraries are
   supposed to be built on top of DPDK.

   Note: these libraries are not supposed to be a 'complete' network stack.

   Implementation of ARP, IP, ETHER, etc. layers and related routing tables,
   code for setup, manage and perform actual IO over underlying devices are
   all out of scope of these libraries.

   The only information these libraries need about the underlying devices:
   - supported HW offloads
   - MTU and L3/L2 addresses
   That allows the libraries to fill L2/L3 headers and mbuf metadata
   for the outgoing packets.

   These libraries should be developed in such manner, that they could be used
   independently from implementations of 2) and 3).

   2) To create VPP graph nodes, plugins etc using those libraries to implement
   a host stack.

   3) To create such mechanisms (netlink agents, packaging, etc) necessary
   to make the resulting host stack easily usable by existing non-vpp aware
   software.

   The library uses siphash logic from the below source
   https://github.com/veorq/SipHash

2. APIs

   TLDK provides three series of APIs:
   - TLDK native APIs, provided by libtle_l4p.
   - Posix APIs, provided by libtle_glue with PRELOAD compile macro.
   - Posix APIs with changed symbol names, provided by libtle_glue without PRELOAD macro.


3. INSTALLATION GUIDE

   - Original guide
   ----------------

   1) Obtain latest supported DPDK version and build it.
   (refer to http://dpdk.org for information how to download and build it).
   Currently supported(tested) DPDK versions: 16.11 LTS, 17.11 LTS, 18.02.
   2) Make sure that RTE_SDK and RTE_TARGET DPDK related environment variables
   are setup correctly.
   3) Go to the TLDK root directory and type: 'make all'.
   4) Run sample applications.

   For enabling unit tests application using GoogleTest please see:
   ./test/gtest/README

   As an example:
   export RTE_SDK=/opt/DPDK
   export RTE_TARGET=x86_64-native-linuxapp-gcc

   cd tldk
   make all
   ./x86_64-native-linuxapp-gcc/app/l4fwd ...


   - For preload use
   -----------------

   Debug:

     $ make DPDK_DEBUG=y EXTRA_CFLAGS="-g -O0 -fPIC -DPRELOAD" all

   Release:

     $ make EXTRA_CFLAGS="-g -fPIC -DPRELOAD" all

   - For TLDK API use
   ------------------

   Debug:

     $ make DPDK_DEBUG=y EXTRA_CFLAGS="-g -O0 -fPIC" all

   Release:

     $ make EXTRA_CFLAGS="-g -O3 -fPIC" all

3. CONTENTS

   $(TLDK_ROOT)
   |
   +----app
   |      |
   |      +-- nginx - a clone of nginx integrated with TLDK
   |          (refer to app/nginx/README.TLDK for more information)
   |
   +----lib
   |      |
   |      +--libtle_dring - dring library
   |      |
   |      +--libtle_l4p - implementation of the TCP/UDP packet processing
   |      |
   |      +--libtle_timer - implementation of the timer library
   |      |
   |      +--libtle_glue - socket glue layer with arp, icmp, epoll, etc
   |
   +----examples
   |      |
   |      +--l4fwd - sample app to demonstrate and test libtle_l4p TCP/UDP
   |            usage (refer to examples/l4fwd/README for more information)
   |
   +----test - unit-tests
   |      |
   |      +--dring - UT for libtle_dring (standalone app)
   |      |
   |      +--gtest - UT for libtle_dring, libtle_l4p and libtle_timer
   |      |     (googletest)
   |      |
   |      +--timer - UT for libtle_timer (standalone app)
   |      |
   |      +--packetdrill - UT for stack (standalone app)


5. Features

   Done:
   - posix interface
   - loopback device
   - regression test
   - multi-thread
   - lightweight mem
   - tcp_info (paritial)
   - fd management
   - arp request/reply
   - icmp reply
   - interrupt mode
   - blocking recv/send
   - TSO
   - UFO

   TODO:
   - fuzzing
   - SACK
   - RACK
   - zerocopy APIs
   - batching APIs
   - multi-process
   - numa awareness
   - context recycle on thread exit

5. Thread model

   - Multi-process is still not fully supported.

   - Symmetric multi-thread

     (app thread)         (app thread)          (app thread)
           \                    \                     \
           /                    /                     /
           \                    \                     \
      --------------------------------------------------------
      |            FD management, Socket APIs (FE)           |
      --------------------------------------------------------

      -----------           -----------           -----------
      |         |           |         |           |         |
      |   ctx   |           |   ctx   |           |   ctx   |
      |         |           |         |           |         |
      -----------           -----------           -----------
            \__                  |                   __/
               \__               |                __/
                  \__            |             __/
                     \__         |          __/
                       -------------------------
                       |    (RSS) NIC (FDIR)   |
                       -------------------------

   - Lookaside multi-thread

      (app thread)         (app thread)          (io thread)
           \                     \                    \
           /                     /                    /
           \                     \                    \
      ------------------------------------------------------
      |            FD management, Socket APIs (FE)         |
      ------------------------------------------------------
                                                      /
                                                      \
                                                      /
      ------------------------------------------------------
      |                                                    |
      |                        ctx                         |
      |                                                    |
      ------------------------------------------------------
                                |
                                |
                    -------------------------
                    |          NIC          |
                    -------------------------

6. How to run

   We have two setups which need their own preparation.

    - virtio-user: test with virtio-user + vhost-kernel;
    - physical NIC: test with physical NIC bound to vfio.

   If you are using physical NIC:
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   a. Set Linux boot options (Only needed if you will use physical NIC)
     a1. Add below configuration into GRUB_CMDLINE_LINUX in /etc/default/grub
       "intel_iommu=on iommu=pt"

     a2. Update grub
       $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

     If you want to use 1GB hugepages, you can also add below content in the
     boot cmdline:
       "default_hugepagesz=1G hugepagesz=1G hugepages=2"

   b. Adjust RLIMIT_MEMLOCK (Only needed if you will use physical NIC)
     Add below two lines into /etc/security/limits.conf
       "* soft memlock 4194304
        * hard memlock 4194304"

   c. Reboot system

   d. Bind NIC to vfio-pci

     $ sudo modprobe vfio-pci
     $ sudo ./usertools/dpdk-devbind.py -b vfio-pci 0000:01:00.1
     $ sudo chmod 666 /dev/vfio/16 (16 is just an example)

   If you are using virtio-user:
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   a. Prepare vhost-kernel

     $ sudo modprobe vhost-net
     (if you don't have those modules, have to compile by yourself)
     $ sudo chmod 666 /dev/vhost-net
     $ sudo tunctl -u <your username>

   b. Prepare the vNIC

     $ export DPDK_VNIC="--vdev=virtio_user0,path=/dev/vhost-net,queue_size=1024,iface=tap0"

   For both cases, we need to:
   ~~~~~~~~~~~~~~~~~~~~~~~~~~

     $ sudo chmod 777 /dev/hugepages
     $ export DPDK_IP=1.1.1.1

   Note: for specific test example, you can refer to the example commit log.

7. How to run packetdrill tests:

  Compile it in LOOK_ASIDE_BACKEND mode:

    $ make PACKETDRILL=y EXTRA_CFLAGS="-g -O0 -fPIC -march=native -DLOOK_ASIDE_BACKEND " all

  To run it:

    $ test/packetdrill/packetdrill --local_ip=192.168.0.2 \
        --remote_ip=192.0.2.1 --so_filename=`pwd`/libtldk.so \
        test/packetdrill/tests/tldk/fast_retransmit/fr-4pkt-tldk.pkt

8. Tested Examples

   - examples/client
   - examples/server
   - wget (epoll)
   - curl (poll)
   - haproxy (multi-thread mode)