aboutsummaryrefslogtreecommitdiffstats
path: root/docs/usecases/container_test.rst
blob: 8ad6285f8ed84fb9657641b544f348160f43999e (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
Simulating networks with VPP
============================

The “make test” framework provides a good way to test individual
features. However, when testing several features at once - or validating
nontrivial configurations - it may prove difficult or impossible to use
the unit-test framework.

This note explains how to set up lxc/lxd, and a 5-container testbed to
test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation
scenario.

OS / Distro test results
------------------------

This setup has been tested on an Ubuntu 18.04 LTS system. If you’re
feeling adventurous, the same scenario also worked on a recent Ubuntu
20.04 “preview” daily build.

Other distros may work fine, or not at all.

Proxy Server
------------

If you need to use a proxy server e.g. from a lab system, you’ll
probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and https_proxy
in /etc/environment. Directly setting variables in the environment
doesn’t work. The lxd snap *daemon* needs the proxy settings, not the
user interface.

Something like so:

::

       HTTP_PROXY=http://my.proxy.server:8080
       HTTPS_PROXY=http://my.proxy.server:4333
       http_proxy=http://my.proxy.server:8080
       https_proxy=http://my.proxy.server:4333

Install and configure lxd
-------------------------

Install the lxd snap. The lxd snap is up to date, as opposed to the
results of “sudo apt-get install lxd”.

::

       # snap install lxd
       # lxd init

“lxd init” asks several questions. With the exception of the storage
pool, take the defaults. To match the configs shown below, create a
storage pool named “vpp.” Storage pools of type “zfs” and “files” have
been tested successfully.

zfs is more space-efficient. “lxc copy” is infinitely faster with zfs.
The path for the zfs storage pool is under /var. Do not replace it with
a symbolic link, unless you want to rebuild all of your containers from
scratch. Ask me how I know that.

Create three network segments
-----------------------------

Aka, linux bridges.

::

       # lxc network create respond
       # lxc network create internet
       # lxc network create initiate

We’ll explain the test topology in a bit. Stay tuned.

Set up the default container profile
------------------------------------

Execute “lxc profile edit default”, and install the following
configuration. Note that the “shared” directory should mount your vpp
workspaces. With that trick, you can edit code from any of the
containers, run vpp without installing it, etc.

::

       config: {}
       description: Default LXD profile
       devices:
         eth0:
           name: eth0
           network: lxdbr0
           type: nic
         eth1:
           name: eth1
           nictype: bridged
           parent: internet
           type: nic
         eth2:
           name: eth2
           nictype: bridged
           parent: respond
           type: nic
         eth3:
           name: eth3
           nictype: bridged
           parent: initiate
           type: nic
         root:
           path: /
           pool: vpp
           type: disk
         shared:
           path: /scratch
           source: /scratch
           type: disk
       name: default

Set up the network configurations
---------------------------------

Edit the fake “internet” backbone:

::

     # lxc network edit internet

Install the ip addresses shown below, to avoid having to rebuild the vpp
and host configuration:

::

       config:
         ipv4.address: 10.26.68.1/24
         ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50
         ipv4.nat: "true"
         ipv6.address: none
         ipv6.nat: "false"
       description: ""
       name: internet
       type: bridge
       used_by:
       managed: true
       status: Created
       locations:
       - none

Repeat the process with the “respond” and “initiate” networks, using
these configurations:

respond network configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

       config:
         ipv4.address: 10.166.14.1/24
         ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50
         ipv4.nat: "true"
         ipv6.address: none
         ipv6.nat: "false"
       description: ""
       name: respond
       type: bridge
       used_by:
       managed: true
       status: Created
       locations:
       - none

initiate network configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

       config:
         ipv4.address: 10.219.188.1/24
         ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50
         ipv4.nat: "true"
         ipv6.address: none
         ipv6.nat: "false"
       description: ""
       name: initiate
       type: bridge
       used_by:
       managed: true
       status: Created
       locations:
       - none

Create a “master” container image
---------------------------------

The master container image should be set up so that you can build vpp,
ssh into the container, edit source code, run gdb, etc.

Make sure that e.g. public key auth ssh works.

::

       # lxd launch ubuntu:18.04 respond
       <spew>
       # lxc exec respond bash
       respond# cd /scratch/my-vpp-workspace
       respond# apt-get install make ssh
       respond# make install-dep
       respond# exit
       # lxc stop respond

Mark the container image privileged. If you forget this step, you’ll
trip over a netlink error (-11) aka EAGAIN when you try to roll in the
vpp configurations.

::

       # lxc config set respond security.privileged "true"

Duplicate the “master” container image
--------------------------------------

To avoid having to configure N containers, be sure that the master
container image is fully set up before you help it have children:

::

       # lxc copy respond respondhost
       # lxc copy respond initiate
       # lxc copy respond initiatehost
       # lxc copy respond dhcpserver    # optional, to test ipv6 prefix delegation

Install handy script
--------------------

See below for a handy script which executes lxc commands across the
current set of running containers. I call it “lxc-foreach,” feel free to
call the script Ishmael if you like.

Examples:

::

       $ lxc-foreach start
       <issues "lxc start" for each container in the list>

After a few seconds, use this one to open an ssh connection to each
container. The ssh command parses the output of “lxc info,” which
displays container ip addresses.

::

       $ lxc-foreach ssh

Here’s the script:

::

       #!/bin/bash

       set -u
       export containers="respond respondhost initiate initiatehost dhcpserver"

       if [ x$1 = "x" ] ; then
           echo missing command
           exit 1
       fi

       if [ $1 = "ssh" ] ; then
           for c in $containers
           do
               inet=`lxc info $c | grep eth0 | grep -v inet6 | head -1 | cut -f 3`
               if [ x$inet = "x" ] ; then
                   echo $c not started
               else
                   gnome-terminal --command "/usr/bin/ssh $inet"
               fi
           done
       exit 0
       fi

       for c in $containers
       do
           echo lxc $1 $c
           lxc $1 $c
       done

       exit 0

Test topology
-------------

Finally, we’re ready to describe a test topology. First, a picture:

::

       ===+======== management lan/bridge lxdbr0 (dhcp) ===========+===
          |                             |                          |
          |                             |                          |
          |                             |                          |
          v                             |                          v
         eth0                           |                         eth0
       +------+ eth1                                       eth1 +------+
       | respond | 10.26.88.100 <= internet bridge => 10.26.88.101 | initiate |
       +------+                                                 +------+
         eth2 / bvi0 10.166.14.2        |       10.219.188.2 eth3 / bvi0
          |                             |                          |
          | ("respond" bridge)             |          ("initiate" bridge) |
          |                             |                          |
          v                             |                          v
         eth2 10.166.14.3               |           eth3 10.219.188.3
       +----------+                     |                   +----------+
       | respondhost |                     |                   | respondhost |
       +----------+                     |                   +----------+
         eth0 (management lan) <========+========> eth0 (management lan)

Test topology discussion
~~~~~~~~~~~~~~~~~~~~~~~~

This topology is suitable for testing almost any tunnel encap/decap
scenario. The two containers “respondhost” and “initiatehost” are
end-stations connected to two vpp instances running on “respond” and
“initiate”.

We leverage the Linux end-station network stacks to generate traffic of
all sorts.

The so-called “internet” bridge models the public internet. The
“respond” and “initiate” bridges connect vpp instances to local hosts

End station configs
-------------------

The end-station Linux configurations set up the eth2 and eth3 ip
addresses shown above, and add tunnel routes to the opposite end-station
networks.

respondhost configuration
~~~~~~~~~~~~~~~~~~~~~~~~~

::

       ifconfig eth2 10.166.14.3/24 up
       route add -net 10.219.188.0/24 gw 10.166.14.2

initiatehost configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~

::

       sudo ifconfig eth3 10.219.188.3/24 up
       sudo route add -net 10.166.14.0/24 gw 10.219.188.2

VPP configs
-----------

Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in
the “respond” config.

respond configuration
~~~~~~~~~~~~~~~~~~~~~

::

       set term pag off

       comment { "internet" }
       create host-interface name eth1
       set int ip address host-eth1 10.26.68.100/24
       set int ip6 table host-eth1 0
       set int state host-eth1 up

       comment { default route via initiate }
       ip route add 0.0.0.0/0 via 10.26.68.101

       comment { "respond-private-net" }
       create host-interface name eth2
       bvi create instance 0
       set int l2 bridge bvi0 1 bvi
       set int ip address bvi0 10.166.14.2/24
       set int state bvi0 up
       set int l2 bridge host-eth2 1
       set int state host-eth2 up


       nat44 add interface address host-eth1
       set interface nat44 in host-eth2 out host-eth1
       nat44 add identity mapping external host-eth1 udp 500
       nat44 add identity mapping external host-eth1 udp 4500
       comment { nat44 untranslated subnet 10.219.188.0/24 }

       comment { responder profile }
       ikev2 profile add initiate
       ikev2 profile set initiate udp-encap
       ikev2 profile set initiate auth rsa-sig cert-file /scratch/setups/respondcert.pem
       set ikev2 local key /scratch/setups/initiatekey.pem
       ikev2 profile set initiate id local fqdn initiator.my.net
       ikev2 profile set initiate id remote fqdn responder.my.net
       ikev2 profile set initiate traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
       ikev2 profile set initiate traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
       create ipip tunnel src 10.26.68.100 dst 10.26.68.101
       ikev2 profile set initiate tunnel ipip0

       comment { ipv6 prefix delegation }
       ip6 nd address autoconfig host-eth1 default-route
       dhcp6 client host-eth1
       dhcp6 pd client host-eth1 prefix group hgw
       set ip6 address bvi0 prefix group hgw ::2/56
       ip6 nd address autoconfig bvi0 default-route
       ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180

       set int mtu packet 1390 ipip0
       set int unnum ipip0 use host-eth1
       ip route add 10.219.188.0/24 via ipip0

initiate configuration
~~~~~~~~~~~~~~~~~~~~~~

::

       set term pag off

       comment { "internet" }
       create host-interface name eth1
       comment { set dhcp client intfc host-eth1 hostname initiate }
       set int ip address host-eth1 10.26.68.101/24
       set int state host-eth1 up

       comment { default route via "internet gateway" }
       comment { ip route add 0.0.0.0/0 via 10.26.68.1 }

       comment { "initiate-private-net" }
       create host-interface name eth3
       bvi create instance 0
       set int l2 bridge bvi0 1 bvi
       set int ip address bvi0 10.219.188.2/24
       set int state bvi0 up
       set int l2 bridge host-eth3 1
       set int state host-eth3 up

       nat44 add interface address host-eth1
       set interface nat44 in bvi0 out host-eth1
       nat44 add identity mapping external host-eth1 udp 500
       nat44 add identity mapping external host-eth1 udp 4500
       comment { nat44 untranslated subnet 10.166.14.0/24 }

       comment { initiator profile }
       ikev2 profile add respond
       ikev2 profile set respond udp-encap
       ikev2 profile set respond auth rsa-sig cert-file /scratch/setups/initiatecert.pem
       set ikev2 local key /scratch/setups/respondkey.pem
       ikev2 profile set respond id local fqdn responder.my.net
       ikev2 profile set respond id remote fqdn initiator.my.net

       ikev2 profile set respond traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
       ikev2 profile set respond traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0

       ikev2 profile set respond responder host-eth1 10.26.68.100
       ikev2 profile set respond ike-crypto-alg aes-cbc 256  ike-integ-alg sha1-96  ike-dh modp-2048
       ikev2 profile set respond esp-crypto-alg aes-cbc 256  esp-integ-alg sha1-96  esp-dh ecp-256
       ikev2 profile set respond sa-lifetime 3600 10 5 0

       create ipip tunnel src 10.26.68.101 dst 10.26.68.100
       ikev2 profile set respond tunnel ipip0
       ikev2 initiate sa-init respond

       set int mtu packet 1390 ipip0
       set int unnum ipip0 use host-eth1
       ip route add 10.166.14.0/24 via ipip0

IKEv2 certificate setup
-----------------------

In both of the vpp configurations, you’ll see “/scratch/setups/xxx.pem”
mentioned. These certificates are used in the ikev2 key exchange.

Here’s how to generate the certificates:

::

       openssl req -x509 -nodes -newkey rsa:4096 -keyout respondkey.pem -out respondcert.pem -days 3560
       openssl x509 -text -noout -in respondcert.pem
       openssl req -x509 -nodes -newkey rsa:4096 -keyout initiatekey.pem -out initiatecert.pem -days 3560
       openssl x509 -text -noout -in initiatecert.pem

Make sure that the “respond” and “initiate” configurations point to the
certificates.

DHCPv6 server setup
-------------------

If you need an ipv6 dhcp server to test ipv6 prefix delegation, create
the “dhcpserver” container as shown above.

Install the “isc-dhcp-server” Debian package:

::

       sudo apt-get install isc-dhcp-server

/etc/dhcp/dhcpd6.conf
~~~~~~~~~~~~~~~~~~~~~

Edit the dhcpv6 configuration and add an ipv6 subnet with prefix
delegation. For example:

::

       subnet6 2001:db01:0:1::/64 {
               range6 2001:db01:0:1::1 2001:db01:0:1::9;
               prefix6 2001:db01:0:100:: 2001:db01:0:200::/56;
       }

Add an ipv6 address on eth1, which is connected to the “internet”
bridge, and start the dhcp server. I use the following trivial bash
script, which runs the dhcp6 server in the foreground and produces dhcp
traffic spew:

::

       #!/bin/bash
       ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true
       dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf

The “\|\| true” bit keeps going if eth1 already has the indicated ipv6
address.

Container / Host Interoperation
-------------------------------

Host / container interoperation is highly desirable. If the host and a
set of containers don’t run the same distro *and distro version*, it’s
reasonably likely that the glibc versions won’t match. That, in turn,
makes vpp binaries built in one environment fail in the other.

Trying to install multiple versions of glibc - especially at the host
level - often ends very badly and is *not recommended*. It’s not just
glibc, either. The dynamic loader ld-linux-xxx-so.2 is glibc version
specific.

Fortunately, it’s reasonable easy to build lxd container images based on
specific Ubuntu or Debian versions.

Create a custom root filesystem image
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

First, install the “debootstrap” tool:

::

       sudo apt-get install debootstrap

Make a temp directory, and use debootstrap to populate it. In this
example, we create an Ubuntu 20.04 (focal fossa) base image:

::

       # mkdir /tmp/myroot
       # debootstrap focal /tmp/myroot http://archive.ubuntu.com/ubuntu

To tinker with the base image (if desired):

::

       # chroot /tmp/myroot
       <add packages, etc.>
       # exit

Make a compressed tarball of the base image:

::

       # tar zcf /tmp/rootfs.tar.gz -C /tmp/myroot .

Create a “metadata.yaml” file which describes the base image:

::

       architecture: "x86_64"
       # To get current date in Unix time, use `date +%s` command
       creation_date: 1458040200
       properties:
       architecture: "x86_64"
       description: "My custom Focal Fossa image"
       os: "Ubuntu"
       release: "focal"

Make a compressed tarball of metadata.yaml:

::

       # tar zcf metadata.tar.gz metadata.yaml

Import the image into lxc / lxd:

::

       $ lxc image import metadata.tar.gz rootfd.tar.gz --alias focal-base

Create a container which uses the customized base image:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

       $ lxc launch focal-base focaltest
       $ lxc exec focaltest bash

The next several steps should be executed in the container, in the bash
shell spun up by “lxc exec…”

Configure container networking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In the container, create /etc/netplan/50-cloud-init.yaml:

::

       network:
           version: 2
           ethernets:
               eth0:
                   dhcp4: true

Use “cat > /etc/netplan/50-cloud-init.yaml”, and cut-’n-paste if your
favorite text editor is AWOL.

Apply the configuration:

::

       # netplan apply

At this point, eth0 should have an ip address, and you should see a
default route with “route -n”.

Configure apt
~~~~~~~~~~~~~

Again, in the container, set up /etc/apt/sources.list via cut-’n-paste
from a recently update “focal fossa” host. Something like so:

::

       deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
       deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
       deb http://us.archive.ubuntu.com/ubuntu/ focal universe
       deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
       deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
       deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
       deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
       deb http://security.ubuntu.com/ubuntu focal-security main restricted
       deb http://security.ubuntu.com/ubuntu focal-security universe
       deb http://security.ubuntu.com/ubuntu focal-security multiverse

“apt-get update” and “apt-install” should produce reasonable results.
Suggest “apt-get install make git”.

At this point, you can use the “/scratch” sharepoint (or similar) to
execute “make install-dep install-ext-deps” to set up the container with
the vpp toolchain; proceed as desired.