summaryrefslogtreecommitdiffstats
path: root/docs/usecases/container_test.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/usecases/container_test.rst')
-rw-r--r--docs/usecases/container_test.rst655
1 files changed, 655 insertions, 0 deletions
diff --git a/docs/usecases/container_test.rst b/docs/usecases/container_test.rst
new file mode 100644
index 00000000000..8ad6285f8ed
--- /dev/null
+++ b/docs/usecases/container_test.rst
@@ -0,0 +1,655 @@
+Simulating networks with VPP
+============================
+
+The “make test” framework provides a good way to test individual
+features. However, when testing several features at once - or validating
+nontrivial configurations - it may prove difficult or impossible to use
+the unit-test framework.
+
+This note explains how to set up lxc/lxd, and a 5-container testbed to
+test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation
+scenario.
+
+OS / Distro test results
+------------------------
+
+This setup has been tested on an Ubuntu 18.04 LTS system. If you’re
+feeling adventurous, the same scenario also worked on a recent Ubuntu
+20.04 “preview” daily build.
+
+Other distros may work fine, or not at all.
+
+Proxy Server
+------------
+
+If you need to use a proxy server e.g. from a lab system, you’ll
+probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and https_proxy
+in /etc/environment. Directly setting variables in the environment
+doesn’t work. The lxd snap *daemon* needs the proxy settings, not the
+user interface.
+
+Something like so:
+
+::
+
+ HTTP_PROXY=http://my.proxy.server:8080
+ HTTPS_PROXY=http://my.proxy.server:4333
+ http_proxy=http://my.proxy.server:8080
+ https_proxy=http://my.proxy.server:4333
+
+Install and configure lxd
+-------------------------
+
+Install the lxd snap. The lxd snap is up to date, as opposed to the
+results of “sudo apt-get install lxd”.
+
+::
+
+ # snap install lxd
+ # lxd init
+
+“lxd init” asks several questions. With the exception of the storage
+pool, take the defaults. To match the configs shown below, create a
+storage pool named “vpp.” Storage pools of type “zfs” and “files” have
+been tested successfully.
+
+zfs is more space-efficient. “lxc copy” is infinitely faster with zfs.
+The path for the zfs storage pool is under /var. Do not replace it with
+a symbolic link, unless you want to rebuild all of your containers from
+scratch. Ask me how I know that.
+
+Create three network segments
+-----------------------------
+
+Aka, linux bridges.
+
+::
+
+ # lxc network create respond
+ # lxc network create internet
+ # lxc network create initiate
+
+We’ll explain the test topology in a bit. Stay tuned.
+
+Set up the default container profile
+------------------------------------
+
+Execute “lxc profile edit default”, and install the following
+configuration. Note that the “shared” directory should mount your vpp
+workspaces. With that trick, you can edit code from any of the
+containers, run vpp without installing it, etc.
+
+::
+
+ config: {}
+ description: Default LXD profile
+ devices:
+ eth0:
+ name: eth0
+ network: lxdbr0
+ type: nic
+ eth1:
+ name: eth1
+ nictype: bridged
+ parent: internet
+ type: nic
+ eth2:
+ name: eth2
+ nictype: bridged
+ parent: respond
+ type: nic
+ eth3:
+ name: eth3
+ nictype: bridged
+ parent: initiate
+ type: nic
+ root:
+ path: /
+ pool: vpp
+ type: disk
+ shared:
+ path: /scratch
+ source: /scratch
+ type: disk
+ name: default
+
+Set up the network configurations
+---------------------------------
+
+Edit the fake “internet” backbone:
+
+::
+
+ # lxc network edit internet
+
+Install the ip addresses shown below, to avoid having to rebuild the vpp
+and host configuration:
+
+::
+
+ config:
+ ipv4.address: 10.26.68.1/24
+ ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50
+ ipv4.nat: "true"
+ ipv6.address: none
+ ipv6.nat: "false"
+ description: ""
+ name: internet
+ type: bridge
+ used_by:
+ managed: true
+ status: Created
+ locations:
+ - none
+
+Repeat the process with the “respond” and “initiate” networks, using
+these configurations:
+
+respond network configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ config:
+ ipv4.address: 10.166.14.1/24
+ ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50
+ ipv4.nat: "true"
+ ipv6.address: none
+ ipv6.nat: "false"
+ description: ""
+ name: respond
+ type: bridge
+ used_by:
+ managed: true
+ status: Created
+ locations:
+ - none
+
+initiate network configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ config:
+ ipv4.address: 10.219.188.1/24
+ ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50
+ ipv4.nat: "true"
+ ipv6.address: none
+ ipv6.nat: "false"
+ description: ""
+ name: initiate
+ type: bridge
+ used_by:
+ managed: true
+ status: Created
+ locations:
+ - none
+
+Create a “master” container image
+---------------------------------
+
+The master container image should be set up so that you can build vpp,
+ssh into the container, edit source code, run gdb, etc.
+
+Make sure that e.g. public key auth ssh works.
+
+::
+
+ # lxd launch ubuntu:18.04 respond
+ <spew>
+ # lxc exec respond bash
+ respond# cd /scratch/my-vpp-workspace
+ respond# apt-get install make ssh
+ respond# make install-dep
+ respond# exit
+ # lxc stop respond
+
+Mark the container image privileged. If you forget this step, you’ll
+trip over a netlink error (-11) aka EAGAIN when you try to roll in the
+vpp configurations.
+
+::
+
+ # lxc config set respond security.privileged "true"
+
+Duplicate the “master” container image
+--------------------------------------
+
+To avoid having to configure N containers, be sure that the master
+container image is fully set up before you help it have children:
+
+::
+
+ # lxc copy respond respondhost
+ # lxc copy respond initiate
+ # lxc copy respond initiatehost
+ # lxc copy respond dhcpserver # optional, to test ipv6 prefix delegation
+
+Install handy script
+--------------------
+
+See below for a handy script which executes lxc commands across the
+current set of running containers. I call it “lxc-foreach,” feel free to
+call the script Ishmael if you like.
+
+Examples:
+
+::
+
+ $ lxc-foreach start
+ <issues "lxc start" for each container in the list>
+
+After a few seconds, use this one to open an ssh connection to each
+container. The ssh command parses the output of “lxc info,” which
+displays container ip addresses.
+
+::
+
+ $ lxc-foreach ssh
+
+Here’s the script:
+
+::
+
+ #!/bin/bash
+
+ set -u
+ export containers="respond respondhost initiate initiatehost dhcpserver"
+
+ if [ x$1 = "x" ] ; then
+ echo missing command
+ exit 1
+ fi
+
+ if [ $1 = "ssh" ] ; then
+ for c in $containers
+ do
+ inet=`lxc info $c | grep eth0 | grep -v inet6 | head -1 | cut -f 3`
+ if [ x$inet = "x" ] ; then
+ echo $c not started
+ else
+ gnome-terminal --command "/usr/bin/ssh $inet"
+ fi
+ done
+ exit 0
+ fi
+
+ for c in $containers
+ do
+ echo lxc $1 $c
+ lxc $1 $c
+ done
+
+ exit 0
+
+Test topology
+-------------
+
+Finally, we’re ready to describe a test topology. First, a picture:
+
+::
+
+ ===+======== management lan/bridge lxdbr0 (dhcp) ===========+===
+ | | |
+ | | |
+ | | |
+ v | v
+ eth0 | eth0
+ +------+ eth1 eth1 +------+
+ | respond | 10.26.88.100 <= internet bridge => 10.26.88.101 | initiate |
+ +------+ +------+
+ eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0
+ | | |
+ | ("respond" bridge) | ("initiate" bridge) |
+ | | |
+ v | v
+ eth2 10.166.14.3 | eth3 10.219.188.3
+ +----------+ | +----------+
+ | respondhost | | | respondhost |
+ +----------+ | +----------+
+ eth0 (management lan) <========+========> eth0 (management lan)
+
+Test topology discussion
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+This topology is suitable for testing almost any tunnel encap/decap
+scenario. The two containers “respondhost” and “initiatehost” are
+end-stations connected to two vpp instances running on “respond” and
+“initiate”.
+
+We leverage the Linux end-station network stacks to generate traffic of
+all sorts.
+
+The so-called “internet” bridge models the public internet. The
+“respond” and “initiate” bridges connect vpp instances to local hosts
+
+End station configs
+-------------------
+
+The end-station Linux configurations set up the eth2 and eth3 ip
+addresses shown above, and add tunnel routes to the opposite end-station
+networks.
+
+respondhost configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ ifconfig eth2 10.166.14.3/24 up
+ route add -net 10.219.188.0/24 gw 10.166.14.2
+
+initiatehost configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ sudo ifconfig eth3 10.219.188.3/24 up
+ sudo route add -net 10.166.14.0/24 gw 10.219.188.2
+
+VPP configs
+-----------
+
+Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in
+the “respond” config.
+
+respond configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ set term pag off
+
+ comment { "internet" }
+ create host-interface name eth1
+ set int ip address host-eth1 10.26.68.100/24
+ set int ip6 table host-eth1 0
+ set int state host-eth1 up
+
+ comment { default route via initiate }
+ ip route add 0.0.0.0/0 via 10.26.68.101
+
+ comment { "respond-private-net" }
+ create host-interface name eth2
+ bvi create instance 0
+ set int l2 bridge bvi0 1 bvi
+ set int ip address bvi0 10.166.14.2/24
+ set int state bvi0 up
+ set int l2 bridge host-eth2 1
+ set int state host-eth2 up
+
+
+ nat44 add interface address host-eth1
+ set interface nat44 in host-eth2 out host-eth1
+ nat44 add identity mapping external host-eth1 udp 500
+ nat44 add identity mapping external host-eth1 udp 4500
+ comment { nat44 untranslated subnet 10.219.188.0/24 }
+
+ comment { responder profile }
+ ikev2 profile add initiate
+ ikev2 profile set initiate udp-encap
+ ikev2 profile set initiate auth rsa-sig cert-file /scratch/setups/respondcert.pem
+ set ikev2 local key /scratch/setups/initiatekey.pem
+ ikev2 profile set initiate id local fqdn initiator.my.net
+ ikev2 profile set initiate id remote fqdn responder.my.net
+ ikev2 profile set initiate traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
+ ikev2 profile set initiate traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
+ create ipip tunnel src 10.26.68.100 dst 10.26.68.101
+ ikev2 profile set initiate tunnel ipip0
+
+ comment { ipv6 prefix delegation }
+ ip6 nd address autoconfig host-eth1 default-route
+ dhcp6 client host-eth1
+ dhcp6 pd client host-eth1 prefix group hgw
+ set ip6 address bvi0 prefix group hgw ::2/56
+ ip6 nd address autoconfig bvi0 default-route
+ ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180
+
+ set int mtu packet 1390 ipip0
+ set int unnum ipip0 use host-eth1
+ ip route add 10.219.188.0/24 via ipip0
+
+initiate configuration
+~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ set term pag off
+
+ comment { "internet" }
+ create host-interface name eth1
+ comment { set dhcp client intfc host-eth1 hostname initiate }
+ set int ip address host-eth1 10.26.68.101/24
+ set int state host-eth1 up
+
+ comment { default route via "internet gateway" }
+ comment { ip route add 0.0.0.0/0 via 10.26.68.1 }
+
+ comment { "initiate-private-net" }
+ create host-interface name eth3
+ bvi create instance 0
+ set int l2 bridge bvi0 1 bvi
+ set int ip address bvi0 10.219.188.2/24
+ set int state bvi0 up
+ set int l2 bridge host-eth3 1
+ set int state host-eth3 up
+
+ nat44 add interface address host-eth1
+ set interface nat44 in bvi0 out host-eth1
+ nat44 add identity mapping external host-eth1 udp 500
+ nat44 add identity mapping external host-eth1 udp 4500
+ comment { nat44 untranslated subnet 10.166.14.0/24 }
+
+ comment { initiator profile }
+ ikev2 profile add respond
+ ikev2 profile set respond udp-encap
+ ikev2 profile set respond auth rsa-sig cert-file /scratch/setups/initiatecert.pem
+ set ikev2 local key /scratch/setups/respondkey.pem
+ ikev2 profile set respond id local fqdn responder.my.net
+ ikev2 profile set respond id remote fqdn initiator.my.net
+
+ ikev2 profile set respond traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0
+ ikev2 profile set respond traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0
+
+ ikev2 profile set respond responder host-eth1 10.26.68.100
+ ikev2 profile set respond ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048
+ ikev2 profile set respond esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256
+ ikev2 profile set respond sa-lifetime 3600 10 5 0
+
+ create ipip tunnel src 10.26.68.101 dst 10.26.68.100
+ ikev2 profile set respond tunnel ipip0
+ ikev2 initiate sa-init respond
+
+ set int mtu packet 1390 ipip0
+ set int unnum ipip0 use host-eth1
+ ip route add 10.166.14.0/24 via ipip0
+
+IKEv2 certificate setup
+-----------------------
+
+In both of the vpp configurations, you’ll see “/scratch/setups/xxx.pem”
+mentioned. These certificates are used in the ikev2 key exchange.
+
+Here’s how to generate the certificates:
+
+::
+
+ openssl req -x509 -nodes -newkey rsa:4096 -keyout respondkey.pem -out respondcert.pem -days 3560
+ openssl x509 -text -noout -in respondcert.pem
+ openssl req -x509 -nodes -newkey rsa:4096 -keyout initiatekey.pem -out initiatecert.pem -days 3560
+ openssl x509 -text -noout -in initiatecert.pem
+
+Make sure that the “respond” and “initiate” configurations point to the
+certificates.
+
+DHCPv6 server setup
+-------------------
+
+If you need an ipv6 dhcp server to test ipv6 prefix delegation, create
+the “dhcpserver” container as shown above.
+
+Install the “isc-dhcp-server” Debian package:
+
+::
+
+ sudo apt-get install isc-dhcp-server
+
+/etc/dhcp/dhcpd6.conf
+~~~~~~~~~~~~~~~~~~~~~
+
+Edit the dhcpv6 configuration and add an ipv6 subnet with prefix
+delegation. For example:
+
+::
+
+ subnet6 2001:db01:0:1::/64 {
+ range6 2001:db01:0:1::1 2001:db01:0:1::9;
+ prefix6 2001:db01:0:100:: 2001:db01:0:200::/56;
+ }
+
+Add an ipv6 address on eth1, which is connected to the “internet”
+bridge, and start the dhcp server. I use the following trivial bash
+script, which runs the dhcp6 server in the foreground and produces dhcp
+traffic spew:
+
+::
+
+ #!/bin/bash
+ ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true
+ dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf
+
+The “\|\| true” bit keeps going if eth1 already has the indicated ipv6
+address.
+
+Container / Host Interoperation
+-------------------------------
+
+Host / container interoperation is highly desirable. If the host and a
+set of containers don’t run the same distro *and distro version*, it’s
+reasonably likely that the glibc versions won’t match. That, in turn,
+makes vpp binaries built in one environment fail in the other.
+
+Trying to install multiple versions of glibc - especially at the host
+level - often ends very badly and is *not recommended*. It’s not just
+glibc, either. The dynamic loader ld-linux-xxx-so.2 is glibc version
+specific.
+
+Fortunately, it’s reasonable easy to build lxd container images based on
+specific Ubuntu or Debian versions.
+
+Create a custom root filesystem image
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, install the “debootstrap” tool:
+
+::
+
+ sudo apt-get install debootstrap
+
+Make a temp directory, and use debootstrap to populate it. In this
+example, we create an Ubuntu 20.04 (focal fossa) base image:
+
+::
+
+ # mkdir /tmp/myroot
+ # debootstrap focal /tmp/myroot http://archive.ubuntu.com/ubuntu
+
+To tinker with the base image (if desired):
+
+::
+
+ # chroot /tmp/myroot
+ <add packages, etc.>
+ # exit
+
+Make a compressed tarball of the base image:
+
+::
+
+ # tar zcf /tmp/rootfs.tar.gz -C /tmp/myroot .
+
+Create a “metadata.yaml” file which describes the base image:
+
+::
+
+ architecture: "x86_64"
+ # To get current date in Unix time, use `date +%s` command
+ creation_date: 1458040200
+ properties:
+ architecture: "x86_64"
+ description: "My custom Focal Fossa image"
+ os: "Ubuntu"
+ release: "focal"
+
+Make a compressed tarball of metadata.yaml:
+
+::
+
+ # tar zcf metadata.tar.gz metadata.yaml
+
+Import the image into lxc / lxd:
+
+::
+
+ $ lxc image import metadata.tar.gz rootfd.tar.gz --alias focal-base
+
+Create a container which uses the customized base image:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+ $ lxc launch focal-base focaltest
+ $ lxc exec focaltest bash
+
+The next several steps should be executed in the container, in the bash
+shell spun up by “lxc exec…”
+
+Configure container networking
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the container, create /etc/netplan/50-cloud-init.yaml:
+
+::
+
+ network:
+ version: 2
+ ethernets:
+ eth0:
+ dhcp4: true
+
+Use “cat > /etc/netplan/50-cloud-init.yaml”, and cut-’n-paste if your
+favorite text editor is AWOL.
+
+Apply the configuration:
+
+::
+
+ # netplan apply
+
+At this point, eth0 should have an ip address, and you should see a
+default route with “route -n”.
+
+Configure apt
+~~~~~~~~~~~~~
+
+Again, in the container, set up /etc/apt/sources.list via cut-’n-paste
+from a recently update “focal fossa” host. Something like so:
+
+::
+
+ deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted
+ deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted
+ deb http://us.archive.ubuntu.com/ubuntu/ focal universe
+ deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe
+ deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse
+ deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse
+ deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse
+ deb http://security.ubuntu.com/ubuntu focal-security main restricted
+ deb http://security.ubuntu.com/ubuntu focal-security universe
+ deb http://security.ubuntu.com/ubuntu focal-security multiverse
+
+“apt-get update” and “apt-install” should produce reasonable results.
+Suggest “apt-get install make git”.
+
+At this point, you can use the “/scratch” sharepoint (or similar) to
+execute “make install-dep install-ext-deps” to set up the container with
+the vpp toolchain; proceed as desired.