diff options
Diffstat (limited to 'docs/usecases/container_test.rst')
| -rw-r--r-- | docs/usecases/container_test.rst | 661 |
1 files changed, 661 insertions, 0 deletions
diff --git a/docs/usecases/container_test.rst b/docs/usecases/container_test.rst new file mode 100644 index 00000000000..9fd4bc0c5db --- /dev/null +++ b/docs/usecases/container_test.rst @@ -0,0 +1,661 @@ +Simulating networks with VPP +============================ + +The “make test” framework provides a good way to test individual +features. However, when testing several features at once - or validating +nontrivial configurations - it may prove difficult or impossible to use +the unit-test framework. + +This note explains how to set up lxc/lxd, and a 5-container testbed to +test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation +scenario. + +OS / Distro test results +------------------------ + +This setup has been tested on Ubuntu 18.04 - 22.04 LTS systems. Other +distros may work fine, or not at all. + +Proxy Server +------------ + +If you need to use a proxy server e.g. from a lab system, you’ll +probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and https_proxy +in /etc/environment. Directly setting variables in the environment +doesn’t work. The lxd snap *daemon* needs the proxy settings, not the +user interface. + +Something like so: + +:: + + HTTP_PROXY=http://my.proxy.server:8080 + HTTPS_PROXY=http://my.proxy.server:4333 + http_proxy=http://my.proxy.server:8080 + https_proxy=http://my.proxy.server:4333 + +Install and configure lxd +------------------------- + +Install the lxd snap. The lxd snap is up to date, as opposed to the +results of “sudo apt-get install lxd”. + +:: + + # snap install lxd + # lxd init + +“lxd init” asks several questions. With the exception of the storage +pool, take the defaults. To match the configs shown below, create a +storage pool named “vpp.” Storage pools of type “zfs” and “files” have +been tested successfully. + +zfs is more space-efficient. “lxc copy” is infinitely faster with zfs. +The path for the zfs storage pool is under /var. Do not replace it with +a symbolic link, unless you want to rebuild all of your containers from +scratch. Ask me how I know that. + +Create three network segments +----------------------------- + +Aka, linux bridges. + +:: + + # lxc network create respond + # lxc network create internet + # lxc network create initiate + +We’ll explain the test topology in a bit. Stay tuned. + +Set up the default container profile +------------------------------------ + +Execute “lxc profile edit default”, and install the following +configuration. Note that the “shared” directory should mount your vpp +workspaces. With that trick, you can edit code from any of the +containers, run vpp without installing it, etc. + +:: + + config: {} + description: Default LXD profile + devices: + eth0: + name: eth0 + network: lxdbr0 + type: nic + eth1: + name: eth1 + nictype: bridged + parent: internet + type: nic + eth2: + name: eth2 + nictype: bridged + parent: respond + type: nic + eth3: + name: eth3 + nictype: bridged + parent: initiate + type: nic + root: + path: / + pool: vpp + type: disk + shared: + path: /scratch + source: /scratch + type: disk + name: default + + +Set up the network configurations +--------------------------------- + +Edit the fake “internet” backbone: + +:: + + # lxc network edit internet + +Install the ip addresses shown below, to avoid having to rebuild the vpp +and host configuration: + +:: + + config: + ipv4.address: 10.26.68.1/24 + ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50 + ipv4.nat: "true" + ipv6.address: none + ipv6.nat: "false" + description: "" + name: internet + type: bridge + used_by: + managed: true + status: Created + locations: + - none + +Repeat the process with the “respond” and “initiate” networks, using +these configurations: + +respond network configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + config: + ipv4.address: 10.166.14.1/24 + ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50 + ipv4.nat: "true" + ipv6.address: none + ipv6.nat: "false" + description: "" + name: respond + type: bridge + used_by: + managed: true + status: Created + locations: + - none + +initiate network configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + config: + ipv4.address: 10.219.188.1/24 + ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50 + ipv4.nat: "true" + ipv6.address: none + ipv6.nat: "false" + description: "" + name: initiate + type: bridge + used_by: + managed: true + status: Created + locations: + +Create a “master” container image +--------------------------------- + +The master container image should be set up so that you can build vpp, +ssh into the container, edit source code, run gdb, etc. + +Make sure that e.g. public key auth ssh works. + +:: + + # lxd launch ubuntu:22.04 respond + <spew> + # lxc exec respond bash + respond# cd /scratch/my-vpp-workspace + respond# apt-get install make ssh + respond# make install-dep + respond# exit + # lxc stop respond + +Mark the container image privileged. If you forget this step, you’ll +trip over a netlink error (-11) aka EAGAIN when you try to roll in the +vpp configurations. + +:: + + # lxc config set respond security.privileged "true" + +Duplicate the “master” container image +-------------------------------------- + +To avoid having to configure N containers, be sure that the master +container image is fully set up before you help it have children: + +:: + + # lxc copy respond respondhost + # lxc copy respond initiate + # lxc copy respond initiatehost + # lxc copy respond dhcpserver # optional, to test ipv6 prefix delegation + +Install handy script +-------------------- + +See below for a handy host script which executes lxc commands across +the current set of running containers. I call it “lxc-foreach,” feel +free to call the script Ishmael if you like. + +Examples: + +:: + + $ lxc-foreach start + <issues "lxc start" for each container in the list> + +After a few seconds, use this one to open an ssh connection to each +container. The ssh command parses the output of “lxc info,” which +displays container ip addresses. + +:: + + $ lxc-foreach ssh + +Here’s the script: + +:: + + #!/bin/bash + + set -u + export containers="respond initiate initiatehost respondhost" + + if [ x$1 = "x" ] ; then + echo missing command + exit 1 + fi + + if [ $1 = "ssh" ] ; then + for c in $containers + do + + inet=`lxc info $c | grep 10.38.33 | sed "s/.*inet://" | sed "s/\/24.*//" | tr -d " "` + if [ x$inet != "x" ] ; then + gnome-terminal --title "$c(ssh)" --command "/usr/bin/ssh -Y root@$inet" + fi + done + exit 0 + fi + + for c in $containers + do + inet=`lxc info $c | grep 10.38.33 | sed "s/.*inet://" | sed "s/\/24.*//" | tr -d " "` + if [ x$1 = "xstart" ] ; then + echo Starting $c + lxc start $c + elif [ x$1 = "xstop" ] ; then + echo Stopping $c + lxc stop $c + elif [ x$inet != "x" ] ; then + echo lxc $1 $c + lxc $1 $c + fi + done + + exit 0 + +Test topology +------------- + +Finally, we’re ready to describe a test topology. First, a picture: + +:: + + ===+======== management lan/bridge lxdbr0 (dhcp) ==============+=== + | | | + | | | + | | | + v | v + eth0 | eth0 + +---------+ eth1 eth1 +----------+ + | respond | 10.26.88.100 <= internet bridge => 10.26.88.101 | initiate | + +---------+ +----------+ + eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0 + | | | + | ("respond" bridge) | ("initiate" bridge) | + | | | + v | v + eth2 10.166.14.3 | eth3 10.219.188.3 + +-------------+ | +-------------+ + | respondhost | | | respondhost | + +-------------+ | +-------------+ + v + eth0 (management lan) <========+========> eth0 (management lan) + +Test topology discussion +~~~~~~~~~~~~~~~~~~~~~~~~ + +This topology is suitable for testing almost any tunnel encap/decap +scenario. The two containers “respondhost” and “initiatehost” are +end-stations connected to two vpp instances running on “respond” and +“initiate”. + +We leverage the Linux end-station network stacks to generate traffic of +all sorts. + +The so-called “internet” bridge models the public internet. The +“respond” and “initiate” bridges connect vpp instances to local hosts + +End station configs +------------------- + +The end-station Linux configurations set up the eth2 and eth3 ip +addresses shown above, and add tunnel routes to the opposite end-station +networks. + +respondhost configuration +~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + ifconfig eth2 10.166.14.3/24 up + route add -net 10.219.188.0/24 gw 10.166.14.2 + +initiatehost configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + sudo ifconfig eth3 10.219.188.3/24 up + sudo route add -net 10.166.14.0/24 gw 10.219.188.2 + +VPP configs +----------- + +Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in +the “respond” config. + +respond configuration +~~~~~~~~~~~~~~~~~~~~~ + +:: + + set term pag off + + comment { "internet" } + create host-interface name eth1 + set int ip address host-eth1 10.26.68.100/24 + set int ip6 table host-eth1 0 + set int state host-eth1 up + + comment { default route via initiate } + ip route add 0.0.0.0/0 via 10.26.68.101 + + comment { "respond-private-net" } + create host-interface name eth2 + bvi create instance 0 + set int l2 bridge bvi0 1 bvi + set int ip address bvi0 10.166.14.2/24 + set int state bvi0 up + set int l2 bridge host-eth2 1 + set int state host-eth2 up + + + nat44 add interface address host-eth1 + set interface nat44 in host-eth2 out host-eth1 + nat44 add identity mapping external host-eth1 udp 500 + nat44 add identity mapping external host-eth1 udp 4500 + comment { nat44 untranslated subnet 10.219.188.0/24 } + + comment { responder profile } + ikev2 profile add initiate + ikev2 profile set initiate udp-encap + ikev2 profile set initiate auth rsa-sig cert-file /scratch/setups/respondcert.pem + set ikev2 local key /scratch/setups/initiatekey.pem + ikev2 profile set initiate id local fqdn initiator.my.net + ikev2 profile set initiate id remote fqdn responder.my.net + ikev2 profile set initiate traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 + ikev2 profile set initiate traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 + create ipip tunnel src 10.26.68.100 dst 10.26.68.101 + ikev2 profile set initiate tunnel ipip0 + + comment { ipv6 prefix delegation } + ip6 nd address autoconfig host-eth1 default-route + dhcp6 client host-eth1 + dhcp6 pd client host-eth1 prefix group hgw + set ip6 address bvi0 prefix group hgw ::2/56 + ip6 nd address autoconfig bvi0 default-route + ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180 + + set int mtu packet 1390 ipip0 + set int unnum ipip0 use host-eth1 + ip route add 10.219.188.0/24 via ipip0 + +initiate configuration +~~~~~~~~~~~~~~~~~~~~~~ + +:: + + set term pag off + + comment { "internet" } + create host-interface name eth1 + comment { set dhcp client intfc host-eth1 hostname initiate } + set int ip address host-eth1 10.26.68.101/24 + set int state host-eth1 up + + comment { default route via "internet gateway" } + comment { ip route add 0.0.0.0/0 via 10.26.68.1 } + + comment { "initiate-private-net" } + create host-interface name eth3 + bvi create instance 0 + set int l2 bridge bvi0 1 bvi + set int ip address bvi0 10.219.188.2/24 + set int state bvi0 up + set int l2 bridge host-eth3 1 + set int state host-eth3 up + + nat44 add interface address host-eth1 + set interface nat44 in bvi0 out host-eth1 + nat44 add identity mapping external host-eth1 udp 500 + nat44 add identity mapping external host-eth1 udp 4500 + comment { nat44 untranslated subnet 10.166.14.0/24 } + + comment { initiator profile } + ikev2 profile add respond + ikev2 profile set respond udp-encap + ikev2 profile set respond auth rsa-sig cert-file /scratch/setups/initiatecert.pem + set ikev2 local key /scratch/setups/respondkey.pem + ikev2 profile set respond id local fqdn responder.my.net + ikev2 profile set respond id remote fqdn initiator.my.net + + ikev2 profile set respond traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 + ikev2 profile set respond traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 + + ikev2 profile set respond responder host-eth1 10.26.68.100 + ikev2 profile set respond ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048 + ikev2 profile set respond esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256 + ikev2 profile set respond sa-lifetime 3600 10 5 0 + + create ipip tunnel src 10.26.68.101 dst 10.26.68.100 + ikev2 profile set respond tunnel ipip0 + ikev2 initiate sa-init respond + + set int mtu packet 1390 ipip0 + set int unnum ipip0 use host-eth1 + ip route add 10.166.14.0/24 via ipip0 + +IKEv2 certificate setup +----------------------- + +In both of the vpp configurations, you’ll see “/scratch/setups/xxx.pem” +mentioned. These certificates are used in the ikev2 key exchange. + +Here’s how to generate the certificates: + +:: + + openssl req -x509 -nodes -newkey rsa:4096 -keyout respondkey.pem -out respondcert.pem -days 3560 + openssl x509 -text -noout -in respondcert.pem + openssl req -x509 -nodes -newkey rsa:4096 -keyout initiatekey.pem -out initiatecert.pem -days 3560 + openssl x509 -text -noout -in initiatecert.pem + +Make sure that the “respond” and “initiate” configurations point to the +certificates. + +DHCPv6 server setup +------------------- + +If you need an ipv6 dhcp server to test ipv6 prefix delegation, create +the “dhcpserver” container as shown above. + +Install the “isc-dhcp-server” Debian package: + +:: + + sudo apt-get install isc-dhcp-server + +/etc/dhcp/dhcpd6.conf +~~~~~~~~~~~~~~~~~~~~~ + +Edit the dhcpv6 configuration and add an ipv6 subnet with prefix +delegation. For example: + +:: + + subnet6 2001:db01:0:1::/64 { + range6 2001:db01:0:1::1 2001:db01:0:1::9; + prefix6 2001:db01:0:100:: 2001:db01:0:200::/56; + } + +Add an ipv6 address on eth1, which is connected to the “internet” +bridge, and start the dhcp server. I use the following trivial bash +script, which runs the dhcp6 server in the foreground and produces dhcp +traffic spew: + +:: + + #!/bin/bash + ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true + dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf + +The “\|\| true” bit keeps going if eth1 already has the indicated ipv6 +address. + +Container / Host Interoperation +------------------------------- + +Host / container interoperation is highly desirable. If the host and a +set of containers don’t run the same distro *and distro version*, it’s +reasonably likely that the glibc versions won’t match. That, in turn, +makes vpp binaries built in one environment fail in the other. + +Trying to install multiple versions of glibc - especially at the host +level - often ends very badly and is *not recommended*. It’s not just +glibc, either. The dynamic loader ld-linux-xxx-so.2 is glibc version +specific. + +Fortunately, it’s reasonable easy to build lxd container images based on +specific Ubuntu or Debian versions. + +Create a custom root filesystem image +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First, install the “debootstrap” tool: + +:: + + sudo apt-get install debootstrap + +Make a temp directory, and use debootstrap to populate it. In this +example, we create an Ubuntu 20.04 (focal fossa) base image: + +:: + + # mkdir /tmp/myroot + # debootstrap focal /tmp/myroot http://archive.ubuntu.com/ubuntu + +To tinker with the base image (if desired): + +:: + + # chroot /tmp/myroot + <add packages, etc.> + # exit + +Make a compressed tarball of the base image: + +:: + + # tar zcf /tmp/rootfs.tar.gz -C /tmp/myroot . + +Create a “metadata.yaml” file which describes the base image: + +:: + + architecture: "x86_64" + # To get current date in Unix time, use `date +%s` command + creation_date: 1458040200 + properties: + architecture: "x86_64" + description: "My custom Focal Fossa image" + os: "Ubuntu" + release: "focal" + +Make a compressed tarball of metadata.yaml: + +:: + + # tar zcf metadata.tar.gz metadata.yaml + +Import the image into lxc / lxd: + +:: + + $ lxc image import metadata.tar.gz rootfd.tar.gz --alias focal-base + +Create a container which uses the customized base image: +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:: + + $ lxc launch focal-base focaltest + $ lxc exec focaltest bash + +The next several steps should be executed in the container, in the bash +shell spun up by “lxc exec…” + +Configure container networking +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the container, create /etc/netplan/50-cloud-init.yaml: + +:: + + network: + version: 2 + ethernets: + eth0: + dhcp4: true + +Use “cat > /etc/netplan/50-cloud-init.yaml”, and cut-’n-paste if your +favorite text editor is AWOL. + +Apply the configuration: + +:: + + # netplan apply + +At this point, eth0 should have an ip address, and you should see a +default route with “route -n”. + +Configure apt +~~~~~~~~~~~~~ + +Again, in the container, set up /etc/apt/sources.list via cut-’n-paste +from a recently update “focal fossa” host. Something like so: + +:: + + deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted + deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted + deb http://us.archive.ubuntu.com/ubuntu/ focal universe + deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe + deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse + deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse + deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse + deb http://security.ubuntu.com/ubuntu focal-security main restricted + deb http://security.ubuntu.com/ubuntu focal-security universe + deb http://security.ubuntu.com/ubuntu focal-security multiverse + +“apt-get update” and “apt-install” should produce reasonable results. +Suggest “apt-get install make git”. + +At this point, you can use the “/scratch” sharepoint (or similar) to +execute “make install-dep install-ext-deps” to set up the container with +the vpp toolchain; proceed as desired. |