diff options
Diffstat (limited to 'docs/usecases/container_test.md')
| -rw-r--r-- | docs/usecases/container_test.md | 640 |
1 files changed, 0 insertions, 640 deletions
diff --git a/docs/usecases/container_test.md b/docs/usecases/container_test.md deleted file mode 100644 index ad0bc2ea098..00000000000 --- a/docs/usecases/container_test.md +++ /dev/null @@ -1,640 +0,0 @@ -Container-based network simulation -================================== - -The "make test" framework provides a good way to test individual -features. However, when testing several features at once - or -validating nontrivial configurations - it may prove difficult or -impossible to use the unit-test framework. - -This note explains how to set up lxc/lxd, and a 5-container testbed to -test a split-tunnel nat + ikev2 + ipsec + ipv6 prefix-delegation -scenario. - -OS / Distro test results ------------------------- - -This setup has been tested on an Ubuntu 18.04 LTS system. If you're -feeling adventurous, the same scenario also worked on a recent Ubuntu -20.04 "preview" daily build. - -Other distros may work fine, or not at all. - -Proxy Server ------------- - -If you need to use a proxy server e.g. from a lab system, you'll -probably need to set HTTP_PROXY, HTTPS_PROXY, http_proxy and -https_proxy in /etc/environment. Directly setting variables in the -environment doesn't work. The lxd snap _daemon_ needs the proxy settings, -not the user interface. - -Something like so: - -``` - HTTP_PROXY=http://my.proxy.server:8080 - HTTPS_PROXY=http://my.proxy.server:4333 - http_proxy=http://my.proxy.server:8080 - https_proxy=http://my.proxy.server:4333 -``` - -Install and configure lxd -------------------------- - -Install the lxd snap. The lxd snap is up to date, as opposed to the -results of "sudo apt-get install lxd". - -``` - # snap install lxd - # lxd init -``` - -"lxd init" asks several questions. With the exception of the storage -pool, take the defaults. To match the configs shown below, create a -storage pool named "vpp." Storage pools of type "zfs" and "files" have -been tested successfully. - -zfs is more space-efficient. "lxc copy" is infinitely faster with -zfs. The path for the zfs storage pool is under /var. Do not replace -it with a symbolic link, unless you want to rebuild all of your -containers from scratch. Ask me how I know that. - -Create three network segments ------------------------------ - -Aka, linux bridges. - -``` - # lxc network create respond - # lxc network create internet - # lxc network create initiate -``` - -We'll explain the test topology in a bit. Stay tuned. - -Set up the default container profile ------------------------------------- - -Execute "lxc profile edit default", and install the following -configuration. Note that the "shared" directory should mount your vpp -workspaces. With that trick, you can edit code from any of the -containers, run vpp without installing it, etc. - -``` - config: {} - description: Default LXD profile - devices: - eth0: - name: eth0 - network: lxdbr0 - type: nic - eth1: - name: eth1 - nictype: bridged - parent: internet - type: nic - eth2: - name: eth2 - nictype: bridged - parent: respond - type: nic - eth3: - name: eth3 - nictype: bridged - parent: initiate - type: nic - root: - path: / - pool: vpp - type: disk - shared: - path: /scratch - source: /scratch - type: disk - name: default -``` - -Set up the network configurations ---------------------------------- - -Edit the fake "internet" backbone: - -``` - # lxc network edit internet -``` - -Install the ip addresses shown below, to avoid having to rebuild the vpp -and host configuration: - -``` - config: - ipv4.address: 10.26.68.1/24 - ipv4.dhcp.ranges: 10.26.68.10-10.26.68.50 - ipv4.nat: "true" - ipv6.address: none - ipv6.nat: "false" - description: "" - name: internet - type: bridge - used_by: - managed: true - status: Created - locations: - - none -``` - -Repeat the process with the "respond" and "initiate" networks, using these -configurations: - -### respond network configuration - -``` - config: - ipv4.address: 10.166.14.1/24 - ipv4.dhcp.ranges: 10.166.14.10-10.166.14.50 - ipv4.nat: "true" - ipv6.address: none - ipv6.nat: "false" - description: "" - name: respond - type: bridge - used_by: - managed: true - status: Created - locations: - - none -``` -### initiate network configuration - -``` - config: - ipv4.address: 10.219.188.1/24 - ipv4.dhcp.ranges: 10.219.188.10-10.219.188.50 - ipv4.nat: "true" - ipv6.address: none - ipv6.nat: "false" - description: "" - name: initiate - type: bridge - used_by: - managed: true - status: Created - locations: - - none -``` - -Create a "master" container image ---------------------------------- - -The master container image should be set up so that you can -build vpp, ssh into the container, edit source code, run gdb, etc. - -Make sure that e.g. public key auth ssh works. - -``` - # lxd launch ubuntu:18.04 respond - <spew> - # lxc exec respond bash - respond# cd /scratch/my-vpp-workspace - respond# apt-get install make ssh - respond# make install-dep - respond# exit - # lxc stop respond -``` - -Mark the container image privileged. If you forget this step, you'll -trip over a netlink error (-11) aka EAGAIN when you try to roll in the -vpp configurations. - -``` - # lxc config set respond security.privileged "true" -``` - -Duplicate the "master" container image --------------------------------------- - -To avoid having to configure N containers, be sure that the master -container image is fully set up before you help it have children: - -``` - # lxc copy respond respondhost - # lxc copy respond initiate - # lxc copy respond initiatehost - # lxc copy respond dhcpserver # optional, to test ipv6 prefix delegation -``` - -Install handy script --------------------- - -See below for a handly script which executes lxc commands across the -current set of running containers. I call it "lxc-foreach," feel free -to call the script Ishmael if you like. - -Examples: - -``` - $ lxc-foreach start - <issues "lxc start" for each container in the list> -``` - -After a few seconds, use this one to open an ssh connection to each -container. The ssh command parses the output of "lxc info," which -displays container ip addresses. - -``` - $ lxc-foreach ssh -``` - -Here's the script: - -``` - #!/bin/bash - - set -u - export containers="respond respondhost initiate initiatehost dhcpserver" - - if [ x$1 = "x" ] ; then - echo missing command - exit 1 - fi - - if [ $1 = "ssh" ] ; then - for c in $containers - do - inet=`lxc info $c | grep eth0 | grep -v inet6 | head -1 | cut -f 3` - if [ x$inet = "x" ] ; then - echo $c not started - else - gnome-terminal --command "/usr/bin/ssh $inet" - fi - done - exit 0 - fi - - for c in $containers - do - echo lxc $1 $c - lxc $1 $c - done - - exit 0 -``` - -Test topology -------------- - -Finally, we're ready to describe a test topology. First, a picture: - -``` - ===+======== management lan/bridge lxdbr0 (dhcp) ===========+=== - | | | - | | | - | | | - v | v - eth0 | eth0 - +------+ eth1 eth1 +------+ - | respond | 10.26.88.100 <= internet bridge => 10.26.88.101 | initiate | - +------+ +------+ - eth2 / bvi0 10.166.14.2 | 10.219.188.2 eth3 / bvi0 - | | | - | ("respond" bridge) | ("initiate" bridge) | - | | | - v | v - eth2 10.166.14.3 | eth3 10.219.188.3 - +----------+ | +----------+ - | respondhost | | | respondhost | - +----------+ | +----------+ - eth0 (management lan) <========+========> eth0 (management lan) -``` - -### Test topology discussion - -This topology is suitable for testing almost any tunnel encap/decap -scenario. The two containers "respondhost" and "initiatehost" are end-stations -connected to two vpp instances running on "respond" and "initiate". - -We leverage the Linux end-station network stacks to generate traffic -of all sorts. - -The so-called "internet" bridge models the public internet. The "respond" and -"initiate" bridges connect vpp instances to local hosts - -End station configs -------------------- - -The end-station Linux configurations set up the eth2 and eth3 ip -addresses shown above, and add tunnel routes to the opposite -end-station networks. - -### respondhost configuration - -``` - ifconfig eth2 10.166.14.3/24 up - route add -net 10.219.188.0/24 gw 10.166.14.2 -``` - -### initiatehost configuration - -``` - sudo ifconfig eth3 10.219.188.3/24 up - sudo route add -net 10.166.14.0/24 gw 10.219.188.2 -``` - -VPP configs ------------ - -Split nat44 / ikev2 + ipsec tunneling, with ipv6 prefix delegation in -the "respond" config. - -### respond configuration - -``` - set term pag off - - comment { "internet" } - create host-interface name eth1 - set int ip address host-eth1 10.26.68.100/24 - set int ip6 table host-eth1 0 - set int state host-eth1 up - - comment { default route via initiate } - ip route add 0.0.0.0/0 via 10.26.68.101 - - comment { "respond-private-net" } - create host-interface name eth2 - bvi create instance 0 - set int l2 bridge bvi0 1 bvi - set int ip address bvi0 10.166.14.2/24 - set int state bvi0 up - set int l2 bridge host-eth2 1 - set int state host-eth2 up - - - nat44 add interface address host-eth1 - set interface nat44 in host-eth2 out host-eth1 - nat44 add identity mapping external host-eth1 udp 500 - nat44 add identity mapping external host-eth1 udp 4500 - comment { nat44 untranslated subnet 10.219.188.0/24 } - - comment { responder profile } - ikev2 profile add initiate - ikev2 profile set initiate udp-encap - ikev2 profile set initiate auth rsa-sig cert-file /scratch/setups/respondcert.pem - set ikev2 local key /scratch/setups/initiatekey.pem - ikev2 profile set initiate id local fqdn initiator.my.net - ikev2 profile set initiate id remote fqdn responder.my.net - ikev2 profile set initiate traffic-selector remote ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 - ikev2 profile set initiate traffic-selector local ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 - create ipip tunnel src 10.26.68.100 dst 10.26.68.101 - ikev2 profile set initiate tunnel ipip0 - - comment { ipv6 prefix delegation } - ip6 nd address autoconfig host-eth1 default-route - dhcp6 client host-eth1 - dhcp6 pd client host-eth1 prefix group hgw - set ip6 address bvi0 prefix group hgw ::2/56 - ip6 nd address autoconfig bvi0 default-route - ip6 nd bvi0 ra-interval 5 3 ra-lifetime 180 - - set int mtu packet 1390 ipip0 - set int unnum ipip0 use host-eth1 - ip route add 10.219.188.0/24 via ipip0 -``` - -### initiate configuration - -``` - set term pag off - - comment { "internet" } - create host-interface name eth1 - comment { set dhcp client intfc host-eth1 hostname initiate } - set int ip address host-eth1 10.26.68.101/24 - set int state host-eth1 up - - comment { default route via "internet gateway" } - comment { ip route add 0.0.0.0/0 via 10.26.68.1 } - - comment { "initiate-private-net" } - create host-interface name eth3 - bvi create instance 0 - set int l2 bridge bvi0 1 bvi - set int ip address bvi0 10.219.188.2/24 - set int state bvi0 up - set int l2 bridge host-eth3 1 - set int state host-eth3 up - - nat44 add interface address host-eth1 - set interface nat44 in bvi0 out host-eth1 - nat44 add identity mapping external host-eth1 udp 500 - nat44 add identity mapping external host-eth1 udp 4500 - comment { nat44 untranslated subnet 10.166.14.0/24 } - - comment { initiator profile } - ikev2 profile add respond - ikev2 profile set respond udp-encap - ikev2 profile set respond auth rsa-sig cert-file /scratch/setups/initiatecert.pem - set ikev2 local key /scratch/setups/respondkey.pem - ikev2 profile set respond id local fqdn responder.my.net - ikev2 profile set respond id remote fqdn initiator.my.net - - ikev2 profile set respond traffic-selector remote ip-range 10.166.14.0 - 10.166.14.255 port-range 0 - 65535 protocol 0 - ikev2 profile set respond traffic-selector local ip-range 10.219.188.0 - 10.219.188.255 port-range 0 - 65535 protocol 0 - - ikev2 profile set respond responder host-eth1 10.26.68.100 - ikev2 profile set respond ike-crypto-alg aes-cbc 256 ike-integ-alg sha1-96 ike-dh modp-2048 - ikev2 profile set respond esp-crypto-alg aes-cbc 256 esp-integ-alg sha1-96 esp-dh ecp-256 - ikev2 profile set respond sa-lifetime 3600 10 5 0 - - create ipip tunnel src 10.26.68.101 dst 10.26.68.100 - ikev2 profile set respond tunnel ipip0 - ikev2 initiate sa-init respond - - set int mtu packet 1390 ipip0 - set int unnum ipip0 use host-eth1 - ip route add 10.166.14.0/24 via ipip0 -``` - -IKEv2 certificate setup ------------------------ - -In both of the vpp configurations, you'll see "/scratch/setups/xxx.pem" -mentioned. These certificates are used in the ikev2 key exchange. - -Here's how to generate the certificates: - -``` - openssl req -x509 -nodes -newkey rsa:4096 -keyout respondkey.pem -out respondcert.pem -days 3560 - openssl x509 -text -noout -in respondcert.pem - openssl req -x509 -nodes -newkey rsa:4096 -keyout initiatekey.pem -out initiatecert.pem -days 3560 - openssl x509 -text -noout -in initiatecert.pem -``` - -Make sure that the "respond" and "initiate" configurations point to the certificates. - -DHCPv6 server setup -------------------- - -If you need an ipv6 dhcp server to test ipv6 prefix delegation, -create the "dhcpserver" container as shown above. - -Install the "isc-dhcp-server" Debian package: - -``` - sudo apt-get install isc-dhcp-server -``` - -### /etc/dhcp/dhcpd6.conf - -Edit the dhcpv6 configuration and add an ipv6 subnet with prefix -delegation. For example: - -``` - subnet6 2001:db01:0:1::/64 { - range6 2001:db01:0:1::1 2001:db01:0:1::9; - prefix6 2001:db01:0:100:: 2001:db01:0:200::/56; - } -``` - -Add an ipv6 address on eth1, which is connected to the "internet" -bridge, and start the dhcp server. I use the following trivial bash -script, which runs the dhcp6 server in the foreground and produces -dhcp traffic spew: - -``` - #!/bin/bash - ifconfig eth1 inet6 add 2001:db01:0:1::10/64 || true - dhcpd -6 -d -cf /etc/dhcp/dhcpd6.conf -``` - -The "|| true" bit keeps going if eth1 already has the indicated ipv6 -address. - -Container / Host Interoperation -------------------------------- - -Host / container interoperation is highly desirable. If the host and a -set of containers don't run the same distro _and distro version_, it's -reasonably likely that the glibc versions won't match. That, in turn, -makes vpp binaries built in one environment fail in the other. - -Trying to install multiple versions of glibc - especially at the host -level - often ends very badly and is _not recommended_. It's not just -glibc, either. The dynamic loader ld-linux-xxx-so.2 is glibc version -specific. - -Fortunately, it's reasonable easy to build lxd container images based on -specific Ubuntu or Debian versions. - -### Create a custom root filesystem image - -First, install the "debootstrap" tool: - -``` - sudo apt-get install debootstrap -``` - -Make a temp directory, and use debootstrap to populate it. In this -example, we create an Ubuntu 20.04 (focal fossa) base image: - -``` - # mkdir /tmp/myroot - # debootstrap focal /tmp/myroot http://archive.ubuntu.com/ubuntu -``` - -To tinker with the base image (if desired): - -``` - # chroot /tmp/myroot - <add packages, etc.> - # exit -``` - -Make a compressed tarball of the base image: - -``` - # tar zcf /tmp/rootfs.tar.gz -C /tmp/myroot . -``` - -Create a "metadata.yaml" file which describes the base image: - -``` - architecture: "x86_64" - # To get current date in Unix time, use `date +%s` command - creation_date: 1458040200 - properties: - architecture: "x86_64" - description: "My custom Focal Fossa image" - os: "Ubuntu" - release: "focal" -``` - -Make a compressed tarball of metadata.yaml: - -``` - # tar zcf metadata.tar.gz metadata.yaml -``` - -Import the image into lxc / lxd: - -``` - $ lxc image import metadata.tar.gz rootfd.tar.gz --alias focal-base -``` - -### Create a container which uses the customized base image: - -``` - $ lxc launch focal-base focaltest - $ lxc exec focaltest bash -``` - -The next several steps should be executed in the container, in the -bash shell spun up by "lxc exec..." - -### Configure container networking - -In the container, create /etc/netplan/50-cloud-init.yaml: - -``` - network: - version: 2 - ethernets: - eth0: - dhcp4: true -``` - -Use "cat > /etc/netplan/50-cloud-init.yaml", and cut-'n-paste if your -favorite text editor is AWOL. - -Apply the configuration: - -``` - # netplan apply -``` - -At this point, eth0 should have an ip address, and you should see -a default route with "route -n". - -### Configure apt - -Again, in the container, set up /etc/apt/sources.list via cut-'n-paste -from a recently update "focal fossa" host. Something like so: - -``` - deb http://us.archive.ubuntu.com/ubuntu/ focal main restricted - deb http://us.archive.ubuntu.com/ubuntu/ focal-updates main restricted - deb http://us.archive.ubuntu.com/ubuntu/ focal universe - deb http://us.archive.ubuntu.com/ubuntu/ focal-updates universe - deb http://us.archive.ubuntu.com/ubuntu/ focal multiverse - deb http://us.archive.ubuntu.com/ubuntu/ focal-updates multiverse - deb http://us.archive.ubuntu.com/ubuntu/ focal-backports main restricted universe multiverse - deb http://security.ubuntu.com/ubuntu focal-security main restricted - deb http://security.ubuntu.com/ubuntu focal-security universe - deb http://security.ubuntu.com/ubuntu focal-security multiverse -``` - -"apt-get update" and "apt-install" should produce reasonable results. -Suggest "apt-get install make git". - -At this point, you can use the "/scratch" sharepoint (or similar) to -execute "make install-dep install-ext-deps" to set up the container -with the vpp toolchain; proceed as desired. |