aboutsummaryrefslogtreecommitdiffstats
path: root/docs/usecases/contiv/MANUAL_INSTALL.rst
diff options
context:
space:
mode:
authorNathan Skrzypczak <nathan.skrzypczak@gmail.com>2021-08-19 11:38:06 +0200
committerDave Wallace <dwallacelf@gmail.com>2021-10-13 23:22:32 +0000
commit9ad39c026c8a3c945a7003c4aa4f5cb1d4c80160 (patch)
tree3cca19635417e28ae381d67ae31c75df2925032d /docs/usecases/contiv/MANUAL_INSTALL.rst
parentf47122e07e1ecd0151902a3cabe46c60a99bee8e (diff)
docs: better docs, mv doxygen to sphinx
This patch refactors the VPP sphinx docs in order to make it easier to consume for external readers as well as VPP developers. It also makes sphinx the single source of documentation, which simplifies maintenance and operation. Most important updates are: - reformat the existing documentation as rst - split RELEASE.md and move it into separate rst files - remove section 'events' - remove section 'archive' - remove section 'related projects' - remove section 'feature by release' - remove section 'Various links' - make (Configuration reference, CLI docs, developer docs) top level items in the list - move 'Use Cases' as part of 'About VPP' - move 'Troubleshooting' as part of 'Getting Started' - move test framework docs into 'Developer Documentation' - add a 'Contributing' section for gerrit, docs and other contributer related infos - deprecate doxygen and test-docs targets - redirect the "make doxygen" target to "make docs" Type: refactor Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com> Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
Diffstat (limited to 'docs/usecases/contiv/MANUAL_INSTALL.rst')
-rw-r--r--docs/usecases/contiv/MANUAL_INSTALL.rst609
1 files changed, 609 insertions, 0 deletions
diff --git a/docs/usecases/contiv/MANUAL_INSTALL.rst b/docs/usecases/contiv/MANUAL_INSTALL.rst
new file mode 100644
index 00000000000..4e0d0c6c52b
--- /dev/null
+++ b/docs/usecases/contiv/MANUAL_INSTALL.rst
@@ -0,0 +1,609 @@
+Manual Installation
+===================
+
+This document describes how to clone the Contiv repository and then use
+`kubeadm <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/>`__
+to manually install Kubernetes with Contiv-VPP networking on one or more
+bare metal or VM hosts.
+
+Clone the Contiv Repository
+---------------------------
+
+To clone the Contiv repository enter the following command:
+
+::
+
+ git clone https://github.com/contiv/vpp/<repository-name>
+
+**Note:** Replace ** with the name you want assigned to your cloned
+contiv repository.
+
+The cloned repository has important folders that contain content that
+are referenced in this Contiv documentation; those folders are noted
+below:
+
+::
+
+ vpp-contiv2$ ls
+ build build-root doxygen gmod LICENSE Makefile RELEASE.md src
+ build-data docs extras INFO.yaml MAINTAINERS README.md sphinx_venv test
+
+Preparing Your Hosts
+--------------------
+
+Host-specific Configurations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- **VmWare VMs**: the vmxnet3 driver is required on each interface that
+ will be used by VPP. Please see
+ `here <https://github.com/contiv/vpp/tree/master/docs/VMWARE_FUSION_HOST.md>`__
+ for instructions how to install the vmxnet3 driver on VmWare Fusion.
+
+Setting up Network Adapter(s)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Setting up DPDK
+^^^^^^^^^^^^^^^
+
+DPDK setup must be completed **on each node** as follows:
+
+- Load the PCI UIO driver:
+
+ ::
+
+ $ sudo modprobe uio_pci_generic
+
+- Verify that the PCI UIO driver has loaded successfully:
+
+ ::
+
+ $ lsmod | grep uio
+ uio_pci_generic 16384 0
+ uio 20480 1 uio_pci_generic
+
+ Please note that this driver needs to be loaded upon each server
+ bootup, so you may want to add ``uio_pci_generic`` into the
+ ``/etc/modules`` file, or a file in the ``/etc/modules-load.d/``
+ directory. For example, the ``/etc/modules`` file could look as
+ follows:
+
+ ::
+
+ # /etc/modules: kernel modules to load at boot time.
+ #
+ # This file contains the names of kernel modules that should be loaded
+ # at boot time, one per line. Lines beginning with "#" are ignored.
+ uio_pci_generic
+
+ .. rubric:: Determining Network Adapter PCI Addresses
+ :name: determining-network-adapter-pci-addresses
+
+ You need the PCI address of the network interface that VPP will use
+ for the multi-node pod interconnect. On Debian-based distributions,
+ you can use ``lshw``\ (*):
+
+::
+
+ $ sudo lshw -class network -businfo
+ Bus info Device Class Description
+ ====================================================
+ pci@0000:00:03.0 ens3 network Virtio network device
+ pci@0000:00:04.0 ens4 network Virtio network device
+
+**Note:** On CentOS/RedHat/Fedora distributions, ``lshw`` may not be
+available by default, install it by issuing the following command:
+``yum -y install lshw``
+
+Configuring vswitch to Use Network Adapters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Finally, you need to set up the vswitch to use the network adapters:
+
+- `Setup on a node with a single
+ NIC <https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md>`__
+- `Setup a node with multiple
+ NICs <https://github.com/contiv/vpp/tree/master/docs/MULTI_NIC_SETUP.md>`__
+
+Using a Node Setup Script
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can perform the above steps using the `node setup
+script <https://github.com/contiv/vpp/tree/master/k8s/README.md#setup-node-sh>`__.
+
+Installing Kubernetes with Contiv-VPP CNI plugin
+------------------------------------------------
+
+After the nodes you will be using in your K8s cluster are prepared, you
+can install the cluster using
+`kubeadm <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/>`__.
+
+(1/4) Installing Kubeadm on Your Hosts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For first-time installation, see `Installing
+kubeadm <https://kubernetes.io/docs/setup/independent/install-kubeadm/>`__.
+To update an existing installation, you should do a
+``apt-get update && apt-get upgrade`` or ``yum update`` to get the
+latest version of kubeadm.
+
+On each host with multiple NICs where the NIC that will be used for
+Kubernetes management traffic is not the one pointed to by the default
+route out of the host, a `custom management
+network <https://github.com/contiv/vpp/tree/master/docs/CUSTOM_MGMT_NETWORK.md>`__
+for Kubernetes must be configured.
+
+Using Kubernetes 1.10 and Above
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In K8s 1.10, support for huge pages in a pod has been introduced. For
+now, this feature must be either disabled or memory limit must be
+defined for vswitch container.
+
+To disable huge pages, perform the following steps as root: \* Using
+your favorite editor, disable huge pages in the kubelet configuration
+file (``/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`` or
+``/etc/default/kubelet`` for version 1.11+):
+
+::
+
+ Environment="KUBELET_EXTRA_ARGS=--feature-gates HugePages=false"
+
+- Restart the kubelet daemon:
+
+::
+
+ systemctl daemon-reload
+ systemctl restart kubelet
+
+To define memory limit, append the following snippet to vswitch
+container in deployment yaml file:
+
+::
+
+ resources:
+ limits:
+ hugepages-2Mi: 1024Mi
+ memory: 1024Mi
+
+or set ``contiv.vswitch.defineMemoryLimits`` to ``true`` in `helm
+values <https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/README.md>`__.
+
+(2/4) Initializing Your Master
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Before initializing the master, you may want to
+`remove <#tearing-down-kubernetes>`__ any previously installed K8s
+components. Then, proceed with master initialization as described in the
+`kubeadm
+manual <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#initializing-your-master>`__.
+Execute the following command as root:
+
+::
+
+ kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16
+
+**Note:** ``kubeadm init`` will autodetect the network interface to
+advertise the master on as the interface with the default gateway. If
+you want to use a different interface (i.e. a custom management network
+setup), specify the ``--apiserver-advertise-address=<ip-address>``
+argument to kubeadm init. For example:
+
+::
+
+ kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16 --apiserver-advertise-address=192.168.56.106
+
+**Note:** The CIDR specified with the flag ``--pod-network-cidr`` is
+used by kube-proxy, and it **must include** the ``PodSubnetCIDR`` from
+the ``IPAMConfig`` section in the Contiv-vpp config map in Contiv-vpp’s
+deployment file
+`contiv-vpp.yaml <https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/values.yaml>`__.
+Pods in the host network namespace are a special case; they share their
+respective interfaces and IP addresses with the host. For proxying to
+work properly it is therefore required for services with backends
+running on the host to also **include the node management IP** within
+the ``--pod-network-cidr`` subnet. For example, with the default
+``PodSubnetCIDR=10.1.0.0/16`` and ``PodIfIPCIDR=10.2.1.0/24``, the
+subnet ``10.3.0.0/16`` could be allocated for the management network and
+``--pod-network-cidr`` could be defined as ``10.0.0.0/8``, so as to
+include IP addresses of all pods in all network namespaces:
+
+::
+
+ kubeadm init --token-ttl 0 --pod-network-cidr=10.0.0.0/8 --apiserver-advertise-address=10.3.1.1
+
+If Kubernetes was initialized successfully, it prints out this message:
+
+::
+
+ Your Kubernetes master has initialized successfully!
+
+After successful initialization, don’t forget to set up your .kube
+directory as a regular user (as instructed by ``kubeadm``):
+
+.. code:: bash
+
+ mkdir -p $HOME/.kube
+ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+ sudo chown $(id -u):$(id -g) $HOME/.kube/config
+
+(3/4) Installing the Contiv-VPP Pod Network
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have already used the Contiv-VPP plugin before, you may need to
+pull the most recent Docker images on each node:
+
+::
+
+ bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/pull-images.sh)
+
+Install the Contiv-VPP network for your cluster as follows:
+
+- If you do not use the STN feature, install Contiv-vpp as follows:
+
+ ::
+
+ kubectl apply -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+
+- If you use the STN feature, download the ``contiv-vpp.yaml`` file:
+
+ ::
+
+ wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+
+ Then edit the STN configuration as described
+ `here <https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md#configuring-stn-in-contiv-vpp-k8s-deployment-files>`__.
+ Finally, create the Contiv-vpp deployment from the edited file:
+
+ ::
+
+ kubectl apply -f ./contiv-vpp.yaml
+
+Beware contiv-etcd data is persisted in ``/var/etcd`` by default. It has
+to be cleaned up manually after ``kubeadm reset``. Otherwise outdated
+data will be loaded by a subsequent deployment.
+
+You can also generate random subfolder, alternatively:
+
+::
+
+ curl --silent https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml | sed "s/\/var\/etcd\/contiv-data/\/var\/etcd\/contiv-data\/$RANDOM/g" | kubectl apply -f -
+
+Deployment Verification
+^^^^^^^^^^^^^^^^^^^^^^^
+
+After some time, all contiv containers should enter the running state:
+
+::
+
+ root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
+ NAME READY STATUS RESTARTS AGE IP NODE
+ ...
+ contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
+ contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
+ contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
+
+In particular, make sure that the Contiv-VPP pod IP addresses are the
+same as the IP address specified in the
+``--apiserver-advertise-address=<ip-address>`` argument to kubeadm init.
+
+Verify that the VPP successfully grabbed the network interface specified
+in the VPP startup config (``GigabitEthernet0/4/0`` in our case):
+
+::
+
+ $ sudo vppctl
+ vpp# sh inter
+ Name Idx State Counter Count
+ GigabitEthernet0/4/0 1 up rx packets 1294
+ rx bytes 153850
+ tx packets 512
+ tx bytes 21896
+ drops 962
+ ip4 1032
+ host-40df9b44c3d42f4 3 up rx packets 126601
+ rx bytes 44628849
+ tx packets 132155
+ tx bytes 27205450
+ drops 24
+ ip4 126585
+ ip6 16
+ host-vppv2 2 up rx packets 132162
+ rx bytes 27205824
+ tx packets 126658
+ tx bytes 44634963
+ drops 15
+ ip4 132147
+ ip6 14
+ local0 0 down
+
+You should also see the interface to kube-dns (``host-40df9b44c3d42f4``)
+and to the node’s IP stack (``host-vppv2``).
+
+Master Isolation (Optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, your cluster will not schedule pods on the master for
+security reasons. If you want to be able to schedule pods on the master,
+(e.g., for a single-machine Kubernetes cluster for development), then
+run:
+
+::
+
+ kubectl taint nodes --all node-role.kubernetes.io/master-
+
+More details about installing the pod network can be found in the
+`kubeadm
+manual <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network>`__.
+
+(4/4) Joining Your Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+To add a new node to your cluster, run as root the command that was
+output by kubeadm init. For example:
+
+::
+
+ kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
+
+More details can be found int the `kubeadm
+manual <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes>`__.
+
+.. _deployment-verification-1:
+
+Deployment Verification
+^^^^^^^^^^^^^^^^^^^^^^^
+
+After some time, all contiv containers should enter the running state:
+
+::
+
+ root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
+ NAME READY STATUS RESTARTS AGE IP NODE
+ contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
+ contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
+ contiv-vswitch-h6759 2/2 Running 0 14h 192.168.56.105 cvpp-slave2
+ contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
+ etcd-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-apiserver-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-controller-manager-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-dns-545bc4bfd4-fr6j9 3/3 Running 0 14h 10.1.134.2 cvpp
+ kube-proxy-q8sv2 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-proxy-s8kv9 1/1 Running 0 14h 192.168.56.105 cvpp-slave2
+ kube-scheduler-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+
+In particular, verify that a vswitch pod and a kube-proxy pod is running
+on each joined node, as shown above.
+
+On each joined node, verify that the VPP successfully grabbed the
+network interface specified in the VPP startup config
+(``GigabitEthernet0/4/0`` in our case):
+
+::
+
+ $ sudo vppctl
+ vpp# sh inter
+ Name Idx State Counter Count
+ GigabitEthernet0/4/0 1 up
+ ...
+
+From the vpp CLI on a joined node you can also ping kube-dns to verify
+node-to-node connectivity. For example:
+
+::
+
+ vpp# ping 10.1.134.2
+ 64 bytes from 10.1.134.2: icmp_seq=1 ttl=64 time=.1557 ms
+ 64 bytes from 10.1.134.2: icmp_seq=2 ttl=64 time=.1339 ms
+ 64 bytes from 10.1.134.2: icmp_seq=3 ttl=64 time=.1295 ms
+ 64 bytes from 10.1.134.2: icmp_seq=4 ttl=64 time=.1714 ms
+ 64 bytes from 10.1.134.2: icmp_seq=5 ttl=64 time=.1317 ms
+
+ Statistics: 5 sent, 5 received, 0% packet loss
+
+Deploying Example Applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Simple Deployment
+^^^^^^^^^^^^^^^^^
+
+You can go ahead and create a simple deployment:
+
+::
+
+ $ kubectl run nginx --image=nginx --replicas=2
+
+Use ``kubectl describe pod`` to get the IP address of a pod, e.g.:
+
+::
+
+ $ kubectl describe pod nginx | grep IP
+
+You should see two ip addresses, for example:
+
+::
+
+ IP: 10.1.1.3
+ IP: 10.1.1.4
+
+You can check the pods’ connectivity in one of the following ways: \*
+Connect to the VPP debug CLI and ping any pod:
+
+::
+
+ sudo vppctl
+ vpp# ping 10.1.1.3
+
+- Start busybox and ping any pod:
+
+::
+
+ kubectl run busybox --rm -ti --image=busybox /bin/sh
+ If you don't see a command prompt, try pressing enter.
+ / #
+ / # ping 10.1.1.3
+
+- You should be able to ping any pod from the host:
+
+::
+
+ ping 10.1.1.3
+
+Deploying Pods on Different Nodes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+to enable pod deployment on the master, untaint the master first:
+
+::
+
+ kubectl taint nodes --all node-role.kubernetes.io/master-
+
+In order to verify inter-node pod connectivity, we need to tell
+Kubernetes to deploy one pod on the master node and one POD on the
+worker. For this, we can use node selectors.
+
+In your deployment YAMLs, add the ``nodeSelector`` sections that refer
+to preferred node hostnames, e.g.:
+
+::
+
+ nodeSelector:
+ kubernetes.io/hostname: vm5
+
+Example of whole JSONs:
+
+::
+
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ name: nginx1
+ spec:
+ nodeSelector:
+ kubernetes.io/hostname: vm5
+ containers:
+ - name: nginx
+
+ : nginx
+
+::
+
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ name: nginx2
+ spec:
+ nodeSelector:
+ kubernetes.io/hostname: vm6
+ containers:
+ - name: nginx
+ image: nginx
+
+After deploying the JSONs, verify they were deployed on different hosts:
+
+::
+
+ $ kubectl get pods -o wide
+ NAME READY STATUS RESTARTS AGE IP NODE
+ nginx1 1/1 Running 0 13m 10.1.36.2 vm5
+ nginx2 1/1 Running 0 13m 10.1.219.3 vm6
+
+Now you can verify the connectivity to both nginx PODs from a busybox
+POD:
+
+::
+
+ kubectl run busybox --rm -it --image=busybox /bin/sh
+
+ / # wget 10.1.36.2
+ Connecting to 10.1.36.2 (10.1.36.2:80)
+ index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
+
+ / # rm index.html
+
+ / # wget 10.1.219.3
+ Connecting to 10.1.219.3 (10.1.219.3:80)
+ index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
+
+Uninstalling Contiv-VPP
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To uninstall the network plugin itself, use ``kubectl``:
+
+::
+
+ kubectl delete -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+
+Tearing down Kubernetes
+~~~~~~~~~~~~~~~~~~~~~~~
+
+- First, drain the node and make sure that the node is empty before
+ shutting it down:
+
+::
+
+ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
+ kubectl delete node <node name>
+
+- Next, on the node being removed, reset all kubeadm installed state:
+
+::
+
+ rm -rf $HOME/.kube
+ sudo su
+ kubeadm reset
+
+- If you added environment variable definitions into
+ ``/etc/systemd/system/kubelet.service.d/10-kubeadm.conf``, this would
+ have been a process from the `Custom Management Network
+ file <https://github.com/contiv/vpp/blob/master/docs/CUSTOM_MGMT_NETWORK.md#setting-up-a-custom-management-network-on-multi-homed-nodes>`__,
+ then remove the definitions now.
+
+Troubleshooting
+~~~~~~~~~~~~~~~
+
+Some of the issues that can occur during the installation are:
+
+- Forgetting to create and initialize the ``.kube`` directory in your
+ home directory (As instructed by ``kubeadm init --token-ttl 0``).
+ This can manifest itself as the following error:
+
+ ::
+
+ W1017 09:25:43.403159 2233 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://192.168.209.128:6443/swagger-2.0.0.pb-v1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")), falling back to swagger
+ Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
+
+- Previous installation lingering on the file system.
+ ``'kubeadm init --token-ttl 0`` fails to initialize kubelet with one
+ or more of the following error messages:
+
+ ::
+
+ ...
+ [kubelet-check] It seems like the kubelet isn't running or healthy.
+ [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
+ ...
+
+If you run into any of the above issues, try to clean up and reinstall
+as root:
+
+::
+
+ sudo su
+ rm -rf $HOME/.kube
+ kubeadm reset
+ kubeadm init --token-ttl 0
+ rm -rf /var/etcd/contiv-data
+ rm -rf /var/bolt/bolt.db
+
+Contiv-specific kubeadm installation on Aarch64
+-----------------------------------------------
+
+Supplemental instructions apply when using Contiv-VPP for Aarch64. Most
+installation steps for Aarch64 are the same as that described earlier in
+this chapter, so you should firstly read it before you start the
+installation on Aarch64 platform.
+
+Use the `Aarch64-specific kubeadm install
+instructions <https://github.com/contiv/vpp/blob/master/docs/arm64/MANUAL_INSTALL_ARM64.md>`__
+to manually install Kubernetes with Contiv-VPP networking on one or more
+bare-metals of Aarch64 platform.