summaryrefslogtreecommitdiffstats
path: root/docs/usecases/contiv/MANUAL_INSTALL.rst
diff options
context:
space:
mode:
Diffstat (limited to 'docs/usecases/contiv/MANUAL_INSTALL.rst')
-rw-r--r--docs/usecases/contiv/MANUAL_INSTALL.rst609
1 files changed, 609 insertions, 0 deletions
diff --git a/docs/usecases/contiv/MANUAL_INSTALL.rst b/docs/usecases/contiv/MANUAL_INSTALL.rst
new file mode 100644
index 00000000000..4e0d0c6c52b
--- /dev/null
+++ b/docs/usecases/contiv/MANUAL_INSTALL.rst
@@ -0,0 +1,609 @@
+Manual Installation
+===================
+
+This document describes how to clone the Contiv repository and then use
+`kubeadm <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/>`__
+to manually install Kubernetes with Contiv-VPP networking on one or more
+bare metal or VM hosts.
+
+Clone the Contiv Repository
+---------------------------
+
+To clone the Contiv repository enter the following command:
+
+::
+
+ git clone https://github.com/contiv/vpp/<repository-name>
+
+**Note:** Replace ** with the name you want assigned to your cloned
+contiv repository.
+
+The cloned repository has important folders that contain content that
+are referenced in this Contiv documentation; those folders are noted
+below:
+
+::
+
+ vpp-contiv2$ ls
+ build build-root doxygen gmod LICENSE Makefile RELEASE.md src
+ build-data docs extras INFO.yaml MAINTAINERS README.md sphinx_venv test
+
+Preparing Your Hosts
+--------------------
+
+Host-specific Configurations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- **VmWare VMs**: the vmxnet3 driver is required on each interface that
+ will be used by VPP. Please see
+ `here <https://github.com/contiv/vpp/tree/master/docs/VMWARE_FUSION_HOST.md>`__
+ for instructions how to install the vmxnet3 driver on VmWare Fusion.
+
+Setting up Network Adapter(s)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Setting up DPDK
+^^^^^^^^^^^^^^^
+
+DPDK setup must be completed **on each node** as follows:
+
+- Load the PCI UIO driver:
+
+ ::
+
+ $ sudo modprobe uio_pci_generic
+
+- Verify that the PCI UIO driver has loaded successfully:
+
+ ::
+
+ $ lsmod | grep uio
+ uio_pci_generic 16384 0
+ uio 20480 1 uio_pci_generic
+
+ Please note that this driver needs to be loaded upon each server
+ bootup, so you may want to add ``uio_pci_generic`` into the
+ ``/etc/modules`` file, or a file in the ``/etc/modules-load.d/``
+ directory. For example, the ``/etc/modules`` file could look as
+ follows:
+
+ ::
+
+ # /etc/modules: kernel modules to load at boot time.
+ #
+ # This file contains the names of kernel modules that should be loaded
+ # at boot time, one per line. Lines beginning with "#" are ignored.
+ uio_pci_generic
+
+ .. rubric:: Determining Network Adapter PCI Addresses
+ :name: determining-network-adapter-pci-addresses
+
+ You need the PCI address of the network interface that VPP will use
+ for the multi-node pod interconnect. On Debian-based distributions,
+ you can use ``lshw``\ (*):
+
+::
+
+ $ sudo lshw -class network -businfo
+ Bus info Device Class Description
+ ====================================================
+ pci@0000:00:03.0 ens3 network Virtio network device
+ pci@0000:00:04.0 ens4 network Virtio network device
+
+**Note:** On CentOS/RedHat/Fedora distributions, ``lshw`` may not be
+available by default, install it by issuing the following command:
+``yum -y install lshw``
+
+Configuring vswitch to Use Network Adapters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Finally, you need to set up the vswitch to use the network adapters:
+
+- `Setup on a node with a single
+ NIC <https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md>`__
+- `Setup a node with multiple
+ NICs <https://github.com/contiv/vpp/tree/master/docs/MULTI_NIC_SETUP.md>`__
+
+Using a Node Setup Script
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can perform the above steps using the `node setup
+script <https://github.com/contiv/vpp/tree/master/k8s/README.md#setup-node-sh>`__.
+
+Installing Kubernetes with Contiv-VPP CNI plugin
+------------------------------------------------
+
+After the nodes you will be using in your K8s cluster are prepared, you
+can install the cluster using
+`kubeadm <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/>`__.
+
+(1/4) Installing Kubeadm on Your Hosts
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For first-time installation, see `Installing
+kubeadm <https://kubernetes.io/docs/setup/independent/install-kubeadm/>`__.
+To update an existing installation, you should do a
+``apt-get update && apt-get upgrade`` or ``yum update`` to get the
+latest version of kubeadm.
+
+On each host with multiple NICs where the NIC that will be used for
+Kubernetes management traffic is not the one pointed to by the default
+route out of the host, a `custom management
+network <https://github.com/contiv/vpp/tree/master/docs/CUSTOM_MGMT_NETWORK.md>`__
+for Kubernetes must be configured.
+
+Using Kubernetes 1.10 and Above
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In K8s 1.10, support for huge pages in a pod has been introduced. For
+now, this feature must be either disabled or memory limit must be
+defined for vswitch container.
+
+To disable huge pages, perform the following steps as root: \* Using
+your favorite editor, disable huge pages in the kubelet configuration
+file (``/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`` or
+``/etc/default/kubelet`` for version 1.11+):
+
+::
+
+ Environment="KUBELET_EXTRA_ARGS=--feature-gates HugePages=false"
+
+- Restart the kubelet daemon:
+
+::
+
+ systemctl daemon-reload
+ systemctl restart kubelet
+
+To define memory limit, append the following snippet to vswitch
+container in deployment yaml file:
+
+::
+
+ resources:
+ limits:
+ hugepages-2Mi: 1024Mi
+ memory: 1024Mi
+
+or set ``contiv.vswitch.defineMemoryLimits`` to ``true`` in `helm
+values <https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/README.md>`__.
+
+(2/4) Initializing Your Master
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Before initializing the master, you may want to
+`remove <#tearing-down-kubernetes>`__ any previously installed K8s
+components. Then, proceed with master initialization as described in the
+`kubeadm
+manual <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#initializing-your-master>`__.
+Execute the following command as root:
+
+::
+
+ kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16
+
+**Note:** ``kubeadm init`` will autodetect the network interface to
+advertise the master on as the interface with the default gateway. If
+you want to use a different interface (i.e. a custom management network
+setup), specify the ``--apiserver-advertise-address=<ip-address>``
+argument to kubeadm init. For example:
+
+::
+
+ kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16 --apiserver-advertise-address=192.168.56.106
+
+**Note:** The CIDR specified with the flag ``--pod-network-cidr`` is
+used by kube-proxy, and it **must include** the ``PodSubnetCIDR`` from
+the ``IPAMConfig`` section in the Contiv-vpp config map in Contiv-vpp’s
+deployment file
+`contiv-vpp.yaml <https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/values.yaml>`__.
+Pods in the host network namespace are a special case; they share their
+respective interfaces and IP addresses with the host. For proxying to
+work properly it is therefore required for services with backends
+running on the host to also **include the node management IP** within
+the ``--pod-network-cidr`` subnet. For example, with the default
+``PodSubnetCIDR=10.1.0.0/16`` and ``PodIfIPCIDR=10.2.1.0/24``, the
+subnet ``10.3.0.0/16`` could be allocated for the management network and
+``--pod-network-cidr`` could be defined as ``10.0.0.0/8``, so as to
+include IP addresses of all pods in all network namespaces:
+
+::
+
+ kubeadm init --token-ttl 0 --pod-network-cidr=10.0.0.0/8 --apiserver-advertise-address=10.3.1.1
+
+If Kubernetes was initialized successfully, it prints out this message:
+
+::
+
+ Your Kubernetes master has initialized successfully!
+
+After successful initialization, don’t forget to set up your .kube
+directory as a regular user (as instructed by ``kubeadm``):
+
+.. code:: bash
+
+ mkdir -p $HOME/.kube
+ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+ sudo chown $(id -u):$(id -g) $HOME/.kube/config
+
+(3/4) Installing the Contiv-VPP Pod Network
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have already used the Contiv-VPP plugin before, you may need to
+pull the most recent Docker images on each node:
+
+::
+
+ bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/pull-images.sh)
+
+Install the Contiv-VPP network for your cluster as follows:
+
+- If you do not use the STN feature, install Contiv-vpp as follows:
+
+ ::
+
+ kubectl apply -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+
+- If you use the STN feature, download the ``contiv-vpp.yaml`` file:
+
+ ::
+
+ wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+
+ Then edit the STN configuration as described
+ `here <https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md#configuring-stn-in-contiv-vpp-k8s-deployment-files>`__.
+ Finally, create the Contiv-vpp deployment from the edited file:
+
+ ::
+
+ kubectl apply -f ./contiv-vpp.yaml
+
+Beware contiv-etcd data is persisted in ``/var/etcd`` by default. It has
+to be cleaned up manually after ``kubeadm reset``. Otherwise outdated
+data will be loaded by a subsequent deployment.
+
+You can also generate random subfolder, alternatively:
+
+::
+
+ curl --silent https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml | sed "s/\/var\/etcd\/contiv-data/\/var\/etcd\/contiv-data\/$RANDOM/g" | kubectl apply -f -
+
+Deployment Verification
+^^^^^^^^^^^^^^^^^^^^^^^
+
+After some time, all contiv containers should enter the running state:
+
+::
+
+ root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
+ NAME READY STATUS RESTARTS AGE IP NODE
+ ...
+ contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
+ contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
+ contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
+
+In particular, make sure that the Contiv-VPP pod IP addresses are the
+same as the IP address specified in the
+``--apiserver-advertise-address=<ip-address>`` argument to kubeadm init.
+
+Verify that the VPP successfully grabbed the network interface specified
+in the VPP startup config (``GigabitEthernet0/4/0`` in our case):
+
+::
+
+ $ sudo vppctl
+ vpp# sh inter
+ Name Idx State Counter Count
+ GigabitEthernet0/4/0 1 up rx packets 1294
+ rx bytes 153850
+ tx packets 512
+ tx bytes 21896
+ drops 962
+ ip4 1032
+ host-40df9b44c3d42f4 3 up rx packets 126601
+ rx bytes 44628849
+ tx packets 132155
+ tx bytes 27205450
+ drops 24
+ ip4 126585
+ ip6 16
+ host-vppv2 2 up rx packets 132162
+ rx bytes 27205824
+ tx packets 126658
+ tx bytes 44634963
+ drops 15
+ ip4 132147
+ ip6 14
+ local0 0 down
+
+You should also see the interface to kube-dns (``host-40df9b44c3d42f4``)
+and to the node’s IP stack (``host-vppv2``).
+
+Master Isolation (Optional)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, your cluster will not schedule pods on the master for
+security reasons. If you want to be able to schedule pods on the master,
+(e.g., for a single-machine Kubernetes cluster for development), then
+run:
+
+::
+
+ kubectl taint nodes --all node-role.kubernetes.io/master-
+
+More details about installing the pod network can be found in the
+`kubeadm
+manual <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network>`__.
+
+(4/4) Joining Your Nodes
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+To add a new node to your cluster, run as root the command that was
+output by kubeadm init. For example:
+
+::
+
+ kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
+
+More details can be found int the `kubeadm
+manual <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes>`__.
+
+.. _deployment-verification-1:
+
+Deployment Verification
+^^^^^^^^^^^^^^^^^^^^^^^
+
+After some time, all contiv containers should enter the running state:
+
+::
+
+ root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
+ NAME READY STATUS RESTARTS AGE IP NODE
+ contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
+ contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
+ contiv-vswitch-h6759 2/2 Running 0 14h 192.168.56.105 cvpp-slave2
+ contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
+ etcd-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-apiserver-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-controller-manager-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-dns-545bc4bfd4-fr6j9 3/3 Running 0 14h 10.1.134.2 cvpp
+ kube-proxy-q8sv2 1/1 Running 0 14h 192.168.56.106 cvpp
+ kube-proxy-s8kv9 1/1 Running 0 14h 192.168.56.105 cvpp-slave2
+ kube-scheduler-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
+
+In particular, verify that a vswitch pod and a kube-proxy pod is running
+on each joined node, as shown above.
+
+On each joined node, verify that the VPP successfully grabbed the
+network interface specified in the VPP startup config
+(``GigabitEthernet0/4/0`` in our case):
+
+::
+
+ $ sudo vppctl
+ vpp# sh inter
+ Name Idx State Counter Count
+ GigabitEthernet0/4/0 1 up
+ ...
+
+From the vpp CLI on a joined node you can also ping kube-dns to verify
+node-to-node connectivity. For example:
+
+::
+
+ vpp# ping 10.1.134.2
+ 64 bytes from 10.1.134.2: icmp_seq=1 ttl=64 time=.1557 ms
+ 64 bytes from 10.1.134.2: icmp_seq=2 ttl=64 time=.1339 ms
+ 64 bytes from 10.1.134.2: icmp_seq=3 ttl=64 time=.1295 ms
+ 64 bytes from 10.1.134.2: icmp_seq=4 ttl=64 time=.1714 ms
+ 64 bytes from 10.1.134.2: icmp_seq=5 ttl=64 time=.1317 ms
+
+ Statistics: 5 sent, 5 received, 0% packet loss
+
+Deploying Example Applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Simple Deployment
+^^^^^^^^^^^^^^^^^
+
+You can go ahead and create a simple deployment:
+
+::
+
+ $ kubectl run nginx --image=nginx --replicas=2
+
+Use ``kubectl describe pod`` to get the IP address of a pod, e.g.:
+
+::
+
+ $ kubectl describe pod nginx | grep IP
+
+You should see two ip addresses, for example:
+
+::
+
+ IP: 10.1.1.3
+ IP: 10.1.1.4
+
+You can check the pods’ connectivity in one of the following ways: \*
+Connect to the VPP debug CLI and ping any pod:
+
+::
+
+ sudo vppctl
+ vpp# ping 10.1.1.3
+
+- Start busybox and ping any pod:
+
+::
+
+ kubectl run busybox --rm -ti --image=busybox /bin/sh
+ If you don't see a command prompt, try pressing enter.
+ / #
+ / # ping 10.1.1.3
+
+- You should be able to ping any pod from the host:
+
+::
+
+ ping 10.1.1.3
+
+Deploying Pods on Different Nodes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+to enable pod deployment on the master, untaint the master first:
+
+::
+
+ kubectl taint nodes --all node-role.kubernetes.io/master-
+
+In order to verify inter-node pod connectivity, we need to tell
+Kubernetes to deploy one pod on the master node and one POD on the
+worker. For this, we can use node selectors.
+
+In your deployment YAMLs, add the ``nodeSelector`` sections that refer
+to preferred node hostnames, e.g.:
+
+::
+
+ nodeSelector:
+ kubernetes.io/hostname: vm5
+
+Example of whole JSONs:
+
+::
+
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ name: nginx1
+ spec:
+ nodeSelector:
+ kubernetes.io/hostname: vm5
+ containers:
+ - name: nginx
+
+ : nginx
+
+::
+
+ apiVersion: v1
+ kind: Pod
+ metadata:
+ name: nginx2
+ spec:
+ nodeSelector:
+ kubernetes.io/hostname: vm6
+ containers:
+ - name: nginx
+ image: nginx
+
+After deploying the JSONs, verify they were deployed on different hosts:
+
+::
+
+ $ kubectl get pods -o wide
+ NAME READY STATUS RESTARTS AGE IP NODE
+ nginx1 1/1 Running 0 13m 10.1.36.2 vm5
+ nginx2 1/1 Running 0 13m 10.1.219.3 vm6
+
+Now you can verify the connectivity to both nginx PODs from a busybox
+POD:
+
+::
+
+ kubectl run busybox --rm -it --image=busybox /bin/sh
+
+ / # wget 10.1.36.2
+ Connecting to 10.1.36.2 (10.1.36.2:80)
+ index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
+
+ / # rm index.html
+
+ / # wget 10.1.219.3
+ Connecting to 10.1.219.3 (10.1.219.3:80)
+ index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
+
+Uninstalling Contiv-VPP
+~~~~~~~~~~~~~~~~~~~~~~~
+
+To uninstall the network plugin itself, use ``kubectl``:
+
+::
+
+ kubectl delete -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+
+Tearing down Kubernetes
+~~~~~~~~~~~~~~~~~~~~~~~
+
+- First, drain the node and make sure that the node is empty before
+ shutting it down:
+
+::
+
+ kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
+ kubectl delete node <node name>
+
+- Next, on the node being removed, reset all kubeadm installed state:
+
+::
+
+ rm -rf $HOME/.kube
+ sudo su
+ kubeadm reset
+
+- If you added environment variable definitions into
+ ``/etc/systemd/system/kubelet.service.d/10-kubeadm.conf``, this would
+ have been a process from the `Custom Management Network
+ file <https://github.com/contiv/vpp/blob/master/docs/CUSTOM_MGMT_NETWORK.md#setting-up-a-custom-management-network-on-multi-homed-nodes>`__,
+ then remove the definitions now.
+
+Troubleshooting
+~~~~~~~~~~~~~~~
+
+Some of the issues that can occur during the installation are:
+
+- Forgetting to create and initialize the ``.kube`` directory in your
+ home directory (As instructed by ``kubeadm init --token-ttl 0``).
+ This can manifest itself as the following error:
+
+ ::
+
+ W1017 09:25:43.403159 2233 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://192.168.209.128:6443/swagger-2.0.0.pb-v1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")), falling back to swagger
+ Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
+
+- Previous installation lingering on the file system.
+ ``'kubeadm init --token-ttl 0`` fails to initialize kubelet with one
+ or more of the following error messages:
+
+ ::
+
+ ...
+ [kubelet-check] It seems like the kubelet isn't running or healthy.
+ [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
+ ...
+
+If you run into any of the above issues, try to clean up and reinstall
+as root:
+
+::
+
+ sudo su
+ rm -rf $HOME/.kube
+ kubeadm reset
+ kubeadm init --token-ttl 0
+ rm -rf /var/etcd/contiv-data
+ rm -rf /var/bolt/bolt.db
+
+Contiv-specific kubeadm installation on Aarch64
+-----------------------------------------------
+
+Supplemental instructions apply when using Contiv-VPP for Aarch64. Most
+installation steps for Aarch64 are the same as that described earlier in
+this chapter, so you should firstly read it before you start the
+installation on Aarch64 platform.
+
+Use the `Aarch64-specific kubeadm install
+instructions <https://github.com/contiv/vpp/blob/master/docs/arm64/MANUAL_INSTALL_ARM64.md>`__
+to manually install Kubernetes with Contiv-VPP networking on one or more
+bare-metals of Aarch64 platform.