summaryrefslogtreecommitdiffstats
path: root/docs/usecases/contiv/MANUAL_INSTALL.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/usecases/contiv/MANUAL_INSTALL.md')
-rw-r--r--docs/usecases/contiv/MANUAL_INSTALL.md482
1 files changed, 0 insertions, 482 deletions
diff --git a/docs/usecases/contiv/MANUAL_INSTALL.md b/docs/usecases/contiv/MANUAL_INSTALL.md
deleted file mode 100644
index 35506db0d16..00000000000
--- a/docs/usecases/contiv/MANUAL_INSTALL.md
+++ /dev/null
@@ -1,482 +0,0 @@
-# Manual Installation
-This document describes how to clone the Contiv repository and then use [kubeadm][1] to manually install Kubernetes
-with Contiv-VPP networking on one or more bare metal or VM hosts.
-
-## Clone the Contiv Repository
-To clone the Contiv repository enter the following command:
-```
-git clone https://github.com/contiv/vpp/<repository-name>
-```
-**Note:** Replace *<repository-name>* with the name you want assigned to your cloned contiv repository.
-
-The cloned repository has important folders that contain content that are referenced in this Contiv documentation; those folders are noted below:
-```
-vpp-contiv2$ ls
-build build-root doxygen gmod LICENSE Makefile RELEASE.md src
-build-data docs extras INFO.yaml MAINTAINERS README.md sphinx_venv test
-```
-## Preparing Your Hosts
-
-### Host-specific Configurations
-- **VmWare VMs**: the vmxnet3 driver is required on each interface that will
- be used by VPP. Please see [here][13] for instructions how to install the
- vmxnet3 driver on VmWare Fusion.
-
-### Setting up Network Adapter(s)
-#### Setting up DPDK
-DPDK setup must be completed **on each node** as follows:
-
-- Load the PCI UIO driver:
- ```
- $ sudo modprobe uio_pci_generic
- ```
-
-- Verify that the PCI UIO driver has loaded successfully:
- ```
- $ lsmod | grep uio
- uio_pci_generic 16384 0
- uio 20480 1 uio_pci_generic
- ```
-
- Please note that this driver needs to be loaded upon each server bootup,
- so you may want to add `uio_pci_generic` into the `/etc/modules` file,
- or a file in the `/etc/modules-load.d/` directory. For example, the
- `/etc/modules` file could look as follows:
- ```
- # /etc/modules: kernel modules to load at boot time.
- #
- # This file contains the names of kernel modules that should be loaded
- # at boot time, one per line. Lines beginning with "#" are ignored.
- uio_pci_generic
- ```
-#### Determining Network Adapter PCI Addresses
-You need the PCI address of the network interface that VPP will use for the multi-node pod interconnect. On Debian-based
-distributions, you can use `lshw`(*):
-
-```
-$ sudo lshw -class network -businfo
-Bus info Device Class Description
-====================================================
-pci@0000:00:03.0 ens3 network Virtio network device
-pci@0000:00:04.0 ens4 network Virtio network device
-```
-**Note:** On CentOS/RedHat/Fedora distributions, `lshw` may not be available by default, install it by issuing the following command:
- ```
- yum -y install lshw
- ```
-
-#### Configuring vswitch to Use Network Adapters
-Finally, you need to set up the vswitch to use the network adapters:
-
-- [Setup on a node with a single NIC][14]
-- [Setup a node with multiple NICs][15]
-
-### Using a Node Setup Script
-You can perform the above steps using the [node setup script][17].
-
-## Installing Kubernetes with Contiv-VPP CNI plugin
-After the nodes you will be using in your K8s cluster are prepared, you can
-install the cluster using [kubeadm][1].
-
-### (1/4) Installing Kubeadm on Your Hosts
-For first-time installation, see [Installing kubeadm][6]. To update an
-existing installation, you should do a `apt-get update && apt-get upgrade`
-or `yum update` to get the latest version of kubeadm.
-
-On each host with multiple NICs where the NIC that will be used for Kubernetes
-management traffic is not the one pointed to by the default route out of the
-host, a [custom management network][12] for Kubernetes must be configured.
-
-#### Using Kubernetes 1.10 and Above
-In K8s 1.10, support for huge pages in a pod has been introduced. For now, this
-feature must be either disabled or memory limit must be defined for vswitch container.
-
-To disable huge pages, perform the following
-steps as root:
-* Using your favorite editor, disable huge pages in the kubelet configuration
- file (`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` or `/etc/default/kubelet` for version 1.11+):
-```
- Environment="KUBELET_EXTRA_ARGS=--feature-gates HugePages=false"
-```
-* Restart the kubelet daemon:
-```
- systemctl daemon-reload
- systemctl restart kubelet
-```
-
-To define memory limit, append the following snippet to vswitch container in deployment yaml file:
-```
- resources:
- limits:
- hugepages-2Mi: 1024Mi
- memory: 1024Mi
-
-```
-or set `contiv.vswitch.defineMemoryLimits` to `true` in [helm values](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/README.md).
-
-### (2/4) Initializing Your Master
-Before initializing the master, you may want to [remove][8] any
-previously installed K8s components. Then, proceed with master initialization
-as described in the [kubeadm manual][3]. Execute the following command as
-root:
-```
-kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16
-```
-**Note:** `kubeadm init` will autodetect the network interface to advertise
-the master on as the interface with the default gateway. If you want to use a
-different interface (i.e. a custom management network setup), specify the
-`--apiserver-advertise-address=<ip-address>` argument to kubeadm init. For
-example:
-```
-kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16 --apiserver-advertise-address=192.168.56.106
-```
-**Note:** The CIDR specified with the flag `--pod-network-cidr` is used by
-kube-proxy, and it **must include** the `PodSubnetCIDR` from the `IPAMConfig`
-section in the Contiv-vpp config map in Contiv-vpp's deployment file
-[contiv-vpp.yaml](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/values.yaml). Pods in the host network namespace
-are a special case; they share their respective interfaces and IP addresses with
-the host. For proxying to work properly it is therefore required for services
-with backends running on the host to also **include the node management IP**
-within the `--pod-network-cidr` subnet. For example, with the default
-`PodSubnetCIDR=10.1.0.0/16` and `PodIfIPCIDR=10.2.1.0/24`, the subnet
-`10.3.0.0/16` could be allocated for the management network and
-`--pod-network-cidr` could be defined as `10.0.0.0/8`, so as to include IP
-addresses of all pods in all network namespaces:
-```
-kubeadm init --token-ttl 0 --pod-network-cidr=10.0.0.0/8 --apiserver-advertise-address=10.3.1.1
-```
-
-If Kubernetes was initialized successfully, it prints out this message:
-```
-Your Kubernetes master has initialized successfully!
-```
-
-After successful initialization, don't forget to set up your .kube directory
-as a regular user (as instructed by `kubeadm`):
-```bash
-mkdir -p $HOME/.kube
-sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
-sudo chown $(id -u):$(id -g) $HOME/.kube/config
-```
-
-### (3/4) Installing the Contiv-VPP Pod Network
-If you have already used the Contiv-VPP plugin before, you may need to pull
-the most recent Docker images on each node:
-```
-bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/pull-images.sh)
-```
-
-Install the Contiv-VPP network for your cluster as follows:
-
-- If you do not use the STN feature, install Contiv-vpp as follows:
- ```
- kubectl apply -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
- ```
-
-- If you use the STN feature, download the `contiv-vpp.yaml` file:
- ```
- wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
- ```
- Then edit the STN configuration as described [here][16]. Finally, create
- the Contiv-vpp deployment from the edited file:
- ```
- kubectl apply -f ./contiv-vpp.yaml
- ```
-
-Beware contiv-etcd data is persisted in `/var/etcd` by default. It has to be cleaned up manually after `kubeadm reset`.
-Otherwise outdated data will be loaded by a subsequent deployment.
-
-You can also generate random subfolder, alternatively:
-
-```
-curl --silent https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml | sed "s/\/var\/etcd\/contiv-data/\/var\/etcd\/contiv-data\/$RANDOM/g" | kubectl apply -f -
-```
-
-#### Deployment Verification
-After some time, all contiv containers should enter the running state:
-```
-root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
-NAME READY STATUS RESTARTS AGE IP NODE
-...
-contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
-contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
-contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
-```
-In particular, make sure that the Contiv-VPP pod IP addresses are the same as
-the IP address specified in the `--apiserver-advertise-address=<ip-address>`
-argument to kubeadm init.
-
-Verify that the VPP successfully grabbed the network interface specified in
-the VPP startup config (`GigabitEthernet0/4/0` in our case):
-```
-$ sudo vppctl
-vpp# sh inter
- Name Idx State Counter Count
-GigabitEthernet0/4/0 1 up rx packets 1294
- rx bytes 153850
- tx packets 512
- tx bytes 21896
- drops 962
- ip4 1032
-host-40df9b44c3d42f4 3 up rx packets 126601
- rx bytes 44628849
- tx packets 132155
- tx bytes 27205450
- drops 24
- ip4 126585
- ip6 16
-host-vppv2 2 up rx packets 132162
- rx bytes 27205824
- tx packets 126658
- tx bytes 44634963
- drops 15
- ip4 132147
- ip6 14
-local0 0 down
-```
-
-You should also see the interface to kube-dns (`host-40df9b44c3d42f4`) and to the
-node's IP stack (`host-vppv2`).
-
-#### Master Isolation (Optional)
-By default, your cluster will not schedule pods on the master for security
-reasons. If you want to be able to schedule pods on the master, (e.g., for a
-single-machine Kubernetes cluster for development), then run:
-
-```
-kubectl taint nodes --all node-role.kubernetes.io/master-
-```
-More details about installing the pod network can be found in the
-[kubeadm manual][4].
-
-### (4/4) Joining Your Nodes
-To add a new node to your cluster, run as root the command that was output
-by kubeadm init. For example:
-```
-kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
-```
-More details can be found int the [kubeadm manual][5].
-
-#### Deployment Verification
-After some time, all contiv containers should enter the running state:
-```
-root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
-NAME READY STATUS RESTARTS AGE IP NODE
-contiv-etcd-gwc84 1/1 Running 0 14h 192.168.56.106 cvpp
-contiv-ksr-5c2vk 1/1 Running 2 14h 192.168.56.106 cvpp
-contiv-vswitch-h6759 2/2 Running 0 14h 192.168.56.105 cvpp-slave2
-contiv-vswitch-l59nv 2/2 Running 0 14h 192.168.56.106 cvpp
-etcd-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
-kube-apiserver-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
-kube-controller-manager-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
-kube-dns-545bc4bfd4-fr6j9 3/3 Running 0 14h 10.1.134.2 cvpp
-kube-proxy-q8sv2 1/1 Running 0 14h 192.168.56.106 cvpp
-kube-proxy-s8kv9 1/1 Running 0 14h 192.168.56.105 cvpp-slave2
-kube-scheduler-cvpp 1/1 Running 0 14h 192.168.56.106 cvpp
-```
-In particular, verify that a vswitch pod and a kube-proxy pod is running on
-each joined node, as shown above.
-
-On each joined node, verify that the VPP successfully grabbed the network
-interface specified in the VPP startup config (`GigabitEthernet0/4/0` in
-our case):
-```
-$ sudo vppctl
-vpp# sh inter
- Name Idx State Counter Count
-GigabitEthernet0/4/0 1 up
-...
-```
-From the vpp CLI on a joined node you can also ping kube-dns to verify
-node-to-node connectivity. For example:
-```
-vpp# ping 10.1.134.2
-64 bytes from 10.1.134.2: icmp_seq=1 ttl=64 time=.1557 ms
-64 bytes from 10.1.134.2: icmp_seq=2 ttl=64 time=.1339 ms
-64 bytes from 10.1.134.2: icmp_seq=3 ttl=64 time=.1295 ms
-64 bytes from 10.1.134.2: icmp_seq=4 ttl=64 time=.1714 ms
-64 bytes from 10.1.134.2: icmp_seq=5 ttl=64 time=.1317 ms
-
-Statistics: 5 sent, 5 received, 0% packet loss
-```
-### Deploying Example Applications
-#### Simple Deployment
-You can go ahead and create a simple deployment:
-```
-$ kubectl run nginx --image=nginx --replicas=2
-```
-
-Use `kubectl describe pod` to get the IP address of a pod, e.g.:
-```
-$ kubectl describe pod nginx | grep IP
-```
-You should see two ip addresses, for example:
-```
-IP: 10.1.1.3
-IP: 10.1.1.4
-```
-
-You can check the pods' connectivity in one of the following ways:
-* Connect to the VPP debug CLI and ping any pod:
-```
- sudo vppctl
- vpp# ping 10.1.1.3
-```
-* Start busybox and ping any pod:
-```
- kubectl run busybox --rm -ti --image=busybox /bin/sh
- If you don't see a command prompt, try pressing enter.
- / #
- / # ping 10.1.1.3
-
-```
-* You should be able to ping any pod from the host:
-```
- ping 10.1.1.3
-```
-
-#### Deploying Pods on Different Nodes
-to enable pod deployment on the master, untaint the master first:
-```
-kubectl taint nodes --all node-role.kubernetes.io/master-
-```
-
-In order to verify inter-node pod connectivity, we need to tell Kubernetes
-to deploy one pod on the master node and one POD on the worker. For this,
-we can use node selectors.
-
-In your deployment YAMLs, add the `nodeSelector` sections that refer to
-preferred node hostnames, e.g.:
-```
- nodeSelector:
- kubernetes.io/hostname: vm5
-```
-
-Example of whole JSONs:
-```
-apiVersion: v1
-kind: Pod
-metadata:
- name: nginx1
-spec:
- nodeSelector:
- kubernetes.io/hostname: vm5
- containers:
- - name: nginx
-
- : nginx
-```
-
-```
-apiVersion: v1
-kind: Pod
-metadata:
- name: nginx2
-spec:
- nodeSelector:
- kubernetes.io/hostname: vm6
- containers:
- - name: nginx
- image: nginx
-```
-
-After deploying the JSONs, verify they were deployed on different hosts:
-```
-$ kubectl get pods -o wide
-NAME READY STATUS RESTARTS AGE IP NODE
-nginx1 1/1 Running 0 13m 10.1.36.2 vm5
-nginx2 1/1 Running 0 13m 10.1.219.3 vm6
-```
-
-Now you can verify the connectivity to both nginx PODs from a busybox POD:
-```
-kubectl run busybox --rm -it --image=busybox /bin/sh
-
-/ # wget 10.1.36.2
-Connecting to 10.1.36.2 (10.1.36.2:80)
-index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
-
-/ # rm index.html
-
-/ # wget 10.1.219.3
-Connecting to 10.1.219.3 (10.1.219.3:80)
-index.html 100% |*******************************************************************************************************************************************************************| 612 0:00:00 ETA
-```
-
-### Uninstalling Contiv-VPP
-To uninstall the network plugin itself, use `kubectl`:
-```
-kubectl delete -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
-```
-
-### Tearing down Kubernetes
-* First, drain the node and make sure that the node is empty before
-shutting it down:
-```
- kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
- kubectl delete node <node name>
-```
-* Next, on the node being removed, reset all kubeadm installed state:
-```
- rm -rf $HOME/.kube
- sudo su
- kubeadm reset
-```
-
-* If you added environment variable definitions into
- `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`, this would have been a process from the [Custom Management Network file][10], then remove the definitions now.
-
-### Troubleshooting
-Some of the issues that can occur during the installation are:
-
-- Forgetting to create and initialize the `.kube` directory in your home
- directory (As instructed by `kubeadm init --token-ttl 0`). This can manifest
- itself as the following error:
- ```
- W1017 09:25:43.403159 2233 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://192.168.209.128:6443/swagger-2.0.0.pb-v1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")), falling back to swagger
- Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
- ```
-- Previous installation lingering on the file system.
- `'kubeadm init --token-ttl 0` fails to initialize kubelet with one or more
- of the following error messages:
- ```
- ...
- [kubelet-check] It seems like the kubelet isn't running or healthy.
- [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
- ...
- ```
-
-If you run into any of the above issues, try to clean up and reinstall as root:
-```
-sudo su
-rm -rf $HOME/.kube
-kubeadm reset
-kubeadm init --token-ttl 0
-rm -rf /var/etcd/contiv-data
-rm -rf /var/bolt/bolt.db
-```
-
-## Contiv-specific kubeadm installation on Aarch64
-Supplemental instructions apply when using Contiv-VPP for Aarch64. Most
-installation steps for Aarch64 are the same as that described earlier in this
-chapter, so you should firstly read it before you start the installation on
-Aarch64 platform.
-
-Use the [Aarch64-specific kubeadm install instructions][18] to manually install
-Kubernetes with Contiv-VPP networking on one or more bare-metals of Aarch64 platform.
-
-[1]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
-[3]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#initializing-your-master
-[4]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
-[5]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes
-[6]: https://kubernetes.io/docs/setup/independent/install-kubeadm/
-[8]: #tearing-down-kubernetes
-[10]: https://github.com/contiv/vpp/blob/master/docs/CUSTOM_MGMT_NETWORK.md#setting-up-a-custom-management-network-on-multi-homed-nodes
-[11]: ../vagrant/README.md
-[12]: https://github.com/contiv/vpp/tree/master/docs/CUSTOM_MGMT_NETWORK.md
-[13]: https://github.com/contiv/vpp/tree/master/docs/VMWARE_FUSION_HOST.md
-[14]: https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md
-[15]: https://github.com/contiv/vpp/tree/master/docs/MULTI_NIC_SETUP.md
-[16]: https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md#configuring-stn-in-contiv-vpp-k8s-deployment-files
-[17]: https://github.com/contiv/vpp/tree/master/k8s/README.md#setup-node-sh
-[18]: https://github.com/contiv/vpp/blob/master/docs/arm64/MANUAL_INSTALL_ARM64.md