path: root/docs/usecases/contiv/MANUAL_INSTALL.md

# Manual Installation
This document describes how to clone the Contiv respository and then use [kubeadm][1] to manually install Kubernetes
with Contiv-VPP networking on one or more bare metal or VM hosts. 

## Clone the Contiv Respository
To clone the Contiv respository enter the following command:
```
git clone https://github.com/contiv/vpp/<repository-name>
```
**Note:** Replace *<repository-name>* with the name you want assigned to your cloned contiv repository.

The cloned repository has important folders that contain content that are referenced in this Contiv documentation; those folders are noted below:
```
vpp-contiv2$ ls
build       build-root  doxygen  gmod       LICENSE      Makefile   RELEASE.md   src
build-data  docs        extras   INFO.yaml  MAINTAINERS  README.md  sphinx_venv  test
```
## Preparing Your Hosts

### Host-specific Configurations
- **VmWare VMs**: the vmxnet3 driver is required on each interface that will
  be used by VPP. Please see [here][13] for instructions how to install the 
  vmxnet3 driver on VmWare Fusion.
  
### Setting up Network Adapter(s)
#### Setting up DPDK
DPDK setup must be completed **on each node** as follows:

- Load the PCI UIO driver:
  ```
  $ sudo modprobe uio_pci_generic
  ```

- Verify that the PCI UIO driver has loaded successfully:
  ```
  $ lsmod | grep uio
  uio_pci_generic        16384  0
  uio                    20480  1 uio_pci_generic
  ```

  Please note that this driver needs to be loaded upon each server bootup,
  so you may want to add `uio_pci_generic` into the `/etc/modules` file, 
  or a file in the `/etc/modules-load.d/` directory. For example, the 
  `/etc/modules` file could look as follows:
  ```
  # /etc/modules: kernel modules to load at boot time.
  #
  # This file contains the names of kernel modules that should be loaded
  # at boot time, one per line. Lines beginning with "#" are ignored.
  uio_pci_generic
  ```
#### Determining Network Adapter PCI Addresses
You need the PCI address of the network interface that VPP will use for the multi-node pod interconnect. On Debian-based
distributions, you can use `lshw`(*):

```
$ sudo lshw -class network -businfo
Bus info          Device      Class      Description
====================================================
pci@0000:00:03.0  ens3        network    Virtio network device
pci@0000:00:04.0  ens4        network    Virtio network device
```
**Note:** On CentOS/RedHat/Fedora distributions, `lshw` may not be available by default, install it by issuing the following command:
    ```
    yum -y install lshw
    ```

#### Configuring vswitch to Use Network Adapters
Finally, you need to set up the vswitch to use the network adapters:

- [Setup on a node with a single NIC][14]
- [Setup a node with multiple NICs][15]

### Using a Node Setup Script
You can perform the above steps using the [node setup script][17].

## Installing Kubernetes with Contiv-VPP CNI plugin
After the nodes you will be using in your K8s cluster are prepared, you can 
install the cluster using [kubeadm][1].

### (1/4) Installing Kubeadm on Your Hosts
For first-time installation, see [Installing kubeadm][6]. To update an
existing installation,  you should do a `apt-get update && apt-get upgrade`
or `yum update` to get the latest version of kubeadm.

On each host with multiple NICs where the NIC that will be used for Kubernetes
management traffic is not the one pointed to by the default route out of the 
host, a [custom management network][12] for Kubernetes must be configured.

#### Using Kubernetes 1.10 and Above
In K8s 1.10, support for huge pages in a pod has been introduced. For now, this
feature must be either disabled or memory limit must be defined for vswitch container.

To disable huge pages, perform the following
steps as root:
* Using your favorite editor, disable huge pages in the kubelet configuration 
  file (`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` or `/etc/default/kubelet` for version 1.11+):
```
  Environment="KUBELET_EXTRA_ARGS=--feature-gates HugePages=false"
```
* Restart the kubelet daemon:
```
  systemctl daemon-reload
  systemctl restart kubelet
```

To define memory limit, append the following snippet to vswitch container in deployment yaml file:
```
			resources:
              limits:
                hugepages-2Mi: 1024Mi
                memory: 1024Mi

```
or set `contiv.vswitch.defineMemoryLimits` to `true` in [helm values](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/README.md).

### (2/4) Initializing Your Master
Before initializing the master, you may want to [remove][8] any
previously installed K8s components. Then, proceed with master initialization
as described in the [kubeadm manual][3]. Execute the following command as
root:
```
kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16
```
**Note:** `kubeadm init` will autodetect the network interface to advertise
the master on as the interface with the default gateway. If you want to use a
different interface (i.e. a custom management network setup), specify the
`--apiserver-advertise-address=<ip-address>` argument to kubeadm init. For
example:
```
kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16 --apiserver-advertise-address=192.168.56.106
```
**Note:** The CIDR specified with the flag `--pod-network-cidr` is used by
kube-proxy, and it **must include** the `PodSubnetCIDR` from the `IPAMConfig`
section in the Contiv-vpp config map in Contiv-vpp's deployment file 
[contiv-vpp.yaml](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/values.yaml). Pods in the host network namespace
are a special case; they share their respective interfaces and IP addresses with
the host. For proxying to work properly it is therefore required for services
with backends running on the host to also **include the node management IP** 
within the `--pod-network-cidr` subnet. For example, with the default 
`PodSubnetCIDR=10.1.0.0/16` and `PodIfIPCIDR=10.2.1.0/24`, the subnet 
`10.3.0.0/16` could be allocated for the management network and 
`--pod-network-cidr` could be defined as `10.0.0.0/8`, so as to include IP 
addresses of all pods in all network namespaces:
```
kubeadm init --token-ttl 0 --pod-network-cidr=10.0.0.0/8 --apiserver-advertise-address=10.3.1.1
```

If Kubernetes was initialized successfully, it prints out this message:
```
Your Kubernetes master has initialized successfully!
```

After successful initialization, don't forget to set up your .kube directory
as a regular user (as instructed by `kubeadm`):
```bash
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

### (3/4) Installing the Contiv-VPP Pod Network
If you have already used the Contiv-VPP plugin before, you may need to pull
the most recent Docker images on each node:
```
bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/pull-images.sh)
```

Install the Contiv-VPP network for your cluster as follows:

- If you do not use the STN feature, install Contiv-vpp as follows: 
  ```
  kubectl apply -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
  ```
  
- If you use the STN feature, download the `contiv-vpp.yaml` file:
  ```
  wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
  ```
  Then edit the STN configuration as described [here][16]. Finally, create 
  the Contiv-vpp deployment from the edited file:
  ```
  kubectl apply -f ./contiv-vpp.yaml
  ``` 

Beware contiv-etcd data is persisted in `/var/etcd` by default. It has to be cleaned up manually after `kubeadm reset`.
Otherwise outdated data will be loaded by a subsequent deployment.

You can also generate random subfolder, alternatively:

```
curl --silent https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml | sed "s/\/var\/etcd\/contiv-data/\/var\/etcd\/contiv-data\/$RANDOM/g" | kubectl apply -f -
```

#### Deployment Verification
After some time, all contiv containers should enter the running state:
```
root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
NAME                           READY     STATUS    RESTARTS   AGE       IP               NODE
...
contiv-etcd-gwc84              1/1       Running   0          14h       192.168.56.106   cvpp
contiv-ksr-5c2vk               1/1       Running   2          14h       192.168.56.106   cvpp
contiv-vswitch-l59nv           2/2       Running   0          14h       192.168.56.106   cvpp
```
In particular, make sure that the Contiv-VPP pod IP addresses are the same as
the IP address specified in the `--apiserver-advertise-address=<ip-address>`
argument to kubeadm init.

Verify that the VPP successfully grabbed the network interface specified in
the VPP startup config (`GigabitEthernet0/4/0` in our case):
```
$ sudo vppctl
vpp# sh inter
              Name               Idx       State          Counter          Count
GigabitEthernet0/4/0              1         up       rx packets                  1294
                                                     rx bytes                  153850
                                                     tx packets                   512
                                                     tx bytes                   21896
                                                     drops                        962
                                                     ip4                         1032
host-40df9b44c3d42f4              3         up       rx packets                126601
                                                     rx bytes                44628849
                                                     tx packets                132155
                                                     tx bytes                27205450
                                                     drops                         24
                                                     ip4                       126585
                                                     ip6                           16
host-vppv2                        2         up       rx packets                132162
                                                     rx bytes                27205824
                                                     tx packets                126658
                                                     tx bytes                44634963
                                                     drops                         15
                                                     ip4                       132147
                                                     ip6                           14
local0                            0        down
```

You should also see the interface to kube-dns (`host-40df9b44c3d42f4`) and to the
node's IP stack (`host-vppv2`).

#### Master Isolation (Optional)
By default, your cluster will not schedule pods on the master for security
reasons. If you want to be able to schedule pods on the master, (e.g., for a
single-machine Kubernetes cluster for development), then run:

```
kubectl taint nodes --all node-role.kubernetes.io/master-
```
More details about installing the pod network can be found in the
[kubeadm manual][4].

### (4/4) Joining Your Nodes
To add a new node to your cluster, run as root the command that was output
by kubeadm init. For example:
```
kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
```
More details can be found int the [kubeadm manual][5].

#### Deployment Verification
After some time, all contiv containers should enter the running state:
```
root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
NAME                           READY     STATUS    RESTARTS   AGE       IP               NODE
contiv-etcd-gwc84              1/1       Running   0          14h       192.168.56.106   cvpp
contiv-ksr-5c2vk               1/1       Running   2          14h       192.168.56.106   cvpp
contiv-vswitch-h6759           2/2       Running   0          14h       192.168.56.105   cvpp-slave2
contiv-vswitch-l59nv           2/2       Running   0          14h       192.168.56.106   cvpp
etcd-cvpp                      1/1       Running   0          14h       192.168.56.106   cvpp
kube-apiserver-cvpp            1/1       Running   0          14h       192.168.56.106   cvpp
kube-controller-manager-cvpp   1/1       Running   0          14h       192.168.56.106   cvpp
kube-dns-545bc4bfd4-fr6j9      3/3       Running   0          14h       10.1.134.2       cvpp
kube-proxy-q8sv2               1/1       Running   0          14h       192.168.56.106   cvpp
kube-proxy-s8kv9               1/1       Running   0          14h       192.168.56.105   cvpp-slave2
kube-scheduler-cvpp            1/1       Running   0          14h       192.168.56.106   cvpp
```
In particular, verify that a vswitch pod and a kube-proxy pod is running on
each joined node, as shown above.

On each joined node, verify that the VPP successfully grabbed the network
interface specified in the VPP startup config (`GigabitEthernet0/4/0` in
our case):
```
$ sudo vppctl
vpp# sh inter
              Name               Idx       State          Counter          Count
GigabitEthernet0/4/0              1         up
...
```
From the vpp CLI on a joined node you can also ping kube-dns to verify
node-to-node connectivity. For example:
```
vpp# ping 10.1.134.2
64 bytes from 10.1.134.2: icmp_seq=1 ttl=64 time=.1557 ms
64 bytes from 10.1.134.2: icmp_seq=2 ttl=64 time=.1339 ms
64 bytes from 10.1.134.2: icmp_seq=3 ttl=64 time=.1295 ms
64 bytes from 10.1.134.2: icmp_seq=4 ttl=64 time=.1714 ms
64 bytes from 10.1.134.2: icmp_seq=5 ttl=64 time=.1317 ms

Statistics: 5 sent, 5 received, 0% packet loss
```
### Deploying Example Applications
#### Simple Deployment
You can go ahead and create a simple deployment:
```
$ kubectl run nginx --image=nginx --replicas=2
```

Use `kubectl describe pod` to get the IP address of a pod, e.g.:
```
$ kubectl describe pod nginx | grep IP
```
You should see two ip addresses, for example:
```
IP:		10.1.1.3
IP:		10.1.1.4
```

You can check the pods' connectivity in one of the following ways:
* Connect to the VPP debug CLI and ping any pod:
```
  sudo vppctl
  vpp# ping 10.1.1.3
```
* Start busybox and ping any pod:
```
  kubectl run busybox --rm -ti --image=busybox /bin/sh
  If you don't see a command prompt, try pressing enter.
  / #
  / # ping 10.1.1.3

```
* You should be able to ping any pod from the host:
```
  ping 10.1.1.3
```

#### Deploying Pods on Different Nodes
to enable pod deployment on the master, untaint the master first:
```
kubectl taint nodes --all node-role.kubernetes.io/master-
```

In order to verify inter-node pod connectivity, we need to tell Kubernetes
to deploy one pod on the master node and one POD on the worker. For this,
we can use node selectors.

In your deployment YAMLs, add the `nodeSelector` sections that refer to
preferred node hostnames, e.g.:
```
  nodeSelector:
    kubernetes.io/hostname: vm5
```

Example of whole JSONs:
```
apiVersion: v1
kind: Pod
metadata:
  name: nginx1
spec:
  nodeSelector:
    kubernetes.io/hostname: vm5
  containers:
    - name: nginx
      
	  : nginx
```

```
apiVersion: v1
kind: Pod
metadata:
  name: nginx2
spec:
  nodeSelector:
    kubernetes.io/hostname: vm6
  containers:
    - name: nginx
      image: nginx
```

After deploying the JSONs, verify they were deployed on different hosts:
```
$ kubectl get pods -o wide
NAME      READY     STATUS    RESTARTS   AGE       IP           NODE
nginx1    1/1       Running   0          13m       10.1.36.2    vm5
nginx2    1/1       Running   0          13m       10.1.219.3   vm6
```

Now you can verify the connectivity to both nginx PODs from a busybox POD:
```
kubectl run busybox --rm -it --image=busybox /bin/sh

/ # wget 10.1.36.2
Connecting to 10.1.36.2 (10.1.36.2:80)
index.html           100% |*******************************************************************************************************************************************************************|   612   0:00:00 ETA

/ # rm index.html

/ # wget 10.1.219.3
Connecting to 10.1.219.3 (10.1.219.3:80)
index.html           100% |*******************************************************************************************************************************************************************|   612   0:00:00 ETA
```

### Uninstalling Contiv-VPP
To uninstall the network plugin itself, use `kubectl`:
```
kubectl delete -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
```

### Tearing down Kubernetes
* First, drain the node and make sure that the node is empty before
shutting it down:
```
  kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
  kubectl delete node <node name>
```
* Next, on the node being removed, reset all kubeadm installed state:
```
  rm -rf $HOME/.kube
  sudo su
  kubeadm reset
```

* If you added environment variable definitions into
  `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`, this would have been a process from the [Custom Management Network file][10], then remove the definitions now.

### Troubleshooting
Some of the issues that can occur during the installation are:

- Forgetting to create and initialize the `.kube` directory in your home
  directory (As instructed by `kubeadm init --token-ttl 0`). This can manifest
  itself as the following error:
  ```
  W1017 09:25:43.403159    2233 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://192.168.209.128:6443/swagger-2.0.0.pb-v1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")), falling back to swagger
  Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
  ```
- Previous installation lingering on the file system.
  `'kubeadm init --token-ttl 0` fails to initialize kubelet with one or more
  of the following error messages:
  ```
  ...
  [kubelet-check] It seems like the kubelet isn't running or healthy.
  [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
  ...
  ```

If you run into any of the above issues, try to clean up and reinstall as root:
```
sudo su
rm -rf $HOME/.kube
kubeadm reset
kubeadm init --token-ttl 0
rm -rf /var/etcd/contiv-data
rm -rf /var/bolt/bolt.db
```

[1]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
[3]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#initializing-your-master
[4]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
[5]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes
[6]: https://kubernetes.io/docs/setup/independent/install-kubeadm/
[8]: #tearing-down-kubernetes
[10]: https://github.com/contiv/vpp/blob/master/docs/CUSTOM_MGMT_NETWORK.md#setting-up-a-custom-management-network-on-multi-homed-nodes
[11]: ../vagrant/README.md
[12]: https://github.com/contiv/vpp/tree/master/docs/CUSTOM_MGMT_NETWORK.md
[13]: https://github.com/contiv/vpp/tree/master/docs/VMWARE_FUSION_HOST.md
[14]: https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md
[15]: https://github.com/contiv/vpp/tree/master/docs/MULTI_NIC_SETUP.md
[16]: https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md#configuring-stn-in-contiv-vpp-k8s-deployment-files
[17]: https://github.com/contiv/vpp/tree/master/k8s/README.md#setup-node-sh