docs: add contiv vpp

Change-Id: I92227fc4968fc6a478beb7f38707b91e9f0635ec
Signed-off-by: Scott Keeler <skeeler@cisco.com>

1 files changed, 472 insertions, 0 deletions

diff --git a/docs/usecases/contiv/MANUAL_INSTALL.md b/docs/usecases/contiv/MANUAL_INSTALL.md
new file mode 100644
index 00000000000..672a7b3fc8b
--- /dev/null
+++ b/docs/usecases/contiv/MANUAL_INSTALL.md
@@ -0,0 +1,472 @@
+# Manual Installation
+This document describes how to clone the Contiv respository and then use [kubeadm][1] to manually install Kubernetes
+with Contiv-VPP networking on one or more bare metal or VM hosts. 
+
+## Clone the Contiv Respository
+To clone the Contiv respository enter the following command:
+```
+git clone https://github.com/contiv/vpp/<repository-name>
+```
+**Note:** Replace *<repository-name>* with the name you want assigned to your cloned contiv repository.
+
+The cloned repository has important folders that contain content that are referenced in this Contiv documentation; those folders are noted below:
+```
+vpp-contiv2$ ls
+build       build-root  doxygen  gmod       LICENSE      Makefile   RELEASE.md   src
+build-data  docs        extras   INFO.yaml  MAINTAINERS  README.md  sphinx_venv  test
+```
+## Preparing Your Hosts
+
+### Host-specific Configurations
+- **VmWare VMs**: the vmxnet3 driver is required on each interface that will
+  be used by VPP. Please see [here][13] for instructions how to install the 
+  vmxnet3 driver on VmWare Fusion.
+  
+### Setting up Network Adapter(s)
+#### Setting up DPDK
+DPDK setup must be completed **on each node** as follows:
+
+- Load the PCI UIO driver:
+  ```
+  $ sudo modprobe uio_pci_generic
+  ```
+
+- Verify that the PCI UIO driver has loaded successfully:
+  ```
+  $ lsmod | grep uio
+  uio_pci_generic        16384  0
+  uio                    20480  1 uio_pci_generic
+  ```
+
+  Please note that this driver needs to be loaded upon each server bootup,
+  so you may want to add `uio_pci_generic` into the `/etc/modules` file, 
+  or a file in the `/etc/modules-load.d/` directory. For example, the 
+  `/etc/modules` file could look as follows:
+  ```
+  # /etc/modules: kernel modules to load at boot time.
+  #
+  # This file contains the names of kernel modules that should be loaded
+  # at boot time, one per line. Lines beginning with "#" are ignored.
+  uio_pci_generic
+  ```
+#### Determining Network Adapter PCI Addresses
+You need the PCI address of the network interface that VPP will use for the multi-node pod interconnect. On Debian-based
+distributions, you can use `lshw`(*):
+
+```
+$ sudo lshw -class network -businfo
+Bus info          Device      Class      Description
+====================================================
+pci@0000:00:03.0  ens3        network    Virtio network device
+pci@0000:00:04.0  ens4        network    Virtio network device
+```
+**Note:** On CentOS/RedHat/Fedora distributions, `lshw` may not be available by default, install it by issuing the following command:
+    ```
+    yum -y install lshw
+    ```
+
+#### Configuring vswitch to Use Network Adapters
+Finally, you need to set up the vswitch to use the network adapters:
+
+- [Setup on a node with a single NIC][14]
+- [Setup a node with multiple NICs][15]
+
+### Using a Node Setup Script
+You can perform the above steps using the [node setup script][17].
+
+## Installing Kubernetes with Contiv-VPP CNI plugin
+After the nodes you will be using in your K8s cluster are prepared, you can 
+install the cluster using [kubeadm][1].
+
+### (1/4) Installing Kubeadm on Your Hosts
+For first-time installation, see [Installing kubeadm][6]. To update an
+existing installation,  you should do a `apt-get update && apt-get upgrade`
+or `yum update` to get the latest version of kubeadm.
+
+On each host with multiple NICs where the NIC that will be used for Kubernetes
+management traffic is not the one pointed to by the default route out of the 
+host, a [custom management network][12] for Kubernetes must be configured.
+
+#### Using Kubernetes 1.10 and Above
+In K8s 1.10, support for huge pages in a pod has been introduced. For now, this
+feature must be either disabled or memory limit must be defined for vswitch container.
+
+To disable huge pages, perform the following
+steps as root:
+* Using your favorite editor, disable huge pages in the kubelet configuration 
+  file (`/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` or `/etc/default/kubelet` for version 1.11+):
+```
+  Environment="KUBELET_EXTRA_ARGS=--feature-gates HugePages=false"
+```
+* Restart the kubelet daemon:
+```
+  systemctl daemon-reload
+  systemctl restart kubelet
+```
+
+To define memory limit, append the following snippet to vswitch container in deployment yaml file:
+```
+			resources:
+              limits:
+                hugepages-2Mi: 1024Mi
+                memory: 1024Mi
+
+```
+or set `contiv.vswitch.defineMemoryLimits` to `true` in [helm values](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/README.md).
+
+### (2/4) Initializing Your Master
+Before initializing the master, you may want to [remove][8] any
+previously installed K8s components. Then, proceed with master initialization
+as described in the [kubeadm manual][3]. Execute the following command as
+root:
+```
+kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16
+```
+**Note:** `kubeadm init` will autodetect the network interface to advertise
+the master on as the interface with the default gateway. If you want to use a
+different interface (i.e. a custom management network setup), specify the
+`--apiserver-advertise-address=<ip-address>` argument to kubeadm init. For
+example:
+```
+kubeadm init --token-ttl 0 --pod-network-cidr=10.1.0.0/16 --apiserver-advertise-address=192.168.56.106
+```
+**Note:** The CIDR specified with the flag `--pod-network-cidr` is used by
+kube-proxy, and it **must include** the `PodSubnetCIDR` from the `IPAMConfig`
+section in the Contiv-vpp config map in Contiv-vpp's deployment file 
+[contiv-vpp.yaml](https://github.com/contiv/vpp/blob/master/k8s/contiv-vpp/values.yaml). Pods in the host network namespace
+are a special case; they share their respective interfaces and IP addresses with
+the host. For proxying to work properly it is therefore required for services
+with backends running on the host to also **include the node management IP** 
+within the `--pod-network-cidr` subnet. For example, with the default 
+`PodSubnetCIDR=10.1.0.0/16` and `PodIfIPCIDR=10.2.1.0/24`, the subnet 
+`10.3.0.0/16` could be allocated for the management network and 
+`--pod-network-cidr` could be defined as `10.0.0.0/8`, so as to include IP 
+addresses of all pods in all network namespaces:
+```
+kubeadm init --token-ttl 0 --pod-network-cidr=10.0.0.0/8 --apiserver-advertise-address=10.3.1.1
+```
+
+If Kubernetes was initialized successfully, it prints out this message:
+```
+Your Kubernetes master has initialized successfully!
+```
+
+After successful initialization, don't forget to set up your .kube directory
+as a regular user (as instructed by `kubeadm`):
+```bash
+mkdir -p $HOME/.kube
+sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
+sudo chown $(id -u):$(id -g) $HOME/.kube/config
+```
+
+### (3/4) Installing the Contiv-VPP Pod Network
+If you have already used the Contiv-VPP plugin before, you may need to pull
+the most recent Docker images on each node:
+```
+bash <(curl -s https://raw.githubusercontent.com/contiv/vpp/master/k8s/pull-images.sh)
+```
+
+Install the Contiv-VPP network for your cluster as follows:
+
+- If you do not use the STN feature, install Contiv-vpp as follows: 
+  ```
+  kubectl apply -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+  ```
+  
+- If you use the STN feature, download the `contiv-vpp.yaml` file:
+  ```
+  wget https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+  ```
+  Then edit the STN configuration as described [here][16]. Finally, create 
+  the Contiv-vpp deployment from the edited file:
+  ```
+  kubectl apply -f ./contiv-vpp.yaml
+  ``` 
+
+Beware contiv-etcd data is persisted in `/var/etcd` by default. It has to be cleaned up manually after `kubeadm reset`.
+Otherwise outdated data will be loaded by a subsequent deployment.
+
+You can also generate random subfolder, alternatively:
+
+```
+curl --silent https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml | sed "s/\/var\/etcd\/contiv-data/\/var\/etcd\/contiv-data\/$RANDOM/g" | kubectl apply -f -
+```
+
+#### Deployment Verification
+After some time, all contiv containers should enter the running state:
+```
+root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
+NAME                           READY     STATUS    RESTARTS   AGE       IP               NODE
+...
+contiv-etcd-gwc84              1/1       Running   0          14h       192.168.56.106   cvpp
+contiv-ksr-5c2vk               1/1       Running   2          14h       192.168.56.106   cvpp
+contiv-vswitch-l59nv           2/2       Running   0          14h       192.168.56.106   cvpp
+```
+In particular, make sure that the Contiv-VPP pod IP addresses are the same as
+the IP address specified in the `--apiserver-advertise-address=<ip-address>`
+argument to kubeadm init.
+
+Verify that the VPP successfully grabbed the network interface specified in
+the VPP startup config (`GigabitEthernet0/4/0` in our case):
+```
+$ sudo vppctl
+vpp# sh inter
+              Name               Idx       State          Counter          Count
+GigabitEthernet0/4/0              1         up       rx packets                  1294
+                                                     rx bytes                  153850
+                                                     tx packets                   512
+                                                     tx bytes                   21896
+                                                     drops                        962
+                                                     ip4                         1032
+host-40df9b44c3d42f4              3         up       rx packets                126601
+                                                     rx bytes                44628849
+                                                     tx packets                132155
+                                                     tx bytes                27205450
+                                                     drops                         24
+                                                     ip4                       126585
+                                                     ip6                           16
+host-vppv2                        2         up       rx packets                132162
+                                                     rx bytes                27205824
+                                                     tx packets                126658
+                                                     tx bytes                44634963
+                                                     drops                         15
+                                                     ip4                       132147
+                                                     ip6                           14
+local0                            0        down
+```
+
+You should also see the interface to kube-dns (`host-40df9b44c3d42f4`) and to the
+node's IP stack (`host-vppv2`).
+
+#### Master Isolation (Optional)
+By default, your cluster will not schedule pods on the master for security
+reasons. If you want to be able to schedule pods on the master, (e.g., for a
+single-machine Kubernetes cluster for development), then run:
+
+```
+kubectl taint nodes --all node-role.kubernetes.io/master-
+```
+More details about installing the pod network can be found in the
+[kubeadm manual][4].
+
+### (4/4) Joining Your Nodes
+To add a new node to your cluster, run as root the command that was output
+by kubeadm init. For example:
+```
+kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
+```
+More details can be found int the [kubeadm manual][5].
+
+#### Deployment Verification
+After some time, all contiv containers should enter the running state:
+```
+root@cvpp:/home/jan# kubectl get pods -n kube-system -o wide | grep contiv
+NAME                           READY     STATUS    RESTARTS   AGE       IP               NODE
+contiv-etcd-gwc84              1/1       Running   0          14h       192.168.56.106   cvpp
+contiv-ksr-5c2vk               1/1       Running   2          14h       192.168.56.106   cvpp
+contiv-vswitch-h6759           2/2       Running   0          14h       192.168.56.105   cvpp-slave2
+contiv-vswitch-l59nv           2/2       Running   0          14h       192.168.56.106   cvpp
+etcd-cvpp                      1/1       Running   0          14h       192.168.56.106   cvpp
+kube-apiserver-cvpp            1/1       Running   0          14h       192.168.56.106   cvpp
+kube-controller-manager-cvpp   1/1       Running   0          14h       192.168.56.106   cvpp
+kube-dns-545bc4bfd4-fr6j9      3/3       Running   0          14h       10.1.134.2       cvpp
+kube-proxy-q8sv2               1/1       Running   0          14h       192.168.56.106   cvpp
+kube-proxy-s8kv9               1/1       Running   0          14h       192.168.56.105   cvpp-slave2
+kube-scheduler-cvpp            1/1       Running   0          14h       192.168.56.106   cvpp
+```
+In particular, verify that a vswitch pod and a kube-proxy pod is running on
+each joined node, as shown above.
+
+On each joined node, verify that the VPP successfully grabbed the network
+interface specified in the VPP startup config (`GigabitEthernet0/4/0` in
+our case):
+```
+$ sudo vppctl
+vpp# sh inter
+              Name               Idx       State          Counter          Count
+GigabitEthernet0/4/0              1         up
+...
+```
+From the vpp CLI on a joined node you can also ping kube-dns to verify
+node-to-node connectivity. For example:
+```
+vpp# ping 10.1.134.2
+64 bytes from 10.1.134.2: icmp_seq=1 ttl=64 time=.1557 ms
+64 bytes from 10.1.134.2: icmp_seq=2 ttl=64 time=.1339 ms
+64 bytes from 10.1.134.2: icmp_seq=3 ttl=64 time=.1295 ms
+64 bytes from 10.1.134.2: icmp_seq=4 ttl=64 time=.1714 ms
+64 bytes from 10.1.134.2: icmp_seq=5 ttl=64 time=.1317 ms
+
+Statistics: 5 sent, 5 received, 0% packet loss
+```
+### Deploying Example Applications
+#### Simple Deployment
+You can go ahead and create a simple deployment:
+```
+$ kubectl run nginx --image=nginx --replicas=2
+```
+
+Use `kubectl describe pod` to get the IP address of a pod, e.g.:
+```
+$ kubectl describe pod nginx | grep IP
+```
+You should see two ip addresses, for example:
+```
+IP:		10.1.1.3
+IP:		10.1.1.4
+```
+
+You can check the pods' connectivity in one of the following ways:
+* Connect to the VPP debug CLI and ping any pod:
+```
+  sudo vppctl
+  vpp# ping 10.1.1.3
+```
+* Start busybox and ping any pod:
+```
+  kubectl run busybox --rm -ti --image=busybox /bin/sh
+  If you don't see a command prompt, try pressing enter.
+  / #
+  / # ping 10.1.1.3
+
+```
+* You should be able to ping any pod from the host:
+```
+  ping 10.1.1.3
+```
+
+#### Deploying Pods on Different Nodes
+to enable pod deployment on the master, untaint the master first:
+```
+kubectl taint nodes --all node-role.kubernetes.io/master-
+```
+
+In order to verify inter-node pod connectivity, we need to tell Kubernetes
+to deploy one pod on the master node and one POD on the worker. For this,
+we can use node selectors.
+
+In your deployment YAMLs, add the `nodeSelector` sections that refer to
+preferred node hostnames, e.g.:
+```
+  nodeSelector:
+    kubernetes.io/hostname: vm5
+```
+
+Example of whole JSONs:
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx1
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: vm5
+  containers:
+    - name: nginx
+      
+	  : nginx
+```
+
+```
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nginx2
+spec:
+  nodeSelector:
+    kubernetes.io/hostname: vm6
+  containers:
+    - name: nginx
+      image: nginx
+```
+
+After deploying the JSONs, verify they were deployed on different hosts:
+```
+$ kubectl get pods -o wide
+NAME      READY     STATUS    RESTARTS   AGE       IP           NODE
+nginx1    1/1       Running   0          13m       10.1.36.2    vm5
+nginx2    1/1       Running   0          13m       10.1.219.3   vm6
+```
+
+Now you can verify the connectivity to both nginx PODs from a busybox POD:
+```
+kubectl run busybox --rm -it --image=busybox /bin/sh
+
+/ # wget 10.1.36.2
+Connecting to 10.1.36.2 (10.1.36.2:80)
+index.html           100% |*******************************************************************************************************************************************************************|   612   0:00:00 ETA
+
+/ # rm index.html
+
+/ # wget 10.1.219.3
+Connecting to 10.1.219.3 (10.1.219.3:80)
+index.html           100% |*******************************************************************************************************************************************************************|   612   0:00:00 ETA
+```
+
+### Uninstalling Contiv-VPP
+To uninstall the network plugin itself, use `kubectl`:
+```
+kubectl delete -f https://raw.githubusercontent.com/contiv/vpp/master/k8s/contiv-vpp.yaml
+```
+
+### Tearing down Kubernetes
+* First, drain the node and make sure that the node is empty before
+shutting it down:
+```
+  kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
+  kubectl delete node <node name>
+```
+* Next, on the node being removed, reset all kubeadm installed state:
+```
+  rm -rf $HOME/.kube
+  sudo su
+  kubeadm reset
+```
+
+* If you added environment variable definitions into
+  `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf`, this would have been a process from the [Custom Management Network file][10], then remove the definitions now.
+
+### Troubleshooting
+Some of the issues that can occur during the installation are:
+
+- Forgetting to create and initialize the `.kube` directory in your home
+  directory (As instructed by `kubeadm init --token-ttl 0`). This can manifest
+  itself as the following error:
+  ```
+  W1017 09:25:43.403159    2233 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://192.168.209.128:6443/swagger-2.0.0.pb-v1: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")), falling back to swagger
+  Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
+  ```
+- Previous installation lingering on the file system.
+  `'kubeadm init --token-ttl 0` fails to initialize kubelet with one or more
+  of the following error messages:
+  ```
+  ...
+  [kubelet-check] It seems like the kubelet isn't running or healthy.
+  [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp [::1]:10255: getsockopt: connection refused.
+  ...
+  ```
+
+If you run into any of the above issues, try to clean up and reinstall as root:
+```
+sudo su
+rm -rf $HOME/.kube
+kubeadm reset
+kubeadm init --token-ttl 0
+rm -rf /var/etcd/contiv-data
+rm -rf /var/bolt/bolt.db
+```
+
+[1]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
+[3]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#initializing-your-master
+[4]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
+[5]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#joining-your-nodes
+[6]: https://kubernetes.io/docs/setup/independent/install-kubeadm/
+[8]: #tearing-down-kubernetes
+[10]: https://github.com/contiv/vpp/blob/master/docs/CUSTOM_MGMT_NETWORK.md#setting-up-a-custom-management-network-on-multi-homed-nodes
+[11]: ../vagrant/README.md
+[12]: https://github.com/contiv/vpp/tree/master/docs/CUSTOM_MGMT_NETWORK.md
+[13]: https://github.com/contiv/vpp/tree/master/docs/VMWARE_FUSION_HOST.md
+[14]: https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md
+[15]: https://github.com/contiv/vpp/tree/master/docs/MULTI_NIC_SETUP.md
+[16]: https://github.com/contiv/vpp/tree/master/docs/SINGLE_NIC_SETUP.md#configuring-stn-in-contiv-vpp-k8s-deployment-files
+[17]: https://github.com/contiv/vpp/tree/master/k8s/README.md#setup-node-sh