docs: Rework the VPP progressive Tutorial.

Change-Id: If5b0d07ea90d978c6b1f11210a661876b7929653
Signed-off-by: John DeNisco <jdenisco@cisco.com>

14 files changed, 1314 insertions, 16 deletions

diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst
index 8c39069032b..b56fec8635c 100644
--- a/docs/gettingstarted/developers/index.rst
+++ b/docs/gettingstarted/developers/index.rst
@@ -1,8 +1,8 @@
 .. _gstarteddevel:
 
-##########
-Developers
-##########
+###############
+For Developers
+###############
 
 The Developers section covers the following areas:
 
diff --git a/docs/gettingstarted/developers/vnet.md b/docs/gettingstarted/developers/vnet.md
index 191a2a16969..602ffb7e782 100644
--- a/docs/gettingstarted/developers/vnet.md
+++ b/docs/gettingstarted/developers/vnet.md
@@ -17,10 +17,6 @@ classification, feeds ipv4-input, ipv6-input, arp-input etc.\] and
 "ipv4-input-no-checksum" \[if hardware can classify, perform ipv4 header
 checksum\].
 
-![image](/_images/VNET_Features.png)
-
-List of features and layer areas that VNET works with:
-
 Effective graph dispatch function coding
 ----------------------------------------
 
diff --git a/docs/gettingstarted/index.rst b/docs/gettingstarted/index.rst
index 4fcc51348fc..f29d17942e5 100644
--- a/docs/gettingstarted/index.rst
+++ b/docs/gettingstarted/index.rst
@@ -33,6 +33,7 @@ The Writing VPP Documentation section covers the following topics:
 .. toctree::
    :maxdepth: 2
 
+   progressivevpp/index.rst
    users/index.rst
    developers/index.rst
    writingdocs/index.rst
diff --git a/docs/gettingstarted/progressivevpp/index.rst b/docs/gettingstarted/progressivevpp/index.rst
new file mode 100644
index 00000000000..458beda57bc
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/index.rst
@@ -0,0 +1,27 @@
+.. _progressivevpp:
+
+.. toctree::
+
+########################
+Progressive VPP Tutorial
+########################
+
+Learn to run FD.io VPP on a single Ubuntu 16.04 VM using Vagrant with this walkthrough
+covering basic FD.io VPP senarios. Useful FD.io VPP commands will be used, and
+will discuss basic operations, and the state of a running FD.io VPP on a system.
+
+.. note::
+
+    This is **not** intended to be a 'How to Run in a Production Environment' set of instructions.
+
+For more information on using VPP with Virtual Box/Vagrant, please refer to :ref:`vppvagrant`
+
+.. toctree::
+
+   settingupenvironment.rst
+   runningvpp.rst
+   interface.rst
+   traces.rst
+   twovppinstances.rst
+   routing.rst
+   switching.rst
diff --git a/docs/gettingstarted/progressivevpp/interface.rst b/docs/gettingstarted/progressivevpp/interface.rst
new file mode 100644
index 00000000000..476fc5dda37
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/interface.rst
@@ -0,0 +1,157 @@
+.. _interface:
+
+.. toctree::
+
+Creating an Interface
+======================
+
+Skills to be Learned
+--------------------
+
+#. Create a veth interface in Linux host
+#. Assign an IP address to one end of the veth interface in the Linux host
+#. Create a vpp host-interface that connected to one end of a veth interface via AF_PACKET
+#. Add an ip address to a vpp interface
+
+VPP commands learned in this exercise
+--------------------------------------
+
+#. `create host-interface <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_devices_af_packet.html#clicmd_create_host-interface>`_
+#. `set int state <https://docs.fd.io/vpp/17.04/clicmd_src_vnet.html#clicmd_set_interface_state>`_
+#. `set int ip address <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_ip.html#clicmd_set_interface_ip_address>`_
+#. `show hardware <https://docs.fd.io/vpp/17.04/clicmd_src_vnet.html#clicmd_show_hardware-interfaces>`_
+#. `show int <https://docs.fd.io/vpp/17.04/clicmd_src_vnet.html#clicmd_show_interfaces>`_
+#. `show int addr <https://docs.fd.io/vpp/17.04/clicmd_src_vnet.html#clicmd_show_interfaces>`_
+#. `trace add <https://docs.fd.io/vpp/17.04/clicmd_src_vlib.html#clicmd_trace_add>`_
+#. `clear trace <https://docs.fd.io/vpp/17.04/clicmd_src_vlib.html#clicmd_clear_trace>`_
+#. `ping <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_ip.html#clicmd_ping>`_
+#. `show ip arp <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_ethernet.html#clicmd_show_ip_arp>`_
+#. `show ip fib <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_fib.html#clicmd_show_ip_fib>`_
+
+Topology
+---------
+
+.. figure:: /_images/Create_Interface_Topology.jpg
+  :alt: Figure: Create Interface Topology
+
+  Figure: Create Interface Topology
+
+Initial State
+--------------
+
+The initial state here is presumed to be the final state from the previous sections
+of the tutorial.
+ 
+Create veth interfaces on host
+-------------------------------
+
+In Linux, there is a type of interface call 'veth'. Think of a 'veth'
+interface as being an interface that has two ends to it (rather than
+one).
+
+Create a veth interface with one end named **vpp1out** and the other
+named **vpp1host**
+
+.. code-block:: console
+
+  $ sudo ip link add name vpp1out type veth peer name vpp1host
+
+Turn up both ends:
+
+.. code-block:: console
+
+  $ sudo ip link set dev vpp1out up
+  $ sudo ip link set dev vpp1host up
+
+Assign an IP address
+---------------------
+
+.. code-block:: console
+
+  $ sudo ip addr add 10.10.1.1/24 dev vpp1host
+
+Display the result:
+
+.. code-block:: console
+
+  $ ip addr show vpp1host
+  5: vpp1host@vpp1out: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
+    link/ether e2:0f:1e:59:ec:f7 brd ff:ff:ff:ff:ff:ff
+    inet 10.10.1.1/24 scope global vpp1host
+       valid_lft forever preferred_lft forever
+    inet6 fe80::e00f:1eff:fe59:ecf7/64 scope link
+       valid_lft forever preferred_lft forever
+
+Create vpp host-interface
+--------------------------
+
+Make sure VPP is running, if not start it.
+
+.. code-block:: console
+
+    $ ps -eaf | grep vpp
+    vagrant   2141   903  0 05:28 pts/0    00:00:00 grep --color=auto vpp
+    # vpp is not running, so start it
+    $ sudo /usr/bin/vpp -c startup1.conf
+
+These commands are run from the vpp shell. Enter the VPP shell with the following
+command:
+
+.. code-block:: console
+
+    $ sudo vppctl -s /run/vpp/cli-vpp1.sock
+        _______    _        _   _____  ___
+     __/ __/ _ \  (_)__    | | / / _ \/ _ \
+     _/ _// // / / / _ \   | |/ / ___/ ___/
+     /_/ /____(_)_/\___/   |___/_/  /_/
+    
+    vpp#
+
+Create a host interface attached to **vpp1out**.
+
+.. code-block:: console
+
+  vpp# create host-interface name vpp1out
+  host-vpp1out
+
+Confirm the interface:
+
+.. code-block:: console
+
+  vpp# show hardware
+                Name                Idx   Link  Hardware
+  host-vpp1out                       1     up   host-vpp1out
+  Ethernet address 02:fe:d9:75:d5:b4
+  Linux PACKET socket interface
+  local0                             0    down  local0
+  local
+
+Turn up the interface:
+
+.. code-block:: console
+
+  vpp# set int state host-vpp1out up
+
+Confirm the interface is up:
+
+.. code-block:: console
+
+  vpp# show int
+                Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count
+  host-vpp1out                      1      up          9000/0/0/0
+  local0                            0     down          0/0/0/0
+
+Assign ip address 10.10.1.2/24
+
+.. code-block:: console
+
+  vpp# set int ip address host-vpp1out 10.10.1.2/24
+
+Confirm the ip address is assigned:
+
+.. code-block:: console
+
+  vpp# show int addr
+  host-vpp1out (up):
+    L3 10.10.1.2/24
+  local0 (dn):
diff --git a/docs/gettingstarted/progressivevpp/routing.rst b/docs/gettingstarted/progressivevpp/routing.rst
new file mode 100644
index 00000000000..c9586c1c85d
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/routing.rst
@@ -0,0 +1,76 @@
+.. _routing:
+
+.. toctree::
+
+Routing
+=======
+
+Skills to be Learned
+---------------------
+
+In this exercise you will learn these new skills:
+
+#. Add route to Linux Host routing table
+#. Add route to FD.io VPP routing table
+
+And revisit the old ones:
+
+#. Examine FD.io VPP routing table
+#. Enable trace on vpp1 and vpp2
+#. ping from host to FD.io VPP
+#. Examine and clear trace on vpp1 and vpp2
+#. ping from FD.io VPP to host
+#. Examine and clear trace on vpp1 and vpp2
+
+VPP command learned in this exercise
+-------------------------------------
+
+#. `ip route
+   add <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_ip.html#clicmd_ip_route>`__
+
+Topology
+---------
+
+.. figure:: /_images/Connecting_two_vpp_instances_with_memif.png
+   :alt: Connect two FD.io VPP topology
+
+   Connect two FD.io VPP topology
+
+Initial State
+--------------
+
+The initial state here is presumed to be the final state from the
+exercise `Connecting two FD.io VPP
+instances <VPP/Progressive_VPP_Tutorial#Connecting_two_vpp_instances>`__
+
+Setup host route
+-----------------
+
+.. code-block:: console
+
+   $ sudo ip route add 10.10.2.0/24 via 10.10.1.2
+   $ ip route
+   default via 10.0.2.2 dev enp0s3 
+   10.0.2.0/24 dev enp0s3  proto kernel  scope link  src 10.0.2.15 
+   10.10.1.0/24 dev vpp1host  proto kernel  scope link  src 10.10.1.1 
+   10.10.2.0/24 via 10.10.1.2 dev vpp1host 
+
+Setup return route on vpp2
+---------------------------
+
+.. code-block:: console
+
+   $ sudo vppctl -s /run/vpp/cli-vpp2.sock
+    vpp# ip route add 10.10.1.0/24  via 10.10.2.1
+
+Ping from host through vpp1 to vpp2
+------------------------------------
+
+The connection from vpp1 to vpp2 uses the **memif** driver, the connection to the host
+uses the **af-packet** driver. To trace packets from the host we use **af-packet-input** from
+vpp1 to vpp2 we use **memif-input**.
+
+#. Setup a trace on vpp1 and vpp2
+#. Ping 10.10.2.2 from the host
+#. Examine the trace on vpp1 and vpp2
+#. Clear the trace on vpp1 and vpp2
diff --git a/docs/gettingstarted/progressivevpp/runningvpp.rst b/docs/gettingstarted/progressivevpp/runningvpp.rst
new file mode 100644
index 00000000000..ff20064b5eb
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/runningvpp.rst
@@ -0,0 +1,74 @@
+.. _runningvpp:
+
+Running VPP
+===========
+
+Using the files we create in :ref`settingupenvironment` we will now start and
+run VPP.
+
+VPP runs in userspace. In a production environment you will often run it
+with DPDK to connect to real NICs or vhost to connect to VMs. In those
+circumstances you usually run a single instance of VPP.
+
+For purposes of this tutorial, it is going to be extremely useful to run
+multiple instances of VPP, and connect them to each other to form a
+topology. Fortunately, VPP supports this.
+
+
+Using the files we created in setup we will start VPP.
+
+.. code-block:: console
+
+   $ sudo /usr/bin/vpp -c startup1.conf
+   vlib_plugin_early_init:361: plugin path /usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
+   load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
+   load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
+   load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
+   .........
+   $
+
+If VPP does not start you can try adding **nodaemon** to the startup.conf file in the
+**unix** section. This should provide more information in the output.
+
+startup.conf example with nodaemon:
+
+.. code-block:: console
+
+   unix {nodaemon cli-listen /run/vpp/cli-vpp1.sock}
+   api-segment { prefix vpp1 }
+   plugins { plugin dpdk_plugin.so { disable } }
+
+The command **vppctl** will launch a VPP shell with which you can run
+VPP commands interactively.
+
+We should now be able to execute the VPP shell and show the version.
+
+.. code-block:: console
+
+   $ sudo vppctl -s /run/vpp/cli-vpp1.sock
+       _______    _        _   _____  ___
+    __/ __/ _ \  (_)__    | | / / _ \/ _ \
+    _/ _// // / / / _ \   | |/ / ___/ ___/
+    /_/ /____(_)_/\___/   |___/_/  /_/
+   
+   vpp# show version
+   vpp v18.07-release built by root on c469eba2a593 at Mon Jul 30 23:27:03 UTC 2018
+   vpp#
+
+.. note::
+
+   Use ctrl-d or q to exit from the VPP shell.
+
+If you are going to run several instances of VPP this way be sure to kill them
+when you are finished.
+
+You can use something like the following:
+
+.. code-block:: console
+
+    $ ps -eaf | grep vpp
+    root      2067     1  2 05:12 ?        00:00:00 /usr/bin/vpp -c startup1.conf
+    vagrant   2070   903  0 05:12 pts/0    00:00:00 grep --color=auto vpp
+    $ kill -9 2067
+    $ ps -eaf | grep vpp
+    vagrant   2074   903  0 05:13 pts/0    00:00:00 grep --color=auto vpp
diff --git a/docs/gettingstarted/progressivevpp/settingupenvironment.rst b/docs/gettingstarted/progressivevpp/settingupenvironment.rst
new file mode 100644
index 00000000000..b22f79b3085
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/settingupenvironment.rst
@@ -0,0 +1,154 @@
+.. _settingupenvironment:
+
+Setting up your environment
+===========================
+
+All of these exercises are designed to be performed on an Ubuntu 16.04 (Xenial) box.
+
+* If you have an Ubuntu 16.04 box on which you have sudo or root access, you can feel free to use that.
+* If you do not, a Vagrantfile is provided to setup a basic Ubuntu 16.04 box for you in the the steps below.
+
+Install Virtual Box and Vagrant
+-------------------------------
+
+You will need to install Virtual Box and Vagrant. If you have not installed Virtual Box or Vagrant please
+refer to :ref:`installingVboxVagrant` to install Virtual Box and Vagrant.
+
+Create a Vagrant Directory
+---------------------------
+
+To get started create a directory for vagrant
+
+.. code-block:: console
+
+   $ mkdir vpp-tutorial
+   $ cd vpp-tutorial
+
+Create a file called **Vagrantfile** with the following contents:
+
+.. code-block:: ruby
+
+    # -*- mode: ruby -*-
+    # vi: set ft=ruby :
+    
+    Vagrant.configure(2) do |config|
+    
+      config.vm.box = "puppetlabs/ubuntu-16.04-64-nocm"
+      config.vm.box_check_update = false
+    
+      vmcpu=(ENV['VPP_VAGRANT_VMCPU'] || 2)
+      vmram=(ENV['VPP_VAGRANT_VMRAM'] || 4096)
+    
+      config.ssh.forward_agent = true
+    
+      config.vm.provider "virtualbox" do |vb|
+          vb.customize ["modifyvm", :id, "--ioapic", "on"]
+          vb.memory = "#{vmram}"
+          vb.cpus = "#{vmcpu}"
+          #support for the SSE4.x instruction is required in some versions of VB.
+          vb.customize ["setextradata", :id, "VBoxInternal/CPUM/SSE4.1", "1"]
+          vb.customize ["setextradata", :id, "VBoxInternal/CPUM/SSE4.2", "1"]
+      end
+    end
+
+
+Running Vagrant
+---------------
+
+VPP runs in userspace.  In a production environment you will often run it with
+DPDK to connect to real NICs or vhost to connect to VMs.mIn those circumstances
+you usually run a single instance of VPP.
+
+For purposes of this tutorial, it is going to be extremely useful to run multiple
+instances of vpp, and connect them to each other to form a topology.  Fortunately,
+VPP supports this.
+
+When running multiple VPP instances, each instance needs to have specified a 'name'
+or 'prefix'.  In the example below, the 'name' or 'prefix' is "vpp1". Note that only
+one instance can use the dpdk plugin, since this plugin is trying to acquire a lock
+on a file.
+
+Setting up VPP environment with Vagrant
+---------------------------------------------
+
+After setting up Vagrant, use these commands on your Vagrant directory to boot the VM:
+
+.. code-block:: console
+
+    $ vagrant up
+    $ vagrant ssh
+    $ sudo apt-get update
+    $ sudo reboot -n
+    $ # Wait for the VM to reboot
+    $ vagrant ssh
+
+Install VPP
+------------
+
+Now that the VM is updated, we will install the VPP packages.
+
+For more on installing VPP please refer to :ref:`installingVPP`.
+
+For this tutorial we need to install VPP by modifying the file
+**/etc/apt/sources.list.d/99fd.io.list**.
+
+Write this file with the following contents:
+
+.. code-block:: console
+
+   deb [trusted=yes] https://nexus.fd.io/content/repositories/fd.io.ubuntu.xenial.main/ ./
+
+Then execute the following commands.
+
+.. code-block:: console
+
+   $ sudo bash
+   # apt-get update
+   # apt-get install vpp-lib vpp vpp-plugins
+   #
+
+Stop VPP for this tutorial. We will be creating our own instances of VPP.
+
+.. code-block:: console
+
+   # service vpp stop
+   #
+
+
+Create some startup files
+--------------------------
+
+We will create some startup files for the use of this tutorial. Typically you will
+modify the startup.conf file found in /etc/vpp/startup.conf. For more information
+on this file refer to :ref:`startup`.
+
+When running multiple VPP instances, each instance needs to have
+specified a 'name' or 'prefix'. In the example below, the 'name' or 'prefix'
+is "vpp1". Note that only one instance can use the dpdk plugin, since this
+plugin is trying to acquire a lock on a file. These startup files we create will
+disable the dpdk plugin.
+
+Also in our startup files notice **api-segment**. **api-segment {prefix vpp1}**
+tells FD.io VPP how to name the files in /dev/shm/ for your VPP instance
+differently from the default. **unix {cli-listen /run/vpp/cli-vpp1.sock}**
+tells vpp to use a non-default socket file when being addressed by vppctl.
+
+Now create 2 files named startup1.conf and startup2.conf with the following
+content. These files can be located anywhere. We specify the location when we
+start VPP.
+
+startup1.conf:
+
+.. code-block:: console
+
+   unix {cli-listen /run/vpp/cli-vpp1.sock}
+   api-segment { prefix vpp1 }
+   plugins { plugin dpdk_plugin.so { disable } }
+
+startup2.conf:
+
+.. code-block:: console
+
+   unix {cli-listen /run/vpp/cli-vpp2.sock}
+   api-segment { prefix vpp2 }
+   plugins { plugin dpdk_plugin.so { disable } }
diff --git a/docs/gettingstarted/progressivevpp/switching.rst b/docs/gettingstarted/progressivevpp/switching.rst
new file mode 100644
index 00000000000..72005080b08
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/switching.rst
@@ -0,0 +1,189 @@
+.. _switching:
+
+.. toctree::
+
+Switching
+=========
+
+Skills to be Learned
+----------------------
+
+#. Associate an interface with a bridge domain
+#. Create a loopback interaface
+#. Create a BVI (Bridge Virtual Interface) for a bridge domain
+#. Examine a bridge domain
+
+FD.io VPP command learned in this exercise
+--------------------------------------------
+
+#. `show
+   bridge <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_l2.html#clicmd_show_bridge-domain>`__
+#. `show bridge
+   detail <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_l2.html#clicmd_show_bridge-domain>`__
+#. `set int l2
+   bridge <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_l2.html#clicmd_set_interface_l2_bridge>`__
+#. `show l2fib
+   verbose <https://docs.fd.io/vpp/17.04/clicmd_src_vnet_l2.html#clicmd_show_l2fib>`__
+
+Topology
+---------
+
+.. figure:: /_images/Switching_Topology.jpg
+   :alt: Switching Topology
+
+   Switching Topology
+
+Initial state
+---------------
+
+Unlike previous exercises, for this one you want to start tabula rasa.
+
+Note: You will lose all your existing config in your FD.io VPP instances!
+
+To clear existing config from previous exercises run:
+
+.. code-block:: console
+
+   $ ps -ef | grep vpp | awk '{print $2}'| xargs sudo kill
+   $ sudo ip link del dev vpp1host
+   $ # do the next command if you are cleaing up from this example
+   $ sudo ip link del dev vpp1vpp2
+
+Run FD.io VPP instances
+------------------------
+
+#. Run a vpp instance named **vpp1**
+#. Run a vpp instance named **vpp2**
+
+Connect vpp1 to host
+---------------------
+
+#. Create a veth with one end named vpp1host and the other named
+   vpp1out.
+#. Connect vpp1out to vpp1
+#. Add ip address 10.10.1.1/24 on vpp1host
+
+Connect vpp1 to vpp2
+---------------------
+
+#. Create a veth with one end named vpp1vpp2 and the other named
+   vpp2vpp1.
+#. Connect vpp1vpp2 to vpp1.
+#. Connect vpp2vpp1 to vpp2.
+
+Configure Bridge Domain on vpp1
+--------------------------------
+
+Check to see what bridge domains already exist, and select the first
+bridge domain number not in use:
+
+.. code-block:: console
+
+   vpp# show bridge-domain
+    ID   Index   Learning   U-Forwrd   UU-Flood   Flooding   ARP-Term     BVI-Intf   
+    0      0        off        off        off        off        off        local0    
+
+In the example above, there is bridge domain ID '0' already. Even though
+sometimes we might get feedback as below:
+
+.. code-block:: console
+
+   no bridge-domains in use
+
+the bridge domain ID '0' still exists, where no operations are
+supported. For instance, if we try to add host-vpp1out and host-vpp1vpp2
+to bridge domain ID 0, we will get nothing setup.
+
+.. code-block:: console
+
+   vpp# set int l2 bridge host-vpp1out 0
+   vpp# set int l2 bridge host-vpp1vpp2 0
+   vpp# show bridge-domain 0 detail
+   show bridge-domain: No operations on the default bridge domain are supported
+
+So we will create bridge domain 1 instead of playing with the default
+bridge domain ID 0.
+
+Add host-vpp1out to bridge domain ID 1
+
+.. code-block:: console
+
+   vpp# set int l2 bridge host-vpp1out 1
+
+Add host-vpp1vpp2 to bridge domain ID1
+
+.. code-block:: console
+
+   vpp# set int l2 bridge host-vpp1vpp2  1
+
+Examine bridge domain 1:
+
+.. code-block:: console
+
+    vpp# show bridge-domain 1 detail
+    BD-ID   Index   BSN  Age(min)  Learning  U-Forwrd  UU-Flood  Flooding  ARP-Term  BVI-Intf
+    1       1      0     off        on        on        on        on       off       N/A
+
+            Interface           If-idx ISN  SHG  BVI  TxFlood        VLAN-Tag-Rewrite
+        host-vpp1out            1     1    0    -      *                 none
+        host-vpp1vpp2           2     1    0    -      *                 none
+
+Configure loopback interface on vpp2
+-------------------------------------
+
+.. code-block:: console
+
+    vpp# create loopback interface
+    loop0
+
+Add the ip address 10.10.1.2/24 to vpp2 interface loop0. Set the state
+of interface loop0 on vpp2 to 'up'
+
+Configure bridge domain on vpp2
+--------------------------------
+
+Check to see the first available bridge domain ID (it will be 1 in this
+case)
+
+Add interface loop0 as a bridge virtual interface (bvi) to bridge domain
+1
+
+.. code-block:: console
+
+   vpp# set int l2 bridge loop0 1 bvi
+
+Add interface vpp2vpp1 to bridge domain 1
+
+.. code-block:: console
+
+   vpp# set int l2 bridge host-vpp2vpp1  1
+
+Examine the bridge domain and interfaces.
+
+Ping from host to vpp and vpp to host
+--------------------------------------
+
+#. Add trace on vpp1 and vpp2
+#. ping from host to 10.10.1.2
+#. Examine and clear trace on vpp1 and vpp2
+#. ping from vpp2 to 10.10.1.1
+#. Examine and clear trace on vpp1 and vpp2
+
+Examine l2 fib
+---------------
+
+.. code-block:: console
+
+    vpp# show l2fib verbose
+    Mac Address     BD Idx           Interface           Index  static  filter  bvi   Mac Age (min) 
+    de:ad:00:00:00:00    1            host-vpp1vpp2           2       0       0     0      disabled    
+    c2:f6:88:31:7b:8e    1            host-vpp1out            1       0       0     0      disabled    
+    2 l2fib entries
+
+.. code-block:: console
+
+    vpp# show l2fib verbose
+    Mac Address     BD Idx           Interface           Index  static  filter  bvi   Mac Age (min) 
+    de:ad:00:00:00:00    1                loop0               2       1       0     1      disabled    
+    c2:f6:88:31:7b:8e    1            host-vpp2vpp1           1       0       0     0      disabled    
+    2 l2fib entries
diff --git a/docs/gettingstarted/progressivevpp/traces.rst b/docs/gettingstarted/progressivevpp/traces.rst
new file mode 100644
index 00000000000..b8238db2cd8
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/traces.rst
@@ -0,0 +1,498 @@
+.. _traces:
+
+.. toctree::
+
+Using the trace command 
+========================
+
+Skills to be Learned
+---------------------
+
+#. Setup a 'trace'
+#. View a 'trace'
+#. Clear a 'trace'
+#. Verify using ping from host
+#. Ping from vpp
+#. Examine Arp Table
+#. Examine ip fib
+
+Basic Trace Commands
+---------------------
+
+Show trace buffer [max COUNT].
+
+.. code-block:: console 
+
+  vpp# show trace
+
+Clear trace buffer and free memory.
+
+.. code-block:: console 
+
+  vpp# clear trace
+
+filter trace output - include NODE COUNT | exclude NODE COUNT | none.
+
+.. code-block:: console 
+
+  vpp# trace filter <include NODE COUNT | exclude NODE COUNT | none>
+
+Add Trace
+----------
+
+.. code-block:: console 
+
+  vpp# trace add af-packet-input 10
+
+Ping from Host to VPP
+----------------------
+
+.. code-block:: console 
+
+  vpp# q
+  $ ping -c 1 10.10.1.2
+  PING 10.10.1.2 (10.10.1.2) 56(84) bytes of data.
+  64 bytes from 10.10.1.2: icmp_seq=1 ttl=64 time=0.283 ms
+
+  --- 10.10.1.2 ping statistics ---
+  1 packets transmitted, 1 received, 0% packet loss, time 0ms
+  rtt min/avg/max/mdev = 0.283/0.283/0.283/0.000 ms
+
+Examine Trace of ping from host to VPP 
+----------------------------------------
+
+.. code-block:: console 
+
+  $ sudo vppctl -s /run/vpp/cli-vpp1.sock
+  vpp# show trace
+  ------------------- Start of thread 0 vpp_main -------------------
+  Packet 1
+
+  00:17:04:099260: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e370 nsec 0x3af2736f vlan 0 vlan_tpid 0
+  00:17:04:099269: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:17:04:099285: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099290: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099296: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3f7c
+      fragment id 0xe516, flags DONT_FRAGMENT
+    ICMP echo_request checksum 0xc043
+  00:17:04:099300: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099301: ip4-icmp-echo-request
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099303: ip4-load-balance
+  fib 0 dpo-idx 13 flow hash: 0x00000000
+  ICMP: 10.10.1.2 -> 10.10.1.1
+    tos 0x00, ttl 64, length 84, checksum 0x4437
+    fragment id 0xe05b, flags DONT_FRAGMENT
+  ICMP echo_reply checksum 0xc843
+  00:17:04:099305: ip4-rewrite
+  tx_sw_if_index 1 dpo-idx 1 : ipv4 via 10.10.1.1 host-vpp1out: mtu:9000 e20f1e59ecf702fed975d5b40800 flow hash: 0x00000000
+  00000000: e20f1e59ecf702fed975d5b4080045000054e05b4000400144370a0a01020a0a
+  00000020: 01010000c8437c92000170e3605b000000001c170f00000000001011
+  00:17:04:099307: host-vpp1out-output
+  host-vpp1out
+  IP4: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
+  ICMP: 10.10.1.2 -> 10.10.1.1
+    tos 0x00, ttl 64, length 84, checksum 0x4437
+    fragment id 0xe05b, flags DONT_FRAGMENT
+  ICMP echo_reply checksum 0xc843
+
+Clear trace buffer
+-------------------
+
+.. code-block:: console 
+
+  vpp# clear trace
+
+Ping from VPP to Host 
+-----------------------
+
+.. code-block:: console 
+
+  vpp# ping 10.10.1.1
+  64 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=.0789 ms
+  64 bytes from 10.10.1.1: icmp_seq=2 ttl=64 time=.0619 ms
+  64 bytes from 10.10.1.1: icmp_seq=3 ttl=64 time=.0519 ms
+  64 bytes from 10.10.1.1: icmp_seq=4 ttl=64 time=.0514 ms
+  64 bytes from 10.10.1.1: icmp_seq=5 ttl=64 time=.0526 ms
+
+  Statistics: 5 sent, 5 received, 0% packet loss
+
+Examine Trace of ping from VPP to host
+---------------------------------------
+
+The output will demonstrate FD.io VPP's trace of ping for all packets.
+
+.. code-block:: console 
+
+  vpp# show trace
+  ------------------- Start of thread 0 vpp_main -------------------
+  Packet 1
+
+  00:17:04:099260: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e370 nsec 0x3af2736f vlan 0 vlan_tpid 0
+  00:17:04:099269: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:17:04:099285: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099290: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099296: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3f7c
+      fragment id 0xe516, flags DONT_FRAGMENT
+    ICMP echo_request checksum 0xc043
+  00:17:04:099300: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099301: ip4-icmp-echo-request
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f7c
+    fragment id 0xe516, flags DONT_FRAGMENT
+  ICMP echo_request checksum 0xc043
+  00:17:04:099303: ip4-load-balance
+  fib 0 dpo-idx 13 flow hash: 0x00000000
+  ICMP: 10.10.1.2 -> 10.10.1.1
+    tos 0x00, ttl 64, length 84, checksum 0x4437
+    fragment id 0xe05b, flags DONT_FRAGMENT
+  ICMP echo_reply checksum 0xc843
+  00:17:04:099305: ip4-rewrite
+  tx_sw_if_index 1 dpo-idx 1 : ipv4 via 10.10.1.1 host-vpp1out: mtu:9000 e20f1e59ecf702fed975d5b40800 flow hash: 0x00000000
+  00000000: e20f1e59ecf702fed975d5b4080045000054e05b4000400144370a0a01020a0a
+  00000020: 01010000c8437c92000170e3605b000000001c170f00000000001011
+  00:17:04:099307: host-vpp1out-output
+  host-vpp1out
+  IP4: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
+  ICMP: 10.10.1.2 -> 10.10.1.1
+    tos 0x00, ttl 64, length 84, checksum 0x4437
+    fragment id 0xe05b, flags DONT_FRAGMENT
+  ICMP echo_reply checksum 0xc843
+
+  Packet 2
+
+  00:17:09:113964: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 42 snaplen 42 mac 66 net 80
+      sec 0x5b60e375 nsec 0x3b3bd57d vlan 0 vlan_tpid 0
+  00:17:09:113974: ethernet-input
+  ARP: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:17:09:113986: arp-input
+  request, type ethernet/IP4, address size 6/4
+  e2:0f:1e:59:ec:f7/10.10.1.1 -> 00:00:00:00:00:00/10.10.1.2
+  00:17:09:114003: host-vpp1out-output
+  host-vpp1out
+  ARP: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
+  reply, type ethernet/IP4, address size 6/4
+  02:fe:d9:75:d5:b4/10.10.1.2 -> e2:0f:1e:59:ec:f7/10.10.1.1
+
+  Packet 3
+
+  00:18:16:407079: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e3b9 nsec 0x90b7566 vlan 0 vlan_tpid 0
+  00:18:16:407085: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:18:16:407090: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3fe8
+    fragment id 0x24ab
+  ICMP echo_reply checksum 0x37eb
+  00:18:16:407094: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3fe8
+    fragment id 0x24ab
+  ICMP echo_reply checksum 0x37eb
+  00:18:16:407097: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3fe8
+      fragment id 0x24ab
+    ICMP echo_reply checksum 0x37eb
+  00:18:16:407101: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3fe8
+    fragment id 0x24ab
+  ICMP echo_reply checksum 0x37eb
+  00:18:16:407104: ip4-icmp-echo-reply
+  ICMP echo id 7531 seq 1
+  00:18:16:407108: ip4-drop
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3fe8
+      fragment id 0x24ab
+    ICMP echo_reply checksum 0x37eb
+  00:18:16:407111: error-drop
+  ip4-icmp-input: unknown type
+
+  Packet 4
+
+  00:18:17:409084: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e3ba nsec 0x90b539f vlan 0 vlan_tpid 0
+  00:18:17:409088: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:18:17:409092: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f40
+    fragment id 0x2553
+  ICMP echo_reply checksum 0xcc6d
+  00:18:17:409095: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f40
+    fragment id 0x2553
+  ICMP echo_reply checksum 0xcc6d
+  00:18:17:409097: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3f40
+      fragment id 0x2553
+    ICMP echo_reply checksum 0xcc6d
+  00:18:17:409099: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3f40
+    fragment id 0x2553
+  ICMP echo_reply checksum 0xcc6d
+  00:18:17:409101: ip4-icmp-echo-reply
+  ICMP echo id 7531 seq 2
+  00:18:17:409104: ip4-drop
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3f40
+      fragment id 0x2553
+    ICMP echo_reply checksum 0xcc6d
+  00:18:17:409104: error-drop
+  ip4-icmp-input: unknown type
+
+  Packet 5
+
+  00:18:18:409082: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e3bb nsec 0x8ecad24 vlan 0 vlan_tpid 0
+  00:18:18:409087: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:18:18:409091: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e66
+    fragment id 0x262d
+  ICMP echo_reply checksum 0x8e59
+  00:18:18:409093: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e66
+    fragment id 0x262d
+  ICMP echo_reply checksum 0x8e59
+  00:18:18:409096: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3e66
+      fragment id 0x262d
+    ICMP echo_reply checksum 0x8e59
+  00:18:18:409098: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e66
+    fragment id 0x262d
+  ICMP echo_reply checksum 0x8e59
+  00:18:18:409099: ip4-icmp-echo-reply
+  ICMP echo id 7531 seq 3
+  00:18:18:409102: ip4-drop
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3e66
+      fragment id 0x262d
+    ICMP echo_reply checksum 0x8e59
+  00:18:18:409102: error-drop
+  ip4-icmp-input: unknown type
+
+  Packet 6
+
+  00:18:19:414750: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e3bc nsec 0x92450f2 vlan 0 vlan_tpid 0
+  00:18:19:414754: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:18:19:414757: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e52
+    fragment id 0x2641
+  ICMP echo_reply checksum 0x9888
+  00:18:19:414760: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e52
+    fragment id 0x2641
+  ICMP echo_reply checksum 0x9888
+  00:18:19:414762: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3e52
+      fragment id 0x2641
+    ICMP echo_reply checksum 0x9888
+  00:18:19:414764: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e52
+    fragment id 0x2641
+  ICMP echo_reply checksum 0x9888
+  00:18:19:414765: ip4-icmp-echo-reply
+  ICMP echo id 7531 seq 4
+  00:18:19:414768: ip4-drop
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3e52
+      fragment id 0x2641
+    ICMP echo_reply checksum 0x9888
+  00:18:19:414769: error-drop
+  ip4-icmp-input: unknown type
+
+  Packet 7
+
+  00:18:20:418038: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 98 snaplen 98 mac 66 net 80
+      sec 0x5b60e3bd nsec 0x937bcc2 vlan 0 vlan_tpid 0
+  00:18:20:418042: ethernet-input
+  IP4: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:18:20:418045: ip4-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e47
+    fragment id 0x264c
+  ICMP echo_reply checksum 0xc0e8
+  00:18:20:418048: ip4-lookup
+  fib 0 dpo-idx 5 flow hash: 0x00000000
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e47
+    fragment id 0x264c
+  ICMP echo_reply checksum 0xc0e8
+  00:18:20:418049: ip4-local
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3e47
+      fragment id 0x264c
+    ICMP echo_reply checksum 0xc0e8
+  00:18:20:418054: ip4-icmp-input
+  ICMP: 10.10.1.1 -> 10.10.1.2
+    tos 0x00, ttl 64, length 84, checksum 0x3e47
+    fragment id 0x264c
+  ICMP echo_reply checksum 0xc0e8
+  00:18:20:418054: ip4-icmp-echo-reply
+  ICMP echo id 7531 seq 5
+  00:18:20:418057: ip4-drop
+    ICMP: 10.10.1.1 -> 10.10.1.2
+      tos 0x00, ttl 64, length 84, checksum 0x3e47
+      fragment id 0x264c
+    ICMP echo_reply checksum 0xc0e8
+  00:18:20:418058: error-drop
+  ip4-icmp-input: unknown type
+
+  Packet 8
+
+  00:18:21:419208: af-packet-input
+  af_packet: hw_if_index 1 next-index 4
+    tpacket2_hdr:
+      status 0x20000001 len 42 snaplen 42 mac 66 net 80
+      sec 0x5b60e3be nsec 0x92a9429 vlan 0 vlan_tpid 0
+  00:18:21:419876: ethernet-input
+  ARP: e2:0f:1e:59:ec:f7 -> 02:fe:d9:75:d5:b4
+  00:18:21:419881: arp-input
+  request, type ethernet/IP4, address size 6/4
+  e2:0f:1e:59:ec:f7/10.10.1.1 -> 00:00:00:00:00:00/10.10.1.2
+  00:18:21:419896: host-vpp1out-output
+  host-vpp1out
+  ARP: 02:fe:d9:75:d5:b4 -> e2:0f:1e:59:ec:f7
+  reply, type ethernet/IP4, address size 6/4
+  02:fe:d9:75:d5:b4/10.10.1.2 -> e2:0f:1e:59:ec:f7/10.10.1.1
+
+After examining the trace, clear it again using vpp# clear trace.
+
+Examine arp tables
+-------------------
+
+.. code-block:: console 
+
+  vpp# show ip arp
+  Time           IP4       Flags      Ethernet              Interface
+  1101.5636    10.10.1.1      D    e2:0f:1e:59:ec:f7 host-vpp1out        
+
+Examine routing tables
+-----------------------
+
+.. code-block:: console 
+
+  vpp# show ip fib
+    ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] locks:[src:plugin-hi:2, src:adjacency:1, src:default-route:1, ]
+  0.0.0.0/0
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
+  0.0.0.0/32
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
+  10.10.1.0/32
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
+  10.10.1.1/32
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:12 to:[5:420] via:[2:168]]
+    [0] [@5]: ipv4 via 10.10.1.1 host-vpp1out: mtu:9000 e20f1e59ecf702fed975d5b40800
+  10.10.1.0/24
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:8 to:[0:0]]
+    [0] [@4]: ipv4-glean: host-vpp1out: mtu:9000 ffffffffffff02fed975d5b40806
+  10.10.1.2/32
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[7:588]]
+    [0] [@2]: dpo-receive: 10.10.1.2 on host-vpp1out
+  10.10.1.255/32
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
+  224.0.0.0/4
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
+  240.0.0.0/4
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
+  255.255.255.255/32
+  unicast-ip4-chain
+  [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
+    [0] [@0]: dpo-drop ip4
diff --git a/docs/gettingstarted/progressivevpp/twovppinstances.rst b/docs/gettingstarted/progressivevpp/twovppinstances.rst
new file mode 100644
index 00000000000..7a8bbe0fd1c
--- /dev/null
+++ b/docs/gettingstarted/progressivevpp/twovppinstances.rst
@@ -0,0 +1,130 @@
+.. _twovppinstances:
+
+.. toctree::
+
+Connecting Two FD.io VPP Instances
+==================================
+
+.. _background-1:
+
+memif is a very high performance, direct memory interface type which can
+be used between FD.io VPP instances. It uses a file socket for a control channel
+to set up shared memory.
+
+.. _skills-to-be-learned-1:
+
+Skills to be Learned
+---------------------
+
+You will learn the following new skill in this exercise:
+
+#. Create a memif interface between two FD.io VPP instances
+
+You should be able to perform this exercise with the following skills
+learned in previous exercises:
+
+#. Run a second FD.io VPP instance
+#. Add an ip address to a FD.io VPP interface
+#. Ping from FD.io VPP
+
+.. _topology-1:
+
+Topology
+---------
+
+.. figure:: /_images/Connecting_two_vpp_instances_with_memif.png
+   :alt: Connect two FD.io VPP topology
+
+   Connect two FD.io VPP topology
+
+.. _initial-state-1:
+
+Initial state
+--------------
+
+The initial state here is presumed to be the final state from the
+exercise `Create an
+Interface <VPP/Progressive_VPP_Tutorial#Exercise:_Create_an_Interface>`__
+
+.. _action-running-a-second-vpp-instances-1:
+
+Running a second FD.io VPP instances
+-------------------------------------
+
+You should already have a FD.io VPP instance running named: vpp1.
+
+Run a second FD.io VPP instance named: vpp2.
+
+.. code-block:: console
+
+    $ sudo /usr/bin/vpp -c startup2.conf
+    ....
+    $ sudo vppctl -s /run/vpp/cli-vpp2.sock
+        _______    _        _   _____  ___
+     __/ __/ _ \  (_)__    | | / / _ \/ _ \
+     _/ _// // / / / _ \   | |/ / ___/ ___/
+     /_/ /____(_)_/\___/   |___/_/  /_/
+    
+    vpp# show version
+    vpp v18.07-release built by root on c469eba2a593 at Mon Jul 30 23:27:03 UTC 2018
+    vpp# quit
+
+.. _action-create-memif-interface-on-vpp1-1:
+
+Create memif interface on vpp1
+-------------------------------
+
+Create a memif interface on vpp1. To connect to the instance vpp1 use the
+socket **/run/vpp/cli-vpp1.sock**
+
+.. code-block:: console
+
+   $ sudo vppctl -s /run/vpp/cli-vpp1.sock
+   vpp# create interface memif id 0 master
+
+This will create an interface on vpp1 memif0/0 using /run/vpp/memif as
+its socket file. The role of vpp1 for this memif inteface is 'master'.
+
+With what you have learned:
+
+#. Set the memif0/0 state to up.
+#. Assign IP address 10.10.2.1/24 to memif0/0
+#. Examine memif0/0 via show commands
+
+.. _action-create-memif-interface-on-vpp2-1:
+
+Create memif interface on vpp2
+--------------------------------
+
+We want vpp2 to pick up the 'slave' role using the same
+run/vpp/memif-vpp1vpp2 socket file
+
+.. code-block:: console
+
+   vpp# create interface memif id 0 slave
+
+This will create an interface on vpp2 memif0/0 using /run/vpp/memif as
+its socket file. The role of vpp1 for this memif inteface is 'slave'.
+
+Use your previously used skills to:
+
+#. Set the memif0/0 state to up.
+#. Assign IP address 10.10.2.2/24 to memif0/0
+#. Examine memif0/0 via show commands
+
+.. _action-ping-from-vpp1-to-vpp2-1:
+
+Ping from vpp1 to vpp2
+------------------------
+
+Ping 10.10.2.2 from vpp1
+
+.. code-block:: console
+
+    $ ping 10.10.2.2 
+
+Ping 10.10.2.1 from vpp2
+
+.. code-block:: console
+
+    $ ping 10.10.2.1
diff --git a/docs/gettingstarted/users/configuring/startup.rst b/docs/gettingstarted/users/configuring/startup.rst
index b756dd4f3cd..0bc6ca4ae8a 100644
--- a/docs/gettingstarted/users/configuring/startup.rst
+++ b/docs/gettingstarted/users/configuring/startup.rst
@@ -1,9 +1,5 @@
 .. _startup:
 
-
-.. toctree::
-
-
 =======================================
 VPP Configuration File - 'startup.conf'
 =======================================
diff --git a/docs/gettingstarted/users/index.rst b/docs/gettingstarted/users/index.rst
index 7bac586d65a..4ec261a9937 100644
--- a/docs/gettingstarted/users/index.rst
+++ b/docs/gettingstarted/users/index.rst
@@ -1,8 +1,8 @@
 .. _users:
 
-########
-Users
-########
+###########
+For Users
+###########
 
 The Users section describe a basic VPP installation and configuration operation.
 The  installation and configuration of VPP can be done either manually, or by
@@ -21,7 +21,7 @@ section covers the following areas:
 .. toctree::
    :maxdepth: 2
 
+   configutil/index.rst
    installing/index.rst
    configuring/index.rst
    running/index.rst
-   configutil/index.rst
diff --git a/docs/gettingstarted/users/installing/ubuntu.rst b/docs/gettingstarted/users/installing/ubuntu.rst
index cb7a1c6097d..d1daed14ddb 100644
--- a/docs/gettingstarted/users/installing/ubuntu.rst
+++ b/docs/gettingstarted/users/installing/ubuntu.rst
@@ -64,7 +64,7 @@ Install the mandatory packages by running the following commands:
 .. code-block:: console
 
   sudo apt-get update
-  sudo apt-get install vpp vpp-lib vpp-plugin
+  sudo apt-get install vpp-lib vpp vpp-plugins
 
 
 Install the Optional Packages