DOCS: Moved multiarch and build system, Incorprated Scott's changes

Change-Id: I5a57846db2d4faac1aa24db4629b612657f59afb
Signed-off-by: John DeNisco <jdenisco@cisco.com>

21 files changed, 661 insertions, 90 deletions

diff --git a/docs/gettingstarted/developers/building.rst b/docs/gettingstarted/developers/building.rst
index a408b510ca0..15754b53cb9 100644
--- a/docs/gettingstarted/developers/building.rst
+++ b/docs/gettingstarted/developers/building.rst
@@ -6,6 +6,7 @@ Building VPP
 ============
 
 To get started developing with VPP you need to get the sources and build the packages.
+For more information on the build system please refer to :ref:`buildsystem`.
 
 .. _setupproxies:
 
@@ -24,7 +25,7 @@ You may run these commands:
 Get the VPP Sources
 -----------------------------------
 
-To get the VPP sources and get ready to build execute the following:
+To get the VPP sources that are used to create the build, run the following commands:
 
 .. code-block:: console
 
@@ -44,8 +45,9 @@ commands:
 
 There should be no output, or packages showing after each of the above commands.
 
-Run these commands to install the dependencies for FD.io VPP. 
-If it hangs during downloading at any point, you may need to set up :ref:`proxies for this to work <setupproxies>`.
+Run the following **make** command to install the dependencies for FD.io VPP. 
+If it hangs at any point during the download, then you may need to set up
+:ref:`proxies for this to work <setupproxies>`.
 
 .. code-block:: console
 
@@ -74,9 +76,9 @@ If it hangs during downloading at any point, you may need to set up :ref:`proxie
 Build VPP (Debug)
 ----------------------------
 
-This build version contains debug symbols which is useful to modify VPP. The command
-below will build debug version of VPP. The binaries when building the debug images
-can be found in /build-root/vpp_debug-native.
+This build version contains debug symbols which are useful for modifying VPP. The
+**make** command below builds a debug version of VPP. The binaries, when building the
+debug images, can be found in /build-root/vpp_debug-native.
 
 .. code-block:: console
 
@@ -105,6 +107,10 @@ Build VPP (Release Version)
 To build the release version of FD.io VPP. This build is optimized and will not create debug symbols.
 The binaries when building the release images can be found in /build-root/vpp-native.
 
+Use the following **make** command below to build the release version of FD.io VPP. This build is
+optimized and will not create debug symbols. When building the release images, the binaries can
+be found in /build-root/vpp-native.
+
 .. code-block:: console
 
     $ make build-release
@@ -113,19 +119,20 @@ The binaries when building the release images can be found in /build-root/vpp-na
 Building Necessary Packages
 --------------------------------------------
 
-To build the debian packages, one of the following commands below depending on the system:
-
 Building Debian Packages
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
+To build the debian packages, use one of the following commands below, depending on the system:
+
 .. code-block:: console
 
     $ make pkg-deb 
 
-
 Building RPM Packages
 ^^^^^^^^^^^^^^^^^^^^^^^
 
+To build the rpm packages, use one of the following commands below, depending on the system:
+
 .. code-block:: console
 
     $ make pkg-rpm
diff --git a/docs/gettingstarted/developers/buildsystem/buildrootmakefile.rst b/docs/gettingstarted/developers/buildsystem/buildrootmakefile.rst
new file mode 100644
index 00000000000..ef736220b25
--- /dev/null
+++ b/docs/gettingstarted/developers/buildsystem/buildrootmakefile.rst
@@ -0,0 +1,353 @@
+Introduction to build-root/Makefile
+===================================
+
+The vpp build system consists of a top-level Makefile, a data-driven
+build-root/Makefile, and a set of makefile fragments. The various parts
+come together as the result of a set of well-thought-out conventions.
+
+This section describes build-root/Makefile in some detail.
+
+Repository Groups and Source Paths
+----------------------------------
+
+Current vpp workspaces comprise a single repository group. The file
+.../build-root/build-config.mk defines a key variable called
+SOURCE\_PATH. The SOURCE\_PATH variable names the set of repository
+groups. At the moment, there is only one repository group.
+
+Single pass build system, dependencies and components
+-----------------------------------------------------
+
+The vpp build system caters to components built with GNU autoconf /
+automake. Adding such components is a simple process. Dealing with
+components which use BSD-style raw Makefiles is a more difficult.
+Dealing with toolchain components such as gcc, glibc, and binutils can
+be considerably more complicated.
+
+The vpp build system is a **single-pass** build system. A partial order
+must exist for any set of components: the set of (a before b) tuples
+must resolve to an ordered list. If you create a circular dependency of
+the form; (a,b) (b,c) (c,a), gmake will try to build the target list,
+but there’s a 0.0% chance that the results will be pleasant. Cut-n-paste
+mistakes in .../build-data/packages/.mk can produce confusing failures.
+
+In a single-pass build system, it’s best to separate libraries and
+applications which instantiate them. For example, if vpp depends on
+libfoo.a, and myapp depends on both vpp and libfoo.a, it's best to place
+libfoo.a and myapp in separate components. The build system will build
+libfoo.a, vpp, and then (as a separate component) myapp. If you try to
+build libfoo.a and myapp from the same component, it won’t work.
+
+If you absolutely, positively insist on having myapp and libfoo.a in the
+same source tree, you can create a pseudo-component in a separate .mk
+file in the .../build-data/packages/ directory. Define the code
+phoneycomponent\_source = realcomponent, and provide manual
+configure/build/install targets.
+
+Separate components for myapp, libfoo.a, and vpp is the best and easiest
+solution. However, the “mumble\_source = realsource” degree of freedom
+exists to solve intractable circular dependencies, such as: to build
+gcc-bootstrap, followed by glibc, followed by “real” gcc/g++ [which
+depends on glibc too].
+
+.../build-root
+--------------
+
+The .../build-root directory contains the repository group specification
+build-config.mk, the main Makefile, and the system-wide set of
+autoconf/automake variable overrides in config.site. We'll describe
+these files in some detail. To be clear about expectations: the main
+Makefile and config.site file are subtle and complex. It's unlikely that
+you'll need or want to modify them. Poorly planned changes in either
+place typically cause bugs that are difficult to solve.
+
+.../build-root/build-config.mk
+------------------------------
+
+As described above, the build-config.mk file is straightforward: it sets
+the make variable SOURCE\_PATH to a list of repository group absolute
+paths.
+
+The SOURCE\_PATH variable If you choose to move a workspace, make sure
+to modify the paths defined by the SOURCE\_PATH variable. Those paths
+need to match changes you make in the workspace paths. For example, if
+you place the vpp directory in the workspace of a user named jsmith, you
+might change the SOURCE\_PATH to:
+
+SOURCE\_PATH = /home/jsmithuser/workspace/vpp
+
+The "out of the box" setting should work 99.5% of the time:
+
+::
+
+        SOURCE_PATH = $(CURDIR)/..
+
+.../vpp/build-root/Makefile
+---------------------------
+
+The main Makefile is complex in a number of dimensions. If you think you
+need to modify it, it's a good idea to do some research, or ask for
+advice before you change it.
+
+The main Makefile was organized and designed to provide the following
+characteristics: excellent performance, accurate dependency processing,
+cache enablement, timestamp optimizations, git integration,
+extensibility, builds with cross-compilation tool chains, and builds
+with embedded Linux distributions.
+
+If you really need to do so, you can build double-cross tools with it,
+with a minimum amount of fuss. For example, you could: compile gdb on
+x86\_64, to run on PowerPC, to debug the Xtensa instruction set.
+
+The PLATFORM variable
+---------------------
+
+The PLATFORM make/environment variable controls a number of important
+characteristics, primarily:
+
+-  CPU architecture
+-  The list of images to build.
+
+With respect to .../build-root/Makefile, the list of images to build is
+specified by the target. For example:
+
+::
+
+       make PLATFORM=vpp TAG=vpp_debug install-deb
+
+builds vpp debug Debian packages.
+
+The main Makefile interprets $PLATFORM by attempting to "-include" the
+file /build-data/platforms.mk:
+
+::
+
+        $(foreach d,$(FULL_SOURCE_PATH), \
+          $(eval -include $(d)/platforms.mk))
+
+By convention, we don't define **platforms** in the
+...//build-data/platforms.mk file.
+
+In the vpp case, we search for platform definition makefile fragments in
+.../vpp/build-data/platforms.mk, as follows:
+
+::
+
+        $(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS), \
+             $(eval -include $(d)/platforms/*.mk))
+
+With vpp, which uses the "vpp" platform as discussed above, we end up
+"-include"-ing .../vpp/build-data/platforms/vpp.mk.
+
+The platform-specific .mk fragment
+----------------------------------
+
+Here are the contents of .../build-data/platforms/vpp.mk:
+
+::
+
+        MACHINE=$(shell uname -m)
+     
+        vpp_arch = native
+        ifeq ($(TARGET_PLATFORM),thunderx)
+        vpp_dpdk_target = arm64-thunderx-linuxapp-gcc
+        endif
+        vpp_native_tools = vppapigen
+     
+        vpp_uses_dpdk = yes
+     
+        # Uncoment to enable building unit tests
+        # vpp_enable_tests = yes
+     
+        vpp_root_packages = vpp vom japi
+     
+        # DPDK configuration parameters
+        # vpp_uses_dpdk_mlx4_pmd = yes
+        # vpp_uses_dpdk_mlx5_pmd = yes
+        # vpp_uses_external_dpdk = yes
+        # vpp_dpdk_inc_dir = /usr/include/dpdk
+        # vpp_dpdk_lib_dir = /usr/lib
+        # vpp_dpdk_shared_lib = yes
+     
+        # Use '--without-libnuma' for non-numa aware architecture
+        # Use '--enable-dlmalloc' to use dlmalloc instead of mheap
+        vpp_configure_args_vpp = --enable-dlmalloc
+        sample-plugin_configure_args_vpp = --enable-dlmalloc
+     
+        # load balancer plugin is not portable on 32 bit platform
+        ifeq ($(MACHINE),i686)
+        vpp_configure_args_vpp += --disable-lb-plugin
+        endif
+     
+        vpp_debug_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 \
+            -fstack-protector-all -fPIC -Werror
+        vpp_debug_TAG_CXXFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 \
+            -fstack-protector-all -fPIC -Werror
+        vpp_debug_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -DFORTIFY_SOURCE=2 \
+            -fstack-protector-all -fPIC -Werror
+
+        vpp_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -fstack-protector -fPIC -Werror
+        vpp_TAG_CXXFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -fstack-protector -fPIC -Werror
+        vpp_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -fstack-protector -fPIC -Werror -pie -Wl,-z,now
+
+        vpp_clang_TAG_CFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -fstack-protector -fPIC -Werror
+        vpp_clang_TAG_LDFLAGS = -g -O2 -DFORTIFY_SOURCE=2 -fstack-protector -fPIC -Werror
+
+        vpp_gcov_TAG_CFLAGS = -g -O0 -DCLIB_DEBUG -fPIC -Werror -fprofile-arcs -ftest-coverage
+        vpp_gcov_TAG_LDFLAGS = -g -O0 -DCLIB_DEBUG -fPIC -Werror -coverage
+
+        vpp_coverity_TAG_CFLAGS = -g -O2 -fPIC -Werror -D__COVERITY__
+        vpp_coverity_TAG_LDFLAGS = -g -O2 -fPIC -Werror -D__COVERITY__
+
+Note the following variable settings:
+
+-  The variable \_arch sets the CPU architecture used to build the
+   per-platform cross-compilation toolchain. With the exception of the
+   "native" architecture - used in our example - the vpp build system
+   produces cross-compiled binaries.
+
+-  The variable \_native\_tools lists the required set of self-compiled
+   build tools.
+
+-  The variable \_root\_packages lists the set of images to build when
+   specifying the target: make PLATFORM= TAG= [install-deb \|
+   install-rpm].
+
+The TAG variable
+----------------
+
+The TAG variable indirectly sets CFLAGS and LDFLAGS, as well as the
+build and install directory names in the .../vpp/build-root directory.
+See definitions above.
+
+Important targets build-root/Makefile
+-------------------------------------
+
+The main Makefile and the various makefile fragments implement the
+following user-visible targets:
+
++------------------+----------------------+--------------------------------------------------------------------------------------+
+| Target           | ENV Variable Settings| Notes                                                                                |
+|                  |                      |                                                                                      |
++==================+======================+======================================================================================+
+| foo              |      bar             | mumble                                                                               |
++------------------+----------------------+--------------------------------------------------------------------------------------+
+| bootstrap-tools  | none                 |  Builds the set of native tools needed by the vpp build system to                    |
+|                  |                      |  build images. Example: vppapigen. In a full cross compilation case might include    |
+|                  |                      |  include "make", "git", "find", and "tar                                             |  
++------------------+----------------------+--------------------------------------------------------------------------------------+  
+| install-tools    | PLATFORM             | Builds the tool chain for the indicated <platform>. Not used in vpp builds           |
++------------------+----------------------+--------------------------------------------------------------------------------------+  
+| distclean        | none                 | Roto-rooters everything in sight: toolchains, images, and so forth.                  |
++------------------+----------------------+--------------------------------------------------------------------------------------+  
+| install-deb      | PLATFORM and TAG     | Build Debian packages comprising components listed in <platform>_root_packages,      |
+|                  |                      | using compile / link options defined by TAG.                                         |
++------------------+----------------------+--------------------------------------------------------------------------------------+  
+| install-rpm      | PLATFORM and TAG     | Build RPMs comprising components listed in <platform>_root_packages,                 |
+|                  |                      | using compile / link options defined by TAG.                                         |
++------------------+----------------------+--------------------------------------------------------------------------------------+  
+
+Additional build-root/Makefile environment variable settings
+------------------------------------------------------------
+
+These variable settings may be of use:
+
++----------------------+------------------------------------------------------------------------------------------------------------+
+| ENV Variable         | Notes                                                                                                      |
++======================+======================+=====================================================================================+
+| BUILD_DEBUG=vx       | Directs Makefile et al. to make a good-faith effort to show what's going on in excruciating detail.        |
+|                      | Use it as follows: "make ... BUILD_DEBUG=vx". Fairly effective in Makefile debug situations.               |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+| V=1                  | print detailed cc / ld command lines. Useful for discovering if -DFOO=11 is in the command line or not     |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+| CC=mygcc             | Override the configured C-compiler                                                                         |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+
+.../build-root/config.site
+--------------------------
+
+The contents of .../build-root/config.site override individual autoconf /
+automake default variable settings. Here are a few sample settings related to
+building a full toolchain:
+
+::
+
+    # glibc needs these setting for cross compiling 
+    libc_cv_forced_unwind=yes
+    libc_cv_c_cleanup=yes
+    libc_cv_ssp=no
+
+Determining the set of variables which need to be overridden, and the
+override values is a matter of trial and error. It should be
+unnecessary to modify this file for use with fd.io vpp.
+
+.../build-data/platforms.mk
+---------------------------
+
+Each repo group includes the platforms.mk file, which is included by
+the main Makefile. The vpp/build-data/platforms.mk file is not terribly
+complex. As of this writing, .../build-data/platforms.mk file accomplishes two
+tasks.
+
+First, it includes vpp/build-data/platforms/\*.mk:
+
+::
+
+    # Pick up per-platform makefile fragments
+    $(foreach d,$(SOURCE_PATH_BUILD_DATA_DIRS),	\
+      $(eval -include $(d)/platforms/*.mk))
+
+This collects the set of platform definition makefile fragments, as discussed above.
+
+Second, platforms.mk implements the user-visible "install-deb" target.
+
+.../build-data/packages/\*.mk
+-----------------------------
+
+Each component needs a makefile fragment in order for the build system
+to recognize it. The per-component makefile fragments vary
+considerably in complexity. For a component built with GNU autoconf /
+automake which does not depend on other components, the make fragment
+can be empty. See .../build-data/packages/vpp.mk for an uncomplicated
+but fully realistic example.
+
+Here are some of the important variable settings in per-component makefile fragments:
+
++----------------------+------------------------------------------------------------------------------------------------------------+
+| Variable             | Notes                                                                                                      |
++======================+======================+=====================================================================================+
+| xxx_configure_depend |  Lists the set of component build dependencies for the xxx component. In plain English: don't try to       |
+|                      |  configure this component until you've successfully built the indicated targets. Almost always,            |
+|                      |  xxx_configure_depend will list a set of "yyy-install" targets. Note the pattern:                          |
+|                      |  "variable names contain underscores, make target names contain hyphens"                                   |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+| xxx_configure_args   | (optional) Lists any additional arguments to pass to the xxx component "configure" script.                 |
+|                      | The main Makefile %-configure rule adds the required settings for --libdir, --prefix, and                  |
+|                      | --host (when cross-compiling)                                                                              |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+| xxx_CPPFLAGS         | Adds -I stanzas to CPPFLAGS for components upon which xxx depends.                                         |
+|                      | Almost invariably "xxx_CPPFLAGS = $(call installed_includes_fn, dep1 dep2 dep3)", where dep1, dep2, and    |
+|                      | dep3 are listed in xxx_configure_depend. It is bad practice to set "-g -O3" here. Those settings           |
+|                      | belong in a TAG.                                                                                           |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+| xxx_LDFLAGS          | Adds -Wl,-rpath -Wl,depN stanzas to LDFLAGS for components upon which xxx depends.                         |
+|                      | Almost invariably "xxx_LDFLAGS = $(call installed_lib_fn, dep1 dep2 dep3)", where dep1, dep2, and          |
+|                      | dep3 are listed in xxx_configure_depend. It is bad manners to set "-liberty-or-death" here.                |
+|                      | Those settings belong in Makefile.am.                                                                      |
++----------------------+------------------------------------------------------------------------------------------------------------+  
+
+When dealing with "irritating" components built with raw Makefiles
+which only work when building in the source tree, we use a specific
+strategy in the xxx.mk file. 
+
+The strategy is simple for those components: We copy the source tree
+into .../vpp/build-root/build-xxx. This works, but completely defeats
+dependency processing. This strategy is acceptable only for 3rd party
+software which won't need extensive (or preferably any) modifications.
+
+Take a look at .../vpp/build-data/packages/dpdk.mk. When invoked, the
+dpdk_configure variable copies source code into $(PACKAGE_BUILD_DIR),
+and performs the BSD equivalent of "autoreconf -i -f" to configure the
+build area. The rest of the file is similar: a bunch of hand-rolled
+glue code which manages to make the dpdk act like a good vpp build
+citizen even though it is not. 
diff --git a/docs/gettingstarted/developers/buildsystem/index.rst b/docs/gettingstarted/developers/buildsystem/index.rst
new file mode 100644
index 00000000000..38ba9724c68
--- /dev/null
+++ b/docs/gettingstarted/developers/buildsystem/index.rst
@@ -0,0 +1,11 @@
+.. _buildsystem:
+
+Build System
+============
+
+This guide describes the vpp build system in detail.
+
+.. toctree::
+
+    mainmakefile
+    buildrootmakefile
diff --git a/docs/gettingstarted/developers/buildsystem/mainmakefile.md b/docs/gettingstarted/developers/buildsystem/mainmakefile.md
new file mode 100644
index 00000000000..ddf0661942f
--- /dev/null
+++ b/docs/gettingstarted/developers/buildsystem/mainmakefile.md
@@ -0,0 +1,3 @@
+Introduction to the top-level Makefile
+======================================
+
diff --git a/docs/gettingstarted/developers/gdb_examples.rst b/docs/gettingstarted/developers/gdb_examples.rst
index b2ef65bcd83..8ea34cbc276 100644
--- a/docs/gettingstarted/developers/gdb_examples.rst
+++ b/docs/gettingstarted/developers/gdb_examples.rst
@@ -10,7 +10,7 @@ In this section we have a few useful gdb commands.
 Starting GDB
 ----------------------------
 
-Once at the gdb prompt VPP can be started with the following:
+Once at the gdb prompt, VPP can be started by isuuing the following commands:
 
 .. code-block:: console
 
@@ -25,7 +25,7 @@ Backtrace
 ----------------------------
 
 If you encounter issues when running VPP, such as VPP terminating due to a segfault
-or abort signal, you can run the VPP debug binary and then execute **backtrace** or **bt**.
+or abort signal, then you can run the VPP debug binary and then execute **backtrace** or **bt**.
 
 .. code-block:: console
 
@@ -38,12 +38,12 @@ or abort signal, you can run the VPP debug binary and then execute **backtrace**
 Get to the GDB prompt
 ---------------------------------------
 
-When running VPP is running get to the command prompt with CTRL-c.
+When VPP is running, you can get to the command prompt by entering CTRL-c.
 
 Breakpoints
 ---------------------------------------
 
-When at the GDB prompt set a breakpoint like so:
+When at the GDB prompt, set a breakpoint by using the commands below:
 
 .. code-block:: console
 
diff --git a/docs/gettingstarted/developers/gitreview.rst b/docs/gettingstarted/developers/gitreview.rst
index f727fb16756..a9e1b02c764 100644
--- a/docs/gettingstarted/developers/gitreview.rst
+++ b/docs/gettingstarted/developers/gitreview.rst
@@ -57,20 +57,20 @@ Otherwise, clone with:
     $ git clone ssh://YOUR_GERRIT_USERNAME@gerrit.fd.io:29418/vpp
     $ cd vpp
 
-When attempting to clone the repo it will ask if you want to add the Server Host Key to the list of known hosts. Type **yes** and hit enter.
+When attempting to clone the repo Git will prompt you asking if you want to add the Server Host Key to the list of known hosts. Enter **yes** and press the **Enter** key.
 
 Git Review
 ===========
 
-The VPP documents use the gerrit server and git review for submitting and fetching patches.
+The VPP documents use the gerrit server, and git review for submitting and fetching patches.
 
 
 New patch
 -----------------
 
-When working with new patch use the following to get your patch reviewed.
+When working with a new patch, use the following commands to get your patch reviewed.
 
-Make sure you have modified the correct files with:
+Make sure you have modified the correct files by issuing the following commands:
 
 .. code-block:: console
 
@@ -78,7 +78,7 @@ Make sure you have modified the correct files with:
     $ git diff
 
 Then add and commit the patch. You may want to add a tag to the commit comments.
-For example for document only patches you should add the tag **DOCS:**.
+For example for a document with only patches you should add the tag **DOCS:**.
 
 .. code-block:: console
 
diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst
index 76feb057552..8c39069032b 100644
--- a/docs/gettingstarted/developers/index.rst
+++ b/docs/gettingstarted/developers/index.rst
@@ -25,8 +25,10 @@ The Developers section covers the following areas:
    plugins
    vnet
    featurearcs
+   multiarch/index.rst
    bihash
    vpp_api_module
    binary_api_support
+   buildsystem/index.rst
    sample_plugin
 
diff --git a/docs/gettingstarted/developers/multiarch/index.rst b/docs/gettingstarted/developers/multiarch/index.rst
new file mode 100644
index 00000000000..48065b8a8e2
--- /dev/null
+++ b/docs/gettingstarted/developers/multiarch/index.rst
@@ -0,0 +1,11 @@
+.. _multiarch:
+
+Multi-architecture support
+==========================
+
+This reference guide describes how to use the vpp muli-architecture support scheme
+
+.. toctree::
+   :maxdepth: 1
+
+   nodefns
diff --git a/docs/gettingstarted/developers/multiarch/nodefns.rst b/docs/gettingstarted/developers/multiarch/nodefns.rst
new file mode 100644
index 00000000000..ad68385d6c5
--- /dev/null
+++ b/docs/gettingstarted/developers/multiarch/nodefns.rst
@@ -0,0 +1,160 @@
+Multi-Architecture Graph Node Cookbook
+======================================
+
+In the context of graph node dispatch functions, it's easy enough to
+use the vpp multi-architecture support setup. The point of the scheme
+is simple: for performance-critical nodes, generate multiple CPU
+hardware-dependent versions of the node dispatch functions, and pick
+the best one at runtime.
+
+The vpp scheme is simple enough to use, but details matter.
+
+100,000 foot view
+-----------------
+
+We compile entire graph node dispatch function implementation files
+multiple times. These compilations give rise to multiple versions of
+the graph node dispatch functions. Per-node constructor-functions
+interrogate CPU hardware, select the node dispatch function variant to
+use, and set the vlib_node_registration_t ".function" member to the
+address of the selected variant.
+
+Details
+-------
+
+Declare the node dispatch function as shown, using the VLIB\_NODE\_FN macro. The
+name of the node function **MUST** match the name of the graph node. 
+
+:: 
+
+    VLIB_NODE_FN (ip4_sdp_node) (vlib_main_t * vm, vlib_node_runtime_t * node,
+                                 vlib_frame_t * frame)
+    {
+      if (PREDICT_FALSE (node->flags & VLIB_NODE_FLAG_TRACE))
+        return ip46_sdp_inline (vm, node, frame, 1 /* is_ip4 */ ,
+    			    1 /* is_trace */ );
+      else
+        return ip46_sdp_inline (vm, node, frame, 1 /* is_ip4 */ ,
+    			    0 /* is_trace */ );
+    }   
+
+We need to generate *precisely one copy* of the
+vlib_node_registration_t, error strings, and packet trace decode function.
+
+Simply bracket these items with "#ifndef CLIB_MARCH_VARIANT...#endif":
+
+::
+
+    #ifndef CLIB_MARCH_VARIANT
+    static u8 *
+    format_sdp_trace (u8 * s, va_list * args)
+    {
+       <snip>
+    }
+    #endif
+
+    ...
+
+    #ifndef CLIB_MARCH_VARIANT
+    static char *sdp_error_strings[] = {
+    #define _(sym,string) string,
+      foreach_sdp_error
+    #undef _
+    };
+    #endif
+
+    ...
+
+    #ifndef CLIB_MARCH_VARIANT
+    VLIB_REGISTER_NODE (ip4_sdp_node) =
+    {
+      // DO NOT set the .function structure member.
+      // The multiarch selection __attribute__((constructor)) function
+      // takes care of it at runtime
+      .name = "ip4-sdp",
+      .vector_size = sizeof (u32),
+      .format_trace = format_sdp_trace,
+      .type = VLIB_NODE_TYPE_INTERNAL,
+
+      .n_errors = ARRAY_LEN(sdp_error_strings),
+      .error_strings = sdp_error_strings,
+
+      .n_next_nodes = SDP_N_NEXT,
+
+      /* edit / add dispositions here */
+      .next_nodes =
+      {
+        [SDP_NEXT_DROP] = "ip4-drop",
+      },
+    };
+    #endif
+
+To belabor the point: *do not* set the ".function" member! That's the job of the multi-arch
+selection \_\_attribute\_\_((constructor)) function
+
+Always inline node dispatch functions
+-------------------------------------
+
+It's typical for a graph dispatch function to contain one or more
+calls to an inline function. See above. If your node dispatch function
+is structured that way, make *ABSOLUTELY CERTAIN* to use the
+"always_inline" macro:
+
+::
+
+    always_inline uword
+    ip46_sdp_inline (vlib_main_t * vm, vlib_node_runtime_t * node, 
+                 vlib_frame_t * frame,
+    		 int is_ip4, int is_trace)
+    { ... }
+
+Otherwise, the compiler is highly likely NOT to build multiple
+versions of the guts of your dispatch function. 
+
+It's fairly easy to spot this mistake in "perf top." If you see, for
+example, a bunch of functions with names of the form
+"xxx_node_fn_avx2" in the profile, *BUT* your brand-new node function
+shows up with a name of the form "xxx_inline.isra.1", it's quite likely
+that the inline was declared "static inline" instead of "always_inline".
+
+Add the required Makefile.am content
+------------------------------------
+
+If the component in question already sets a "multiversioning_sources"
+variable, simply add the indicated .c file to the list. If not, add
+the required boilerplate:
+
+::
+
+    if CPU_X86_64
+    sdp_multiversioning_sources =			\
+    	sdp/node.c				\
+    	sdp/sdp_slookup.c
+
+    if CC_SUPPORTS_AVX2
+    ###############################################################
+    # AVX2
+    ###############################################################
+    libsdp_plugin_avx2_la_SOURCES = $(sdp_multiversioning_sources)
+    libsdp_plugin_avx2_la_CFLAGS =					\
+    	$(AM_CFLAGS)  @CPU_AVX2_FLAGS@				\
+    	-DCLIB_MARCH_VARIANT=avx2
+    noinst_LTLIBRARIES += libsdp_plugin_avx2.la
+    sdp_plugin_la_LIBADD += libsdp_plugin_avx2.la
+    endif
+
+    if CC_SUPPORTS_AVX512
+    ###############################################################
+    # AVX512
+    ###############################################################
+    libsdp_plugin_avx512_la_SOURCES = $(sdp_multiversioning_sources)
+    libsdp_plugin_avx512_la_CFLAGS =				\
+    	$(AM_CFLAGS) @CPU_AVX512_FLAGS@				\
+    	-DCLIB_MARCH_VARIANT=avx512
+    noinst_LTLIBRARIES += libsdp_plugin_avx512.la
+    sdp_plugin_la_LIBADD += libsdp_plugin_avx512.la
+    endif
+    endif
+
+A certain amount of cut-paste-modify is currently required. Hopefully
+we'll manage to improve the scheme in the future.
diff --git a/docs/gettingstarted/developers/plugins.md b/docs/gettingstarted/developers/plugins.rst
index ba3a2446306..d18a5a879d7 100644
--- a/docs/gettingstarted/developers/plugins.md
+++ b/docs/gettingstarted/developers/plugins.rst
@@ -1,3 +1,4 @@
+.. _dplugins:
 
 Plugins
 =======
@@ -9,3 +10,4 @@ filter to apply (if desired). VLIB needs to load plug-ins very early.
 Once loaded, the plug-in DLL mechanism uses dlsym to find and verify a
 vlib\_plugin\_registration data structure in the newly-loaded plug-in.
 
+For more on plugins please refer to :ref:`sample_plugin`.
diff --git a/docs/gettingstarted/developers/running_vpp.rst b/docs/gettingstarted/developers/running_vpp.rst
index f0d56fbab2b..a1215765545 100644
--- a/docs/gettingstarted/developers/running_vpp.rst
+++ b/docs/gettingstarted/developers/running_vpp.rst
@@ -5,13 +5,14 @@
 Running VPP
 ===========
 
-After build the VPP binaries, there a several to run the images you've built. These is useful when
-if you need to run VPP without  installing the packages. For instance if you want to run VPP with GDB.
+After building the VPP binaries, you now have several images that you have built.
+These images are useful when you need to run VPP without  installing the packages.
+For instance if you want to run VPP with GDB.
 
-Without GDB
+Running Without GDB
 _________________________
 
-To run the VPP images, that you've build without GDB.
+To run the VPP images that you've built without GDB, run the following commands:
 
 Running the release image:
 
@@ -27,7 +28,7 @@ Running the debug image:
    # make run
    #
 
-With GDB
+Running With GDB
 _________________________
 
 With the following commands you can run VPP and then be dropped into the GDB prompt.
diff --git a/docs/gettingstarted/developers/sample_plugin.rst b/docs/gettingstarted/developers/sample_plugin.rst
index 0426606f4c9..33fea0b4201 100644
--- a/docs/gettingstarted/developers/sample_plugin.rst
+++ b/docs/gettingstarted/developers/sample_plugin.rst
@@ -1,9 +1,9 @@
 .. _sample_plugin:
 
-.. toctree::
+Integrating a plugin
+=====================
 
-Integrating a complete plugin with VPP
-======================================
+.. toctree::
 
 Overview
 ________
diff --git a/docs/gettingstarted/index.rst b/docs/gettingstarted/index.rst
index acee1963af5..4fcc51348fc 100644
--- a/docs/gettingstarted/index.rst
+++ b/docs/gettingstarted/index.rst
@@ -1,17 +1,20 @@
 .. _gettingstarted:
 
 ######################
-Getting Started Guides
+Getting Started
 ######################
 
-The Getting Started Guide is comprised of several different sections; a section for Users,  another for Developers, and a section for Writing VPP documentation.
+The Getting Started Guide is comprised of several different sections; a User section
+that describes a basic installation and configuration of VPP (either manually or using
+a config utility),  another install for Developers, which contains additional code that
+provides tools that are used in a development environment.
 
-The Users section covers basic VPP installation and configuration operations. This section covers the following areas:
+The Users section covers basic VPP installation and configuration operations; this
+section covers the following areas:
 
-* Describes the different types of VPP packages
-* Describes how to install VPP Binaries on different OS platforms (Ubuntu, Centos, openSUSE).
-* Explains how to configure, then use VPP.
-* Explains how to use the Configuration Utility.
+* Describes the different types of VPP packages, which are used in both basic and developer installs.
+* Describes how to manually install VPP Binaries on different OS platforms (Ubuntu, Centos, openSUSE) and then how to configure and use VPP.
+* Explains how to use the Configuration Utility to install, and then configure VPP.
 
 The Developers section covers the following areas:
 
diff --git a/docs/gettingstarted/users/configuring/hugepages.rst b/docs/gettingstarted/users/configuring/hugepages.rst
index 76ac7d0807d..da812e37048 100644
--- a/docs/gettingstarted/users/configuring/hugepages.rst
+++ b/docs/gettingstarted/users/configuring/hugepages.rst
@@ -3,13 +3,14 @@
 Huge Pages
 ----------
 
-VPP requires *'hugepages'* to run. VPP will overwrite existing hugepage settings
-when VPP is installed. By default, VPP sets the number of hugepages on a system
-to 1024 2M hugepages (1G hugepages are no longer supported). This is the number
-of hugepages on the system, not just used by VPP. 
+VPP requires *hugepages* to run during VPP operation, to manage large pages of memory.
+During VPP  installation, VPP will overwrite the existing hugepage settings. By
+default, VPP sets the number of hugepages on a system to 1024 2M hugepages (1G hugepages
+are no longer supported). This is the number of hugepages on the system, not just used by VPP. 
 
-When VPP is installed, the following file is copied to the system and used to apply the
-hugepage settings on VPP installation and system reboot:
+When VPP is installed, the following configuration file is copied to the system. The
+hugepage settings are applied in the VPP installation and on system reboots. To set
+the hugepage settings, perform the following commands:
 
 .. code-block:: console
 
@@ -30,10 +31,10 @@ hugepage settings on VPP installation and system reboot:
     # to current shmmax value.
     kernel.shmmax=2147483648
 
-Depending on how the system is being used, this file can be updated to adjust
+Depending on how the system is being used, this configuration file can be updated to adjust
 the number of hugepages reserved on a system. Below are some examples of
-possible values.
-
+possible settings.
+ 
 For a small VM with minimal workload:
 
 .. code-block:: console
@@ -54,9 +55,9 @@ For a large system running multiple VMs, each needing its own set of hugepages:
 .. note::
 
     If VPP is being run in a Virtual Machine (VM), the VM must have hugepage
-    backing. When VPP is installed, it will attempt to overwrite existing
-    hugepage setting. If the VM does not have hugepage backing, this will fail,
-    but this may go unnoticed. When the VM is rebooted, on system startup,
-    *'vm.nr_hugepages'* will be reapplied, will fail, and the VM will abort kernel
+    backing. When VPP is installed, it will attempt to overwrite existing the
+    hugepage setting. If the VM does not have hugepage backing, the install will fail,
+    but the failure may go unnoticed. When the VM is rebooted, on system startup,
+    *'vm.nr_hugepages'* will be reapplied, and will fail, and the VM will abort kernel
     boot, locking up the VM. To avoid this scenario, ensure the VM has enough
     hugepage backing.
diff --git a/docs/gettingstarted/users/configuring/index.rst b/docs/gettingstarted/users/configuring/index.rst
index f1244c61a4f..0de77adea6c 100644
--- a/docs/gettingstarted/users/configuring/index.rst
+++ b/docs/gettingstarted/users/configuring/index.rst
@@ -3,8 +3,8 @@
 Configuring VPP
 ==================
 
-There is some basic configuration that is needed to run FD.io VPP.  This section
-describes the basic configuration:
+There is some basic configuration that needs to be performed before running
+FD.io VPP.  This section describes the configuration process:
 
 .. toctree::
    :maxdepth: 2
diff --git a/docs/gettingstarted/users/index.rst b/docs/gettingstarted/users/index.rst
index 36640a8b454..7bac586d65a 100644
--- a/docs/gettingstarted/users/index.rst
+++ b/docs/gettingstarted/users/index.rst
@@ -4,13 +4,19 @@
 Users
 ########
 
+The Users section describe a basic VPP installation and configuration operation.
+The  installation and configuration of VPP can be done either manually, or by
+using a configuration utility.
+
+This section covers the following areas:
+
 The Users section covers basic VPP installation and configuration operations. This
 section covers the following areas:
 
 * Describes the different types of VPP packages 
-* Describes how to install VPP Binaries on different OS platforms (Ubuntu, Centos, openSUSE)
-* Explains how to configure, then use VPP
-* Explains how to use the Configuration Utility
+* Describes how to manually install VPP Binaries on different OS platforms (Ubuntu, Centos, openSUSE)
+* Explains how to manually configure, then run VPP
+* Explains how to install, then configure VPP using the Configuration Utility
 
 .. toctree::
    :maxdepth: 2
diff --git a/docs/gettingstarted/users/installing/centos.rst b/docs/gettingstarted/users/installing/centos.rst
index 470ca87be1f..dae24f6b4c8 100644
--- a/docs/gettingstarted/users/installing/centos.rst
+++ b/docs/gettingstarted/users/installing/centos.rst
@@ -8,7 +8,8 @@ Setup the fd.io Repository - Centos 7
 Update the OS
 -------------
 
-Before starting the repository setup, it is a good idea to first update and upgrade the OS.
+Before starting the repository setup, it is a good idea to first update and upgrade
+the OS; run the following command to update the OS:
 
 .. code-block:: console
 
@@ -18,8 +19,7 @@ Before starting the repository setup, it is a good idea to first update and upgr
 Point to the Repository
 -----------------------
 
-For CentOS based systems, there are two respositories to pull VPP binaries
-from.
+For CentOS based systems, there are two respositories to pull VPP binaries from:
 
 * CentOS NFV SIG Repository
 * Nexus Repository
@@ -28,15 +28,14 @@ from.
 CentOS NFV SIG Repository
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-VPP is not in the official CentOS 7 distro. However, CentOS has Special
+VPP is not in the official CentOS 7 distro; however, CentOS has Special
 Interest Groups (SIG), which are smaller groups within the CentOS community that
 focus on a small set of issues. The CentOS NFV (Network Function Virtualization)
 SIG was created to provide a CentOS-based stack that will serve as a platform
-for the deployment and testing of virtual network functions (VNFs). VPP has been
-included in this SIG.
 
 To install released packages from the CentOS NFV SIG Repository on an updated
-Centos 7 system, first, install the CentOS NFV SIG FIDO repo file:
+Centos 7 system, first, install the CentOS NFV SIG FIDO repo file by running the
+following command:
 
 .. code-block:: console
 
@@ -109,7 +108,7 @@ VPP Latest Release
 """"""""""""""""""
 
 To allow *'yum'* access to the official VPP releases, create the file
-*'/etc/yum.repos.d/fdio-release.repo'* with the following content:
+*'/etc/yum.repos.d/fdio-release.repo'* with the following content.
 
 .. code-block:: console
 
@@ -122,7 +121,7 @@ To allow *'yum'* access to the official VPP releases, create the file
 
 The *'yum install vpp'* command will install the most recent release. To
 install older releases, run the following command to get the list of releases
-provided:
+provided.
 
 .. code-block:: console
 
@@ -133,10 +132,10 @@ sample *'yum --showduplicates list'* output and an example of installing a
 particular version of the RPMs.
 
 VPP Stable Branch
-"""""""""""""""""
+"""""""""""""""""""
 
 To allow *yum* access to the build artifacts for a VPP stable branch, create
-the file *'/etc/yum.repos.d/fdio-release.repo'* with the following content:
+the file *'/etc/yum.repos.d/fdio-release.repo'* with the following content.
 
 .. code-block:: console
 
@@ -164,10 +163,10 @@ particular version of the RPMs.
 
 
 VPP Master Branch
-"""""""""""""""""
+"""""""""""""""""""
 
 To allow *yum* access to the nightly builds from the VPP master branch, create
-the file *'/etc/yum.repos.d/fdio-release.repo'* with the following content:
+the file *'/etc/yum.repos.d/fdio-release.repo'* with the following content.
 
 .. code-block:: console
 
@@ -180,7 +179,7 @@ the file *'/etc/yum.repos.d/fdio-release.repo'* with the following content:
 
 The *'yum install vpp'* command will install the most recent build on the
 branch. Run the following command to get the list of images produce by the
-branch:
+branch.
 
 .. code-block:: console
 
@@ -194,18 +193,18 @@ particular version of the RPMs.
 Install VPP RPMs
 ================
 
-To install the VPP packet engine, run the following:
+To install the VPP packet engine, run the following command:
 
 .. code-block:: console
 
    $ sudo yum install vpp
 
-The **'vpp'** RPM depend on the **'vpp-lib'** and **'vpp-selinux-policy'**
+The *vpp* RPM depends on the *vpp-lib* and *vpp-selinux-policy*
 RPMs, so they will be installed as well.
 
 .. note::
 
-    The **'vpp-selinux-policy'** will not enable SELinux on the system. It
+    The *vpp-selinux-policy* will not enable SELinux on the system. It
     will install a Custom VPP SELinux policy that will be used if SELinux is
     enabled at any time.
 
@@ -221,19 +220,18 @@ Starting VPP
 ============
 
 Once VPP is installed on the system, to run VPP as a systemd service on CentOS,
-run:
+run the following command:
 
 .. code-block:: console
 
    $ sudo systemctl start vpp
 
-Then to enable VPP to start on system reboot:
+Then to enable VPP to start on system reboot, run the following command:
 
 .. code-block:: console
 
    $ sudo systemctl enable vpp
 
-
 Outside of running VPP as a systemd service, VPP can be started manually or
 made to run within GDB for debugging. See :ref:`running` for more details and
 ways to tailor VPP to a specific system.
@@ -242,6 +240,8 @@ ways to tailor VPP to a specific system.
 Uninstall the VPP RPMs
 ======================
 
+To uninstall a VPP RPM, run the following command:
+
 .. code-block:: console
 
    $ sudo yum autoremove vpp*
diff --git a/docs/gettingstarted/users/installing/index.rst b/docs/gettingstarted/users/installing/index.rst
index 4a8ac7a6952..12a5ab8e015 100644
--- a/docs/gettingstarted/users/installing/index.rst
+++ b/docs/gettingstarted/users/installing/index.rst
@@ -12,16 +12,8 @@ existing packages. This guide describes how to pull, install and run the VPP pac
 
 .. toctree::
 
-Package Descriptions
---------------------
-The following is a brief description of the packages to be installed with VPP.
-
-.. toctree::
-
-   packages
-
-Installing VPP Binaries
-----------------------------------
+Installing VPP
+---------------
 
 This section provides directions on how to Install VPP binaries on Ubuntu, Centos,
 and openSUSE platforms.
@@ -49,3 +41,11 @@ The following are instructions on how to install VPP on openSUSE.
 .. toctree::
 
    opensuse
+
+Package Descriptions
+--------------------
+The following is a brief description of the packages to be installed with VPP.
+
+.. toctree::
+
+   packages
diff --git a/docs/gettingstarted/users/installing/opensuse.rst b/docs/gettingstarted/users/installing/opensuse.rst
index b5dcbc35820..6d7873820dc 100644
--- a/docs/gettingstarted/users/installing/opensuse.rst
+++ b/docs/gettingstarted/users/installing/opensuse.rst
@@ -4,8 +4,9 @@
 
 Installing
 ==========
-To install VPP on openSUSE, first install the following release and then execute
-the appropriate commands.
+
+To install VPP on openSUSE, first install the following release, and then execute
+the associated commands.
 
 openSUSE Tumbleweed (rolling release)
 ------------------------------------------------------------
@@ -25,6 +26,8 @@ openSUSE Leap 42.3
 Uninstall
 =========
 
+To uninstall the vpp plugins, run the following command:
+
 .. code-block:: console
 
    sudo zypper remove -u vpp vpp-plugins
@@ -32,6 +35,8 @@ Uninstall
 openSUSE Tumbleweed (rolling release)
 -------------------------------------
 
+To uninstall the openSUSE Tumbleweed, run the following command:
+
 .. code-block:: console
 
    sudo zypper remove -u vpp vpp-plugins
diff --git a/docs/gettingstarted/users/installing/ubuntu.rst b/docs/gettingstarted/users/installing/ubuntu.rst
index 5b4a1d86fee..cb7a1c6097d 100644
--- a/docs/gettingstarted/users/installing/ubuntu.rst
+++ b/docs/gettingstarted/users/installing/ubuntu.rst
@@ -5,13 +5,13 @@
 Ubuntu 16.04 - Setup the fd.io Repository
 ==========================================
 
-From the following, choose one of the releases to install.
+Choose one of the following releases to install.
 
 
 Update the OS
 -----------------------
 
-It is probably a good idea to update and upgrade the OS before starting
+It is a good idea to first update and upgrade the OS before starting; run the following command to update the OS:
 
 .. code-block:: console
 
@@ -21,7 +21,7 @@ It is probably a good idea to update and upgrade the OS before starting
 Point to the Repository
 -----------------------------------
 
-Create a file **"/etc/apt/sources.list.d/99fd.io.list"** with the contents that point to
+Create a file **/etc/apt/sources.list.d/99fd.io.list** with contents that point to
 the version needed. The contents needed are shown below.
 
 .. _install_vpp:
@@ -29,7 +29,7 @@ the version needed. The contents needed are shown below.
 VPP latest Release
 ^^^^^^^^^^^^^^^^^^^
 
-Create the file **/etc/apt/sources.list.d/99fd.io.list** with contents:
+Create the file **/etc/apt/sources.list.d/99fd.io.list** that contain the following contents:
 
 .. code-block:: console
 
@@ -39,7 +39,7 @@ Create the file **/etc/apt/sources.list.d/99fd.io.list** with contents:
 VPP stable/1804 Branch
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-Create the file **/etc/apt/sources.list.d/99fd.io.list** with contents:
+Create the file **/etc/apt/sources.list.d/99fd.io.list** that contain the following contents:
 
 .. code-block:: console
 
@@ -49,7 +49,7 @@ Create the file **/etc/apt/sources.list.d/99fd.io.list** with contents:
 VPP master Branch
 ^^^^^^^^^^^^^^^^^^^^
 
-Create the file **/etc/apt/sources.list.d/99fd.io.list** with contents:
+Create the file **/etc/apt/sources.list.d/99fd.io.list** that contain the following contents:
 
 .. code-block:: console
 
@@ -59,6 +59,8 @@ Create the file **/etc/apt/sources.list.d/99fd.io.list** with contents:
 Install the Mandatory Packages
 ===============================
 
+Install the mandatory packages by running the following commands:
+
 .. code-block:: console
 
   sudo apt-get update
@@ -68,6 +70,8 @@ Install the Mandatory Packages
 Install the Optional Packages
 ==============================
 
+Install the optional packages by running the following command:
+
 .. code-block:: console
 
   sudo apt-get install vpp-dbg vpp-dev vpp-api-java vpp-api-python vpp-api-lua
@@ -76,6 +80,8 @@ Install the Optional Packages
 Uninstall the Packages
 ======================
 
+Uninstall the  packages by running the following command:
+
 .. code-block:: console
 
   sudo apt-get remove --purge vpp*
diff --git a/docs/gettingstarted/writingdocs/index.rst b/docs/gettingstarted/writingdocs/index.rst
index c59a39d0e5d..45a746df826 100644
--- a/docs/gettingstarted/writingdocs/index.rst
+++ b/docs/gettingstarted/writingdocs/index.rst
@@ -1,7 +1,7 @@
 .. _writingdocs:
 
 #########################
-VPP Documents
+Writing Documents
 #########################
 
 This section covers the following topics: