DOC ONLY: clean up plugin documentation

The old "sample_plugin" page was stuffed with superceded autotools
build information, so it morphed into an "add a new plugin" page based
on the emacs-lisp plugin generator.

Before sending hate mail about emacs, please *look* at the new
document: you'll find running the plugin generator hard to tell from
running a shell script.

Change-Id: I84da45675e838c05faeca05c8f7be45d8c7bff13
Signed-off-by: Dave Barach <dave@barachs.net>

4 files changed, 277 insertions, 175 deletions

diff --git a/docs/gettingstarted/developers/add_plugin.rst b/docs/gettingstarted/developers/add_plugin.rst
new file mode 100644
index 00000000000..eb9113a2efc
--- /dev/null
+++ b/docs/gettingstarted/developers/add_plugin.rst
@@ -0,0 +1,275 @@
+.. _add_plugin:
+
+Adding a plugin
+===============
+
+.. toctree::
+
+Overview
+________
+ 
+This section shows how a VPP developer can create a new plugin, and
+add it to VPP.
+
+As an example, we will use the **make-plugin.sh** tool found in
+**.../extras/emacs**. make-plugin.sh is a simple wrapper for a comprehensive
+plugin generator constructed from a set of emacs-lisp skeletons.
+
+Create your new plugin
+----------------------
+
+Change directory to **.../src/plugins**, and run the plugin generator:
+
+.. code-block:: console
+    
+    $ cd .../src/plugins
+    $ ../../extras/emacs/make-plugin.sh
+    <snip>
+    Loading /scratch/vpp-docs/extras/emacs/tunnel-c-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/tunnel-decap-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/tunnel-encap-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/tunnel-h-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/elog-4-int-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/elog-4-int-track-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/elog-enum-skel.el (source)...
+    Loading /scratch/vpp-docs/extras/emacs/elog-one-datum-skel.el (source)...
+    Plugin name: myplugin
+    Dispatch type [dual or qs]: dual
+    (Shell command succeeded with no output)
+    
+    OK...
+
+The plugin generator script asks two questions: the name of the
+plugin, and which of two dispatch types to use. Since the plugin name
+finds its way into quite a number of places - filenames, typedef
+names, graph arc names - it pays to think for a moment.
+
+The dispatch type refers to the coding pattern used to construct
+**node.c**, the *pro forma* data-plane node. The **dual** option
+constructs a dual-single loop pair with speculative enqueueing. This
+is the traditional coding pattern for load-store intensive graph
+nodes.
+
+The **qs** option generates a quad-single loop pair which uses
+vlib_get_buffers(...) and vlib_buffer_enqueue_to_next(...). These
+operators make excellent use of available SIMD vector unit
+operations. It's very simple to change a quad-single loop-pair to a
+dual-single loop pair if you decide to do so later.
+
+Generated Files
+---------------
+
+Here are the generated files. We'll go through them in a moment.
+
+.. code-block:: console
+
+    $ cd .../src/plugins/myplugin
+    $ ls
+    CMakeLists.txt        myplugin.c           myplugin_periodic.c  setup.pg
+    myplugin_all_api_h.h  myplugin.h           myplugin_test.c
+    myplugin.api          myplugin_msg_enum.h  node.c
+
+Due to recent build system improvements, you **don't** need to touch
+any other files to integrate your new plugin into the vpp build. Simply
+rebuild your workspace from scratch, and the new plugin will appear.
+
+Rebuild your workspace
+----------------------
+
+This is the straightforward way to reconfigure and rebuild your workspace:
+
+.. code-block:: console
+
+    $ cd <top-of-workspace>
+    $ make rebuild [or rebuild-release]
+
+Thanks to ccache, this operation doesn't take an annoying amount of time.
+
+Sanity check: run vpp
+---------------------
+
+As a quick sanity check, run vpp and make sure that
+"myplugin_plugin.so" and "myplugin_test_plugin.so" are loaded:
+
+.. code-block:: console
+    
+    $ cd <top-of-workspace>
+    $ make run
+    <snip>
+    load_one_plugin:189: Loaded plugin: myplugin_plugin.so (myplugin description goes here)
+    <snip>
+    load_one_vat_plugin:67: Loaded plugin: myplugin_test_plugin.so
+    <snip>
+    DBGvpp#
+
+If this simple test fails, please seek assistance.
+
+Generated Files in Detail
+_________________________
+
+This section discusses the generated files in some detail. It's fine to
+skim this section, and return later for more detail.
+
+CMakeLists.txt
+--------------
+
+This is the build system recipe for building your plugin. Please fix
+the copyright notice:
+
+.. code-block:: console
+
+    # Copyright (c) <current-year> <your-organization>
+
+The rest of the build recipe is pretty simple:
+
+.. code-block:: console
+
+    add_vpp_plugin (myplugin
+    SOURCES
+    myplugin.c 
+    node.c 
+    myplugin_periodic.c
+    myplugin.h
+    
+    MULTIARCH_SOURCES
+    node.c 
+    
+    API_FILES
+    myplugin.api
+    
+    INSTALL_HEADERS
+    myplugin_all_api_h.h
+    myplugin_msg_enum.h
+    
+    API_TEST_SOURCES
+    myplugin_test.c
+    )
+
+As you can see, the build recipe consists of several lists of
+files. **SOURCES** is a list of C source files. **API_FILES** is a
+list of the plugin's binary API definition files [one such file is
+usually plenty], and so forth.
+
+**MULTIARCH_SOURCES** lists data plane graph node dispatch function
+source files considered to be performance-critical. Specific functions
+in these files are compiled multiple times, so that they can leverage
+CPU-specific features. More on this in a moment.
+
+If you add source files, simply add them to the indicated list(s).
+
+myplugin.h
+----------
+
+This is the primary #include file for the new plugin. Among other
+things, it defines the plugin's *main_t* data structure. This is the
+right place to add problem-specific data structures. Please **resist
+the temptation** to create a set of static or [worse yet] global
+variables in your plugin. Refereeing name-collisions between plugins
+is not anyone's idea of a good time.
+
+myplugin.c
+----------
+
+For want of a better way to describe it, myplugin.c is the vpp plugin
+equivalent of "main.c". Its job is to hook the plugin into the vpp
+binary API message dispatcher, and to add its messages to vpp's global
+"message-name_crc" hash table. See "myplugin_init (...")"
+
+Vpp itself uses dlsym(...) to track down the vlib_plugin_registration_t
+generated by the VLIB_PLUGIN_REGISTER macro:
+
+.. code-block:: console
+
+    VLIB_PLUGIN_REGISTER () = 
+      {
+        .version = VPP_BUILD_VER,
+        .description = "myplugin plugin description goes here",
+      };  
+
+Vpp only loads .so files from the plugin directory which contain an
+instance of this data structure.
+
+You can enable or disable specific vpp plugins from the command
+line. By default, plugins are loaded. To change that behavior, set
+default_disabled in the vlib_plugin_macro:
+
+.. code-block:: console
+
+  .default_disabled = 1
+
+The boilerplate generator places the graph node dispatch function
+onto the "device-input" feature arc. This may or may not be useful.
+
+.. code-block:: console
+
+    VNET_FEATURE_INIT (myplugin, static) =
+    {
+      .arc_name = "device-input",
+      .node_name = "myplugin",
+      .runs_before = VNET_FEATURES ("ethernet-input"),
+    };  
+
+As given by the plugin generator, myplugin.c contains the binary API
+message handler for a generic "please enable my feature on such and
+such an interface" binary API message. As you'll see, setting up the
+vpp message API tables is simple. Big fat warning: the scheme is
+intolerant of minor mistakes. Example: forgetting to add
+mainp->msg_id_base can lead to very confusing failures.
+
+If you stick to modifying the generated boilerplate with care -
+instead of trying to build code from first principles - you'll save
+yourself a bunch of time and aggravation
+
+myplugin_test.c
+---------------
+
+This file contains binary API message **generation** code, which is
+compiled into a separate .so file. The "vpp_api_test" program loads
+these plugins, yielding immediate access to your plugin APIs for
+external client binary API testing.
+
+vpp itself loads test plugins, and makes the code available via the
+"binary-api" debug CLI. This is a favorite way to unit-test binary
+APIs prior to integration testing.
+
+node.c
+------
+
+This is the generated graph node dispatch function. You'll need to
+rewrite it to solve the problem at hand. It will save considerable
+time and aggravation to retain the **structure** of the node dispatch
+function. 
+
+Even for an expert, it's a waste of time to reinvent the *loop
+structure*, enqueue patterns, and so forth. Simply tear out and
+replace the specimen 1x, 2x, 4x packet processing code with code
+relevant to the problem you're trying to solve.
+
+Plugin "Friends with Benefits"
+------------------------------
+
+In vpp VLIB_INIT_FUNCTION functions, It's reasonably common to see a
+specific init function invoke other init functions:
+
+.. code-block:: console
+
+    if ((error = vlib_call_init_function (vm, some_other_init_function))
+       return error;
+
+In the case where one plugin needs to call a init function in another
+plugin, use the vlib_call_plugin_init_function macro:
+
+.. code-block:: console
+
+    if ((error = vlib_call_plugin_init_function (vm, "otherpluginname", some_init_function))
+       return error;
+
+This allows sequencing between plugin init functions.
+
+If you wish to obtain a pointer to a symbol in another plugin, use the
+vlib_plugin_get_symbol(...) API:
+
+.. code-block:: console
+
+    void *p = vlib_get_plugin_symbol ("plugin_name", "symbol");
+
diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst
index 8b772a10aeb..3520ed377f5 100644
--- a/docs/gettingstarted/developers/index.rst
+++ b/docs/gettingstarted/developers/index.rst
@@ -34,5 +34,5 @@ The Developers section covers the following areas:
    binary_api_support
    buildsystem/index.rst
    eventviewer
-   sample_plugin
+   add_plugin
    fib20/index.rst
diff --git a/docs/gettingstarted/developers/plugins.rst b/docs/gettingstarted/developers/plugins.rst
index d18a5a879d7..09db1d3459f 100644
--- a/docs/gettingstarted/developers/plugins.rst
+++ b/docs/gettingstarted/developers/plugins.rst
@@ -10,4 +10,4 @@ filter to apply (if desired). VLIB needs to load plug-ins very early.
 Once loaded, the plug-in DLL mechanism uses dlsym to find and verify a
 vlib\_plugin\_registration data structure in the newly-loaded plug-in.
 
-For more on plugins please refer to :ref:`sample_plugin`.
+For more on plugins please refer to :ref:`add_plugin`.
diff --git a/docs/gettingstarted/developers/sample_plugin.rst b/docs/gettingstarted/developers/sample_plugin.rst
deleted file mode 100644
index 33fea0b4201..00000000000
--- a/docs/gettingstarted/developers/sample_plugin.rst
+++ /dev/null
@@ -1,173 +0,0 @@
-.. _sample_plugin:
-
-Integrating a plugin
-=====================
-
-.. toctree::
-
-Overview
-________
- 
-This section shows how a VPP plugin developer can modify VPP scripts to add and load their plugin as a node in VPP. 
-
-As an example we will integrate the **Sample Plugin** found in *vpp/src/examples/sample-plugin/sample* The VPP Sample Plugin is a small plugin that demonstrates simple implementation of a macswap algorithim. Since it is a VPP plugin, it has runtime integration with the VPP graph hierachy, API, and CLI.
-
-This section will not go into the details of the plugin itself. For a deeper dive into the sample plugin see the annotations in `sample.c <https://docs.fd.io/vpp/18.11/da/d30/sample_8c.html>`_,  or go to the next page for general VPP C API usage.
-
-Setup
-_____
-
-Each plugin has their own automake file (\*.am) used by *configure.ac*, as well as a separate directory containing C files for the plugin. The directory containing these for each plugin is *vpp/src/plugins*
-
-To get a basic idea for how a VPP automake plugin file specifies its C files, here is part of the Sample Plugin automake file, *sample.am*
-
-.. code-block:: console
-    
-    sample_plugin_la_SOURCES =      \
-        sample/sample.c             \
-        sample/node.c               \
-        sample/sample_plugin.api.h
-
-    API_FILES += sample/sample.api
-
-    nobase_apiinclude_HEADERS +=            \
-      sample/sample_all_api_h.h             \
-      sample/sample_msg_enum.h              \
-      sample/sample.api.h
-
-
-The Sample Plugin is located in *vpp/src/examples/sample-plugin/sample*, so as mentioned above we will need to copy its contents into *vpp/src/plugins*
-
-In your */vpp* directory, or the directory above */src*, run:
-
-.. code-block:: console
-    
-    $ cp -r src/examples/sample-plugin/sample src/plugins
-    $ cp src/examples/sample-plugin/sample.am src/plugins
-
-Modifying configure.ac and Makefile.am
-______________________________________
-
-We now need to modify the plugin sections of the VPP automake and configuration scripts so that VPP builds correctly with your new plugin.
-
-Using a text editor such as *vi*, add the following entry to the plugins section in *vpp/src/configure.ac*
-
-.. code-block:: console
-    
-    PLUGIN_ENABLED(sample)
-
-For reference, the plugins section of that file looks like this:
-
-.. code-block:: console
-
-    ###############################################################################
-    # Plugins
-    ###############################################################################
-
-    # Please keep alphabetical order
-    PLUGIN_ENABLED(abf)
-    PLUGIN_ENABLED(acl)
-    PLUGIN_ENABLED(avf)
-    PLUGIN_ENABLED(cdp)
-    PLUGIN_ENABLED(dpdk)
-    PLUGIN_ENABLED(flowprobe)
-
-
-Using a text editor such as *vi*, now add the following entry to the plugins section in *vpp/src/plugins/Makefile.am*
-
-.. code-block:: console
-
-    if ENABLE_SAMPLE_PLUGIN
-    include sample.am
-    endif
-
-For reference, the plugins section of that file looks something like this:
-
-.. code-block:: console
-
-    vppapitestpluginsdir = ${libdir}/vpp_api_test_plugins
-    vpppluginsdir = ${libdir}/vpp_plugins
-
-    if ENABLE_ABF_PLUGIN
-    include abf.am
-    endif
-
-    if ENABLE_ACL_PLUGIN
-    include acl.am
-    endif
-
-    if ENABLE_AVF_PLUGIN
-    include avf.am
-    endif
-
-Building and Running
-____________________
-
-
-Build VPP by using the main Makefile found in */vpp/Makefile*
-
-.. code-block:: console
-    
-    $ make build
-
-.. note::
-
-    If you want to have a fresh debug build and compile every VPP file from scratch, you can wipe all compiled files and build VPP with:
-
-    .. code-block:: console
-    
-        $ make rebuild
-
-    However this will take much longer than just running *make build*
-
-Run VPP and make sure the plugin is loaded. Below is the command for running the VPP debug binary, accompanied with sample output.
-
-.. code-block:: console
-    
-    $ make run
-    vlib_plugin_early_init:361: plugin path /vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins:/vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_plugins
-    load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
-    load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
-    load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
-    load_one_plugin:191: Loaded plugin: cdp_plugin.so
-    ...
-    load_one_plugin:189: Loaded plugin: sample_plugin.so (Sample of VPP Plugin)
-    ...
-    load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
-    load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
-    load_one_vat_plugin:67: Loaded plugin: sample_test_plugin.so
-    ...
-        _______    _        _   _____  ___ 
-     __/ __/ _ \  (_)__    | | / / _ \/ _ \
-     _/ _// // / / / _ \   | |/ / ___/ ___/
-     /_/ /____(_)_/\___/   |___/_/  /_/    
-
-    DBGvpp# 
-
-.. note::
-
-    Notice when running the debug build that (\*_test_plugin.so) is also loaded, which is meant for testing your plugin.
-
-To enable the sample plugin, use this command:
-
-.. code-block:: console
-    
-    DBGvpp# sample macswap <interface name>
-
-To disable the sample plugin, use this command:
-
-.. code-block:: console
-    
-    DBGvpp# sample macswap <interface name> disable
-
-
-Great! Now you've successfully added your plugin as a VPP node.
-
-
-Additional remarks 
-__________________
-
-How the build process works for plugins is that the (\*.api) plugin file is automatically translated to a JSON file (\*.api.json) in *vpp/build-root/install-vpp_debug-native/vpp/share/vpp/api/plugins*, which the code generator then parses and generates a C header file (\*.api.h) in *vpp/build-root/install-vpp_debug-native/vpp/include/vpp_plugins/\**.
-
-After the build process is completed you finally end up with two plugin files (\*_plugin.so and \*_test_plugin.so) found in *vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_plugins* and *vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_api_test_plugins* respectively, that are loaded at runtime during a debug binary run of VPP (*make run*).
-