DOC ONLY: Add a new doc for integrating a plugin with VPP

Convert doxygen docs (vpp api module and binary api support) to .rst

Added a brief explanation of binary api message numbering for plugins

Change-Id: If7d37f29aa65a884b51f3d27957f80d1357ed29b
Signed-off-by: andrew <andrew.olechtchouk@gmail.com>
Signed-off-by: Dave Barach <dave@barachs.net>

4 files changed, 710 insertions, 0 deletions

diff --git a/docs/gettingstarted/developers/binary_api_support.rst b/docs/gettingstarted/developers/binary_api_support.rst
new file mode 100644
index 00000000000..a99bf3faa98
--- /dev/null
+++ b/docs/gettingstarted/developers/binary_api_support.rst
@@ -0,0 +1,412 @@
+.. _binary_api_support:
+
+.. toctree::
+
+Binary API Support
+==================
+
+VPP provides a binary API scheme to allow a wide variety of client
+codes to program data-plane tables. As of this writing, there are
+hundreds of binary APIs.
+
+Messages are defined in \*.api files. Today, there are about 80 api
+files, with more arriving as folks add programmable features. The API
+file compiler sources reside in src/tools/vppapigen.
+
+From `src/vnet/interface.api
+<https://docs.fd.io/vpp/18.11/de/d75/interface_8api.html>`_, here's a
+typical request/response message definition:
+
+.. code-block:: console
+
+	autoreply define sw_interface_set_flags
+	{
+	  u32 client_index;
+	  u32 context;
+	  u32 sw_if_index;
+	  /* 1 = up, 0 = down */
+	  u8 admin_up_down;
+	};
+
+To a first approximation, the API compiler renders this definition
+into
+*vpp/build-root/install-vpp_debug-native/vpp/include/vnet/interface.api.h*
+as follows:
+
+.. code-block:: C
+
+	/****** Message ID / handler enum ******/
+
+	#ifdef vl_msg_id
+	vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS, vl_api_sw_interface_set_flags_t_handler)
+	vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, vl_api_sw_interface_set_flags_reply_t_handler)
+	#endif      
+	/****** Message names ******/
+
+	#ifdef vl_msg_name
+	vl_msg_name(vl_api_sw_interface_set_flags_t, 1)
+	vl_msg_name(vl_api_sw_interface_set_flags_reply_t, 1)
+	#endif      
+	/****** Message name, crc list ******/
+
+	#ifdef vl_msg_name_crc_list
+	#define foreach_vl_msg_name_crc_interface \
+	_(VL_API_SW_INTERFACE_SET_FLAGS, sw_interface_set_flags, f890584a) \
+	_(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply, dfbf3afa) \
+	#endif      
+	/****** Typedefs *****/
+
+	#ifdef vl_typedefs
+	#ifndef defined_sw_interface_set_flags
+	#define defined_sw_interface_set_flags
+	typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags {
+	    u16 _vl_msg_id;
+	    u32 client_index;
+	    u32 context;
+	    u32 sw_if_index;
+	    u8 admin_up_down;
+	}) vl_api_sw_interface_set_flags_t;
+	#endif
+
+	#ifndef defined_sw_interface_set_flags_reply
+	#define defined_sw_interface_set_flags_reply
+	typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags_reply {
+	    u16 _vl_msg_id;
+	    u32 context;
+	    i32 retval;
+	}) vl_api_sw_interface_set_flags_reply_t;
+	#endif
+	...
+	#endif /* vl_typedefs */
+
+To change the admin state of an interface, a binary api client sends a
+`vl_api_sw_interface_set_flags_t
+<https://docs.fd.io/vpp/18.11/dc/da3/structvl__api__sw__interface__set__flags__t.html>`_
+to VPP, which will respond with a
+vl_api_sw_interface_set_flags_reply_t message.
+
+Multiple layers of software, transport types, and shared libraries
+implement a variety of features:
+
+* API message allocation, tracing, pretty-printing, and replay.
+* Message transport via global shared memory, pairwise/private shared memory, and sockets.
+* Barrier synchronization of worker threads across thread-unsafe message handlers.
+
+Correctly-coded message handlers know nothing about the transport used
+to deliver messages to/from VPP. It's reasonably straighforward to use
+multiple API message transport types simultaneously.
+
+For historical reasons, binary api messages are (putatively) sent in
+network byte order. As of this writing, we're seriously considering
+whether that choice makes sense.
+
+Message Allocation
+__________________
+
+Since binary API messages are always processed in order, we allocate
+messages using a ring allocator whenever possible. This scheme is
+extremely fast when compared with a traditional memory allocator, and
+doesn't cause heap fragmentation. See `src/vlibmemory/memory_shared.c
+<https://docs.fd.io/vpp/18.11/dd/d0d/memory__shared_8c.html>`_
+`vl_msg_api_alloc_internal()
+<https://docs.fd.io/vpp/18.11/dd/d0d/memory__shared_8c.html#ac6b6797850e1a53bc68b206e6b8413fb>`_.
+
+Regardless of transport, binary api messages always follow a `msgbuf_t <https://docs.fd.io/vpp/18.11/d9/d65/structmsgbuf__.html>`_ header:
+
+.. code-block:: C
+
+	/** Message header structure */
+	typedef struct msgbuf_
+	{
+	  svm_queue_t *q; /**< message allocated in this shmem ring  */
+	  u32 data_len;                  /**< message length not including header  */
+	  u32 gc_mark_timestamp;         /**< message garbage collector mark TS  */
+	  u8 data[0];                    /**< actual message begins here  */
+	} msgbuf_t;
+
+This structure makes it easy to trace messages without having to
+decode them - simply save data_len bytes - and allows
+`vl_msg_api_free()
+<https://docs.fd.io/vpp/18.11/d6/d1b/api__common_8h.html#aff61e777fe5df789121d8e78134867e6>`_
+to rapidly dispose of message buffers:
+
+.. code-block:: C
+
+	void
+	vl_msg_api_free (void *a)
+	{
+	  msgbuf_t *rv;
+	  void *oldheap;
+	  api_main_t *am = &api_main;
+
+	  rv = (msgbuf_t *) (((u8 *) a) - offsetof (msgbuf_t, data));
+
+	  /*
+	   * Here's the beauty of the scheme.  Only one proc/thread has
+	   * control of a given message buffer. To free a buffer, we just clear the
+	   * queue field, and leave. No locks, no hits, no errors...
+	   */
+	  if (rv->q)
+	    {
+	      rv->q = 0;
+	      rv->gc_mark_timestamp = 0;
+	      <more code...>
+	      return;
+	    }
+	  <more code...>
+	}
+
+Message Tracing and Replay
+__________________________
+
+It's extremely important that VPP can capture and replay sizeable
+binary API traces. System-level issues involving hundreds of thousands
+of API transactions can be re-run in a second or less. Partial replay
+allows one to binary-search for the point where the wheels fall
+off. One can add scaffolding to the data plane, to trigger when
+complex conditions obtain.
+
+With binary API trace, print, and replay, system-level bug reports of
+the form "after 300,000 API transactions, the VPP data-plane stopped
+forwarding traffic, FIX IT!" can be solved offline.
+
+More often than not, one discovers that a control-plane client
+misprograms the data plane after a long time or under complex
+circumstances. Without direct evidence, "it's a data-plane problem!"
+
+See `src/vlibmemory/memory_vlib::c
+<https://docs.fd.io/vpp/18.11/dd/d3e/vpp__get__metrics_8c.html#a7c3855ed3c45b48ff92a7e881bfede73>`_
+`vl_msg_api_process_file()
+<https://docs.fd.io/vpp/18.11/d0/d5b/vlib__api__cli_8c.html#a60194e3e91c0dc6a75906ea06f4ec113>`_,
+and `src/vlibapi/api_shared.c
+<https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html>`_. See also
+the debug CLI command "api trace"
+
+Client connection details
+_________________________
+
+Establishing a binary API connection to VPP from a C-language client is easy:
+
+.. code-block:: C
+
+	int
+	connect_to_vpe (char *client_name, int client_message_queue_length)
+	{
+	  vat_main_t *vam = &vat_main;
+	  api_main_t *am = &api_main;
+	  if (vl_client_connect_to_vlib ("/vpe-api", client_name, 
+	                                client_message_queue_length) < 0)
+	    return -1;
+	  /* Memorize vpp's binary API message input queue address */
+	  vam->vl_input_queue = am->shmem_hdr->vl_input_queue;
+	  /* And our client index */
+	  vam->my_client_index = am->my_client_index;
+	  return 0;
+	}       
+
+32 is a typical value for client_message_queue_length. VPP *cannot*
+block when it needs to send an API message to a binary API client. The
+VPP-side binary API message handlers are very fast. So, when sending
+asynchronous messages, make sure to scrape the binary API rx ring with
+some enthusiasm!
+
+**Binary API message RX pthread**
+
+Calling `vl_client_connect_to_vlib
+<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a6654b42c91be33bfb6a4b4bfd2327920>`_
+spins up a binary API message RX pthread:
+
+.. code-block:: C
+
+	static void *
+	rx_thread_fn (void *arg)
+	{
+	  svm_queue_t *q;
+	  memory_client_main_t *mm = &memory_client_main;
+	  api_main_t *am = &api_main;
+	  int i;
+
+	  q = am->vl_input_queue;
+
+	  /* So we can make the rx thread terminate cleanly */
+	  if (setjmp (mm->rx_thread_jmpbuf) == 0)
+	    {
+	      mm->rx_thread_jmpbuf_valid = 1;
+	      /*
+	       * Find an unused slot in the per-cpu-mheaps array,
+	       * and grab it for this thread. We need to be able to
+	       * push/pop the thread heap without affecting other thread(s).
+	       */
+	      if (__os_thread_index == 0)
+	        {
+	          for (i = 0; i < ARRAY_LEN (clib_per_cpu_mheaps); i++)
+	            {
+	              if (clib_per_cpu_mheaps[i] == 0)
+	                {
+	                  /* Copy the main thread mheap pointer */
+	                  clib_per_cpu_mheaps[i] = clib_per_cpu_mheaps[0];
+	                  __os_thread_index = i;
+	                  break;
+	                }
+	            }
+	          ASSERT (__os_thread_index > 0);
+	        }
+	      while (1)
+	        vl_msg_api_queue_handler (q);
+	    }
+	  pthread_exit (0);
+	}
+
+To handle the binary API message queue yourself, use
+`vl_client_connect_to_vlib_no_rx_pthread
+<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a11b9577297106c57c0783b96ab190c36>`_.
+
+**Queue non-empty signalling**
+
+vl_msg_api_queue_handler(...) uses mutex/condvar signalling to wake
+up, process VPP -> client traffic, then sleep. VPP supplies a condvar
+broadcast when the VPP -> client API message queue transitions from
+empty to nonempty.
+
+VPP checks its own binary API input queue at a very high rate. VPP
+invokes message handlers in "process" context [aka cooperative
+multitasking thread context] at a variable rate, depending on
+data-plane packet processing requirements.
+
+Client disconnection details
+____________________________
+
+To disconnect from VPP, call `vl_client_disconnect_from_vlib
+<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a82c9ba6e7ead8362ae2175eefcf2fd12>`_. Please
+arrange to call this function if the client application terminates
+abnormally. VPP makes every effort to hold a decent funeral for dead
+clients, but VPP can't guarantee to free leaked memory in the shared
+binary API segment.
+
+Sending binary API messages to VPP
+__________________________________
+
+The point of the exercise is to send binary API messages to VPP, and
+to receive replies from VPP. Many VPP binary APIs comprise a client
+request message, and a simple status reply. For example, to set the
+admin status of an interface:
+
+.. code-block:: C
+
+	vl_api_sw_interface_set_flags_t *mp;
+	mp = vl_msg_api_alloc (sizeof (*mp));
+	memset (mp, 0, sizeof (*mp));
+	mp->_vl_msg_id = clib_host_to_net_u16 (VL_API_SW_INTERFACE_SET_FLAGS);
+	mp->client_index = api_main.my_client_index;
+	mp->sw_if_index = clib_host_to_net_u32 (<interface-sw-if-index>);
+	vl_msg_api_send (api_main.shmem_hdr->vl_input_queue, (u8 *)mp);
+
+Key points:
+
+* Use `vl_msg_api_alloc <https://docs.fd.io/vpp/18.11/dc/d5a/memory__shared_8h.html#a109ff1e95ebb2c968d43c100c4a1c55a>`_ to allocate message buffers
+* Allocated message buffers are not initialized, and must be presumed to contain trash.
+* Don't forget to set the _vl_msg_id field!
+* As of this writing, binary API message IDs and data are sent in network byte order
+* The client-library global data structure `api_main <https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html#af58e3e46b569573e9622b826b2f47a22>`_ keeps track of sufficient pointers and handles used to communicate with VPP
+
+Receiving binary API messages from VPP
+______________________________________
+
+Unless you've made other arrangements (see
+`vl_client_connect_to_vlib_no_rx_pthread
+<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a11b9577297106c57c0783b96ab190c36>`_),
+*messages are received on a separate rx pthread*. Synchronization with
+the client application main thread is the responsibility of the
+application!
+
+Set up message handlers about as follows:
+
+.. code-block:: C
+
+	#define vl_typedefs         /* define message structures */
+	#include <vpp/api/vpe_all_api_h.h>
+	#undef vl_typedefs
+	/* declare message handlers for each api */
+	#define vl_endianfun                /* define message structures */
+	#include <vpp/api/vpe_all_api_h.h>
+	#undef vl_endianfun
+	/* instantiate all the print functions we know about */
+	#define vl_print(handle, ...)
+	#define vl_printfun
+	#include <vpp/api/vpe_all_api_h.h>
+	#undef vl_printfun
+	/* Define a list of all message that the client handles */
+	#define foreach_vpe_api_reply_msg                            \
+	   _(SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply)           
+	   static clib_error_t *
+	   my_api_hookup (vlib_main_t * vm)
+	   {
+	     api_main_t *am = &api_main;
+	   #define _(N,n)                                                  \
+	       vl_msg_api_set_handlers(VL_API_##N, #n,                     \
+	                              vl_api_##n##_t_handler,              \
+	                              vl_noop_handler,                     \
+	                              vl_api_##n##_t_endian,               \
+	                              vl_api_##n##_t_print,                \
+	                              sizeof(vl_api_##n##_t), 1);
+	     foreach_vpe_api_msg;
+	   #undef _
+	     return 0;
+	    }
+
+The key API used to establish message handlers is
+`vl_msg_api_set_handlers
+<https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html#aa8a8e1f3876ec1a02f283c1862ecdb7a>`_
+, which sets values in multiple parallel vectors in the `api_main_t
+<https://docs.fd.io/vpp/18.11/dd/db2/structapi__main__t.html>`_
+structure. As of this writing: not all vector element values can be
+set through the API. You'll see sporadic API message registrations
+followed by minor adjustments of this form:
+
+.. code-block:: C
+
+	/*
+	 * Thread-safe API messages
+	 */
+	am->is_mp_safe[VL_API_IP_ADD_DEL_ROUTE] = 1;
+	am->is_mp_safe[VL_API_GET_NODE_GRAPH] = 1;
+
+API message numbering in plugins
+--------------------------------
+
+Binary API message numbering in plugins relies on vpp to issue a block 
+of message-ID's for the plugin to use:
+
+.. code-block:: C
+
+        static clib_error_t *
+        my_init (vlib_main_t * vm)
+        {
+          my_main_t *mm = &my_main;
+
+          name = format (0, "myplugin_%08x%c", api_version, 0);
+
+          /* Ask for a correctly-sized block of API message decode slots */
+          mm->msg_id_base = vl_msg_api_get_msg_ids
+            ((char *) name, VL_MSG_FIRST_AVAILABLE);
+
+          }
+
+Control-plane codes use the vl_client_get_first_plugin_msg_id (...) api
+to recover the message ID block base:
+
+.. code-block:: C
+
+          /* Ask the vpp engine for the first assigned message-id */
+          name = format (0, "myplugin_%08x%c", api_version, 0);
+          sm->msg_id_base = vl_client_get_first_plugin_msg_id ((char *) name);
+
+It's a fairly common error to forget to add msg_id_base when
+registering message handlers, or when sending messages. Using macros
+from .../src/vlibapi/api_helper_macros.h can automate the process, but
+remember to #define REPLY_MSG_ID_BASE before #including the file:
+
+.. code-block:: C
+
+          #define REPLY_MSG_ID_BASE mm->msg_id_base
+          #include <vlibapi/api_helper_macros.h>
diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst
index 165488c7a31..a5d71a9a5c3 100644
--- a/docs/gettingstarted/developers/index.rst
+++ b/docs/gettingstarted/developers/index.rst
@@ -23,4 +23,7 @@ The Developers section covers the following areas:
    vnet
    featurearcs
    bihash
+   vpp_api_module
+   binary_api_support
+   sample_plugin
 
diff --git a/docs/gettingstarted/developers/sample_plugin.rst b/docs/gettingstarted/developers/sample_plugin.rst
new file mode 100644
index 00000000000..0426606f4c9
--- /dev/null
+++ b/docs/gettingstarted/developers/sample_plugin.rst
@@ -0,0 +1,173 @@
+.. _sample_plugin:
+
+.. toctree::
+
+Integrating a complete plugin with VPP
+======================================
+
+Overview
+________
+ 
+This section shows how a VPP plugin developer can modify VPP scripts to add and load their plugin as a node in VPP. 
+
+As an example we will integrate the **Sample Plugin** found in *vpp/src/examples/sample-plugin/sample* The VPP Sample Plugin is a small plugin that demonstrates simple implementation of a macswap algorithim. Since it is a VPP plugin, it has runtime integration with the VPP graph hierachy, API, and CLI.
+
+This section will not go into the details of the plugin itself. For a deeper dive into the sample plugin see the annotations in `sample.c <https://docs.fd.io/vpp/18.11/da/d30/sample_8c.html>`_,  or go to the next page for general VPP C API usage.
+
+Setup
+_____
+
+Each plugin has their own automake file (\*.am) used by *configure.ac*, as well as a separate directory containing C files for the plugin. The directory containing these for each plugin is *vpp/src/plugins*
+
+To get a basic idea for how a VPP automake plugin file specifies its C files, here is part of the Sample Plugin automake file, *sample.am*
+
+.. code-block:: console
+    
+    sample_plugin_la_SOURCES =      \
+        sample/sample.c             \
+        sample/node.c               \
+        sample/sample_plugin.api.h
+
+    API_FILES += sample/sample.api
+
+    nobase_apiinclude_HEADERS +=            \
+      sample/sample_all_api_h.h             \
+      sample/sample_msg_enum.h              \
+      sample/sample.api.h
+
+
+The Sample Plugin is located in *vpp/src/examples/sample-plugin/sample*, so as mentioned above we will need to copy its contents into *vpp/src/plugins*
+
+In your */vpp* directory, or the directory above */src*, run:
+
+.. code-block:: console
+    
+    $ cp -r src/examples/sample-plugin/sample src/plugins
+    $ cp src/examples/sample-plugin/sample.am src/plugins
+
+Modifying configure.ac and Makefile.am
+______________________________________
+
+We now need to modify the plugin sections of the VPP automake and configuration scripts so that VPP builds correctly with your new plugin.
+
+Using a text editor such as *vi*, add the following entry to the plugins section in *vpp/src/configure.ac*
+
+.. code-block:: console
+    
+    PLUGIN_ENABLED(sample)
+
+For reference, the plugins section of that file looks like this:
+
+.. code-block:: console
+
+    ###############################################################################
+    # Plugins
+    ###############################################################################
+
+    # Please keep alphabetical order
+    PLUGIN_ENABLED(abf)
+    PLUGIN_ENABLED(acl)
+    PLUGIN_ENABLED(avf)
+    PLUGIN_ENABLED(cdp)
+    PLUGIN_ENABLED(dpdk)
+    PLUGIN_ENABLED(flowprobe)
+
+
+Using a text editor such as *vi*, now add the following entry to the plugins section in *vpp/src/plugins/Makefile.am*
+
+.. code-block:: console
+
+    if ENABLE_SAMPLE_PLUGIN
+    include sample.am
+    endif
+
+For reference, the plugins section of that file looks something like this:
+
+.. code-block:: console
+
+    vppapitestpluginsdir = ${libdir}/vpp_api_test_plugins
+    vpppluginsdir = ${libdir}/vpp_plugins
+
+    if ENABLE_ABF_PLUGIN
+    include abf.am
+    endif
+
+    if ENABLE_ACL_PLUGIN
+    include acl.am
+    endif
+
+    if ENABLE_AVF_PLUGIN
+    include avf.am
+    endif
+
+Building and Running
+____________________
+
+
+Build VPP by using the main Makefile found in */vpp/Makefile*
+
+.. code-block:: console
+    
+    $ make build
+
+.. note::
+
+    If you want to have a fresh debug build and compile every VPP file from scratch, you can wipe all compiled files and build VPP with:
+
+    .. code-block:: console
+    
+        $ make rebuild
+
+    However this will take much longer than just running *make build*
+
+Run VPP and make sure the plugin is loaded. Below is the command for running the VPP debug binary, accompanied with sample output.
+
+.. code-block:: console
+    
+    $ make run
+    vlib_plugin_early_init:361: plugin path /vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins:/vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_plugins
+    load_one_plugin:189: Loaded plugin: abf_plugin.so (ACL based Forwarding)
+    load_one_plugin:189: Loaded plugin: acl_plugin.so (Access Control Lists)
+    load_one_plugin:189: Loaded plugin: avf_plugin.so (Intel Adaptive Virtual Function (AVF) Device Plugin)
+    load_one_plugin:191: Loaded plugin: cdp_plugin.so
+    ...
+    load_one_plugin:189: Loaded plugin: sample_plugin.so (Sample of VPP Plugin)
+    ...
+    load_one_vat_plugin:67: Loaded plugin: avf_test_plugin.so
+    load_one_vat_plugin:67: Loaded plugin: mactime_test_plugin.so
+    load_one_vat_plugin:67: Loaded plugin: sample_test_plugin.so
+    ...
+        _______    _        _   _____  ___ 
+     __/ __/ _ \  (_)__    | | / / _ \/ _ \
+     _/ _// // / / / _ \   | |/ / ___/ ___/
+     /_/ /____(_)_/\___/   |___/_/  /_/    
+
+    DBGvpp# 
+
+.. note::
+
+    Notice when running the debug build that (\*_test_plugin.so) is also loaded, which is meant for testing your plugin.
+
+To enable the sample plugin, use this command:
+
+.. code-block:: console
+    
+    DBGvpp# sample macswap <interface name>
+
+To disable the sample plugin, use this command:
+
+.. code-block:: console
+    
+    DBGvpp# sample macswap <interface name> disable
+
+
+Great! Now you've successfully added your plugin as a VPP node.
+
+
+Additional remarks 
+__________________
+
+How the build process works for plugins is that the (\*.api) plugin file is automatically translated to a JSON file (\*.api.json) in *vpp/build-root/install-vpp_debug-native/vpp/share/vpp/api/plugins*, which the code generator then parses and generates a C header file (\*.api.h) in *vpp/build-root/install-vpp_debug-native/vpp/include/vpp_plugins/\**.
+
+After the build process is completed you finally end up with two plugin files (\*_plugin.so and \*_test_plugin.so) found in *vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_plugins* and *vpp/build-root/install-vpp_debug-native/vpp/lib64/vpp_api_test_plugins* respectively, that are loaded at runtime during a debug binary run of VPP (*make run*).
+
diff --git a/docs/gettingstarted/developers/vpp_api_module.rst b/docs/gettingstarted/developers/vpp_api_module.rst
new file mode 100644
index 00000000000..261c8691dc2
--- /dev/null
+++ b/docs/gettingstarted/developers/vpp_api_module.rst
@@ -0,0 +1,122 @@
+.. _vpp_api_module:
+
+.. toctree::
+
+VPP API module
+==============
+
+Overview
+________
+
+VPP API module allows communicating with VPP over shared memory interface. The API consists of 3 parts:
+
+* common code - low-level API
+* generated code - high-level API
+* code generator - to generate your own high-level API e.g. for custom plugins
+
+Common code
+___________
+
+**C**
+
+C common code represents the basic, low-level API, providing functions to connect/disconnect, perform message discovery and send/receive messages. The C variant is in vapi.h.
+
+**C++**
+
+C++ is provided by vapi.hpp and contains high-level API templates, which are specialized by generated code.
+
+Generated code
+______________
+
+Each API file present in the source tree is automatically translated to JSON file, which the code generator parses and generates either C (vapi_c_gen.py) or C++ (vapi_cpp_gen.py) code.
+
+This can then be included in the client application and provides convenient way to interact with VPP. This includes:
+
+* automatic byte-swapping
+* automatic request-response matching based on context
+* automatic casts to appropriate types (type-safety) when calling callbacks
+* automatic sending of control-pings for dump messages
+
+The API supports two modes of operation:
+
+* blocking
+* non-blocking
+
+In blocking mode, whenever an operation is initiated, the code waits until it can finish. This means that when sending a message, the call blocks until the message can be written to shared memory. Similarly, receiving a message blocks until a message becomes available. On higher level, this also means that when doing a request (e.g. show_version), the call blocks until a response comes back (e.g. show_version_reply).
+
+In non-blocking mode, these are decoupled, the API returns VAPI_EAGAIN whenever an operation cannot be performed and after sending a request, it's up to the client to wait for and process a response.
+
+Code generator
+______________
+
+Python code generator comes in two flavors - C and C++ and generates high-level API headers. All the code is stored in the headers.
+
+C Usage
+_______
+
+**Low-level API**
+
+Refer to inline API documentation in doxygen format in vapi.h header for description of functions. It's recommened to use the safer, high-level API provided by specialized headers (e.g. vpe.api.vapi.h or vpe.api.vapi.hpp).
+
+**C high-level API**
+
+*Callbacks*
+
+The C high-level API is strictly callback-based for maximum efficiency. Whenever an operation is initiated a callback with a callback context is part of that operation. The callback is then invoked when the response (or multiple responses) arrive which are tied to the request. Also, callbacks are invoked whenever an event arrives, if such callback is registered. All the pointers to responses/events point to shared memory and are immediately freed after callback finishes so the client needs to extract/copy any data in which it is interested in.
+
+**Blocking mode**
+
+In simple blocking mode, the whole operation (being a simple request or a dump) is finished and it's callback is called (potentially multiple times for dumps) during function call.
+
+Example pseudo-code for a simple request in this mode:
+
+vapi_show_version(message, callback, callback_context)
+
+#. generate unique internal context and assign it to message.header.context 
+#. byteswap the message to network byte order 
+#. send message to vpp (message is now consumed and vpp will free it) 
+#. create internal "outstanding request context" which stores the callback, callback context and the internal context value 
+#. call dispatch, which in this mode receives and processes responses until the internal "outstanding requests" queue is empty. In blocking mode, this queue always contains at most one item. 
+
+.. note::
+
+	It's possible for different - unrelated callbacks to be called before the response callbacks is called in cases where e.g. events are stored in shared memory queue.
+
+**Non-blocking mode**
+In non-blocking mode, all the requests are only byte-swapped and the context information along with callbacks is stored locally (so in the above example, only steps 1-4 are executed and step 5 is skipped). Calling dispatch is up to the client application. This allows to alternate between sending/receiving messages or have a dedicated thread which calls dispatch.
+
+C++ high level API
+__________________
+
+**Callbacks**
+
+In C++ API, the response is automatically tied to the corresponding Request, Dump or Event_registration object. Optionally a callback might be specified, which then gets called when the response is received.
+
+.. note::
+
+	Responses take up shared memory space and should be freed either manually (in case of result sets) or automatically (by destroying the object owning them) when no longer needed. Once a Request or Dump object was executed, it cannot be re-sent, since the request itself (stores in shared memory) is consumed by vpp and inaccessible (set to nullptr) anymore.
+
+C++ Usage
+_________
+
+**Requests & dumps**
+
+*Create an object of Connection type and call connect() to connect to vpp.*
+
+#. Create an object of Request or Dump type using it's typedef (e.g. Show_version)
+#. Use get_request() to obtain and manipulate the underlying request if required.
+#. Issue execute() to send the request.
+#. Use either wait_for_response() or dispatch() to wait for the response.
+#. Use get_response_state() to get the state and get_response() to read the response.
+
+**Events**
+
+*Create a Connection and execute the appropriate Request to subscribe to events (e.g. Want_stats)*
+
+#. Create an Event_registration with a template argument being the type of event you are insterested in.
+#. Call dispatch() or wait_for_response() to wait for the event. A callback will be called when an event occurs (if passed to Event_registration() constructor). Alternatively, read the result set.
+
+.. note::
+
+	Events stored in the result set take up space in shared memory and should be freed regularly (e.g. in the callback, once the event is processed).
+