docs: convert extras doc md->rst

Type: improvement

Change-Id: Ie3b25a86b99098d2b3a21a11fc73234c8ed589d6
Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com>

24 files changed, 1922 insertions, 1659 deletions

diff --git a/extras/lcov/README.md b/extras/lcov/README.md
deleted file mode 100644
index ae57fe53477..00000000000
--- a/extras/lcov/README.md
+++ /dev/null
@@ -1,48 +0,0 @@
-# Code coverage analysis with lcov  {#lcov_code_coverage}
-
-## Prerequisites
-
-The Linux gcov and lcov tools are fussy about gcc / g++ compiler
-versions. As of this writing, Ubuntu 18.04 gcov / lcov work with
-these toolchain versions:
-
-  $ gcc --version
-  gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
-  $ g++ --version
-  g++ (Ubuntu 8.3.0-6ubuntu1~18.04.1) 8.3.0
-
-Refer to
-https://askubuntu.com/questions/26498/how-to-choose-the-default-gcc-and-g-version for information on how to install multiple gcc / g++ versions, and
-switch between them.
-
-You'll need to install the following additional packages:
-
-  $ sudo apt-get install gcovr ggcov lcov
-
-## Compile an instrumented vpp image
-
-Two ways:
-
-  $ cd <workspace-root>
-  $ make test-gcov
-  $ ## interrupt compilation after building the image
-
-or
-  $ cd <workspace-root>/build-root
-  $ make PLATFORM=vpp TAG=vpp_gcov vpp-install
-
-## Initialize the lcov database
-
-  $ cd <workspace-root>
-  $ ./extras/lcov/lcov_prep
-  $ make test-gcov or make TEST=my_test test-gcov
-  $ # repeat or vary as desired to increase reported coverage
-  $ # Generate the report:
-  $ ./extras/lcov/lcov_post
-
-You can run vpp manually, do anything you like. Results are cumulative
-until you re-run the "prep" script.
-
-## Look at the results
-
-Point a browser at file:///<workspace-root>/build-root/html/index.html
diff --git a/extras/lcov/README.rst b/extras/lcov/README.rst
new file mode 100644
index 00000000000..3dab9510215
--- /dev/null
+++ b/extras/lcov/README.rst
@@ -0,0 +1,48 @@
+.. _lcov_code_coverage:
+
+Code coverage with lcov
+=======================
+
+Prerequisites
+-------------
+
+The Linux gcov and lcov tools are fussy about gcc / g++ compiler
+versions. As of this writing, Ubuntu 18.04 gcov / lcov work with these
+toolchain versions:
+
+$ gcc –version gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 $ g++ –version
+g++ (Ubuntu 8.3.0-6ubuntu1~18.04.1) 8.3.0
+
+Refer to
+https://askubuntu.com/questions/26498/how-to-choose-the-default-gcc-and-g-version
+for information on how to install multiple gcc / g++ versions, and
+switch between them.
+
+You’ll need to install the following additional packages:
+
+$ sudo apt-get install gcovr ggcov lcov
+
+Compile an instrumented vpp image
+---------------------------------
+
+Two ways:
+
+$ cd $ make test-gcov $ ## interrupt compilation after building the
+image
+
+or $ cd /build-root $ make PLATFORM=vpp TAG=vpp_gcov vpp-install
+
+Initialize the lcov database
+----------------------------
+
+$ cd $ ./extras/lcov/lcov_prep $ make test-gcov or make TEST=my_test
+test-gcov $ # repeat or vary as desired to increase reported coverage $
+# Generate the report: $ ./extras/lcov/lcov_post
+
+You can run vpp manually, do anything you like. Results are cumulative
+until you re-run the “prep” script.
+
+Look at the results
+-------------------
+
+Point a browser at file:////build-root/html/index.html
diff --git a/extras/scripts/check_documentation.sh b/extras/scripts/check_documentation.sh
new file mode 100755
index 00000000000..2ab093dfbcb
--- /dev/null
+++ b/extras/scripts/check_documentation.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Copyright (c) 2021 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eEo pipefail
+
+SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+WS_ROOT=$( realpath ${SCRIPTDIR}/../.. )
+
+function red   () { printf "\e[0;31m$1\e[0m\n" ; }
+function green () { printf "\e[0;32m$1\e[0m\n" ; }
+
+find_linked_docs () {
+  find ${WS_ROOT}/docs -type l \
+      \( -name '*.rst' -o -name '*.md' \) \
+      -exec readlink -f {} \; | sort
+}
+
+find_excluded_docs () {
+    cat ${WS_ROOT}/docs/docsignore \
+      | grep -v '#' \
+      | sed s@^@${WS_ROOT}/@ \
+      | sort
+}
+
+find_linked_and_excluded_docs () {
+  cat <( find_linked_docs ) <( find_excluded_docs  ) | sort
+}
+
+find_candidate_docs () {
+  find \
+    ${WS_ROOT}/src \
+    ${WS_ROOT}/test \
+    ${WS_ROOT}/extras \
+    -not -path "${WS_ROOT}/test/venv/*" \
+    \( -name '*.rst' -o -name '*.md' \) \
+    | sort
+}
+
+spellcheck () {
+    make -C ${WS_ROOT} docs-spell
+}
+
+if [ "x$(comm -13 <( find_linked_and_excluded_docs ) <( find_candidate_docs ))" != x ]; then
+    red "The following files need to be linked"
+    red "in the doc folder e.g. :"
+    red "$ cd vpp/docs/developer/plugins"
+    red "$ ln -s ../../../src/plugins/my_plugin/my_plugin.rst"
+    echo ""
+    cat <( comm -13 <( find_linked_and_excluded_docs ) <( find_candidate_docs ) )
+    exit 1
+fi
+spellcheck
+green "**********************************************"
+green "* VPP Docs Checkstyle Successfully Completed *"
+green "**********************************************"
diff --git a/extras/selinux/selinux_doc.md b/extras/selinux/selinux_doc.md
deleted file mode 100644
index 2320dd538d2..00000000000
--- a/extras/selinux/selinux_doc.md
+++ /dev/null
@@ -1,506 +0,0 @@
-# SELinux - VPP Custom SELinux Policy    {#selinux_doc}
-
-## Overview
-
-Security-enhanced Linux (SELinux) is a security feature in the Linux kernel. At
-a very high level, SELinux implements mandatory access controls (MAC), as
-opposed to discretionary access control (DAC) implemented in standard Linux. MAC
-defines how processes can interact with other system components (Files,
-Directories, Other Processes, Pipes, Sockets, Network Ports). Each system
-component is assigned a label, and then the SELinux Policy defines which labels
-and which actions on each label a process is able to perform. The VPP Custom
-SELinux Policy defines the actions VPP is allowed to perform on which labels.
-
-The VPP Custom SELinux Policy is intended to be installed on RPM based platforms
-(tested on CentOS 7 and RHEL 7). Though SELinux can run on Debian platforms, it
-typically is not and therefore is not currently being built for Debian.
-
-The VPP Custom SELinux Policy does not enable or disable SELinux, only allows
-VPP to run when SELinux is enabled. A fresh install of either Fedora, CentOS or
-RHEL will have SELinux enabled by default. To determine if SELinux is enabled on
-a given system and enable it if needed, run:
-
-```
-   $ getenforce
-   Permissive
-
-   $ sudo setenforce 1
-
-   $ getenforce
-   Enforcing
-```
-
-To make the change persistent, modify the following file to set
-`SELINUX=enforcing`:
-
-```
-   $ sudo vi /etc/selinux/config
-   :
-   # This file controls the state of SELinux on the system.
-   # SELINUX= can take one of these three values:
-   #     enforcing - SELinux security policy is enforced.
-   #     permissive - SELinux prints warnings instead of enforcing.
-   #     disabled - No SELinux policy is loaded.
-   SELINUX=enforcing
-   :
-```
-
-## Installation
-
-To install VPP, see the installation instructions on the VPP Wiki
-(https://wiki.fd.io/view/VPP/Installing_VPP_binaries_from_packages). The VPP
-Custom SELinux Policy is packaged in its own RPM starting in 18.04,
-`vpp-selinux-policy-<VERSION>-<RELEASE>.rpm`. It is packaged and installed along
-with the other VPP RPMs.
-
-### Fresh Install of VPP
-
-If VPP has never been installed on a system, then starting in 18.04, the VPP
-Custom SELinux Policy will be installed with the other RPMs and all the system
-components managed by VPP will be labeled properly.
-
-### Fix SELinux Labels for VPP
-In the case where the VPP Custom Policy is being installed for the first time,
-either because VPP has been upgraded or packages were removed and then
-reinstalled, several directories and files will not not be properly labeled. The
-labels on these files will need to be fixed for VPP to run properly with SELinux
-enabled. After the VPP Custom SELinux Policy is installed, run the following
-commands to fix the labels. If VPP is already running, make sure to restart
-VPP after the labels are fixed. This change is persistent for the life of the
-file. Once the VPP Custom Policy is installed on the system, subsequent files
-created by VPP will be labeled properly. This is only to fix files created by
-VPP prior to the VPP Custom Policy being installed.
-
-```
-  $ sudo restorecon -Rv /etc/vpp/
-  $ sudo restorecon -Rv /usr/lib/vpp_api_test_plugins/
-  $ sudo restorecon -Rv /usr/lib/vpp_plugins/
-  $ sudo restorecon -Rv /usr/share/vpp/
-  $ sudo restorecon -Rv /var/run/vpp/
-
-  $ sudo chcon -t vpp_tmp_t /tmp/vpp_*
-  $ sudo chcon -t vpp_var_run_t /var/run/.vpp_*
-```
-
-**NOTE:** Because the VPP APIs allow custom filenames in certain scenarios, the
-above commands may not handle all files. Inspect your system and correct any
-files that are mislabeled. For example, to verify all VPP files in `/tmp/` are
-labeled properly, run:
-
-```
-  $ sudo ls -alZ /tmp/
-```
-
-Any files not properly labeled with `vpp_tmp_t`, run:
-
-```
-  $ sudo chcon -t vpp_tmp_t /tmp/<filename>
-```
-
-## VPP Files
-
-### Recommended Default File Directories
-
-Documentation in the VPP Wiki (https://wiki.fd.io/view/VPP/) and doxygen
-generated documentation have examples with files located in certain directories.
-Some of the recommend file locations have been moved to satisfy SELinux. Most of
-the documentation has been updated, but links to older documentation still exist
-and there may have been instances that were missed. Use the file locations
-described below to allow SELinux to properly label the given files.
-
-File locations that have changed:
-* VPP Debug CLI Script Files
-* vHost Sockets
-* VPP Log Files
-
-#### VPP Debug CLI Script Files
-
-The VPP Debug CLI, `vppctl`, allows a sequence of CLI commands to be read from a
-file and executed. To avoid from having to grant VPP access to all of `/tmp/` and
-possibly `/home/` sub-directories, it is recommended that any VPP Debug CLI script
-files be placed in a common directory such as `/usr/share/vpp/`.
-
-For example:
-```
-$ cat /usr/share/vpp/scripts/gigup.txt
-set interface state GigabitEthernet0/8/0 up
-set interface state GigabitEthernet0/9/0 up
-```
-
-To execute:
-```
-$ vppctl exec /usr/share/vpp/scripts/gigup.txt
-```
-Or
-```
-$ vppctl
-    _______    _        _   _____  ___
- __/ __/ _ \  (_)__    | | / / _ \/ _ \
- _/ _// // / / / _ \   | |/ / ___/ ___/
- /_/ /____(_)_/\___/   |___/_/  /_/
-
-vpp# exec /usr/share/vpp/scripts/gigup.txt
-vpp# quit
-
-```
-
-If the file is not labeled properly, you will see something similar to:
-```
-$ vppctl exec /home/<user>/dev/vpp/scripts/vppctl/gigup.txt
-exec: failed to open `/home/<user>/dev/vpp/scripts/vppctl/gigup.txt': Permission denied
-
-$ ls -alZ
-drwxrwxr-x. <user> <user> unconfined_u:object_r:user_home_t:s0 .
-drwxrwxr-x. <user> <user> unconfined_u:object_r:user_home_t:s0 ..
--rw-r--r--. <user> <user> unconfined_u:object_r:user_home_t:s0 gigup.txt
-```
-
-##### Original Documentation
-
-Some of the original documentation showed script files being executed out of
-`/tmp/`. Convenience also may lead to script files being placed in
-`/home/<user>/` subdirectories. If a file is generated by the VPP process in
-`/tmp/`, for example a trace file or pcap file, it will get properly labeled
-with the SELinux label `vpp_tmp_t`. When a file is created, unless a rule is in
-place for the process that created it, the file will inherit the SELinux label
-of the parent directory. So if a user creates a file themselves in `/tmp/`, it
-will get the SELinux label `tmp_t`, which VPP does not have permission to
-access. Therefore it is recommended that script files are located as described
-above.
-
-#### vHost Sockets
-
-vHost sockets are created from VPP perspective in either Server or Client mode.
-In Server mode, the socket name is provided to VPP and VPP creates the socket.
-In Client mode, the socket name is provided to VPP and the hypervisor creates
-the socket. In order for VPP and hypervisor to share the socket resource with
-SELinux enabled, a rule in the VPP Custom SELinux Policy has been added. This
-rules allows processes with the `svirt_t` label (the hypervisor) to access
-sockets with the `vpp_var_run_t` label. As such, when SELinux is enabled,
-vHost sockets should be created in the directory `/var/run/vpp/`.
-
-##### Original Documentation
-
-Some of the original documentation showed vHost sockets being created in the
-directory `/tmp/`. To work properly with SELinux enabled, vHost sockets should be
-created as described above.
-
-#### VPP Log Files
-
-The VPP log file location is set by updating the `/etc/vpp/startup.conf` file:
-
-```
-vi /etc/vpp/startup.conf
-unix {
-:
-  log /var/log/vpp/vpp.log
-:
-}
-
-```
-
-By moving the log file to `/var/log/vpp/`, it will get the label `vpp_log_t`,
-which indicates that the files are log files so they benefit from the
-associated rules (for example granting rights to logrotate so that it can
-manipulate them).
-
-##### Original Documentation
-
-The default `startup.conf` file creates the VPP log file in `/tmp/vpp.log`. By
-leaving the log file in `/tmp/`, it will get the label `vpp_tmp_t`. Moving it
-to `/var/log/vpp/`, it will get the label `vpp_log_t`.
-
-### Use of Non-default File Directories
-
-VPP installs multiple files on the system.
-Some files have fixed directory and file names:
-- /etc/bash_completion.d/vppctl_completion
-- /etc/sysctl.d/80-vpp.conf
-- /usr/lib/systemd/system/vpp.service
-
-Others files have default directory and file names but the default can be
-overwritten:
-- /etc/vpp/startup.conf
-  - Can be changed via the `/usr/lib/systemd/system/vpp.service` file by
-    changing the -c option on the VPP command line:
-
-```
-ExecStart=/usr/bin/vpp -c /etc/vpp/startup.conf
-```
-
-- /run/vpp/cli.sock
-  - Can be changed via the `/etc/vpp/startup.conf` file by changing the
-    cli-listen setting:
-
-```
-unix {
-:
-  cli-listen /run/vpp/cli.sock
-:
-}
-```
-
-
-- /var/log/vpp/vpp.log
-  - Can be changed via the `/etc/vpp/startup.conf` file by changing the log
-    setting:
-
-```
-unix {
-  :
-  log /var/log/vpp/vpp.log
-  :
-}
-
-```
-
-If the directory of any VPP installed files is changed from the default, ensure
-that the proper SELiunx label is applied. The SELinux label can be determined by
-passing the -Z option to many common Linux commands:
-
-```
-ls -alZ /run/vpp/
-drwxr-xr-x. root vpp  system_u:object_r:vpp_var_run_t:s0 .
-drwxr-xr-x. root root system_u:object_r:var_run_t:s0     ..
-srwxrwxr-x. root vpp  system_u:object_r:vpp_var_run_t:s0 cli.sock
-```
-
-### VPP SELinux Types ###
-
-The following SELinux types are created by the VPP Custom SELinux Policy:
-- `vpp_t` - Applied to:
-  - VPP process and spawned threads.
-
-- `vpp_config_rw_t` - Applied to:
-  - `/etc/vpp/*`
-
-- `vpp_tmp_t` - Applied to:
-  - `/tmp/*`
-
-- `vpp_exec_t` - Applied to:
-  - `/usr/bin/*`
-
-- `vpp_lib_t` - Applied to:
-  - `/usr/lib/vpp_api_test_plugins/*`
-  - `/usr/lib/vpp_plugins/*`
-
-- `vpp_unit_file_t` - Applied to:
-  - `/usr/lib/systemd/system/vpp.*`
-
-- `vpp_log_t` - Applied to:
-  - `/var/log/vpp/*`
-
-- `vpp_var_run_t` - Applied to:
-  - `/var/run/vpp/*`
-
-## Debug SELinux Issues
-
-If SELinux issues are suspected, there are a few steps that can be taken to
-debug the issue. This section provides a few pointers on on those steps. Any
-SELinux JIRAs will need this information to properly address the issue.
-
-### Additional SELinux Packages and Setup
-
-First, install the SELinux troubleshooting packages:
-
-```
-$ sudo yum -y install setroubleshoot setroubleshoot-server setools-console
--- OR --
-$ sudo dnf -y install setroubleshoot setroubleshoot-server setools-console
-```
-
-To enable proper logging, restart auditd:
-
-```
-$ sudo service auditd restart
-```
-
-While debugging issues, it is best to set SELinux to `Permissive` mode. In
-`Permissive` mode, SELinux will still detect and flag errors, but will allow
-processes to continue normal operation. This allows multiple errors to be
-collected at once as opposed to breaking on each individual error. To set
-SELinux to `Permissive` mode (until next reboot or it is set back), use:
-
-```
-$ sudo setenforce 0
-
-$ getenforce
-Permissive
-```
-
-After debugging, to set SELinux back to `Enforcing` mode, use:
-
-```
-$ sudo setenforce 1
-
-$ getenforce
-Enforcing
-```
-
-### Debugging
-
-Once the SELinux troubleshooting packages are installed, perform the actions
-that are suspected to be blocked by SELinux. Either `tail` the log during
-these actions or `grep` the log for additional SELinux logs:
-
-```
-sudo tail -f /var/log/messages
--- OR --
-sudo journalctl -f
-```
-
-Below are some examples of SELinux logs that are generated:
-
-```
-May 14 11:28:34 svr-22 setroubleshoot: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt. For complete SELinux messages run: sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
-May 14 11:28:34 svr-22 python: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp should be allowed read access on the hostCreate.txt file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
-May 14 11:28:34 svr-22 setroubleshoot: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt. For complete SELinux messages run: sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
-May 14 11:28:34 svr-22 python: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp should be allowed read access on the hostCreate.txt file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
-May 14 11:28:37 svr-22 setroubleshoot: SELinux is preventing vpp_main from map access on the packet_socket packet_socket. For complete SELinux messages run: sealert -l ab6667d9-3f14-4dbd-96a0-7a655f7b4eb1
-May 14 11:28:37 svr-22 python: SELinux is preventing vpp_main from map access on the packet_socket packet_socket.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp_main should be allowed map access on the packet_socket packet_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
-May 14 11:28:51 svr-22 setroubleshoot: SELinux is preventing vpp_main from map access on the packet_socket packet_socket. For complete SELinux messages run: sealert -l ab6667d9-3f14-4dbd-96a0-7a655f7b4eb1
-May 14 11:28:51 svr-22 python: SELinux is preventing vpp_main from map access on the packet_socket packet_socket.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp_main should be allowed map access on the packet_socket packet_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
-```
-
-From the logs above, there are two sets of commands that are recommended to be
-run. The first is to run the `sealert` command. The second is to run the
-`ausearch | audit2allow` commands and the `semodule` command.
-
-#### sealert Command
-
-This `sealert` command provides a more detailed output for the given issue
-detected.
-
-```
-$ sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
-SELinux is preventing /usr/bin/vpp from 'read, write' accesses on the chr_file noiommu-0.
-
-*****  Plugin device (91.4 confidence) suggests   ****************************
-
-If you want to allow vpp to have read write access on the noiommu-0 chr_file
-Then you need to change the label on noiommu-0 to a type of a similar device.
-Do
-# semanage fcontext -a -t SIMILAR_TYPE 'noiommu-0'
-# restorecon -v 'noiommu-0'
-
-*****  Plugin catchall (9.59 confidence) suggests   **************************
-
-If you believe that vpp should be allowed read write access on the noiommu-0 chr_file by default.
-Then you should report this as a bug.
-You can generate a local policy module to allow this access.
-Do
-allow this access for now by executing:
-# ausearch -c 'vpp' --raw | audit2allow -M my-vpp
-# semodule -i my-vpp.pp
-
-
-Additional Information:
-Source Context                system_u:system_r:vpp_t:s0
-Target Context                system_u:object_r:device_t:s0
-Target Objects                noiommu-0 [ chr_file ]
-Source                        vpp
-Source Path                   /usr/bin/vpp
-Port                          <Unknown>
-Host                          vpp_centos7_selinux
-Source RPM Packages           vpp-19.01.2-rc0~17_gcfd3086.x86_64
-Target RPM Packages
-Policy RPM                    selinux-policy-3.13.1-229.el7_6.12.noarch
-Selinux Enabled               True
-Policy Type                   targeted
-Enforcing Mode                Permissive
-Host Name                     vpp_centos7_selinux
-Platform                      Linux vpp_centos7_selinux
-                              3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29
-                              14:59:59 UTC 2019 x86_64 x86_64
-Alert Count                   1
-First Seen                    2019-05-13 18:10:50 EDT
-Last Seen                     2019-05-13 18:10:50 EDT
-Local ID                      a418f869-f470-4c8a-b8e9-bdd41f2dd60b
-
-Raw Audit Messages
-type=AVC msg=audit(1557785450.964:257): avc:  denied  { read write } for  pid=5273 comm="vpp" name="noiommu-0" dev="devtmpfs" ino=36022 scontext=system_u:system_r:vpp_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1
-
-
-type=AVC msg=audit(1557785450.964:257): avc:  denied  { open } for  pid=5273 comm="vpp" path="/dev/vfio/noiommu-0" dev="devtmpfs" ino=36022 scontext=system_u:system_r:vpp_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1
-
-
-type=SYSCALL msg=audit(1557785450.964:257): arch=x86_64 syscall=open success=yes exit=ENOTBLK a0=7fb395ffd7f0 a1=2 a2=7fb395ffd803 a3=7fb395ffe2a0 items=0 ppid=1 pid=5273 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=993 sgid=0 fsgid=993 tty=(none) ses=4294967295 comm=vpp exe=/usr/bin/vpp subj=system_u:system_r:vpp_t:s0 key=(null)
-
-Hash: vpp,vpp_t,device_t,chr_file,read,write
-```
-
-In general, this command pumps out too much info and is only needed for
-additional debugging for tougher issues. Also note that once the process being
-tested is restarted, this command loses it's context and will not provide any
-information:
-
-```
-$ sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
-Error
-query_alerts error (1003): id (a418f869-f470-4c8a-b8e9-bdd41f2dd60b) not found
-```
-
-#### ausearch | audit2allow and semodule Commands
-
-These set of commands are more useful for basic debugging. The
-`ausearch | audit2allow` commands generate a set files. It may be worthwhile to
-run the commands in a temporary subdirectory:
-
-```
-$ mkdir test-01/; cd test-01/
-
-$ sudo ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain
-
-$ ls
-my-vpp.pp  my-vpp.te
-
-$ cat my-vpp.te
-module my-vpp 1.0;
-
-require {
-        type user_home_t;
-        type vpp_t;
-        class packet_socket map;
-        class file { open read };
-}
-
-#============= vpp_t ==============
-allow vpp_t self:packet_socket map;
-allow vpp_t user_home_t:file { open read };
-```
-
-As shown above, the file `my-vpp.te` has been generated. This file shows
-possible changes to the SELinux policy that may fix the issue. If an SELinux
-policy was being created from scratch, this policy could be applied using the
-`semodule -i my-vpp.pp` command. HOWEVER, VPP already has a policy in place. So
-these changes need to be incorporated into the existing policy. The VPP SELinux
-policy is located in the following files:
-
-```
-$ ls extras/selinux/
-selinux_doc.md  vpp-custom.fc  vpp-custom.if  vpp-custom.te
-```
-
-In this example, `map` needs to be added to the `packet_socket` class. If the
-`vpp-custom.te` is examined (prior to this fix), then one would see that the
-`packet_socket` class is already defined and just needs to be updated:
-
-```
-$ vi extras/selinux/vpp-custom.te
-:
-allow vpp_t self:process { execmem execstack setsched signal }; # too benevolent
-allow vpp_t self:packet_socket { bind create setopt ioctl };  <---
-allow vpp_t self:tun_socket { create relabelto relabelfrom };
-:
-```
-
-Before blindly applying the changes proposed by the `ausearch | audit2allow`
-commands, try to determine what is being allowed by the policy and determine if
-this is desired, or if the code can be reworked to no longer require the
-suggested permission. In the `my-vpp.te` file from above, it is suggested to
-allow `vpp_t` (i.e. the VPP process) access to all files in the home directory
-(`allow vpp_t user_home_t:file { open read };`). This was because a
-`vppctl exec` command was executed calling a script located in the
-`/home/<user>/` directory. Once this script was run from the `/usr/share/vpp/`
-directory as described in a section above, these permissions were no longer
-needed.
diff --git a/extras/selinux/selinux_doc.rst b/extras/selinux/selinux_doc.rst
new file mode 100644
index 00000000000..a902ec675ce
--- /dev/null
+++ b/extras/selinux/selinux_doc.rst
@@ -0,0 +1,554 @@
+.. _selinux_doc:
+
+SELinux - VPP Custom SELinux Policy
+===================================
+
+Overview
+--------
+
+Security-enhanced Linux (SELinux) is a security feature in the Linux
+kernel. At a very high level, SELinux implements mandatory access
+controls (MAC), as opposed to discretionary access control (DAC)
+implemented in standard Linux. MAC defines how processes can interact
+with other system components (Files, Directories, Other Processes,
+Pipes, Sockets, Network Ports). Each system component is assigned a
+label, and then the SELinux Policy defines which labels and which
+actions on each label a process is able to perform. The VPP Custom
+SELinux Policy defines the actions VPP is allowed to perform on which
+labels.
+
+The VPP Custom SELinux Policy is intended to be installed on RPM based
+platforms (tested on CentOS 7 and RHEL 7). Though SELinux can run on
+Debian platforms, it typically is not and therefore is not currently
+being built for Debian.
+
+The VPP Custom SELinux Policy does not enable or disable SELinux, only
+allows VPP to run when SELinux is enabled. A fresh install of either
+Fedora, CentOS or RHEL will have SELinux enabled by default. To
+determine if SELinux is enabled on a given system and enable it if
+needed, run:
+
+::
+
+      $ getenforce
+      Permissive
+
+      $ sudo setenforce 1
+
+      $ getenforce
+      Enforcing
+
+To make the change persistent, modify the following file to set
+``SELINUX=enforcing``:
+
+::
+
+      $ sudo vi /etc/selinux/config
+      :
+      # This file controls the state of SELinux on the system.
+      # SELINUX= can take one of these three values:
+      #     enforcing - SELinux security policy is enforced.
+      #     permissive - SELinux prints warnings instead of enforcing.
+      #     disabled - No SELinux policy is loaded.
+      SELINUX=enforcing
+      :
+
+Installation
+------------
+
+To install VPP, see the installation instructions on the VPP Wiki
+(https://wiki.fd.io/view/VPP/Installing_VPP_binaries_from_packages). The
+VPP Custom SELinux Policy is packaged in its own RPM starting in 18.04,
+``vpp-selinux-policy-<VERSION>-<RELEASE>.rpm``. It is packaged and
+installed along with the other VPP RPMs.
+
+Fresh Install of VPP
+~~~~~~~~~~~~~~~~~~~~
+
+If VPP has never been installed on a system, then starting in 18.04, the
+VPP Custom SELinux Policy will be installed with the other RPMs and all
+the system components managed by VPP will be labeled properly.
+
+Fix SELinux Labels for VPP
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In the case where the VPP Custom Policy is being installed for the first
+time, either because VPP has been upgraded or packages were removed and
+then reinstalled, several directories and files will not not be properly
+labeled. The labels on these files will need to be fixed for VPP to run
+properly with SELinux enabled. After the VPP Custom SELinux Policy is
+installed, run the following commands to fix the labels. If VPP is
+already running, make sure to restart VPP after the labels are fixed.
+This change is persistent for the life of the file. Once the VPP Custom
+Policy is installed on the system, subsequent files created by VPP will
+be labeled properly. This is only to fix files created by VPP prior to
+the VPP Custom Policy being installed.
+
+::
+
+     $ sudo restorecon -Rv /etc/vpp/
+     $ sudo restorecon -Rv /usr/lib/vpp_api_test_plugins/
+     $ sudo restorecon -Rv /usr/lib/vpp_plugins/
+     $ sudo restorecon -Rv /usr/share/vpp/
+     $ sudo restorecon -Rv /var/run/vpp/
+
+     $ sudo chcon -t vpp_tmp_t /tmp/vpp_*
+     $ sudo chcon -t vpp_var_run_t /var/run/.vpp_*
+
+**NOTE:** Because the VPP APIs allow custom filenames in certain
+scenarios, the above commands may not handle all files. Inspect your
+system and correct any files that are mislabeled. For example, to verify
+all VPP files in ``/tmp/`` are labeled properly, run:
+
+::
+
+     $ sudo ls -alZ /tmp/
+
+Any files not properly labeled with ``vpp_tmp_t``, run:
+
+::
+
+     $ sudo chcon -t vpp_tmp_t /tmp/<filename>
+
+VPP Files
+---------
+
+Recommended Default File Directories
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Documentation in the VPP Wiki (https://wiki.fd.io/view/VPP/) and doxygen
+generated documentation have examples with files located in certain
+directories. Some of the recommend file locations have been moved to
+satisfy SELinux. Most of the documentation has been updated, but links
+to older documentation still exist and there may have been instances
+that were missed. Use the file locations described below to allow
+SELinux to properly label the given files.
+
+File locations that have changed: \* VPP Debug CLI Script Files \* vHost
+Sockets \* VPP Log Files
+
+VPP Debug CLI Script Files
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The VPP Debug CLI, ``vppctl``, allows a sequence of CLI commands to be
+read from a file and executed. To avoid from having to grant VPP access
+to all of ``/tmp/`` and possibly ``/home/`` sub-directories, it is
+recommended that any VPP Debug CLI script files be placed in a common
+directory such as ``/usr/share/vpp/``.
+
+For example:
+
+::
+
+   $ cat /usr/share/vpp/scripts/gigup.txt
+   set interface state GigabitEthernet0/8/0 up
+   set interface state GigabitEthernet0/9/0 up
+
+To execute:
+
+::
+
+   $ vppctl exec /usr/share/vpp/scripts/gigup.txt
+
+Or
+
+::
+
+   $ vppctl
+       _______    _        _   _____  ___
+    __/ __/ _ \  (_)__    | | / / _ \/ _ \
+    _/ _// // / / / _ \   | |/ / ___/ ___/
+    /_/ /____(_)_/\___/   |___/_/  /_/
+
+   vpp# exec /usr/share/vpp/scripts/gigup.txt
+   vpp# quit
+
+If the file is not labeled properly, you will see something similar to:
+
+::
+
+   $ vppctl exec /home/<user>/dev/vpp/scripts/vppctl/gigup.txt
+   exec: failed to open `/home/<user>/dev/vpp/scripts/vppctl/gigup.txt': Permission denied
+
+   $ ls -alZ
+   drwxrwxr-x. <user> <user> unconfined_u:object_r:user_home_t:s0 .
+   drwxrwxr-x. <user> <user> unconfined_u:object_r:user_home_t:s0 ..
+   -rw-r--r--. <user> <user> unconfined_u:object_r:user_home_t:s0 gigup.txt
+
+Original Documentation
+''''''''''''''''''''''
+
+Some of the original documentation showed script files being executed
+out of ``/tmp/``. Convenience also may lead to script files being placed
+in ``/home/<user>/`` subdirectories. If a file is generated by the VPP
+process in ``/tmp/``, for example a trace file or pcap file, it will get
+properly labeled with the SELinux label ``vpp_tmp_t``. When a file is
+created, unless a rule is in place for the process that created it, the
+file will inherit the SELinux label of the parent directory. So if a
+user creates a file themselves in ``/tmp/``, it will get the SELinux
+label ``tmp_t``, which VPP does not have permission to access. Therefore
+it is recommended that script files are located as described above.
+
+vHost Sockets
+^^^^^^^^^^^^^
+
+vHost sockets are created from VPP perspective in either Server or
+Client mode. In Server mode, the socket name is provided to VPP and VPP
+creates the socket. In Client mode, the socket name is provided to VPP
+and the hypervisor creates the socket. In order for VPP and hypervisor
+to share the socket resource with SELinux enabled, a rule in the VPP
+Custom SELinux Policy has been added. This rules allows processes with
+the ``svirt_t`` label (the hypervisor) to access sockets with the
+``vpp_var_run_t`` label. As such, when SELinux is enabled, vHost sockets
+should be created in the directory ``/var/run/vpp/``.
+
+.. _original-documentation-1:
+
+Original Documentation
+''''''''''''''''''''''
+
+Some of the original documentation showed vHost sockets being created in
+the directory ``/tmp/``. To work properly with SELinux enabled, vHost
+sockets should be created as described above.
+
+VPP Log Files
+^^^^^^^^^^^^^
+
+The VPP log file location is set by updating the
+``/etc/vpp/startup.conf`` file:
+
+::
+
+   vi /etc/vpp/startup.conf
+   unix {
+   :
+     log /var/log/vpp/vpp.log
+   :
+   }
+
+By moving the log file to ``/var/log/vpp/``, it will get the label
+``vpp_log_t``, which indicates that the files are log files so they
+benefit from the associated rules (for example granting rights to
+logrotate so that it can manipulate them).
+
+.. _original-documentation-2:
+
+Original Documentation
+''''''''''''''''''''''
+
+The default ``startup.conf`` file creates the VPP log file in
+``/tmp/vpp.log``. By leaving the log file in ``/tmp/``, it will get the
+label ``vpp_tmp_t``. Moving it to ``/var/log/vpp/``, it will get the
+label ``vpp_log_t``.
+
+Use of Non-default File Directories
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+VPP installs multiple files on the system. Some files have fixed
+directory and file names: - /etc/bash_completion.d/vppctl_completion -
+/etc/sysctl.d/80-vpp.conf - /usr/lib/systemd/system/vpp.service
+
+Others files have default directory and file names but the default can
+be overwritten: - /etc/vpp/startup.conf - Can be changed via the
+``/usr/lib/systemd/system/vpp.service`` file by changing the -c option
+on the VPP command line:
+
+::
+
+   ExecStart=/usr/bin/vpp -c /etc/vpp/startup.conf
+
+-  /run/vpp/cli.sock
+
+   -  Can be changed via the ``/etc/vpp/startup.conf`` file by changing
+      the cli-listen setting:
+
+::
+
+   unix {
+   :
+     cli-listen /run/vpp/cli.sock
+   :
+   }
+
+-  /var/log/vpp/vpp.log
+
+   -  Can be changed via the ``/etc/vpp/startup.conf`` file by changing
+      the log setting:
+
+::
+
+   unix {
+     :
+     log /var/log/vpp/vpp.log
+     :
+   }
+
+If the directory of any VPP installed files is changed from the default,
+ensure that the proper SELiunx label is applied. The SELinux label can
+be determined by passing the -Z option to many common Linux commands:
+
+::
+
+   ls -alZ /run/vpp/
+   drwxr-xr-x. root vpp  system_u:object_r:vpp_var_run_t:s0 .
+   drwxr-xr-x. root root system_u:object_r:var_run_t:s0     ..
+   srwxrwxr-x. root vpp  system_u:object_r:vpp_var_run_t:s0 cli.sock
+
+VPP SELinux Types
+~~~~~~~~~~~~~~~~~
+
+The following SELinux types are created by the VPP Custom SELinux
+Policy: - ``vpp_t`` - Applied to: - VPP process and spawned threads.
+
+-  ``vpp_config_rw_t`` - Applied to:
+
+   -  ``/etc/vpp/*``
+
+-  ``vpp_tmp_t`` - Applied to:
+
+   -  ``/tmp/*``
+
+-  ``vpp_exec_t`` - Applied to:
+
+   -  ``/usr/bin/*``
+
+-  ``vpp_lib_t`` - Applied to:
+
+   -  ``/usr/lib/vpp_api_test_plugins/*``
+   -  ``/usr/lib/vpp_plugins/*``
+
+-  ``vpp_unit_file_t`` - Applied to:
+
+   -  ``/usr/lib/systemd/system/vpp.*``
+
+-  ``vpp_log_t`` - Applied to:
+
+   -  ``/var/log/vpp/*``
+
+-  ``vpp_var_run_t`` - Applied to:
+
+   -  ``/var/run/vpp/*``
+
+Debug SELinux Issues
+--------------------
+
+If SELinux issues are suspected, there are a few steps that can be taken
+to debug the issue. This section provides a few pointers on on those
+steps. Any SELinux JIRAs will need this information to properly address
+the issue.
+
+Additional SELinux Packages and Setup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, install the SELinux troubleshooting packages:
+
+::
+
+   $ sudo yum -y install setroubleshoot setroubleshoot-server setools-console
+   -- OR --
+   $ sudo dnf -y install setroubleshoot setroubleshoot-server setools-console
+
+To enable proper logging, restart auditd:
+
+::
+
+   $ sudo service auditd restart
+
+While debugging issues, it is best to set SELinux to ``Permissive``
+mode. In ``Permissive`` mode, SELinux will still detect and flag errors,
+but will allow processes to continue normal operation. This allows
+multiple errors to be collected at once as opposed to breaking on each
+individual error. To set SELinux to ``Permissive`` mode (until next
+reboot or it is set back), use:
+
+::
+
+   $ sudo setenforce 0
+
+   $ getenforce
+   Permissive
+
+After debugging, to set SELinux back to ``Enforcing`` mode, use:
+
+::
+
+   $ sudo setenforce 1
+
+   $ getenforce
+   Enforcing
+
+Debugging
+~~~~~~~~~
+
+Once the SELinux troubleshooting packages are installed, perform the
+actions that are suspected to be blocked by SELinux. Either ``tail`` the
+log during these actions or ``grep`` the log for additional SELinux
+logs:
+
+::
+
+   sudo tail -f /var/log/messages
+   -- OR --
+   sudo journalctl -f
+
+Below are some examples of SELinux logs that are generated:
+
+::
+
+   May 14 11:28:34 svr-22 setroubleshoot: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt. For complete SELinux messages run: sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
+   May 14 11:28:34 svr-22 python: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp should be allowed read access on the hostCreate.txt file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
+   May 14 11:28:34 svr-22 setroubleshoot: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt. For complete SELinux messages run: sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
+   May 14 11:28:34 svr-22 python: SELinux is preventing /usr/bin/vpp from read access on the file hostCreate.txt.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp should be allowed read access on the hostCreate.txt file by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
+   May 14 11:28:37 svr-22 setroubleshoot: SELinux is preventing vpp_main from map access on the packet_socket packet_socket. For complete SELinux messages run: sealert -l ab6667d9-3f14-4dbd-96a0-7a655f7b4eb1
+   May 14 11:28:37 svr-22 python: SELinux is preventing vpp_main from map access on the packet_socket packet_socket.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp_main should be allowed map access on the packet_socket packet_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
+   May 14 11:28:51 svr-22 setroubleshoot: SELinux is preventing vpp_main from map access on the packet_socket packet_socket. For complete SELinux messages run: sealert -l ab6667d9-3f14-4dbd-96a0-7a655f7b4eb1
+   May 14 11:28:51 svr-22 python: SELinux is preventing vpp_main from map access on the packet_socket packet_socket.#012#012*****  Plugin catchall (100. confidence) suggests   **************************#012#012If you believe that vpp_main should be allowed map access on the packet_socket packet_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain#012# semodule -i my-vppmain.pp#012
+
+From the logs above, there are two sets of commands that are recommended
+to be run. The first is to run the ``sealert`` command. The second is to
+run the ``ausearch | audit2allow`` commands and the ``semodule``
+command.
+
+sealert Command
+^^^^^^^^^^^^^^^
+
+This ``sealert`` command provides a more detailed output for the given
+issue detected.
+
+::
+
+   $ sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
+   SELinux is preventing /usr/bin/vpp from 'read, write' accesses on the chr_file noiommu-0.
+
+   *****  Plugin device (91.4 confidence) suggests   ****************************
+
+   If you want to allow vpp to have read write access on the noiommu-0 chr_file
+   Then you need to change the label on noiommu-0 to a type of a similar device.
+   Do
+   # semanage fcontext -a -t SIMILAR_TYPE 'noiommu-0'
+   # restorecon -v 'noiommu-0'
+
+   *****  Plugin catchall (9.59 confidence) suggests   **************************
+
+   If you believe that vpp should be allowed read write access on the noiommu-0 chr_file by default.
+   Then you should report this as a bug.
+   You can generate a local policy module to allow this access.
+   Do
+   allow this access for now by executing:
+   # ausearch -c 'vpp' --raw | audit2allow -M my-vpp
+   # semodule -i my-vpp.pp
+
+
+   Additional Information:
+   Source Context                system_u:system_r:vpp_t:s0
+   Target Context                system_u:object_r:device_t:s0
+   Target Objects                noiommu-0 [ chr_file ]
+   Source                        vpp
+   Source Path                   /usr/bin/vpp
+   Port                          <Unknown>
+   Host                          vpp_centos7_selinux
+   Source RPM Packages           vpp-19.01.2-rc0~17_gcfd3086.x86_64
+   Target RPM Packages
+   Policy RPM                    selinux-policy-3.13.1-229.el7_6.12.noarch
+   Selinux Enabled               True
+   Policy Type                   targeted
+   Enforcing Mode                Permissive
+   Host Name                     vpp_centos7_selinux
+   Platform                      Linux vpp_centos7_selinux
+                                 3.10.0-957.12.1.el7.x86_64 #1 SMP Mon Apr 29
+                                 14:59:59 UTC 2019 x86_64 x86_64
+   Alert Count                   1
+   First Seen                    2019-05-13 18:10:50 EDT
+   Last Seen                     2019-05-13 18:10:50 EDT
+   Local ID                      a418f869-f470-4c8a-b8e9-bdd41f2dd60b
+
+   Raw Audit Messages
+   type=AVC msg=audit(1557785450.964:257): avc:  denied  { read write } for  pid=5273 comm="vpp" name="noiommu-0" dev="devtmpfs" ino=36022 scontext=system_u:system_r:vpp_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1
+
+
+   type=AVC msg=audit(1557785450.964:257): avc:  denied  { open } for  pid=5273 comm="vpp" path="/dev/vfio/noiommu-0" dev="devtmpfs" ino=36022 scontext=system_u:system_r:vpp_t:s0 tcontext=system_u:object_r:device_t:s0 tclass=chr_file permissive=1
+
+
+   type=SYSCALL msg=audit(1557785450.964:257): arch=x86_64 syscall=open success=yes exit=ENOTBLK a0=7fb395ffd7f0 a1=2 a2=7fb395ffd803 a3=7fb395ffe2a0 items=0 ppid=1 pid=5273 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=993 sgid=0 fsgid=993 tty=(none) ses=4294967295 comm=vpp exe=/usr/bin/vpp subj=system_u:system_r:vpp_t:s0 key=(null)
+
+   Hash: vpp,vpp_t,device_t,chr_file,read,write
+
+In general, this command pumps out too much info and is only needed for
+additional debugging for tougher issues. Also note that once the process
+being tested is restarted, this command loses it’s context and will not
+provide any information:
+
+::
+
+   $ sealert -l a418f869-f470-4c8a-b8e9-bdd41f2dd60b
+   Error
+   query_alerts error (1003): id (a418f869-f470-4c8a-b8e9-bdd41f2dd60b) not found
+
+ausearch \| audit2allow and semodule Commands
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+These set of commands are more useful for basic debugging. The
+``ausearch | audit2allow`` commands generate a set files. It may be
+worthwhile to run the commands in a temporary subdirectory:
+
+::
+
+   $ mkdir test-01/; cd test-01/
+
+   $ sudo ausearch -c 'vpp_main' --raw | audit2allow -M my-vppmain
+
+   $ ls
+   my-vpp.pp  my-vpp.te
+
+   $ cat my-vpp.te
+   module my-vpp 1.0;
+
+   require {
+           type user_home_t;
+           type vpp_t;
+           class packet_socket map;
+           class file { open read };
+   }
+
+   #============= vpp_t ==============
+   allow vpp_t self:packet_socket map;
+   allow vpp_t user_home_t:file { open read };
+
+As shown above, the file ``my-vpp.te`` has been generated. This file
+shows possible changes to the SELinux policy that may fix the issue. If
+an SELinux policy was being created from scratch, this policy could be
+applied using the ``semodule -i my-vpp.pp`` command. HOWEVER, VPP
+already has a policy in place. So these changes need to be incorporated
+into the existing policy. The VPP SELinux policy is located in the
+following files:
+
+::
+
+   $ ls extras/selinux/
+   selinux_doc.md  vpp-custom.fc  vpp-custom.if  vpp-custom.te
+
+In this example, ``map`` needs to be added to the ``packet_socket``
+class. If the ``vpp-custom.te`` is examined (prior to this fix), then
+one would see that the ``packet_socket`` class is already defined and
+just needs to be updated:
+
+::
+
+   $ vi extras/selinux/vpp-custom.te
+   :
+   allow vpp_t self:process { execmem execstack setsched signal }; # too benevolent
+   allow vpp_t self:packet_socket { bind create setopt ioctl };  <---
+   allow vpp_t self:tun_socket { create relabelto relabelfrom };
+   :
+
+Before blindly applying the changes proposed by the
+``ausearch | audit2allow`` commands, try to determine what is being
+allowed by the policy and determine if this is desired, or if the code
+can be reworked to no longer require the suggested permission. In the
+``my-vpp.te`` file from above, it is suggested to allow ``vpp_t``
+(i.e. the VPP process) access to all files in the home directory
+(``allow vpp_t user_home_t:file { open read };``). This was because a
+``vppctl exec`` command was executed calling a script located in the
+``/home/<user>/`` directory. Once this script was run from the
+``/usr/share/vpp/`` directory as described in a section above, these
+permissions were no longer needed.
diff --git a/extras/snap/README.md b/extras/snap/README.md
deleted file mode 100644
index 47fdd3c204e..00000000000
--- a/extras/snap/README.md
+++ /dev/null
@@ -1,86 +0,0 @@
-VPP Snap Build   {#snap_doc}
---------------
-
-General
--------
-
-The external dependency package will not build in the snapcraft
-vm. The path of least resistance is to copy it to the root of the
-(original) workspace before running the prep script.
-
-Snapcraft has mount issues except under /home. Run the prep script and
-copy the entire directory (including the .tgz file) under
-/home/yourself.
-
-Run the prep script
--------------------
-
-```
-  $ cd <vpp-workspace>/extras/snap
-  $ ./prep
-```
-
-Copy data to /home (if necessary)
-
-```
-  $ mkdir /home/xxx
-  $ cd <vpp-workspace>/extras/snap
-  $ cp * /home/xxx
-
-Set snapcraft environment variables
------------------------------------
-
-Minimum requirements:
-
-```
-  SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY=16G
-  SNAPCRAFT_BUILD_ENVIRONMENT_DISK=32G
-```
-
-Optional:
-
-```
-  SNAPCRAFT_BUILD_ENVIRONMENT_CPU=8
-  SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=yes
-```
-
-Run snapcraft
--------------
-
-With luck, simply running snapcraft will produce the snap
-
-```
-  $ <environment-variable-settings> snapcraft [--debug]
-```
-
-Rerunning snapcraft phases
---------------------------
-
-Here's how to (re)run individual phases, to avoid starting from
-scratch N times in case of errors:
-
-```
-  snapcraft pull [<part-name>]
-  snapcraft build [<part-name>]
-  snapcraft stage [<part-name>]
-  snapcraft prime [<part-name>]
-  snapcraft snap or snapcraft
-```
-
-Restart without rebuilding VM
------------------------------
-
-To restart from scratch without rebuilding the VM:
-
-```
-  snapcraft clean vpp
-```
-
-Delete (all) snapcraft VMs
---------------------------
-
-```
-  for vm in $(multipass list | awk '{print $1}' | grep ^snapcraft-); do
-  	multipass delete $vm --purge
-  done
-```
diff --git a/extras/snap/README.rst b/extras/snap/README.rst
new file mode 100644
index 00000000000..1a455565814
--- /dev/null
+++ b/extras/snap/README.rst
@@ -0,0 +1,84 @@
+.. _snap_doc:
+
+VPP Snap Build
+==============
+
+The external dependency package will not build in the snapcraft vm. The
+path of least resistance is to copy it to the root of the (original)
+workspace before running the prep script.
+
+Snapcraft has mount issues except under /home. Run the prep script and
+copy the entire directory (including the .tgz file) under
+/home/yourself.
+
+Run the prep script
+-------------------
+
+::
+
+     $ cd <vpp-workspace>/extras/snap
+     $ ./prep
+
+Copy data to /home (if necessary)
+
+::
+
+     $ mkdir /home/xxx
+     $ cd <vpp-workspace>/extras/snap
+     $ cp * /home/xxx
+
+   Set snapcraft environment variables
+   -----------------------------------
+
+   Minimum requirements:
+
+SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY=16G
+SNAPCRAFT_BUILD_ENVIRONMENT_DISK=32G
+
+::
+
+
+   Optional:
+
+SNAPCRAFT_BUILD_ENVIRONMENT_CPU=8 SNAPCRAFT_ENABLE_DEVELOPER_DEBUG=yes
+
+::
+
+
+   Run snapcraft
+   -------------
+
+   With luck, simply running snapcraft will produce the snap
+
+$ snapcraft [–debug]
+
+::
+
+
+   Rerunning snapcraft phases
+   --------------------------
+
+   Here's how to (re)run individual phases, to avoid starting from
+   scratch N times in case of errors:
+
+snapcraft pull [] snapcraft build [] snapcraft stage [] snapcraft prime
+[] snapcraft snap or snapcraft
+
+::
+
+
+   Restart without rebuilding VM
+   -----------------------------
+
+   To restart from scratch without rebuilding the VM:
+
+snapcraft clean vpp
+
+::
+
+
+   Delete (all) snapcraft VMs
+   --------------------------
+
+for vm in $(multipass list \| awk ‘{print $1}’ \| grep ^snapcraft-); do
+multipass delete $vm –purge done \``\`
diff --git a/extras/strongswan/README.md b/extras/strongswan/README.md
deleted file mode 100644
index 48d02244370..00000000000
--- a/extras/strongswan/README.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# VPP Swanstrong Testing Recipe {#strongswan_test_doc}
-
-Simple test framework for VPP and strongSwan scenarios.
-
-## setup and run
-
-`docker` is needed to run the tests.
-
-Create `~/.vpp_sswan` file and set `VPP_BIN` and `VPPCTL` variables that points to vpp and vppctl binaries, like follows:
-```
-export VPP_BIN=/path/to/vpp
-export VPPCTL=/path/to/vppctl
-```
-
-To run all test
-```
-./run.sh
-```
-
-or specific test
-```
-./test_responder.sh
-```
diff --git a/extras/strongswan/README.rst b/extras/strongswan/README.rst
new file mode 100644
index 00000000000..2ba6f6e430f
--- /dev/null
+++ b/extras/strongswan/README.rst
@@ -0,0 +1,31 @@
+.. _strongswan_test_doc:
+
+Strongswan Testing Tool
+=======================
+
+Simple test framework for VPP and strongSwan scenarios.
+
+setup and run
+-------------
+
+``docker`` is needed to run the tests.
+
+Create ``~/.vpp_sswan`` file and set ``VPP_BIN`` and ``VPPCTL``
+variables that points to vpp and vppctl binaries, like follows:
+
+::
+
+   export VPP_BIN=/path/to/vpp
+   export VPPCTL=/path/to/vppctl
+
+To run all test
+
+::
+
+   ./run.sh
+
+or specific test
+
+::
+
+   ./test_responder.sh
diff --git a/extras/vcl-ldpreload/README.md b/extras/vcl-ldpreload/README.md
deleted file mode 100644
index 81c86f648c8..00000000000
--- a/extras/vcl-ldpreload/README.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# vcl-ldpreload: a LD_PRELOAD library that uses the VPP Communications Library (VCL).  {#vcl_ldpreload_doc}
-
-User can LD_PRELOAD any application that uses POSIX socket API.
-
-NOTE: The sources have been moved to .../vpp/src/vcl and
-      libvcl_ldpreload.so is built with VPP and can be found in
-      .../vpp/build-root/install-vpp[_debug]-native/vpp/lib
-
-## HowTo
-
-# 1. Running the demo
-## Run test script without parameters to see help menu:
-
-export WS_ROOT=<top level vpp git repo dir>  (e.g. /scratch/my_name/vpp)
-$WS_ROOT/test/scripts/socket_test.sh
-
-# 2. Docker iPerf examples.
-## These launch xterms. To quit, close xterms and run following docker kill cmd (WARNING: This will kill all docker containers!) 'docker kill $(docker ps -q)'
-
-
-## Docker iPerf using default Linux Bridge
-
-$WS_ROOT/test/scripts/socket_test.sh -bi docker-kernel
-
-## Docker iPerf using VPP
-$WS_ROOT/test/scripts/socket_test.sh -bi docker-preload
-
diff --git a/extras/vcl-ldpreload/README.rst b/extras/vcl-ldpreload/README.rst
new file mode 100644
index 00000000000..7168697ec01
--- /dev/null
+++ b/extras/vcl-ldpreload/README.rst
@@ -0,0 +1,40 @@
+.. _vcl_ldpreload_doc:
+
+LD_PRELOAD the VCL
+==================
+
+vcl-ldpreload is a LD_PRELOAD library that uses the VPP Communications Library (VCL).
+
+User can LD_PRELOAD any application that uses POSIX socket API.
+
+NOTE: The sources have been moved to ``vpp/src/vcl`` and ``libvcl_ldpreload.so`` is built with VPP and can be found in
+``vpp/build-root/install-vpp[_debug]-native/vpp/lib``
+
+1. Running the demo
+-------------------
+
+Run test script without parameters to see help menu:
+
+::
+
+    export WS_ROOT= (e.g. /scratch/my_name/vpp)
+    $WS_ROOT/test/scripts/socket_test.sh
+
+
+2. Docker iPerf examples
+------------------------
+
+These launch xterms. To quit, close xterms and run following docker kill cmd (WARNING: This will kill all docker containers!) ‘docker kill $(docker ps -q)’
+
+Docker iPerf using default Linux Bridge
+
+::
+
+    $WS_ROOT/test/scripts/socket_test.sh -bi docker-kernel
+
+
+Docker iPerf using VPP
+
+::
+
+    $WS_ROOT/test/scripts/socket_test.sh -bi docker-preload
diff --git a/extras/vpp_config/README.rst b/extras/vpp_config/README.rst
index 8995edfce9f..19fc9166dad 100644
--- a/extras/vpp_config/README.rst
+++ b/extras/vpp_config/README.rst
@@ -1,514 +1,517 @@
-Summary:

-

-The purpose of the VPP configuration utility is to allow the user to configure

-VPP in a simple and safe manner. The utility takes input from the user and

-then modifies the key configuration files. The user can then examine these files

-to be sure they are correct and then actually apply the configuration. The user

-can also install a released and stable version of VPP. This is currently

-released with release 17.10.

-

-Use:

-

-The installation and executing of the VPP configuration utility is simple. First

-install the python pip module. Using pip install, then pip install vpp-config.

-Then simply type �vpp-config� and answer the questions. If you are not sure what

-to answer choose the default. For yes or no questions the capital letter

-designates the default. For example, for a question that shows [Y/n] Y is the

-default. For numbers the default is within the brackets for example for a

-question that shows [1024]. 1024 is the default. 

-

-The flow of the utility is to inspect the system, if VPP is not install it,

-create dry run configurations, inspect the files created during the dry run,

-apply the configuration and then inspect the system again and then repeat.  

-

-Caveats:

-

-- Supports Ubuntu, centos7, RedHat is coming shortly.

-

-For Developers:

-

-Modifying the code is reasonable simple. The process would be edit and debug the

-code from the root directory. In order to do this, we need a script that will copy

-or data files to the proper place. This is where they end up with pip install. For

-Ubuntu, this is /usr/local/vpp/vpp-config. I have provided a script that will copy

-the relevant files correctly. I have also provided a script that will clean the

-environment so you can start from scratch. These are the steps to run the utility

-in this environment. The scripts are meant to be run from the root directory.

-

-  ./scripts/clean.sh

-  ./scripts/cp-data.sh

-  ./vpp-config

-

-When the utility is installed with pip the wrapper scripts/vpp-config is written to

-/usr/local/bin. However, the starting point when debugging this script locally is

-vpp-config. Run the utility by executing vpp-config.

-

-The start point in the code is in vpp_config.py. However, most of the work is

-done in

-the files in ./vpplib

-

-Uploading to PyPi:

-

-To upload this utility to PpPi simple do the following. Currently, I have my own account

-when we want everyone to contribute we will need to change that.

-

-  sudo �H bash

-  cd vpp_config

-  python setup.py sdist bdist_wheel

-  twine upload dist/*

-

-Example Run:

-

-# pip install vpp-config

-# vpp-config

-

-Welcome to the VPP system configuration utility

-

-These are the files we will modify:

-    /etc/vpp/startup.conf

-    /etc/sysctl.d/80-vpp.conf

-    /etc/default/grub

-

-Before we change them, we'll create working copies in /usr/local/vpp/vpp-config/dryrun

-Please inspect them carefully before applying the actual configuration (option 3)!

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 1

-

-==============================

-NODE: DUT1

-

-CPU:

-          Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz

-              CPU(s):    32

-  Thread(s) per core:    2

-  Core(s) per socket:    8

-           Socket(s):    2

-   NUMA node0 CPU(s):    0-7,16-23

-   NUMA node1 CPU(s):    8-15,24-31

-         CPU max MHz:    3600.0000

-         CPU min MHz:    1200.0000

-                 SMT:    Enabled

-

-VPP Threads: (Name: Cpu Number)

-

-Grub Command Line:

-  Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro

-  Configured: GRUB_CMDLINE_LINUX_DEFAULT=""

-

-Huge Pages:

-  Total System Memory           : 65863484 kB

-  Total Free Memory             : 41325924 kB

-  Actual Huge Page Total        : 8192

-  Configured Huge Page Total    : 1024

-  Huge Pages Free               : 8192

-  Huge Page Size                : 2048 kB

-

-Devices:

-

-Status:

-  Not Installed

-

-==============================

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 4

-

-There are no VPP packages on node localhost.

-Do you want to install VPP [Y/n]? 

-INFO:root: Local Command: ls /etc/apt/sources.list.d/99fd.io.list.orig

-INFO:root:  /etc/apt/sources.list.d/99fd.io.list.orig

-��..

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 1

-

-==============================

-NODE: DUT1

-

-CPU:

-          Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz

-              CPU(s):    32

-  Thread(s) per core:    2

-  Core(s) per socket:    8

-           Socket(s):    2

-   NUMA node0 CPU(s):    0-7,16-23

-   NUMA node1 CPU(s):    8-15,24-31

-         CPU max MHz:    3600.0000

-         CPU min MHz:    1200.0000

-                 SMT:    Enabled

-

-VPP Threads: (Name: Cpu Number)

-  vpp_main  : 0   

-  vpp_stats : 0   

-

-Grub Command Line:

-  Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro

-  Configured: GRUB_CMDLINE_LINUX_DEFAULT=""

-

-Huge Pages:

-  Total System Memory           : 65863484 kB

-  Total Free Memory             : 55877364 kB

-  Actual Huge Page Total        : 1024

-  Configured Huge Page Total    : 1024

-  Huge Pages Free               : 1024

-  Huge Page Size                : 2048 kB

-

-Devices:

-Name                           Socket RXQs RXDescs TXQs TXDescs

-

-Status:

-  active (running)

-

-==============================

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 2

-

-These device(s) are currently NOT being used by VPP or the OS.

-

-PCI ID          Description                                       

-----------------------------------------------------------------

-0000:02:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:02:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-

-Would you like to give any of these devices back to the OS [y/N]? y

-Would you like to use device 0000:02:00.0 for the OS [y/N]? y

-Would you like to use device 0000:02:00.1 for the OS [y/N]? y

-

-These devices have kernel interfaces, but appear to be safe to use with VPP.

-

-PCI ID          Kernel Interface(s)       Description                                       

-------------------------------------------------------------------------------------------

-0000:90:00.0    enp144s0                  VIC Ethernet NIC                                  

-0000:8f:00.0    enp143s0                  VIC Ethernet NIC                                  

-0000:84:00.0    enp132s0f0,enp132s0f0d1   Ethernet Controller XL710 for 40GbE QSFP+         

-0000:84:00.1    enp132s0f1,enp132s0f1d1   Ethernet Controller XL710 for 40GbE QSFP+         

-0000:08:00.1    enp8s0f1                  I350 Gigabit Network Connection                   

-0000:02:00.0    enp2s0f0                  82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:02:00.1    enp2s0f1                  82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:86:00.0    enp134s0f0                82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:86:00.1    enp134s0f1                82599ES 10-Gigabit SFI/SFP+ Network Connection    

-

-Would you like to use any of these device(s) for VPP [y/N]? y

-Would you like to use device 0000:90:00.0 for VPP [y/N]? 

-Would you like to use device 0000:8f:00.0 for VPP [y/N]? 

-Would you like to use device 0000:84:00.0 for VPP [y/N]? 

-Would you like to use device 0000:84:00.1 for VPP [y/N]? 

-Would you like to use device 0000:08:00.1 for VPP [y/N]? 

-Would you like to use device 0000:02:00.0 for VPP [y/N]? y

-Would you like to use device 0000:02:00.1 for VPP [y/N]? y

-Would you like to use device 0000:86:00.0 for VPP [y/N]? y

-Would you like to use device 0000:86:00.1 for VPP [y/N]? y

-

-These device(s) will be used by VPP.

-

-PCI ID          Description                                       

-----------------------------------------------------------------

-0000:86:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:86:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:02:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:02:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-

-Would you like to remove any of these device(s) [y/N]? 

-

-These device(s) will be used by VPP, please rerun this option if this is incorrect.

-

-PCI ID          Description                                       

-----------------------------------------------------------------

-0000:86:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:86:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:02:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-0000:02:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection    

-

-Your system has 32 core(s) and 2 Numa Nodes.

-To begin, we suggest not reserving any cores for VPP or other processes.

-Then to improve performance try reserving cores as needed. 

-

-How many core(s) do you want to reserve for processes other than VPP? [0-16][0]? 4

-How many core(s) shall we reserve for VPP workers[0-4][0]? 2

-Should we reserve 1 core for the VPP Main thread? [Y/n]? 

-

-There currently 1024 2048 kB huge pages free.

-Do you want to reconfigure the number of huge pages [y/N]? y

-

-There currently a total of 1024 huge pages.

-How many huge pages do you want [1024 - 22511][1024]? 8192

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 3

-

-We are now going to configure your system(s).

-

-Are you sure you want to do this [Y/n]? 

-These are the changes we will apply to

-the huge page file (/etc/sysctl.d/80-vpp.conf).

-

-1,2d0

-< # Number of 2MB hugepages desired

-< vm.nr_hugepages=1024

-4,7c2,3

-< # Must be greater than or equal to (2 * vm.nr_hugepages).

-< vm.max_map_count=3096

-< 

-< # All groups allowed to access hugepages

----

-> vm.nr_hugepages=8192

-> vm.max_map_count=17408

-8a5

-> kernel.shmmax=17179869184

-10,15d6

-< # Shared Memory Max must be greator or equal to the total size of hugepages.

-< # For 2MB pages, TotalHugepageSize = vm.nr_hugepages * 2 * 1024 * 1024

-< # If the existing kernel.shmmax setting  (cat /sys/proc/kernel/shmmax)

-< # is greater than the calculated TotalHugepageSize then set this parameter

-< # to current shmmax value.

-< kernel.shmmax=2147483648

-

-

-Are you sure you want to apply these changes [Y/n]? 

-These are the changes we will apply to

-the VPP startup file (/etc/vpp/startup.conf).

-

-3c3

-<   nodaemon

----

->   interactive

-5a6

->   cli-listen /run/vpp/cli.sock

-17c18,25

-< 	## In the VPP there is one main thread and optionally the user can create worker(s)

----

-> 

->   main-core 8

->   corelist-workers 9-10,5-6

-> 

->   scheduler-policy fifo

->   scheduler-priority 50

-> 

->         ## In the VPP there is one main thread and optionally the user can create worker(s)

-52,53c60,76

-< # dpdk {

-< 	## Change default settings for all intefaces

----

-> dpdk {

-> 

->   dev 0000:86:00.0 { 

->     num-rx-queues 2

->   }

->   dev 0000:86:00.1 { 

->     num-rx-queues 2

->   }

->   dev 0000:02:00.0 { 

->     num-rx-queues 2

->   }

->   dev 0000:02:00.1 { 

->     num-rx-queues 2

->   }

->   num-mbufs 71680

-> 

->         ## Change default settings for all intefaces

-82a106,115

-> 	## Specify bonded interface and its slaves via PCI addresses

-> 	## 

->         ## Bonded interface in XOR load balance mode (mode 2) with L3 and L4 headers 

-> 	# vdev eth_bond0,mode=2,slave=0000:02:00.0,slave=0000:03:00.0,xmit_policy=l34

-> 	# vdev eth_bond1,mode=2,slave=0000:02:00.1,slave=0000:03:00.1,xmit_policy=l34

-> 	##

-> 	## Bonded interface in Active-Back up mode (mode 1)

-> 	# vdev eth_bond0,mode=1,slave=0000:02:00.0,slave=0000:03:00.0

-> 	# vdev eth_bond1,mode=1,slave=0000:02:00.1,slave=0000:03:00.1

-> 

-99c132

-< # }

----

-> }

-108a142

-> 

-

-

-Are you sure you want to apply these changes [Y/n]? 

-

-The configured grub cmdline looks like this:

-GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable isolcpus=1-4,8,9-10,5-6 nohz_full=1-4,8,9-10,5-6 rcu_nocbs=1-4,8,9-10,5-6"

-

-The current boot cmdline looks like this:

-BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro

-

-Do you want to keep the current boot cmdline [Y/n]? 

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 1

-

-==============================

-NODE: DUT1

-

-CPU:

-          Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz

-              CPU(s):    32

-  Thread(s) per core:    2

-  Core(s) per socket:    8

-           Socket(s):    2

-   NUMA node0 CPU(s):    0-7,16-23

-   NUMA node1 CPU(s):    8-15,24-31

-         CPU max MHz:    3600.0000

-         CPU min MHz:    1200.0000

-                 SMT:    Enabled

-

-VPP Threads: (Name: Cpu Number)

-

-Grub Command Line:

-  Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro

-  Configured: GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable isolcpus=1-4,8,9-10,5-6 nohz_full=1-4,8,9-10,5-6 rcu_nocbs=1-4,8,9-10,5-6"

-

-Huge Pages:

-  Total System Memory           : 65863484 kB

-  Total Free Memory             : 41163916 kB

-  Actual Huge Page Total        : 8192

-  Configured Huge Page Total    : 8192

-  Huge Pages Free               : 3108

-  Huge Page Size                : 2048 kB

-

-Devices:

-Total Number of Buffers: 71680

-

-Status:

-  active (running)

-  Sep 27 12:49:59 tf-ucs-3 vpp[13671]: EAL: No free hugepages reported in hugepages-1048576kB

-

-==============================

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: 1

-

-==============================

-NODE: DUT1

-

-CPU:

-          Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz

-              CPU(s):    32

-  Thread(s) per core:    2

-  Core(s) per socket:    8

-           Socket(s):    2

-   NUMA node0 CPU(s):    0-7,16-23

-   NUMA node1 CPU(s):    8-15,24-31

-         CPU max MHz:    3600.0000

-         CPU min MHz:    1200.0000

-                 SMT:    Enabled

-

-VPP Threads: (Name: Cpu Number)

-  vpp_stats : 0   

-  vpp_wk_2  : 9   

-  vpp_wk_3  : 10  

-  vpp_wk_0  : 5   

-  vpp_wk_1  : 6   

-  vpp_main  : 8   

-

-Grub Command Line:

-  Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro

-  Configured: GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable isolcpus=1-4,8,9-10,5-6 nohz_full=1-4,8,9-10,5-6 rcu_nocbs=1-4,8,9-10,5-6"

-

-Huge Pages:

-  Total System Memory           : 65863484 kB

-  Total Free Memory             : 41170684 kB

-  Actual Huge Page Total        : 8192

-  Configured Huge Page Total    : 8192

-  Huge Pages Free               : 7936

-  Huge Page Size                : 2048 kB

-

-Devices:

-Total Number of Buffers: 71680

-Name                           Socket RXQs RXDescs TXQs TXDescs

-TenGigabitEthernet2/0/0             0    2    1024    5    1024

-TenGigabitEthernet2/0/1             0    2    1024    5    1024

-TenGigabitEthernet86/0/0            1    2    1024    5    1024

-TenGigabitEthernet86/0/1            1    2    1024    5    1024

-

-Status:

-  active (running)

-

-==============================

-

-What would you like to do?

-

-1) Show basic system information

-2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)

-       and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml

-3) Full configuration (WARNING: This will change the system configuration)

-4) Install/Uninstall VPP.

-5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).

-6) Install QEMU patch (Needed when running openstack).

-9 or q) Quit

-

-Command: q

-# 

+VPP configuration utility
+=========================
+
+The purpose of the VPP configuration utility is to allow the user to configure
+VPP in a simple and safe manner. The utility takes input from the user and
+then modifies the key configuration files. The user can then examine these files
+to be sure they are correct and then actually apply the configuration. The user
+can also install a released and stable version of VPP. This is currently
+released with release 17.10.
+
+VPP config Usage
+----------------
+
+The installation and executing of the VPP configuration utility is simple. First
+install the python pip module. Using pip install, then pip install vpp-config.
+Then simply type ``vpp-config`` and answer the questions. If you are not sure what
+to answer choose the default. For yes or no questions the capital letter
+designates the default. For example, for a question that shows [Y/n] Y is the
+default. For numbers the default is within the brackets for example for a
+question that shows [1024]. 1024 is the default.
+
+The flow of the utility is to inspect the system, if VPP is not install it,
+create dry run configurations, inspect the files created during the dry run,
+apply the configuration and then inspect the system again and then repeat.
+
+Caveats
+-------
+
+- Supports Ubuntu, centos7, RedHat is coming shortly.
+
+For Developers:
+
+Modifying the code is reasonable simple. The process would be edit and debug the
+code from the root directory. In order to do this, we need a script that will copy
+or data files to the proper place. This is where they end up with pip install. For
+Ubuntu, this is ``/usr/local/vpp/vpp-config``. I have provided a script that will copy
+the relevant files correctly. I have also provided a script that will clean the
+environment so you can start from scratch. These are the steps to run the utility
+in this environment. The scripts are meant to be run from the root directory.
+
+.. code-block:: console
+
+    ./scripts/clean.sh
+    ./scripts/cp-data.sh
+    ./vpp-config
+
+When the utility is installed with pip the wrapper ``scripts/vpp-config`` is written to
+``/usr/local/bin``. However, the starting point when debugging this script locally is
+vpp-config. Run the utility by executing vpp-config.
+
+The start point in the code is in vpp_config.py. However, most of the work is
+done in the files in ``./vpplib``
+
+Uploading to PyPi
+-----------------
+
+To upload this utility to PpPi simple do the following. Currently, I have my own account
+when we want everyone to contribute we will need to change that.
+
+
+.. code-block:: console
+
+    sudo bash
+    cd vpp_config
+    python setup.py sdist bdist_wheel
+    twine upload dist/*
+
+Example Run:
+
+.. code-block:: console
+
+    # pip install vpp-config
+    # vpp-config
+
+    Welcome to the VPP system configuration utility
+
+    These are the files we will modify:
+
+    /etc/vpp/startup.conf
+    /etc/sysctl.d/80-vpp.conf
+    /etc/default/grub
+
+    Before we change them, we'll create working copies in ``/usr/local/vpp/vpp-config/dryrun``
+    Please inspect them carefully before applying the actual configuration (option 3)!
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 1
+
+    NODE: DUT1
+
+    CPU:
+              Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
+                  CPU(s):    32
+      Thread(s) per core:    2
+      Core(s) per socket:    8
+              Socket(s):    2
+      NUMA node0 CPU(s):    0-7,16-23
+      NUMA node1 CPU(s):    8-15,24-31
+            CPU max MHz:    3600.0000
+            CPU min MHz:    1200.0000
+                    SMT:    Enabled
+
+    VPP Threads: (Name: Cpu Number)
+
+    Grub Command Line:
+      Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro
+      Configured: GRUB_CMDLINE_LINUX_DEFAULT=""
+
+    Huge Pages:
+      Total System Memory           : 65863484 kB
+      Total Free Memory             : 41325924 kB
+      Actual Huge Page Total        : 8192
+      Configured Huge Page Total    : 1024
+      Huge Pages Free               : 8192
+      Huge Page Size                : 2048 kB
+
+    Devices:
+
+    Status:
+      Not Installed
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 4
+
+    There are no VPP packages on node localhost.
+    Do you want to install VPP [Y/n]?
+    INFO:root: Local Command: ls /etc/apt/sources.list.d/99fd.io.list.orig
+    INFO:root:  /etc/apt/sources.list.d/99fd.io.list.orig
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 1
+
+    ==============================
+    NODE: DUT1
+
+    CPU:
+              Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
+                  CPU(s):    32
+      Thread(s) per core:    2
+      Core(s) per socket:    8
+              Socket(s):    2
+      NUMA node0 CPU(s):    0-7,16-23
+      NUMA node1 CPU(s):    8-15,24-31
+            CPU max MHz:    3600.0000
+            CPU min MHz:    1200.0000
+                    SMT:    Enabled
+
+    VPP Threads: (Name: Cpu Number)
+      vpp_main  : 0
+      vpp_stats : 0
+
+    Grub Command Line:
+      Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro
+      Configured: GRUB_CMDLINE_LINUX_DEFAULT=""
+
+    Huge Pages:
+      Total System Memory           : 65863484 kB
+      Total Free Memory             : 55877364 kB
+      Actual Huge Page Total        : 1024
+      Configured Huge Page Total    : 1024
+      Huge Pages Free               : 1024
+      Huge Page Size                : 2048 kB
+
+    Devices:
+    Name                           Socket RXQs RXDescs TXQs TXDescs
+
+    Status:
+      active (running)
+
+    ==============================
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 2
+
+    These device(s) are currently NOT being used by VPP or the OS.
+
+    PCI ID          Description
+    ----------------------------------------------------------------
+    0000:02:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:02:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection
+
+    Would you like to give any of these devices back to the OS [y/N]? y
+    Would you like to use device 0000:02:00.0 for the OS [y/N]? y
+    Would you like to use device 0000:02:00.1 for the OS [y/N]? y
+
+    These devices have kernel interfaces, but appear to be safe to use with VPP.
+
+    PCI ID          Kernel Interface(s)       Description
+    ------------------------------------------------------------------------------------------
+    0000:90:00.0    enp144s0                  VIC Ethernet NIC
+    0000:8f:00.0    enp143s0                  VIC Ethernet NIC
+    0000:84:00.0    enp132s0f0,enp132s0f0d1   Ethernet Controller XL710 for 40GbE QSFP+
+    0000:84:00.1    enp132s0f1,enp132s0f1d1   Ethernet Controller XL710 for 40GbE QSFP+
+    0000:08:00.1    enp8s0f1                  I350 Gigabit Network Connection
+    0000:02:00.0    enp2s0f0                  82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:02:00.1    enp2s0f1                  82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:86:00.0    enp134s0f0                82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:86:00.1    enp134s0f1                82599ES 10-Gigabit SFI/SFP+ Network Connection
+
+    Would you like to use any of these device(s) for VPP [y/N]? y
+    Would you like to use device 0000:90:00.0 for VPP [y/N]?
+    Would you like to use device 0000:8f:00.0 for VPP [y/N]?
+    Would you like to use device 0000:84:00.0 for VPP [y/N]?
+    Would you like to use device 0000:84:00.1 for VPP [y/N]?
+    Would you like to use device 0000:08:00.1 for VPP [y/N]?
+    Would you like to use device 0000:02:00.0 for VPP [y/N]? y
+    Would you like to use device 0000:02:00.1 for VPP [y/N]? y
+    Would you like to use device 0000:86:00.0 for VPP [y/N]? y
+    Would you like to use device 0000:86:00.1 for VPP [y/N]? y
+
+    PCI ID          Description
+    ----------------------------------------------------------------
+    0000:86:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:86:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:02:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:02:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection
+
+    Would you like to remove any of these device(s) [y/N]?
+
+    These device(s) will be used by VPP, please rerun this option if this is incorrect.
+
+    PCI ID          Description
+    ----------------------------------------------------------------
+    0000:86:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:86:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:02:00.0    82599ES 10-Gigabit SFI/SFP+ Network Connection
+    0000:02:00.1    82599ES 10-Gigabit SFI/SFP+ Network Connection
+
+    Your system has 32 core(s) and 2 Numa Nodes.
+    To begin, we suggest not reserving any cores for VPP or other processes.
+    Then to improve performance try reserving cores as needed.
+
+    How many core(s) do you want to reserve for processes other than VPP? [0-16][0]? 4
+    How many core(s) shall we reserve for VPP workers[0-4][0]? 2
+    Should we reserve 1 core for the VPP Main thread? [Y/n]?
+
+    There currently 1024 2048 kB huge pages free.
+    Do you want to reconfigure the number of huge pages [y/N]? y
+
+    There currently a total of 1024 huge pages.
+    How many huge pages do you want [1024 - 22511][1024]? 8192
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 3
+
+    We are now going to configure your system(s).
+
+    Are you sure you want to do this [Y/n]?
+    These are the changes we will apply to
+    the huge page file (/etc/sysctl.d/80-vpp.conf).
+
+    1,2d0
+    < # Number of 2MB hugepages desired
+    < vm.nr_hugepages=1024
+    4,7c2,3
+    < # Must be greater than or equal to (2 * vm.nr_hugepages).
+    < vm.max_map_count=3096
+    <
+    < # All groups allowed to access hugepages
+    ---
+    > vm.nr_hugepages=8192
+    > vm.max_map_count=17408
+    8a5
+    > kernel.shmmax=17179869184
+    10,15d6
+    < # Shared Memory Max must be greator or equal to the total size of hugepages.
+    < # For 2MB pages, TotalHugepageSize = vm.nr_hugepages * 2 * 1024 * 1024
+    < # If the existing kernel.shmmax setting  (cat /sys/proc/kernel/shmmax)
+    < # is greater than the calculated TotalHugepageSize then set this parameter
+    < # to current shmmax value.
+    < kernel.shmmax=2147483648
+
+    Are you sure you want to apply these changes [Y/n]?
+    These are the changes we will apply to
+    the VPP startup file (/etc/vpp/startup.conf).
+
+    3c3
+    <   nodaemon
+    ---
+    >   interactive
+    5a6
+    >   cli-listen /run/vpp/cli.sock
+    17c18,25
+    <     ## In the VPP there is one main thread and optionally the user can create worker(s)
+    ---
+    >
+    >   main-core 8
+    >   corelist-workers 9-10,5-6
+    >
+    >   scheduler-policy fifo
+    >   scheduler-priority 50
+    >
+    >         ## In the VPP there is one main thread and optionally the user can create worker(s)
+    52,53c60,76
+    < # dpdk {
+    <     ## Change default settings for all intefaces
+    ---
+    > dpdk {
+    >
+    >   dev 0000:86:00.0 {
+    >     num-rx-queues 2
+    >   }
+    >   dev 0000:86:00.1 {
+    >     num-rx-queues 2
+    >   }
+    >   dev 0000:02:00.0 {
+    >     num-rx-queues 2
+    >   }
+    >   dev 0000:02:00.1 {
+    >     num-rx-queues 2
+    >   }
+    >   num-mbufs 71680
+    >
+    >         ## Change default settings for all intefaces
+    82a106,115
+    >     ## Specify bonded interface and its slaves via PCI addresses
+    >     ##
+    >         ## Bonded interface in XOR load balance mode (mode 2) with L3 and L4 headers
+    >     # vdev eth_bond0,mode=2,slave=0000:02:00.0,slave=0000:03:00.0,xmit_policy=l34
+    >     # vdev eth_bond1,mode=2,slave=0000:02:00.1,slave=0000:03:00.1,xmit_policy=l34
+    >     ##
+    >     ## Bonded interface in Active-Back up mode (mode 1)
+    >     # vdev eth_bond0,mode=1,slave=0000:02:00.0,slave=0000:03:00.0
+    >     # vdev eth_bond1,mode=1,slave=0000:02:00.1,slave=0000:03:00.1
+    >
+    99c132
+    < # }
+    ---
+    > }
+    108a142
+    >
+
+    Are you sure you want to apply these changes [Y/n]?
+
+    The configured grub cmdline looks like this:
+    GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable isolcpus=1-4,8,9-10,5-6 nohz_full=1-4,8,9-10,5-6 rcu_nocbs=1-4,8,9-10,5-6"
+
+    The current boot cmdline looks like this:
+    BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro
+
+    Do you want to keep the current boot cmdline [Y/n]?
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 1
+
+    ==============================
+    NODE: DUT1
+
+    CPU:
+              Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
+                  CPU(s):    32
+      Thread(s) per core:    2
+      Core(s) per socket:    8
+              Socket(s):    2
+      NUMA node0 CPU(s):    0-7,16-23
+      NUMA node1 CPU(s):    8-15,24-31
+            CPU max MHz:    3600.0000
+            CPU min MHz:    1200.0000
+                    SMT:    Enabled
+
+    VPP Threads: (Name: Cpu Number)
+
+    Grub Command Line:
+      Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro
+      Configured: GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable isolcpus=1-4,8,9-10,5-6 nohz_full=1-4,8,9-10,5-6 rcu_nocbs=1-4,8,9-10,5-6"
+
+    Huge Pages:
+      Total System Memory           : 65863484 kB
+      Total Free Memory             : 41163916 kB
+      Actual Huge Page Total        : 8192
+      Configured Huge Page Total    : 8192
+      Huge Pages Free               : 3108
+      Huge Page Size                : 2048 kB
+
+    Devices:
+    Total Number of Buffers: 71680
+
+    Status:
+      active (running)
+      Sep 27 12:49:59 tf-ucs-3 vpp[13671]: EAL: No free hugepages reported in hugepages-1048576kB
+
+    ==============================
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: 1
+
+    ==============================
+    NODE: DUT1
+
+    CPU:
+              Model name:    Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz
+                  CPU(s):    32
+      Thread(s) per core:    2
+      Core(s) per socket:    8
+              Socket(s):    2
+      NUMA node0 CPU(s):    0-7,16-23
+      NUMA node1 CPU(s):    8-15,24-31
+            CPU max MHz:    3600.0000
+            CPU min MHz:    1200.0000
+                    SMT:    Enabled
+
+    VPP Threads: (Name: Cpu Number)
+      vpp_stats : 0
+      vpp_wk_2  : 9
+      vpp_wk_3  : 10
+      vpp_wk_0  : 5
+      vpp_wk_1  : 6
+      vpp_main  : 8
+
+    Grub Command Line:
+      Current: BOOT_IMAGE=/boot/vmlinuz-4.4.0-96-generic root=UUID=d760b82f-f37b-47e2-9815-db8d479a3557 ro
+      Configured: GRUB_CMDLINE_LINUX_DEFAULT="intel_pstate=disable isolcpus=1-4,8,9-10,5-6 nohz_full=1-4,8,9-10,5-6 rcu_nocbs=1-4,8,9-10,5-6"
+
+    Huge Pages:
+      Total System Memory           : 65863484 kB
+      Total Free Memory             : 41170684 kB
+      Actual Huge Page Total        : 8192
+      Configured Huge Page Total    : 8192
+      Huge Pages Free               : 7936
+      Huge Page Size                : 2048 kB
+
+    Devices:
+    Total Number of Buffers: 71680
+    Name                           Socket RXQs RXDescs TXQs TXDescs
+    TenGigabitEthernet2/0/0             0    2    1024    5    1024
+    TenGigabitEthernet2/0/1             0    2    1024    5    1024
+    TenGigabitEthernet86/0/0            1    2    1024    5    1024
+    TenGigabitEthernet86/0/1            1    2    1024    5    1024
+
+    Status:
+      active (running)
+
+    ==============================
+
+    What would you like to do?
+
+    1) Show basic system information
+    2) Dry Run (Will save the configuration files in /usr/local/vpp/vpp-config/dryrun for inspection)
+        and user input in /usr/local/vpp/vpp-config/configs/auto-config.yaml
+    3) Full configuration (WARNING: This will change the system configuration)
+    4) Install/Uninstall VPP.
+    5) Dry Run from /usr/local/vpp/vpp-config/auto-config.yaml (will not ask questions).
+    6) Install QEMU patch (Needed when running openstack).
+    9 or q) Quit
+
+    Command: q
+    #
diff --git a/extras/vpp_if_stats/README.md b/extras/vpp_if_stats/README.md
deleted file mode 100755
index 89d60c5255d..00000000000
--- a/extras/vpp_if_stats/README.md
+++ /dev/null
@@ -1,35 +0,0 @@
-# VPP interface stats client {#if_stats_client_doc}
-
-This is a source code and a binary of a 'thin client' to collect, 
-aggregate and expose VPP interface stats through VPP stats socket API. 
-It also provides some information about the installed VPP version.
-
-This can be used by monitoring systems that needs to grab those details 
-through a simple executable client with no dependencies.
-
-example use case: where VPP runs in a container that can't expose the socket API to the host level
-
-
-## Prerequisites (for building)
-
-**GoVPP** library (compatible with VPP 18.10)
-vpp, vpp-api, vpp-lib
-
-## Building
-
-```bash
-go get git.fd.io/govpp.git
-go build
-``` 
-
-## Using (post-build for example on linux 64bit)
-
-```bash
-./bin/vpp_if_stats_linux_amd64
-```
-
-## Output examples
-
-[JSON schema](./response_schema.json)
-[Example](./response_example.json)
-
diff --git a/extras/vpp_if_stats/README.rst b/extras/vpp_if_stats/README.rst
new file mode 100644
index 00000000000..2e4dcc0fb4d
--- /dev/null
+++ b/extras/vpp_if_stats/README.rst
@@ -0,0 +1,40 @@
+.. _if_stats_client_doc:
+
+VPP interface stats client
+==========================
+
+This is a source code and a binary of a ‘thin client’ to collect,
+aggregate and expose VPP interface stats through VPP stats socket API.
+It also provides some information about the installed VPP version.
+
+This can be used by monitoring systems that needs to grab those details
+through a simple executable client with no dependencies.
+
+example use case: where VPP runs in a container that can’t expose the
+socket API to the host level
+
+Prerequisites (for building)
+----------------------------
+
+**GoVPP** library (compatible with VPP 18.10) vpp, vpp-api, vpp-lib
+
+Building
+--------
+
+.. code:: bash
+
+   go get git.fd.io/govpp.git
+   go build
+
+Using (post-build for example on linux 64bit)
+---------------------------------------------
+
+.. code:: bash
+
+   ./bin/vpp_if_stats_linux_amd64
+
+Output examples
+---------------
+
+`JSON schema <./response_schema.json>`__
+`Example <./response_example.json>`__
diff --git a/extras/vpp_stats_fs/README.md b/extras/vpp_stats_fs/README.md
deleted file mode 100755
index 52610feba94..00000000000
--- a/extras/vpp_stats_fs/README.md
+++ /dev/null
@@ -1,113 +0,0 @@
-# VPP stats segment FUSE filesystem  {#stats_fs_doc}
-
-The statfs binary allows to create a FUSE filesystem to expose and to browse the stats segment.
-It relies on the Go-FUSE library and requires Go-VPP stats bindings to work.
-
-The binary mounts a filesystem on the local machine whith the data from the stats segments.
-The counters can be opened and read as files (e.g. in a Unix shell).
-Note that the value of a counter is determined when the corresponding file is opened (as for /proc/interrupts).
-
-Directories update their contents on epoch changes so that new counters get added to the filesystem.
-
-The script `install.sh` is responsible for buildiing and installing the filesystem.
-
-## Usage
-
-The local Makefile contains targets for all the possible intercations with the stats_f binary.
-
-### Help
-A basic help menu
-```bash
-make help
-```
-
-### Install
-Building the binary
-```bash
-make install
-```
-
-### Start
-Starts the filesystem. Requires a running VPP instance using the default socket /run/vpp/stats.sock.
-
-May require a privileged user (sudo)
-```bash
-make start
-```
-
-### Stop
-Stops and unmounts the filesystem if it is not busy.
-
-May require a privileged user (sudo)
-```bash
-make stop
-```
-
-### Force unmount
-Forces the unmount of the filesystem even if it is busy.
-
-May require a privileged user (sudo)
-```bash
-make force-unmount
-```
-
-### Cleanup
-Cleaning stats_fs binary.
-
-May require a privileged user (sudo).
-```bash
-make clean
-```
-
-## Browsing the filesystem
-
-The default mountpoint is /run/vpp/stats_fs_dir.
-You can browse the filesystem as a regular user.
-Example:
-
-```bash
-cd /run/vpp/stats_fs_dir
-cd sys/node
-ls -al
-cat names
-```
-
-## Building and mounting the filesystem manually
-
-For more modularity, you can build and mount the filesystem manually.
-
-### Building
-Inside the local directory, you can build the go binary:
-```bash
-go build
-```
-
-### Mounting
-Then, ou can mount the filesystem with the local binary.
-
-May require a privileged user (sudo).
-
-The basic usage is:
-```bash
-./stats_fs <MOUNT_POINT>
-```
-
-**Options:**
- - debug \<true|false\> (default is false)
- - socket \<statSocket\> (default is /run/vpp/stats.sock) : VPP socket for stats
-
-
-### Unmounting the file system
-
-You can unmount the filesystem with the fusermount command.
-
-May require a privileged user (sudo)
-
-```bash
-fusermount -u /path/to/mountpoint
-```
-
-To force the unmount even if the resource  is busy, add the -z option:
-```bash
-fusermount -uz /path/to/mountpoint
-```
diff --git a/extras/vpp_stats_fs/README.rst b/extras/vpp_stats_fs/README.rst
new file mode 100644
index 00000000000..d6635d146c0
--- /dev/null
+++ b/extras/vpp_stats_fs/README.rst
@@ -0,0 +1,148 @@
+.. _stats_fs_doc:
+
+VPP stats segment FUSE filesystem
+=================================
+
+The statfs binary allows to create a FUSE filesystem to expose and to
+browse the stats segment. It relies on the Go-FUSE library and requires
+Go-VPP stats bindings to work.
+
+The binary mounts a filesystem on the local machine with the data from
+the stats segments. The counters can be opened and read as files
+(e.g. in a Unix shell). Note that the value of a counter is determined
+when the corresponding file is opened (as for /proc/interrupts).
+
+Directories update their contents on epoch changes so that new counters
+get added to the filesystem.
+
+The script ``install.sh`` is responsible for building and installing
+the filesystem.
+
+Usage
+-----
+
+The local Makefile contains targets for all the possible interactions
+with the stats_f binary.
+
+Help
+~~~~
+
+A basic help menu
+
+.. code:: bash
+
+   make help
+
+Install
+~~~~~~~
+
+Building the binary
+
+.. code:: bash
+
+   make install
+
+Start
+~~~~~
+
+Starts the filesystem. Requires a running VPP instance using the default
+socket /run/vpp/stats.sock.
+
+May require a privileged user (sudo)
+
+.. code:: bash
+
+   make start
+
+Stop
+~~~~
+
+Stops and unmounts the filesystem if it is not busy.
+
+May require a privileged user (sudo)
+
+.. code:: bash
+
+   make stop
+
+Force unmount
+~~~~~~~~~~~~~
+
+Forces the unmount of the filesystem even if it is busy.
+
+May require a privileged user (sudo)
+
+.. code:: bash
+
+   make force-unmount
+
+Cleanup
+~~~~~~~
+
+Cleaning stats_fs binary.
+
+May require a privileged user (sudo).
+
+.. code:: bash
+
+   make clean
+
+Browsing the filesystem
+-----------------------
+
+The default mountpoint is /run/vpp/stats_fs_dir. You can browse the
+filesystem as a regular user. Example:
+
+.. code:: bash
+
+   cd /run/vpp/stats_fs_dir
+   cd sys/node
+   ls -al
+   cat names
+
+Building and mounting the filesystem manually
+---------------------------------------------
+
+For more modularity, you can build and mount the filesystem manually.
+
+Building
+~~~~~~~~
+
+Inside the local directory, you can build the go binary:
+
+.. code:: bash
+
+   go build
+
+Mounting
+~~~~~~~~
+
+Then, you can mount the filesystem with the local binary.
+
+May require a privileged user (sudo).
+
+The basic usage is:
+
+.. code:: bash
+
+   ./stats_fs <MOUNT_POINT>
+
+**Options:** - debug <true|false> (default is false) - socket
+<statSocket> (default is /run/vpp/stats.sock) : VPP socket for stats
+
+Unmounting the file system
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can unmount the filesystem with the fusermount command.
+
+May require a privileged user (sudo)
+
+.. code:: bash
+
+   fusermount -u /path/to/mountpoint
+
+To force the unmount even if the resource is busy, add the -z option:
+
+.. code:: bash
+
+   fusermount -uz /path/to/mountpoint
diff --git a/extras/vpptop/README.md b/extras/vpptop/README.md
deleted file mode 100644
index c0f4dc99b3a..00000000000
--- a/extras/vpptop/README.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# VPP Top Installation {#vpp_top_doc}
-
-[VPPTop]((https://github.com/PANTHEONtech/vpptop)) is a real-time data viewer for VPP interfaces and metrics displayed in dynamic terminal user interface, written in GO.
-
-Following make targets are available:
-
-**install** downloads and installs VPPTop including all external dependencies, binary API generator and latest version of GO. Running `make install-dep` (from the VPP top-level Makefile)
-is recommended.
-
-**cleanup** removes VPPTop repository from the target directory (/build-root/vpptop)
-
-**start** runs the VPPTop if installed
-
-**help** shows information about available commands
-
-The VPPTop is installed to be compatible with the given VPP version and may not work with other versions with different API. In that case, the VPPTop has to be re-installed.
-
-### GO variables management
-
-The installer depends on Golang environment variables GOROOT (for the GO installation) and GOPATH (for other binaries). Those variables are read from the environment and set to following values if not found:
-
-GOROOT=/root/.go/
-GOPATH=/root/go/
-
-If you have the GO already installed and have to run the installer with `sudo`, use the `-E` switch to provide those variables to the installer.
diff --git a/extras/vpptop/README.rst b/extras/vpptop/README.rst
new file mode 100644
index 00000000000..053659bca5f
--- /dev/null
+++ b/extras/vpptop/README.rst
@@ -0,0 +1,36 @@
+.. _vpp_top_doc:
+
+VPP Top Installation
+====================
+
+`VPPTop <(https://github.com/PANTHEONtech/vpptop)>`__ is a real-time
+data viewer for VPP interfaces and metrics displayed in dynamic terminal
+user interface, written in GO.
+
+Following make targets are available:
+
+* ``install`` downloads and installs VPPTop including all external dependencies, binary API generator
+  and latest version of GO. Running ``make install-dep`` (from the VPP top-level Makefile) is recommended.
+* ``cleanup`` removes VPPTop repository from the target directory (``/build-root/vpptop``)
+* ``start`` runs the VPPTop if installed
+* ``help`` shows information about available commands
+
+The VPPTop is installed to be compatible with the given VPP version and
+may not work with other versions with different API. In that case, the
+VPPTop has to be re-installed.
+
+GO variables management
+-----------------------
+
+The installer depends on Golang environment variables GOROOT (for the GO
+installation) and GOPATH (for other binaries). Those variables are read
+from the environment and set to following values if not found:
+
+::
+
+    GOROOT=/root/.go/ GOPATH=/root/go/
+
+
+If you have the GO already installed and have to run the installer with
+``sudo``, use the ``-E`` switch to provide those variables to the
+installer.
diff --git a/src/examples/handoffdemo/README.md b/src/examples/handoffdemo/README.md
deleted file mode 100644
index e38c7b3cca0..00000000000
--- a/src/examples/handoffdemo/README.md
+++ /dev/null
@@ -1,186 +0,0 @@
-# Handoff queue demo plugin {#handoff_queue_demo_plugin}
-
-This plugin provides a simplified example of how to hand off
-packets between threads. I used it to debug packet-tracer handoff
-tracing support.
-
-# Packet generator input script
-
-```
- packet-generator new {
-    name x
-    limit 5
-    size 128-128
-    interface local0
-    node handoffdemo-1
-    data {
-        incrementing 30
-    }
- }
-```
-# Start vpp with 2 worker threads
-
-The demo plugin hands packets from worker 1 to worker 2.
-
-# Enable tracing, and start the packet generator
-
-```
-  trace add pg-input 100
-  packet-generator enable
-```
-
-# Sample Run
-
-```
-  DBGvpp# ex /tmp/pg_input_script
-  DBGvpp# pa en
-  DBGvpp# sh err
-   Count                    Node                  Reason
-         5              handoffdemo-1             packets handed off processed
-         5              handoffdemo-2             completed packets
-  DBGvpp# show run
-  Thread 1 vpp_wk_0 (lcore 0)
-  Time 133.9, average vectors/node 5.00, last 128 main loops 0.00 per node 0.00
-    vector rates in 3.7331e-2, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
-               Name                 State         Calls          Vectors        Suspends         Clocks       Vectors/Call
-  handoffdemo-1                    active                  1               5               0          4.76e3            5.00
-  pg-input                        disabled                 2               5               0          5.58e4            2.50
-  unix-epoll-input                 polling             22760               0               0          2.14e7            0.00
-  ---------------
-  Thread 2 vpp_wk_1 (lcore 2)
-  Time 133.9, average vectors/node 5.00, last 128 main loops 0.00 per node 0.00
-    vector rates in 0.0000e0, out 0.0000e0, drop 3.7331e-2, punt 0.0000e0
-               Name                 State         Calls          Vectors        Suspends         Clocks       Vectors/Call
-  drop                             active                  1               5               0          1.35e4            5.00
-  error-drop                       active                  1               5               0          2.52e4            5.00
-  handoffdemo-2                    active                  1               5               0          2.56e4            5.00
-  unix-epoll-input                 polling             22406               0               0          2.18e7            0.00
-```
-
-Enable the packet tracer and run it again...
-
-```
-  DBGvpp# trace add pg-input 100
-  DBGvpp# pa en
-  DBGvpp# sh trace
-  sh trace
-  ------------------- Start of thread 0 vpp_main -------------------
-  No packets in trace buffer
-  ------------------- Start of thread 1 vpp_wk_0 -------------------
-  Packet 1
-
-  00:06:50:520688: pg-input
-    stream x, 128 bytes, 0 sw_if_index
-    current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000000
-    00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
-    00000020: 0000000000000000000000000000000000000000000000000000000000000000
-    00000040: 0000000000000000000000000000000000000000000000000000000000000000
-    00000060: 0000000000000000000000000000000000000000000000000000000000000000
-  00:06:50:520762: handoffdemo-1
-    HANDOFFDEMO: current thread 1
-
-  Packet 2
-
-  00:06:50:520688: pg-input
-    stream x, 128 bytes, 0 sw_if_index
-    current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000001
-    00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
-    00000020: 0000000000000000000000000000000000000000000000000000000000000000
-    00000040: 0000000000000000000000000000000000000000000000000000000000000000
-    00000060: 0000000000000000000000000000000000000000000000000000000000000000
-  00:06:50:520762: handoffdemo-1
-    HANDOFFDEMO: current thread 1
-
-  Packet 3
-
-  00:06:50:520688: pg-input
-    stream x, 128 bytes, 0 sw_if_index
-    current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000002
-    00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
-    00000020: 0000000000000000000000000000000000000000000000000000000000000000
-    00000040: 0000000000000000000000000000000000000000000000000000000000000000
-    00000060: 0000000000000000000000000000000000000000000000000000000000000000
-  00:06:50:520762: handoffdemo-1
-    HANDOFFDEMO: current thread 1
-
-  Packet 4
-
-  00:06:50:520688: pg-input
-    stream x, 128 bytes, 0 sw_if_index
-    current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000003
-    00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
-    00000020: 0000000000000000000000000000000000000000000000000000000000000000
-    00000040: 0000000000000000000000000000000000000000000000000000000000000000
-    00000060: 0000000000000000000000000000000000000000000000000000000000000000
-  00:06:50:520762: handoffdemo-1
-    HANDOFFDEMO: current thread 1
-
-  Packet 5
-
-  00:06:50:520688: pg-input
-    stream x, 128 bytes, 0 sw_if_index
-    current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000004
-    00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
-    00000020: 0000000000000000000000000000000000000000000000000000000000000000
-    00000040: 0000000000000000000000000000000000000000000000000000000000000000
-    00000060: 0000000000000000000000000000000000000000000000000000000000000000
-  00:06:50:520762: handoffdemo-1
-    HANDOFFDEMO: current thread 1
-
-  ------------------- Start of thread 2 vpp_wk_1 -------------------
-  Packet 1
-
-  00:06:50:520796: handoff_trace
-    HANDED-OFF: from thread 1 trace index 0
-  00:06:50:520796: handoffdemo-2
-    HANDOFFDEMO: current thread 2
-  00:06:50:520867: error-drop
-    rx:local0
-  00:06:50:520914: drop
-    handoffdemo-2: completed packets
-
-  Packet 2
-
-  00:06:50:520796: handoff_trace
-    HANDED-OFF: from thread 1 trace index 1
-  00:06:50:520796: handoffdemo-2
-    HANDOFFDEMO: current thread 2
-  00:06:50:520867: error-drop
-    rx:local0
-  00:06:50:520914: drop
-    handoffdemo-2: completed packets
-
-  Packet 3
-
-  00:06:50:520796: handoff_trace
-    HANDED-OFF: from thread 1 trace index 2
-  00:06:50:520796: handoffdemo-2
-    HANDOFFDEMO: current thread 2
-  00:06:50:520867: error-drop
-    rx:local0
-  00:06:50:520914: drop
-    handoffdemo-2: completed packets
-
-  Packet 4
-
-  00:06:50:520796: handoff_trace
-    HANDED-OFF: from thread 1 trace index 3
-  00:06:50:520796: handoffdemo-2
-    HANDOFFDEMO: current thread 2
-  00:06:50:520867: error-drop
-    rx:local0
-  00:06:50:520914: drop
-    handoffdemo-2: completed packets
-
-  Packet 5
-
-  00:06:50:520796: handoff_trace
-    HANDED-OFF: from thread 1 trace index 4
-  00:06:50:520796: handoffdemo-2
-    HANDOFFDEMO: current thread 2
-  00:06:50:520867: error-drop
-    rx:local0
-  00:06:50:520914: drop
-    handoffdemo-2: completed packets
- DBGvpp#
-```
diff --git a/src/examples/handoffdemo/handoffdemo.rst b/src/examples/handoffdemo/handoffdemo.rst
new file mode 100644
index 00000000000..d44854cc5cc
--- /dev/null
+++ b/src/examples/handoffdemo/handoffdemo.rst
@@ -0,0 +1,194 @@
+.. _handoff_queue_demo_plugin:
+
+Handoff queue in a plugin
+=========================
+
+This plugin provides a simplified example of how to hand off packets
+between threads. I used it to debug packet-tracer handoff tracing
+support.
+
+Packet generator input script
+-----------------------------
+
+::
+
+    packet-generator new {
+       name x
+       limit 5
+       size 128-128
+       interface local0
+       node handoffdemo-1
+       data {
+           incrementing 30
+       }
+    }
+
+Start vpp with 2 worker threads
+-------------------------------
+
+The demo plugin hands packets from worker 1 to worker 2.
+
+Enable tracing, and start the packet generator
+----------------------------------------------
+
+::
+
+     trace add pg-input 100
+     packet-generator enable
+
+Sample Run
+----------
+
+::
+
+     DBGvpp# ex /tmp/pg_input_script
+     DBGvpp# pa en
+     DBGvpp# sh err
+      Count                    Node                  Reason
+            5              handoffdemo-1             packets handed off processed
+            5              handoffdemo-2             completed packets
+     DBGvpp# show run
+     Thread 1 vpp_wk_0 (lcore 0)
+     Time 133.9, average vectors/node 5.00, last 128 main loops 0.00 per node 0.00
+       vector rates in 3.7331e-2, out 0.0000e0, drop 0.0000e0, punt 0.0000e0
+                  Name                 State         Calls          Vectors        Suspends         Clocks       Vectors/Call
+     handoffdemo-1                    active                  1               5               0          4.76e3            5.00
+     pg-input                        disabled                 2               5               0          5.58e4            2.50
+     unix-epoll-input                 polling             22760               0               0          2.14e7            0.00
+     ---------------
+     Thread 2 vpp_wk_1 (lcore 2)
+     Time 133.9, average vectors/node 5.00, last 128 main loops 0.00 per node 0.00
+       vector rates in 0.0000e0, out 0.0000e0, drop 3.7331e-2, punt 0.0000e0
+                  Name                 State         Calls          Vectors        Suspends         Clocks       Vectors/Call
+     drop                             active                  1               5               0          1.35e4            5.00
+     error-drop                       active                  1               5               0          2.52e4            5.00
+     handoffdemo-2                    active                  1               5               0          2.56e4            5.00
+     unix-epoll-input                 polling             22406               0               0          2.18e7            0.00
+
+Enable the packet tracer and run it again…
+
+::
+
+     DBGvpp# trace add pg-input 100
+     DBGvpp# pa en
+     DBGvpp# sh trace
+     sh trace
+     ------------------- Start of thread 0 vpp_main -------------------
+     No packets in trace buffer
+     ------------------- Start of thread 1 vpp_wk_0 -------------------
+     Packet 1
+
+     00:06:50:520688: pg-input
+       stream x, 128 bytes, 0 sw_if_index
+       current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000000
+       00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
+       00000020: 0000000000000000000000000000000000000000000000000000000000000000
+       00000040: 0000000000000000000000000000000000000000000000000000000000000000
+       00000060: 0000000000000000000000000000000000000000000000000000000000000000
+     00:06:50:520762: handoffdemo-1
+       HANDOFFDEMO: current thread 1
+
+     Packet 2
+
+     00:06:50:520688: pg-input
+       stream x, 128 bytes, 0 sw_if_index
+       current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000001
+       00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
+       00000020: 0000000000000000000000000000000000000000000000000000000000000000
+       00000040: 0000000000000000000000000000000000000000000000000000000000000000
+       00000060: 0000000000000000000000000000000000000000000000000000000000000000
+     00:06:50:520762: handoffdemo-1
+       HANDOFFDEMO: current thread 1
+
+     Packet 3
+
+     00:06:50:520688: pg-input
+       stream x, 128 bytes, 0 sw_if_index
+       current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000002
+       00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
+       00000020: 0000000000000000000000000000000000000000000000000000000000000000
+       00000040: 0000000000000000000000000000000000000000000000000000000000000000
+       00000060: 0000000000000000000000000000000000000000000000000000000000000000
+     00:06:50:520762: handoffdemo-1
+       HANDOFFDEMO: current thread 1
+
+     Packet 4
+
+     00:06:50:520688: pg-input
+       stream x, 128 bytes, 0 sw_if_index
+       current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000003
+       00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
+       00000020: 0000000000000000000000000000000000000000000000000000000000000000
+       00000040: 0000000000000000000000000000000000000000000000000000000000000000
+       00000060: 0000000000000000000000000000000000000000000000000000000000000000
+     00:06:50:520762: handoffdemo-1
+       HANDOFFDEMO: current thread 1
+
+     Packet 5
+
+     00:06:50:520688: pg-input
+       stream x, 128 bytes, 0 sw_if_index
+       current data 0, length 128, buffer-pool 0, ref-count 1, trace handle 0x1000004
+       00000000: 000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d0000
+       00000020: 0000000000000000000000000000000000000000000000000000000000000000
+       00000040: 0000000000000000000000000000000000000000000000000000000000000000
+       00000060: 0000000000000000000000000000000000000000000000000000000000000000
+     00:06:50:520762: handoffdemo-1
+       HANDOFFDEMO: current thread 1
+
+     ------------------- Start of thread 2 vpp_wk_1 -------------------
+     Packet 1
+
+     00:06:50:520796: handoff_trace
+       HANDED-OFF: from thread 1 trace index 0
+     00:06:50:520796: handoffdemo-2
+       HANDOFFDEMO: current thread 2
+     00:06:50:520867: error-drop
+       rx:local0
+     00:06:50:520914: drop
+       handoffdemo-2: completed packets
+
+     Packet 2
+
+     00:06:50:520796: handoff_trace
+       HANDED-OFF: from thread 1 trace index 1
+     00:06:50:520796: handoffdemo-2
+       HANDOFFDEMO: current thread 2
+     00:06:50:520867: error-drop
+       rx:local0
+     00:06:50:520914: drop
+       handoffdemo-2: completed packets
+
+     Packet 3
+
+     00:06:50:520796: handoff_trace
+       HANDED-OFF: from thread 1 trace index 2
+     00:06:50:520796: handoffdemo-2
+       HANDOFFDEMO: current thread 2
+     00:06:50:520867: error-drop
+       rx:local0
+     00:06:50:520914: drop
+       handoffdemo-2: completed packets
+
+     Packet 4
+
+     00:06:50:520796: handoff_trace
+       HANDED-OFF: from thread 1 trace index 3
+     00:06:50:520796: handoffdemo-2
+       HANDOFFDEMO: current thread 2
+     00:06:50:520867: error-drop
+       rx:local0
+     00:06:50:520914: drop
+       handoffdemo-2: completed packets
+
+     Packet 5
+
+     00:06:50:520796: handoff_trace
+       HANDED-OFF: from thread 1 trace index 4
+     00:06:50:520796: handoffdemo-2
+       HANDOFFDEMO: current thread 2
+     00:06:50:520867: error-drop
+       rx:local0
+     00:06:50:520914: drop
+       handoffdemo-2: completed packets
+    DBGvpp#
diff --git a/src/examples/sample-plugin/sample_plugin_doc.md b/src/examples/sample-plugin/sample_plugin_doc.md
deleted file mode 100644
index 9aaaefa0061..00000000000
--- a/src/examples/sample-plugin/sample_plugin_doc.md
+++ /dev/null
@@ -1,66 +0,0 @@
-# Sample plugin for VPP    {#sample_plugin_doc}
-
-## Overview
-
-This is the VPP sample plugin demonstrates how to create a new plugin that integrates 
-with VPP. The sample code implements a trival macswap algorithim that demonstrates plugin 
-runtime integration with the VPP graph hierachy, api and cli.
-
-For deeper dive information see the annotations in the  sample code itself. See [sample.c](@ref sample.c)
-
-## How to build and run the sample plugin.
-
-Now (re)build VPP.
-
-	$ make wipe
-
-Define environmental variable 'SAMPLE_PLUGIN=yes' with a process scope
-
-	$ SAMPLE_PLUGIN=yes make build
-
-or a session scope, and build VPP. 
-
-	$ export SAMPLE_PLUGIN=yes
-	$ make build
-
-Now run VPP and make sure the plugin is loaded. 
-
-	$ make run
-	...
-	load_one_plugin:184: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimetal))
-	load_one_plugin:184: Loaded plugin: sample_plugin.so (Sample of VPP Plugin)
-	load_one_plugin:184: Loaded plugin: nat_plugin.so (Network Address Translation)
-	...
-	DBGvpp#
-
-## How to create a new plugin
-
-To create a new plugin based on the sample plugin, copy and rename the sample plugin directory and automake config.
-
-	cp -r src/examples/sample-plugin/sample src/plugins/newplugin
-	cp src/examples/sample-plugin/sample.am src/plugins/newplugin.am
-
-Add the following entry to the plugins section of `src/configure.ac`.
-
-	PLUGIN_ENABLED(newplugin)
-
-Add the following entry to the plugins section of `src/plugins/Makefile.am`
-
-	if ENABLE_NEWPLUGIN
-	include newplugin.am
-	endif
-
-Now (re)build VPP.
-
-	$ make wipe
-	$ make build
-
-## Configuration
-
-To enable the sample plugin
-
-	sample macswap <interface name>
-
-To disable the sample plugin
-
-	sample macswap <interface name> disable
diff --git a/src/examples/sample-plugin/sample_plugin_doc.rst b/src/examples/sample-plugin/sample_plugin_doc.rst
new file mode 100644
index 00000000000..23023e21bfb
--- /dev/null
+++ b/src/examples/sample-plugin/sample_plugin_doc.rst
@@ -0,0 +1,97 @@
+.. _sample_plugin_doc:
+
+Sample plugin for VPP
+=====================
+
+Overview
+--------
+
+This is the VPP sample plugin demonstrates how to create a new plugin
+that integrates with VPP. The sample code implements a trivial macswap
+algorithm that demonstrates plugin runtime integration with the VPP
+graph hierarchy, api and cli.
+
+For deeper dive information see the annotations in the sample code
+itself. See `sample.c <@ref%20sample.c>`__
+
+How to build and run the sample plugin.
+---------------------------------------
+
+Now (re)build VPP.
+
+::
+
+   $ make wipe
+
+Define environmental variable ‘SAMPLE_PLUGIN=yes’ with a process scope
+
+::
+
+   $ SAMPLE_PLUGIN=yes make build
+
+or a session scope, and build VPP.
+
+::
+
+   $ export SAMPLE_PLUGIN=yes
+   $ make build
+
+Now run VPP and make sure the plugin is loaded.
+
+::
+
+   $ make run
+   ...
+   load_one_plugin:184: Loaded plugin: memif_plugin.so (Packet Memory Interface (experimetal))
+   load_one_plugin:184: Loaded plugin: sample_plugin.so (Sample of VPP Plugin)
+   load_one_plugin:184: Loaded plugin: nat_plugin.so (Network Address Translation)
+   ...
+   DBGvpp#
+
+How to create a new plugin
+--------------------------
+
+To create a new plugin based on the sample plugin, copy and rename the
+sample plugin directory and automake config.
+
+::
+
+   cp -r src/examples/sample-plugin/sample src/plugins/newplugin
+   cp src/examples/sample-plugin/sample.am src/plugins/newplugin.am
+
+Add the following entry to the plugins section of ``src/configure.ac``.
+
+::
+
+   PLUGIN_ENABLED(newplugin)
+
+Add the following entry to the plugins section of
+``src/plugins/Makefile.am``
+
+::
+
+   if ENABLE_NEWPLUGIN
+   include newplugin.am
+   endif
+
+Now (re)build VPP.
+
+::
+
+   $ make wipe
+   $ make build
+
+Configuration
+-------------
+
+To enable the sample plugin
+
+::
+
+   sample macswap <interface name>
+
+To disable the sample plugin
+
+::
+
+   sample macswap <interface name> disable
diff --git a/src/examples/srv6-sample-localsid/srv6_sample_localsid_doc.md b/src/examples/srv6-sample-localsid/srv6_sample_localsid_doc.md
deleted file mode 100644
index cd717db8135..00000000000
--- a/src/examples/srv6-sample-localsid/srv6_sample_localsid_doc.md
+++ /dev/null
@@ -1,30 +0,0 @@
-# Sample SRv6 LocalSID documentation    {#srv6_plugin_doc}
-
-## Introduction
-
-This plugin is an example of how an user can create a new SRv6 LocalSID behavior by using VPP plugins with the appropiate API calls to the existing SR code.
-
-This **example** plugin registers a new localsid behavior, with cli keyword 'new_srv6_localsid' which only takes one parameter, a fib-table. Upon recival of a packet, this plugin will enforce the next IP6 lookup in the specific fib-table specified by the user. (Indeed it will do the lookup in the fib_table n+1 (since for the shake of the example we increment the fib-table.)
-
-Notice that the plugin only 'defines' a new SRv6 LocalSID behavior, but the existing SR code in VNET is the one actually instantiating new LocalSIDs. Notice that there are callback functions such that when you create or remove a LocalSID you can actually setup specific parameters through the functions in this plugin.
-
-## Variables to watch for
-
-* srv6_localsid_name: This variable is the name (used as a unique key) identifying this SR LocalSID plugin.
-* keyword_str: This is the CLI keyword to be used for the plugin. In this example 'new_srv6_localsid'. (i.e. sr localsid address cafe::1 behavior new_srv6_localsid <parameters>)
-* def_str: This is a definition of this SR behavior. This is printed when you do 'show sr localsid behaviors'.
-* params_str: This is a definition of the parameters of this localsid. This is printed when you do 'show sr localsid behaviors'.
-
-## Functions to watch for
-
-* srv6_localsid_creation_fn: This function will be called every time a new SR LocalSID is instantiated with the behavior defined in this plugin. 
-* srv6_localsid_removal_fn: This function will be called every time a new SR LocalSID is removed with the behavior defined in this plugin. This function tends to be used for freeing up all the memory created in the previous function.
-* format_srv6_localsid_sample: This function prints nicely the parameters of every SR LocalSID using this behavior.
-* unformat_srv6_localsid_sample: This function parses the CLI command when initialising a new SR LocalSID using this behavior. It parses all the parameters and ensures that the parameters are correct.
-* format_srv6_localsid_sample_dpo: This function formats the 'show ip6 fib' message for the SR LocalSIDs created with this plugin behavior.
-
-## Graph node
-
-The current graph node uses the function 'end_srh_processing' to do the Segment Routing Endpoint behavior. Notice that it does not allow the cleanup of a Segment Routing header (as per the SRv6 behavior specs).
-This function is identical to the one found in /src/vnet/srv6/sr_localsid.c
-In case that by some other reason you want to do decapsulation, or SRH clean_up you can use the functions 'end_decaps_srh_processing' or 'end_psp_srh_processing' respectively.
diff --git a/src/examples/srv6-sample-localsid/srv6_sample_localsid_doc.rst b/src/examples/srv6-sample-localsid/srv6_sample_localsid_doc.rst
new file mode 100644
index 00000000000..a076cd2a6c7
--- /dev/null
+++ b/src/examples/srv6-sample-localsid/srv6_sample_localsid_doc.rst
@@ -0,0 +1,66 @@
+.. _srv6_plugin_doc:
+
+Sample SRv6 LocalSID documentation
+==================================
+
+Introduction
+------------
+
+This plugin is an example of how an user can create a new SRv6 LocalSID
+behavior by using VPP plugins with the appropriate API calls to the
+existing SR code.
+
+This **example** plugin registers a new localsid behavior, with cli
+keyword ‘new_srv6_localsid’ which only takes one parameter, a fib-table.
+Upon receival of a packet, this plugin will enforce the next IP6 lookup
+in the specific fib-table specified by the user. (Indeed it will do the
+lookup in the fib_table n+1 (since for the shake of the example we
+increment the fib-table.)
+
+Notice that the plugin only ‘defines’ a new SRv6 LocalSID behavior, but
+the existing SR code in VNET is the one actually instantiating new
+LocalSIDs. Notice that there are callback functions such that when you
+create or remove a LocalSID you can actually setup specific parameters
+through the functions in this plugin.
+
+Variables to watch for
+----------------------
+
+-  srv6_localsid_name: This variable is the name (used as a unique key)
+   identifying this SR LocalSID plugin.
+-  keyword_str: This is the CLI keyword to be used for the plugin. In
+   this example ‘new_srv6_localsid’. (i.e. sr localsid address cafe::1
+   behavior new_srv6_localsid )
+-  def_str: This is a definition of this SR behavior. This is printed
+   when you do ‘show sr localsid behaviors’.
+-  params_str: This is a definition of the parameters of this localsid.
+   This is printed when you do ‘show sr localsid behaviors’.
+
+Functions to watch for
+----------------------
+
+-  srv6_localsid_creation_fn: This function will be called every time a
+   new SR LocalSID is instantiated with the behavior defined in this
+   plugin.
+-  srv6_localsid_removal_fn: This function will be called every time a
+   new SR LocalSID is removed with the behavior defined in this plugin.
+   This function tends to be used for freeing up all the memory created
+   in the previous function.
+-  format_srv6_localsid_sample: This function prints nicely the
+   parameters of every SR LocalSID using this behavior.
+-  unformat_srv6_localsid_sample: This function parses the CLI command
+   when initializing a new SR LocalSID using this behavior. It parses
+   all the parameters and ensures that the parameters are correct.
+-  format_srv6_localsid_sample_dpo: This function formats the ‘show ip6
+   fib’ message for the SR LocalSIDs created with this plugin behavior.
+
+Graph node
+----------
+
+The current graph node uses the function ‘end_srh_processing’ to do the
+Segment Routing Endpoint behavior. Notice that it does not allow the
+cleanup of a Segment Routing header (as per the SRv6 behavior specs).
+This function is identical to the one found in
+/src/vnet/srv6/sr_localsid.c In case that by some other reason you want
+to do decapsulation, or SRH clean_up you can use the functions
+‘end_decaps_srh_processing’ or ‘end_psp_srh_processing’ respectively.