docs: better docs, mv doxygen to sphinx

This patch refactors the VPP sphinx docs
in order to make it easier to consume
for external readers as well as VPP developers.

It also makes sphinx the single source
of documentation, which simplifies maintenance
and operation.

Most important updates are:

- reformat the existing documentation as rst
- split RELEASE.md and move it into separate rst files
- remove section 'events'
- remove section 'archive'
- remove section 'related projects'
- remove section 'feature by release'
- remove section 'Various links'
- make (Configuration reference, CLI docs,
  developer docs) top level items in the list
- move 'Use Cases' as part of 'About VPP'
- move 'Troubleshooting' as part of 'Getting Started'
- move test framework docs into 'Developer Documentation'
- add a 'Contributing' section for gerrit,
  docs and other contributer related infos
- deprecate doxygen and test-docs targets
- redirect the "make doxygen" target to "make docs"

Type: refactor

Change-Id: I552a5645d5b7964d547f99b1336e2ac24e7c209f
Signed-off-by: Nathan Skrzypczak <nathan.skrzypczak@gmail.com>
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>

4 files changed, 540 insertions, 0 deletions

diff --git a/docs/contributing/gitreview.rst b/docs/contributing/gitreview.rst
new file mode 100644
index 00000000000..b2608d11812
--- /dev/null
+++ b/docs/contributing/gitreview.rst
@@ -0,0 +1,199 @@
+.. _gitreview:
+
+*******************************
+Getting a Patch Reviewed
+*******************************
+
+This section describes how to get FD.io VPP sources reviewed and merged.
+
+Setup
+========
+
+If you don't have a Linux Foundation ID, `create one here. <https://identity.linuxfoundation.org/>`_
+
+With your Linux Foundation ID credentials sign into `Gerrit Code Review at gerrit.fd.io <https://gerrit.fd.io/r/login/%23%2Fq%2Fstatus%3Aopen>`_
+
+`Install git-review, <https://www.mediawiki.org/wiki/Gerrit/git-review>`_ which is a "command-line tool for Git / Gerrit to submit a change or to fetch an existing one."
+
+If you're on Ubuntu, install keychain:
+
+.. code-block:: console
+
+    $ sudo apt-get install keychain
+
+ssh keys
+-------------
+
+To get FD.io VPP documents reviewed the VPP repository should be cloned with ssh. You should be logged into Gerrit Code Review as noted above.
+
+Create your public and private ssh key with:
+
+.. code-block:: console
+
+    $ ssh-keygen -t rsa
+    $ keychain
+    $ cat ~/.ssh/id_rsa.pub
+
+Copy **all** the contents of the public key (id_rsa.pub) output by the above **cat** command. Then go to your `SSH Public keys settings page <https://gerrit.fd.io/r/#/settings/ssh-keys>`_, click **Add Key ...**, paste your public key, and finally click **Add**.
+
+.. _clone-ssh:
+
+Clone with ssh
+==============
+
+Clone the repo with:
+
+.. code-block:: console
+
+    $ git clone ssh://gerrit.fd.io:29418/vpp
+    $ cd vpp
+
+This will only work if the name of the user on your system matches your Gerrit username.
+
+Otherwise, clone with:
+
+.. code-block:: console
+
+    $ git clone ssh://<YOUR_GERRIT_USERNAME>@gerrit.fd.io:29418/vpp
+    $ cd vpp
+
+When attempting to clone the repo Git will prompt you asking if you want to add the Server Host Key to the list of known hosts. Enter **yes** and press the **Enter** key.
+
+Git Review
+===========
+
+The VPP documents use the gerrit server, and git review for submitting and fetching patches.
+
+
+New patch
+-----------------
+
+When working with a new patch, use the following commands to get your patch reviewed.
+
+Make sure you have modified the correct files by issuing the following commands:
+
+.. code-block:: console
+
+    $ git status
+    $ git diff
+
+Then add and commit the patch. You may want to add a tag to the commit comments.
+For example for a document with only patches you should add the tag **docs:**.
+
+.. code-block:: console
+
+    $ git add <filename>
+    $ git commit -s
+
+The commit comment should have something like the following comment:
+
+.. code-block:: console
+
+   docs: A brief description of the commit
+
+   Type: Improvement (The type of commit this could be: Improvement, Fix or Feature)
+
+   A detailed description of the commit could go here.
+
+Push the patch for review.
+
+.. code-block:: console
+
+    $ git review
+
+If you are creating a draft, meaning you do not want your changes reviewed yet, do the following:
+
+.. code-block:: console
+
+    $ git review -D
+
+After submitting a review, reset where the HEAD is pointing to with:
+
+.. code-block:: console
+
+    $ git reset --hard origin/master
+
+Existing patch
+-----------------------
+
+The "change number" used below is in the URL of the review.
+
+After clicking an individual review, the change number can be found in the URL at "https://gerrit.fd.io/r/#/c/<*CHANGE_NUMBER*>/"
+
+To view an existing patch:
+
+.. code-block:: console
+
+    $ git review -d <change number>
+    $ git status
+    $ git diff
+
+.. caution::
+
+    If you have made changes and do "git review -d <change number>", your current
+    changes will try to be stashed so that the working tree can change to the review branch
+    you specified. If you want to make sure you don't lose your changes, clone another Gerrit
+    repo into a new directory using the cloning steps shown in :ref:`clone-ssh`, and perform
+    "git review -d <change number>" in this new directory.
+
+To modify an existing patch, make sure you modified the correct files, and apply the patch with:
+
+.. code-block:: console
+
+    $ git review -d <change number>
+    $ git status
+    $ git diff
+
+    $ git add <filename>
+    $ git commit --amend
+    $ git review
+
+When you're done viewing or modifying a branch, get back to the master branch by entering:
+
+.. code-block:: console
+
+    $ git reset --hard origin/master
+    $ git checkout master
+
+Patch Conflict Resolution
+-------------------------
+
+Two different patch conflict scenarios arise from time to
+time. Sometime after uploading a patch to https://gerrit.fd.io, the
+gerrit UI may show a patch status of "Merge Conflict."
+
+Or, you may attempt to upload a new patch-set via "git review," only to
+discover that the gerrit server won't allow the upload due to an upstream
+merge conflict.
+
+In both cases, it's [usually] fairly simple to fix the problem. You
+need to rebase the patch onto master/latest. Details vary from case to
+case.
+
+Here's how to rebase a patch previously uploaded to the Gerrit server
+which now has a merge conflict. In a fresh workspace cloned from
+master/latest, do the following:
+
+.. code-block:: console
+
+    $ git-review -d <*Gerrit change #*>
+    $ git rebase origin/master
+       while (conflicts)
+          <fix conflicts>
+          $ git rebase --continue
+    $ git review
+
+In the upload-failure case, use caution: carefully **save your work**
+before you do anything else!
+
+Rebase your patch and try again. Please **do not** re-download ["git
+review -d"] the patch from the gerrit server...:
+
+.. code-block:: console
+
+    $ git rebase origin/master
+       while (conflicts)
+          <fix conflicts>
+          $ git rebase --continue
+    $ git review
+
diff --git a/docs/contributing/reportingissues/index.rst b/docs/contributing/reportingissues/index.rst
new file mode 100644
index 00000000000..b16a2a915a8
--- /dev/null
+++ b/docs/contributing/reportingissues/index.rst
@@ -0,0 +1,23 @@
+.. _reportingissues:
+
+Reporting Bugs
+==============
+
+Although every situation is different, this section describes how to
+collect data which will help make efficient use of everyone's time
+when dealing with vpp bugs.
+
+Before you press the Jira button to create a bug report - or email
+vpp-dev@lists.fd.io - please ask yourself whether there's enough
+information for someone else to understand and to reproduce the issue
+given a reasonable amount of effort. **Unicast emails to maintainers,
+committers, and the project PTL are strongly discouraged.**
+
+A good strategy for clear-cut bugs: file a detailed Jira ticket, and
+then send a short description of the issue to vpp-dev@lists.fd.io,
+perhaps from the Jira ticket description. It's fine to send email to
+vpp-dev@lists.fd.io to ask a few questions **before** filing Jira tickets.
+
+.. toctree::
+
+  reportingissues
diff --git a/docs/contributing/reportingissues/reportingissues.rst b/docs/contributing/reportingissues/reportingissues.rst
new file mode 100644
index 00000000000..970f3407c54
--- /dev/null
+++ b/docs/contributing/reportingissues/reportingissues.rst
@@ -0,0 +1,266 @@
+.. _reportingbugs:
+
+.. toctree::
+
+Data to include in bug reports
+==============================
+
+Image version and operating environment
+---------------------------------------
+
+Please make sure to include the vpp image version and command-line arguments.
+
+.. code-block:: console
+
+    $ sudo bash
+    # vppctl show version verbose cmdline
+    Version:                  v18.07-rc0~509-gb9124828
+    Compiled by:              vppuser
+    Compile host:             vppbuild
+    Compile date:             Fri Jul 13 09:05:37 EDT 2018
+    Compile location:         /scratch/vpp-showversion
+    Compiler:                 GCC 7.3.0
+    Current PID:              5211
+    Command line arguments:
+      /scratch/vpp-showversion/build-root/install-vpp_debug-native/vpp/bin/vpp
+      unix
+      interactive
+
+With respect to the operating environment: if misbehavior involving a
+specific VM / container / bare-metal environment is involved, please
+describe the environment in detail:
+
+* Linux Distro (e.g. Ubuntu 18.04.2 LTS, CentOS-7, etc.)
+* NIC type(s) (ixgbe, i40e, enic, etc. etc.), vhost-user, tuntap
+* NUMA configuration if applicable
+
+Please note the CPU architecture (x86_86, aarch64), and hardware platform.
+
+When practicable, please report issues against released software, or
+unmodified master/latest software.
+
+"Show" command output
+---------------------
+
+Every situation is different. If the issue involves a sequence of
+debug CLI command, please enable CLI command logging, and send the
+sequence involved. Note that the debug CLI is a developer's tool -
+**no warranty express or implied** - and that we may choose not to fix
+debug CLI bugs.
+
+Please include "show error" [error counter] output. It's often helpful
+to "clear error", send a bit of traffic, then "show error"
+particularly when running vpp on noisy networks.
+
+Please include ip4 / ip6 / mpls FIB contents ("show ip fib", "show ip6
+fib", "show mpls fib", "show mpls tunnel").
+
+Please include "show hardware", "show interface", and "show interface
+address" output
+
+Here is a consolidated set of commands that are generally useful
+before/after sending traffic.  Before sending traffic:
+
+.. code-block:: console
+
+    vppctl clear hardware
+    vppctl clear interface
+    vppctl clear error
+    vppctl clear run
+
+Send some traffic and then issue the following commands.
+
+.. code-block:: console
+
+    vppctl show version verbose
+    vppctl show hardware
+    vppctl show interface address
+    vppctl show interface
+    vppctl show run
+    vppctl show error
+
+Here are some protocol specific show commands that may also make
+sense.  Only include those features which have been configured.
+
+.. code-block:: console
+
+     vppctl show l2fib
+     vppctl show bridge-domain
+
+     vppctl show ip fib
+     vppctl show ip neighbors
+
+     vppctl show ip6 fib
+     vppctl show ip6 neighbors
+
+     vppctl show mpls fib
+     vppctl show mpls tunnel
+
+Network Topology
+----------------
+
+Please include a crisp description of the network topology, including
+L2 / IP / MPLS / segment-routing addressing details. If you expect
+folks to reproduce and debug issues, this is a must.
+
+At or above a certain level of topological complexity, it becomes
+problematic to reproduce the original setup.
+
+Packet Tracer Output
+--------------------
+
+If you capture packet tracer output which seems relevant, please include it.
+
+.. code-block:: console
+
+    vppctl trace add dpdk-input 100  # or similar
+
+send-traffic
+
+.. code-block:: console
+
+    vppctl show trace
+
+Capturing post-mortem data
+==========================
+
+It should go without saying, but anyhow: **please put post-mortem data
+in obvious, accessible places.** Time wasted trying to acquire
+accounts, credentials, and IP addresses simply delays problem
+resolution.
+
+Please remember to add post-mortem data location information to Jira
+tickets.
+
+Syslog Output
+-------------
+
+The vpp signal handler typically writes a certain amount of data in
+/var/log/syslog before exiting. Make sure to check for evidence, e.g
+via "grep /usr/bin/vpp /var/log/syslog" or similar.
+
+Binary API Trace
+----------------
+
+If the issue involves a sequence of control-plane API messages - even
+a very long sequence - please enable control-plane API
+tracing. Control-plane API post-mortem traces end up in
+/tmp/api_post_mortem.<pid>.
+
+Please remember to put post-mortem binary api traces in accessible
+places.
+
+These API traces are especially helpful in cases where the vpp engine
+is throwing traffic on the floor, e.g. for want of a default route or
+similar.
+
+Make sure to leave the default stanza "... api-trace { on } ... " in
+the vpp startup configuration file /etc/vpp/startup.conf, or to
+include it in the command line arguments passed by orchestration
+software.
+
+Core Files
+----------
+
+Production systems, as well as long-running pre-production soak-test
+systems, **must** arrange to collect core images. There are various
+ways to configure core image capture, including e.g. the Ubuntu
+"corekeeper" package. In a pinch, the following very basic sequence
+will capture usable vpp core files in /tmp/dumps.
+
+.. code-block:: console
+
+    # mkdir -p /tmp/dumps
+    # sysctl -w debug.exception-trace=1
+    # sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
+    # ulimit -c unlimited
+    # echo 2 > /proc/sys/fs/suid_dumpable
+
+If you start VPP from systemd, you also need to edit
+/lib/systemd/system/vpp.service and uncomment the "LimitCORE=infinity"
+line before restarting VPP.
+
+Vpp core files often appear enormous, but they are invariably
+sparse. Gzip compresses them to manageable sizes. A multi-GByte
+corefile often compresses to 10-20 Mbytes.
+
+When decompressing a vpp core file, we suggest using "dd" as shown to
+create a sparse, uncompressed core file:
+
+.. code-block:: console
+
+   $ zcat vpp_core.gz | dd conv=sparse of=vpp_core
+
+Please remember to put compressed core files in accessible places.
+
+Make sure to leave the default stanza "... unix { ... full-coredump
+... } ... " in the vpp startup configuration file
+/etc/vpp/startup.conf, or to include it in the command line arguments
+passed by orchestration software.
+
+Core files from Private Images
+==============================
+
+Core files from private images require special handling. If it's
+necessary to go that route, copy the **exact** Debian packages (or
+RPMs) which correspond to the core file to the same public place as
+the core file. A no-excuses-allowed, hard-and-fast requirement.
+
+In particular:
+
+.. code-block:: console
+
+  libvppinfra_<version>_<arch>.deb # vppinfra library
+  libvppinfra-dev_<version>_<arch>.deb # vppinfra library development pkg
+  vpp_<version>_<arch>.deb         # the vpp executable
+  vpp-dbg_<version>_<arch>.deb     # debug symbols
+  vpp-dev_<version>_<arch>.deb     # vpp development pkg
+  vpp-lib_<version>_<arch>.deb     # shared libraries
+  vpp-plugin-core_<version>_<arch>.deb # core plugins
+  vpp-plugin-dpdk_<version>_<arch>.deb # dpdk plugin
+
+For reference, please include git commit-ID, branch, and git repo
+information [for repos other than gerrit.fd.io] in the Jira ticket.
+
+Note that git commit-ids are crypto sums of the head [latest]
+**merged** patch. They say **nothing whatsoever** about local
+workspace modifications, branching, or the git repo in question.
+
+Even given a byte-for-byte identical source tree, it's easy to build
+dramatically different binary artifacts. All it takes is a different
+toolchain version.
+
+
+On-the-fly Core File Compression
+--------------------------------
+
+Depending on operational requirements, it's possible to compress
+corefiles as they are generated. Please note that it takes several
+seconds' worth of wall-clock time to compress a vpp core file on the
+fly, during which all packet processing activities are suspended.
+
+To create compressed core files on the fly, create the following
+script, e.g. in /usr/local/bin/compressed_corefiles, owned by root,
+executable:
+
+.. code-block:: console
+
+  #!/bin/sh
+  exec /bin/gzip -f - >"/tmp/dumps/core-$1.$2.gz"
+
+Adjust the kernel core file pattern as shown:
+
+.. code-block:: console
+
+  sysctl -w kernel.core_pattern="|/usr/local/bin/compressed_corefiles %e %t"
+
+Core File Summary
+-----------------
+
+Bottom line: please follow core file handling instructions to the
+letter. It's not complicated. Simply copy the exact Debian packages or
+RPMs which correspond to core files to accessible locations.
+
+If we go through the setup process only to discover that the image and
+core files don't match, it will simply delay resolution of the issue;
+to say nothing of irritating the person who just wasted their time.
diff --git a/docs/contributing/writingdocs.rst b/docs/contributing/writingdocs.rst
new file mode 100644
index 00000000000..ba3713045cc
--- /dev/null
+++ b/docs/contributing/writingdocs.rst
@@ -0,0 +1,52 @@
+.. _buildingrst:
+
+Writing VPP Documentation
+=========================
+
+These instructions show how the VPP documentation sources are built.
+
+The VPP Documents are written using `reStructuredText <http://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html>`_ (rst),
+or markdown (md). These files are then built using the Sphinx build system `Sphinx <http://www.sphinx-doc.org/en/master/>`_.
+
+Building the docs
+-----------------
+
+Start with a clone of the vpp repository.
+
+.. code-block:: console
+
+   $ git clone https://gerrit.fd.io/r/vpp
+   $ cd vpp
+
+Build the html **index.html** file:
+
+.. code-block:: console
+
+   $ make docs
+
+Delete all the generated files with the following:
+
+.. code-block:: console
+
+   $ make docs-clean
+
+View the results
+----------------
+
+If there are no errors during the build process, you should now have an ``index.html`` file in your ``vpp/docs/_build/html`` directory, which you can then view in your browser.
+
+Whenever you make changes to your ``.rst`` files that you want to see, repeat this build process.
+
+Writing Docs and merging
+------------------------
+
+Documentation should be added as ``.rst`` file in the ``./src/`` tree next to the code it refers to. A symlink should be added at the relevant place in the ``./docs`` folder and a link in the appropriate place in the tree.
+
+To ensure documentation is correctly inserted, you can run
+
+.. code-block:: console
+
+   $ ./extras/scripts/check_documentation.sh
+
+VPP documents are reviewed and merged like and other source code. Refer to :ref:`gitreview`
+to get your changes reviewed and merged.