Refactor pre-Doxy siphon scripts; VPP-396

- Modularize the code to make the Siphon process easier to
  maintain.
- Move much of the output rendering into Jinja2 templates.
- Add syscfg siphon type for startup config documentation.
- Add sample syscfg documentation.
- Add clicfg and syscfg preamble docs, adapted from their wiki pages.
- Fix sorting of CLI items across multiple directories.

Change-Id: Ib8288fe005adfea68ceed75a38ff8eba25d3cc79
Signed-off-by: Chris Luke <chrisy@flirble.org>

9 files changed, 427 insertions, 0 deletions

diff --git a/doxygen/siphon_templates/clicmd/index_entry.md b/doxygen/siphon_templates/clicmd/index_entry.md
new file mode 100644
index 00000000000..1fa9ec9b25a
--- /dev/null
+++ b/doxygen/siphon_templates/clicmd/index_entry.md
@@ -0,0 +1,17 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+{% set v = item['value'] %}
+{{ "* [%s](@ref %s)" % (v['path'], meta["label"]) }}
diff --git a/doxygen/siphon_templates/clicmd/index_header.md b/doxygen/siphon_templates/clicmd/index_header.md
new file mode 100644
index 00000000000..4167f4dc477
--- /dev/null
+++ b/doxygen/siphon_templates/clicmd/index_header.md
@@ -0,0 +1,130 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+# Debug CLI    {{'{#'}}clicmd}
+
+The VPP network stack comes equipped with a set of commands that are useful
+for debugging.
+
+The easiest way to access the CLI (with proper permissions) is to use the
+vppctl command:
+
+```
+sudo vppctl <cli-command>
+```
+
+The CLI parser matches static keyword strings, eventually invoking an action
+function. Unambiguous partial keyword matching always occurs. The action
+functions consume input until satisfied or until they fail. This model makes
+for easy coding, but does not guarantee useful "help" output. It's up to the
+CLI command writer to add useful help strings.
+
+You can find the source code of CLI commands by searching for instances of the
+@c VLIB_CLI_COMMAND macro in the code source files.
+
+Please help maintain and improve this document to make and keep these commands
+clear and useful!
+
+@todo Document where to modify this CLI intro text.
+
+
+## Debug and Telnet CLI
+
+The debug CLI is enabled with the unix interactive parameter or startup
+configuration option. This causes VPP to start without daemonizing and
+presents a command line interface on the terminal where it is run.
+
+The Telnet CLI is enabled with the `cli-listen localhost:5002` option which
+will cause VPP to listen for TCP connections on the localhost address port
+@c 5002. A Telnet client can then connect to this port (for example, `telnet
+localhost 5002`) and will receive a command line prompt.
+
+This configuration will enable both mechanisms:
+
+```
+unix {
+  interactive
+  cli-listen localhost:5002
+}
+```
+
+The debug CLI can operate in line mode, which may be useful when running
+inside an IDE like Emacs. This is enabled with the option
+`unix cli-line-mode`. Several other options exist that alter how this
+CLI works, see the @ref syscfg section for details.
+
+The CLI starts with a banner graphic (which can be disabled) and a prompt. The
+prompt will typically read `vpp` for a release version of VPP and `DBGvpp#`
+for a development version with debugging enabled, for example:
+
+        _______    _        _   _____  ___ 
+     __/ __/ _ \  (_)__    | | / / _ \/ _ \
+     _/ _// // / / / _ \   | |/ / ___/ ___/
+     /_/ /____(_)_/\___/   |___/_/  /_/    
+    
+    vpp# 
+
+versus:
+
+        _______    _        _   _____  ___ 
+     __/ __/ _ \  (_)__    | | / / _ \/ _ \
+     _/ _// // / / / _ \   | |/ / ___/ ___/
+     /_/ /____(_)_/\___/   |___/_/  /_/    
+    
+    DBGvpp# 
+
+This prompt can be configured with the `unix cli-prompt` setting and the
+banner is disabled with `unix cli-no-banner`.
+
+## CLI features
+
+The CLI has several editing features that make it easy to use.
+
+- Cursor keys left/right will move the cursor within a command line;
+  typing will insert at the cursor; erase will erase at the cursor.
+
+- Ctrl-left/right will search for the start of the next word to
+  the left or right.
+- Home/end will jump the cursor to the start and end of the line.
+- Cursor keys up/down and ^P/^N iterate through the command history
+  buffer. Lines from the history buffer may be edited. New commands
+  are added to the end of the buffer when executed; though
+  duplicates of the previous command are not added.
+- ^U erases the line contents from the left of the cursor to the
+  start.
+- ^K erases the contents from the cursor to the end.
+- ^S/^R will search the command history forwards or in reverse for
+  a command; start typing for matches to auto complete.
+- ^L will clear the screen (if supported by the terminal) and repaint
+  the prompt and any current line. The cursor position is also
+  retained.
+- The CLI can be closed with the quit command. Alternatively, ^D on
+  an empty input line will also close the session. Closing the debug
+  session will also shutdown VPP.
+
+Output that exceeds the length of a terminal page will be buffered, up to a
+limit.
+
+- Space or page-down displays the next page.
+- Enter or down-arrow displays the next line.
+- Page-up goes back a page.
+- Up-arrow goes up a line.
+- Home/end jump to the start/end of the buffered output.
+- The key q quits the pager. Space and enter will also quit the
+  pager if the end of the buffer has been reached.
+
+## Index of CLI commands
+
+[TOC]
diff --git a/doxygen/siphon_templates/clicmd/item_format.md b/doxygen/siphon_templates/clicmd/item_format.md
new file mode 100644
index 00000000000..77d0484bfb7
--- /dev/null
+++ b/doxygen/siphon_templates/clicmd/item_format.md
@@ -0,0 +1,59 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+{% set v = item['value'] %}
+{{ "@section %s %s" % (meta['label'], v['path']) }}
+{% if 'short_help' in v %}
+
+### Summary/usage
+
+{% set str = v['short_help'] %}
+{% set period = "." if str[-1] != "." else "" %}
+{% set prefix = "    " if "[" in str or "&lt;" in str or "|" in str else "" %}
+{% set str = this.unescape(str) %}
+{{ "%s%s%s" % (prefix, str, period) }}
+{% endif %}
+{% if 'long_help' in v %}
+{# This is seldom used and will likely be deprecated #}
+
+### Long help
+
+{{ v['long_help'] }}
+{%- endif %}
+{% if 'siphon_block' in item['meta'] %}
+{% set sb = item["meta"]["siphon_block"] %}
+{% if sb %}
+{# Extracted from the code in /*? ... ?*/ blocks #}
+
+### Description
+
+{{ sb }}
+{% endif %}
+{% endif %}
+{% if "item" in meta or "function" in v %}
+{# Gives some developer-useful linking #}
+
+### Declaration and implementation
+{% if "item" in meta %}
+
+{{ "Declaration: @ref %s (@ref %s line %d)" %
+   (meta['item'], meta["file"], item["meta"]["line_start"]) }}
+{% endif %}
+{% if "function" in v %}
+
+{{ "Implementation: @ref %s." % v["function"] }}
+{% endif %}
+{% endif %}
+
diff --git a/doxygen/siphon_templates/default/index_entry.md b/doxygen/siphon_templates/default/index_entry.md
new file mode 100644
index 00000000000..479dcdb2cc3
--- /dev/null
+++ b/doxygen/siphon_templates/default/index_entry.md
@@ -0,0 +1,16 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+{{ "* [%s](@ref %s)" % (item["name"], meta["label"]) }}
diff --git a/doxygen/siphon_templates/default/index_section.md b/doxygen/siphon_templates/default/index_section.md
new file mode 100644
index 00000000000..3c9d2b474c5
--- /dev/null
+++ b/doxygen/siphon_templates/default/index_section.md
@@ -0,0 +1,18 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+
+@subpage {{ this.page_label(group) }}
+
diff --git a/doxygen/siphon_templates/default/item_format.md b/doxygen/siphon_templates/default/item_format.md
new file mode 100644
index 00000000000..ed1b1bf7eeb
--- /dev/null
+++ b/doxygen/siphon_templates/default/item_format.md
@@ -0,0 +1,16 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+{{ raise NotImplementedError }}
diff --git a/doxygen/siphon_templates/default/item_header.md b/doxygen/siphon_templates/default/item_header.md
new file mode 100644
index 00000000000..0c21e51fa5d
--- /dev/null
+++ b/doxygen/siphon_templates/default/item_header.md
@@ -0,0 +1,18 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+
+{{ "@page %s %s" % (this.page_label(group), this.page_title(group)) }}
+
diff --git a/doxygen/siphon_templates/syscfg/index_header.md b/doxygen/siphon_templates/syscfg/index_header.md
new file mode 100644
index 00000000000..5d338a0472b
--- /dev/null
+++ b/doxygen/siphon_templates/syscfg/index_header.md
@@ -0,0 +1,111 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+# Startup Configuration    {{'{#'}}syscfg}
+
+The VPP network stack comes with several configuration options that can be
+provided either on the command line or in a configuration file.
+
+Specific applications built on the stack have been known to require a dozen
+arguments, depending on requirements. This section describes commonly-used
+options and parameters.
+
+You can find command-line argument parsers in the source code by searching for
+instances of the `VLIB_CONFIG_FUNCTION` macro. The invocation
+`VLIB_CONFIG_FUNCTION(foo_config, "foo")` will cause the function
+`foo_config` to receive all the options and values supplied in a parameter
+block named "`foo`", for example: `foo { arg1 arg2 arg3 ... }`.
+
+@todo Tell the nice people where this document lives so that the might
+help improve it!
+
+## Command-line arguments
+
+Parameters are grouped by a section name. When providing more than one
+parameter to a section all parameters for that section must be wrapped in
+curly braces.
+
+```
+/usr/bin/vpp unix { interactive cli-listen 127.0.0.1:5002 }
+```
+
+Which will produce output similar to this:
+
+    <startup diagnostic messages>
+        _______    _        _   _____  ___ 
+     __/ __/ _ \  (_)__    | | / / _ \/ _ \
+     _/ _// // / / / _ \   | |/ / ___/ ___/
+     /_/ /____(_)_/\___/   |___/_/  /_/    
+    
+    vpp# <start-typing>
+
+When providing only one such parameter the braces are optional. For example,
+the following command argument, `unix interactive` does not have braces:
+
+```
+/usr/bin/vpp unix interactive
+```
+
+The command line can be presented as a single string or as several; anything
+given on the command line is concatenated with spaces into a single string
+before parsing.
+
+VPP applications must be able to locate their own executable images. The
+simplest way to ensure this will work is to invoke a VPP application by giving
+its absolute path; for example: `/usr/bin/vpp <options>`. At startup, VPP
+applications parse through their own ELF-sections (primarily) to make lists
+of init, configuration, and exit handlers.
+
+When developing with VPP, in _gdb_ it's often sufficient to start an application
+like this at the `(gdb)` prompt:
+
+```
+run unix interactive
+```
+
+## Configuration file
+
+It is also possible to supply parameters in a startup configuration file the
+path of which is provided to the VPP application on its command line.
+
+The format of the configuration file is a simple text file with the same
+content as the command line but with the benefit of being able to use newlines
+to make the content easier to read. For example:
+
+```
+unix {
+  nodaemon
+  log /tmp/vpp.log
+  full-coredump
+  cli-listen localhost:5002
+}
+api-trace {
+  on
+}
+dpdk {
+  dev 0000:03:00.0
+}
+```
+
+VPP is then instructed to load this file with the `-c` option:
+
+```
+/usr/bin/vpp -c /etc/vpp/startup.conf
+```
+
+## Index of startup command sections
+
+[TOC]
+
diff --git a/doxygen/siphon_templates/syscfg/item_format.md b/doxygen/siphon_templates/syscfg/item_format.md
new file mode 100644
index 00000000000..5d3bb5c83fb
--- /dev/null
+++ b/doxygen/siphon_templates/syscfg/item_format.md
@@ -0,0 +1,42 @@
+{#
+# Copyright (c) 2016 Comcast Cable Communications Management, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+{% set v = item['value'] %}
+{{ "@section %s %s" % (meta['label'], item['name']) }}
+{% if 'siphon_block' in item['meta'] %}
+{% set sb = item["meta"]["siphon_block"] %}
+{% if sb %}
+{# Extracted from the code in /*? ... ?*/ blocks #}
+
+### Description
+
+{{ sb }}
+{% endif %}
+{% endif %}
+{% if "item" in meta or "function" in item %}
+{# Gives some developer-useful linking #}
+
+### Declaration and implementation
+{% if "item" in meta %}
+
+{{ "Declaration: @ref %s (@ref %s line %d)" %
+   (meta['item'], meta["file"], item["meta"]["line_start"]) }}
+{% endif %}
+{% if "function" in item %}
+
+{{ "Implementation: @ref %s." % item["function"] }}
+{% endif %}
+{% endif %}
+