From 2ba9dcf408d5f861738356dfb43c28a803fcd7fc Mon Sep 17 00:00:00 2001 From: John DeNisco Date: Thu, 23 Aug 2018 14:04:22 -0400 Subject: docs: Finish event logger, viewer and cleanup. Change-Id: I3de038439bf0ab5755777c0f4930aec0514f5b63 Signed-off-by: John DeNisco --- docs/gettingstarted/developers/building.rst | 50 ++-- docs/gettingstarted/developers/eventviewer.rst | 293 +++++++++++++++++++++++ docs/gettingstarted/developers/gdb_examples.rst | 8 +- docs/gettingstarted/developers/gitreview.rst | 9 +- docs/gettingstarted/developers/index.rst | 14 +- docs/gettingstarted/developers/infrastructure.md | 108 --------- docs/gettingstarted/developers/running_vpp.rst | 4 +- docs/gettingstarted/developers/vnet.md | 2 +- 8 files changed, 346 insertions(+), 142 deletions(-) create mode 100644 docs/gettingstarted/developers/eventviewer.rst (limited to 'docs/gettingstarted/developers') diff --git a/docs/gettingstarted/developers/building.rst b/docs/gettingstarted/developers/building.rst index 15754b53cb9..37dacf1e2da 100644 --- a/docs/gettingstarted/developers/building.rst +++ b/docs/gettingstarted/developers/building.rst @@ -5,16 +5,16 @@ Building VPP ============ -To get started developing with VPP you need to get the sources and build the packages. -For more information on the build system please refer to :ref:`buildsystem`. +To get started developing with VPP, you need to get the required VPP sources and then build the packages. +For more detailed information on the build system please refer to :ref:`buildsystem`. .. _setupproxies: Set up Proxies -------------------------- -Depending on the environment, proxies may need to be set. -You may run these commands: +Depending on the environment you are operating in, proxies may need to be set. +Run these proxy commands to specify the *proxy-server-name* and corresponding *port-number*: .. code-block:: console @@ -35,19 +35,20 @@ To get the VPP sources that are used to create the build, run the following comm Build VPP Dependencies -------------------------------------- -Before building, make sure there are no FD.io VPP or DPDK packages installed by entering the following -commands: +Before building a VPP image, make sure there are no FD.io VPP or DPDK packages +installed, by entering the following commands: .. code-block:: console $ dpkg -l | grep vpp $ dpkg -l | grep DPDK -There should be no output, or packages showing after each of the above commands. +There should be no output, or no packages shown after the above commands are run. Run the following **make** command to install the dependencies for FD.io VPP. -If it hangs at any point during the download, then you may need to set up -:ref:`proxies for this to work `. + +If the download hangs at any point, then you may need to +:ref:`set up proxies ` for the download to work. .. code-block:: console @@ -80,6 +81,10 @@ This build version contains debug symbols which are useful for modifying VPP. Th **make** command below builds a debug version of VPP. The binaries, when building the debug images, can be found in /build-root/vpp_debug-native. +The Debug build version contains debug symbols, which are useful for troubleshooting +or modifying VPP. The **make** command below, builds a debug version of VPP. The +binaries used for building the debug image can be found in */build-root/vpp_debug-native*. + .. code-block:: console $ make build @@ -104,12 +109,11 @@ debug images, can be found in /build-root/vpp_debug-native. Build VPP (Release Version) ----------------------------------------- -To build the release version of FD.io VPP. This build is optimized and will not create debug symbols. -The binaries when building the release images can be found in /build-root/vpp-native. +This section describes how to build the regular release version of FD.io VPP. The +release build is optimized and does not create any debug symbols. +The binaries used in building the release images are found in */build-root/vpp-native*. -Use the following **make** command below to build the release version of FD.io VPP. This build is -optimized and will not create debug symbols. When building the release images, the binaries can -be found in /build-root/vpp-native. +Use the following **make** command below to build the release version of FD.io VPP. .. code-block:: console @@ -119,14 +123,23 @@ be found in /build-root/vpp-native. Building Necessary Packages -------------------------------------------- +The package that needs to be built depends on the type system VPP will be running on: + +* The :ref:`Debian package ` is built if VPP is going to run on Ubuntu +* The :ref:`RPM package ` is built if VPP is going to run on Centos or Redhat + +.. _debianpackages: + Building Debian Packages ^^^^^^^^^^^^^^^^^^^^^^^^^ -To build the debian packages, use one of the following commands below, depending on the system: +To build the debian packages, use the following command: .. code-block:: console $ make pkg-deb + +.. _rpmpackages: Building RPM Packages ^^^^^^^^^^^^^^^^^^^^^^^ @@ -137,20 +150,21 @@ To build the rpm packages, use one of the following commands below, depending on $ make pkg-rpm -Once the packages are builty they can be found in the build-root directory. +Once the packages are built they can be found in the build-root directory. .. code-block:: console $ ls *.deb - If packages built correctly, this should be the Output + If the packages are built correctly, then this should be the corresponding output: vpp_18.07-rc0~456-gb361076_amd64.deb vpp-dbg_18.07-rc0~456-gb361076_amd64.deb vpp-api-java_18.07-rc0~456-gb361076_amd64.deb vpp-dev_18.07-rc0~456-gb361076_amd64.deb vpp-api-lua_18.07-rc0~456-gb361076_amd64.deb vpp-lib_18.07-rc0~456-gb361076_amd64.deb vpp-api-python_18.07-rc0~456-gb361076_amd64.deb vpp-plugins_18.07-rc0~456-gb361076_amd64.deb -Finally, the packages can be installed with the following: +Finally, the created packages can be installed using the following commands. Install +the package that correspnds to OS that VPP will be running on: For Ubuntu: diff --git a/docs/gettingstarted/developers/eventviewer.rst b/docs/gettingstarted/developers/eventviewer.rst new file mode 100644 index 00000000000..19d3e7c3c59 --- /dev/null +++ b/docs/gettingstarted/developers/eventviewer.rst @@ -0,0 +1,293 @@ +.. _eventviewer: + +Event-logger +============ + +The vppinfra event logger provides very lightweight (sub-100ns) +precisely time-stamped event-logging services. See +./src/vppinfra/{elog.c, elog.h} + +Serialization support makes it easy to save and ultimately to combine a +set of event logs. In a distributed system running NTP over a local LAN, +we find that event logs collected from multiple system elements can be +combined with a temporal uncertainty no worse than 50us. + +A typical event definition and logging call looks like this: + +.. code-block:: c + + ELOG_TYPE_DECLARE (e) = + { + .format = "tx-msg: stream %d local seq %d attempt %d", + .format_args = "i4i4i4", + }; + struct { u32 stream_id, local_sequence, retry_count; } * ed; + ed = ELOG_DATA (m->elog_main, e); + ed->stream_id = stream_id; + ed->local_sequence = local_sequence; + ed->retry_count = retry_count; + +The ELOG\_DATA macro returns a pointer to 20 bytes worth of arbitrary +event data, to be formatted (offline, not at runtime) as described by +format\_args. Aside from obvious integer formats, the CLIB event logger +provides a couple of interesting additions. The "t4" format +pretty-prints enumerated values: + +.. code-block:: c + + ELOG_TYPE_DECLARE (e) = + { + .format = "get_or_create: %s", + .format_args = "t4", + .n_enum_strings = 2, + .enum_strings = { "old", "new", }, + }; + +The "t" format specifier indicates that the corresponding datum is an +index in the event's set of enumerated strings, as shown in the previous +event type definition. + +The “T” format specifier indicates that the corresponding datum is an +index in the event log’s string heap. This allows the programmer to emit +arbitrary formatted strings. One often combines this facility with a +hash table to keep the event-log string heap from growing arbitrarily +large. + +Noting the 20-octet limit per-log-entry data field, the event log +formatter supports arbitrary combinations of these data types. As in: +the ".format" field may contain one or more instances of the following: + +- i1 - 8-bit unsigned integer +- i2 - 16-bit unsigned integer +- i4 - 32-bit unsigned integer +- i8 - 64-bit unsigned integer +- f4 - float +- f8 - double +- s - NULL-terminated string - be careful +- sN - N-byte character array +- t1,2,4 - per-event enumeration ID +- T4 - Event-log string table offset + +The vpp engine event log is thread-safe, and is shared by all threads. +Take care not to serialize the computation. Although the event-logger is +about as fast as practicable, it's not appropriate for per-packet use in +hard-core data plane code. It's most appropriate for capturing rare +events - link up-down events, specific control-plane events and so +forth. + +The vpp engine has several debug CLI commands for manipulating its event +log: + +.. code-block:: console + + vpp# event-logger clear + vpp# event-logger save # for security, writes into /tmp/. + # must not contain '.' or '/' characters + vpp# show event-logger [all] [] # display the event log + # by default, the last 250 entries + +The event log defaults to 128K entries. The command-line argument "... +vlib { elog-events nnn } ..." configures the size of the event log. + +As described above, the vpp engine event log is thread-safe and shared. +To avoid confusing non-appearance of events logged by worker threads, +make sure to code vlib\_global\_main.elog\_main - instead of +vm->elog\_main. The latter form is correct in the main thread, but +will almost certainly produce bad results in worker threads. + +G2 graphical event viewer +========================== + +The G2 graphical event viewer can display serialized vppinfra event logs +directly, or via the c2cpel tool. G2 is a fine-grained event-log viewer. It's +highly scalable, supporting O(1e7 events) and O(1e3 discrete display "tracks"). +G2 displays binary data generated by the vppinfra "elog.[ch]" logger component, +and also supports the CPEL file format, as described in this section. + +Building +-------------- + +.. code-block:: console + + $ cd build-root + $ make g2-install + $ ./install-native/g2/bin/g2 --help + g2 [--ticks-per-us ][--cpel-input ] [--clib-input + G2 (x86_64 GNU/Linux) major version 3.0 + Built Wed Feb 3 10:58:12 EST 2016 + +Setting the Display Preferences +------------------------------------------------ + +The file $<*HOMEDIR*>/.g2 contains display preferences, which can be overridden. +Simply un-comment one of the stanzas shown below, or experiment as desired. + +.. code-block:: c + + /* + * Property / parameter settings for G2 + * + * Setting for a 1024x768 display: + * event_selector_lines=20 + * drawbox_height=800 + * drawbox_width=600 + * + * new mac w/ no monitor: + * event_selector_lines=20 + * drawbox_height=1200 + * drawbox_width=700 + * + * 1600x1200: + * drawbox_width=1200 + * drawbox_height=1000 + * event_selector_lines=25 + * + * for making screenshots on a Macbook Pro + * drawbox_width=1200 + * drawbox_height=600 + * event_selector_lines=20 + */ + +Screen Taxonomy +---------------------------- + +Here is an annotated G2 viewer screenshot, corresponding to activity during BGP +prefix download. This data was captured on a Cisco IOS-XR system: + +.. figure:: /_images/g21.jpg + :scale: 75% + + +The viewer has two main scrollbars: the horizontal axis scrollbar shifts the main +drawing area in time; the vertical axis changes the set of visible process traces. +The zoomin / zoomout operators change the time scale. + +The event selector PolyCheckMenu changes the set of displayed events. +Using these tools -- and some patience -- you can understand a given event log. + +Mouse Gestures +------------------------- + +G2 has three fairly sophisticated mouse gesture interfaces, which are worth describing +in detail. First, a left mouse click on a display event pops up a per-event detail box. + +.. figure:: /_images/g22.jpg + :scale: 75% + +A left mouse click on an event detail box closes it. +To zoom to a region of the display, press and hold the left mouse button, then drag +right or left until the zoom-fence pair appears: + +.. figure:: /_images/g23.jpg + :scale: 75% + +When the zoom operation completes, the display is as follows: + +.. figure:: /_images/g24.jpg + +A click on any of the figures will show them at full resolution, right-click will open figures in new tabs, + +Time Ruler +------------------ + +To use a time ruler, press and hold the right mouse button; drag right or left +until the ruler measures the region of interest. If the time axis scale is coarse, +event boxes can have significant width in time, so use a "reference point" in +each event box when using the time ruler. + +.. figure:: /_images/g25.jpg + :scale: 75% + +Event Selection +------------------------- + +Changing the Event Selector setup controls the set of points displayed in an +obvious way. Here, we suppress all events except "this thread is now running on the CPU": + +.. figure:: /_images/g26.jpg + :scale: 75% + +Same setup, with all events displayed: + +.. figure:: /_images/g27.jpg + :scale: 75% + +Note that event detail boxes previously shown, but suppressed due to deselection +of the event code will reappear when one reselects the event code. In the example +above, the "THREAD/THREADY pid:491720 tid:12" detail box appears in this fashion. + +Snapshot Ring +----------------------- + +Three buttons in lower left-hand corner of the g2 main window control the snapshot +ring. Snapshots are simply saved views: maneuver the viewer into an "interesting" +configuration, then press the "Snap" button to add a snapshot to the ring. + +Click **Next** to restore the next available snapshot. The **Del** button deletes the current snapshot. + +See the hotkey section below for access to a quick and easy method to save and +restore the snapshot ring. Eventually we may add a safe/portable/supported mechanism +to save/restore the snapshot ring from CPEL and vppinfra event log files. + +Chasing Events +------------------------ + +Event chasing sorts the trace axis by occurrence of the last selected event. For +example, if one selects an event which means "thread running on the CPU" the first +N displayed traces will be the first M threads to run (N <= M; a thread may run +more than once. This feature addresses analytic problems caused by the finite size of the drawing area. + +In standard (NoChaseEvent) mode, it looks like only BGP threads 5 and 9 are active: + +.. figure:: /_images/g28.jpg + :scale: 75% + +After pressing the ChaseEvent button, we see a different picture: + +.. figure:: /_images/g29.jpg + :scale: 75% + +Burying Boring Tracks +----------------------------------- + +The sequence moves the track under the mouse to the end +of the set of tracks, effectively burying it. The sequence +moves the track under the mouse to the beginning of the set of tracks. The latter +function probably isn't precisely right--I think we may eventually provide an "undo" +stack to provide precise thread exhumation. + +Summary Mode +------------------------- + +Summary mode declutters the screen by rendering events as short vertical line +segments instead of numbered boxes. Event detail display is unaffected. G2 starts +in summary mode, zoomed out sufficiently for all events in the trace to be displayed. +Given a large number of events, summary mode reduces initial screen-paint time to a +tolerable value. Once you've zoomed in sufficiently, type "e" - enter event mode, +to enable boxed numeric event display. + +Hotkeys +------------- + +G2 supports the following hotkey actions, supposedly (circa 1996) Quake-like +according to the feature's original author: + ++----------------------+--------------------------------------------------------+ +| Key | Function | ++======================+========================================================+ +| w | Zoom-in | ++----------------------+--------------------------------------------------------+ +| s | Zoom-out | ++----------------------+--------------------------------------------------------+ +| a | Scroll Left | ++----------------------+--------------------------------------------------------+ +| d | Scroll Right | ++----------------------+--------------------------------------------------------+ +| e | Toggle between event and summary-event mode | ++----------------------+--------------------------------------------------------+ +| p | Put (write) snapshot ring to snapshots.g2 | ++----------------------+--------------------------------------------------------+ +| l | Load (read) snapshot ring from snapshots.g2 | ++----------------------+--------------------------------------------------------+ +| -q | quit | ++----------------------+--------------------------------------------------------+ diff --git a/docs/gettingstarted/developers/gdb_examples.rst b/docs/gettingstarted/developers/gdb_examples.rst index 8ea34cbc276..8a0fb33e2a8 100644 --- a/docs/gettingstarted/developers/gdb_examples.rst +++ b/docs/gettingstarted/developers/gdb_examples.rst @@ -10,7 +10,7 @@ In this section we have a few useful gdb commands. Starting GDB ---------------------------- -Once at the gdb prompt, VPP can be started by isuuing the following commands: +Once at the gdb prompt, VPP can be started by running the following commands: .. code-block:: console @@ -24,7 +24,7 @@ Once at the gdb prompt, VPP can be started by isuuing the following commands: Backtrace ---------------------------- -If you encounter issues when running VPP, such as VPP terminating due to a segfault +If you encounter errors when running VPP, such as VPP terminating due to a segfault or abort signal, then you can run the VPP debug binary and then execute **backtrace** or **bt**. .. code-block:: console @@ -38,12 +38,12 @@ or abort signal, then you can run the VPP debug binary and then execute **backtr Get to the GDB prompt --------------------------------------- -When VPP is running, you can get to the command prompt by entering CTRL-c. +When VPP is running, you can get to the command prompt by pressing **CTRL+C**. Breakpoints --------------------------------------- -When at the GDB prompt, set a breakpoint by using the commands below: +When at the GDB prompt, set a breakpoint by running the commands below: .. code-block:: console diff --git a/docs/gettingstarted/developers/gitreview.rst b/docs/gettingstarted/developers/gitreview.rst index a9e1b02c764..e32d8c59991 100644 --- a/docs/gettingstarted/developers/gitreview.rst +++ b/docs/gettingstarted/developers/gitreview.rst @@ -54,7 +54,7 @@ Otherwise, clone with: .. code-block:: console - $ git clone ssh://YOUR_GERRIT_USERNAME@gerrit.fd.io:29418/vpp + $ git clone ssh://@gerrit.fd.io:29418/vpp $ cd vpp When attempting to clone the repo Git will prompt you asking if you want to add the Server Host Key to the list of known hosts. Enter **yes** and press the **Enter** key. @@ -78,7 +78,7 @@ Make sure you have modified the correct files by issuing the following commands: $ git diff Then add and commit the patch. You may want to add a tag to the commit comments. -For example for a document with only patches you should add the tag **DOCS:**. +For example for a document with only patches you should add the tag **docs:**. .. code-block:: console @@ -133,7 +133,7 @@ To modify an existing patch, make sure you modified the correct files, and apply $ git commit --amend $ git review -When you're done viewing or modifying a branch, get back to the master branch with: +When you're done viewing or modifying a branch, get back to the master branch by entering: .. code-block:: console @@ -143,10 +143,11 @@ When you're done viewing or modifying a branch, get back to the master branch wi Resolving a Conflict -------------------------------- -If a change has a conflict it should be resolved with the following:git-review -d +If a change has a conflict it should be resolved by entering: .. code-block:: console + $ git-review -d <*Gerrit change #*> $ git rebase origin/master while (conflicts) diff --git a/docs/gettingstarted/developers/index.rst b/docs/gettingstarted/developers/index.rst index b56fec8635c..d57c954fc5b 100644 --- a/docs/gettingstarted/developers/index.rst +++ b/docs/gettingstarted/developers/index.rst @@ -6,11 +6,14 @@ For Developers The Developers section covers the following areas: -* Building VPP -* Describes the components of the four VPP layers -* How to Create, Add, Enable/Disable features -* Discusses different aspects of Bounded-index Extensible Hashing (bihash) - +* Describes how to build different types of VPP images +* Explains how to run VPP with and without GDB, with some GDB examples +* Describes the steps required to get a patch reviewed and merged +* Describes the VPP software architecture and identifies the associated four VPP layers +* Describes the different components that are associated with each VPP layer +* Explains how to Create, Add, Enable/Disable different ARC features +* Discusses different aspects of Bounded-index Extensible Hashing (bihash), and how it is used in database lookups +* Describes the different types of API support and how to integrate a plugin .. toctree:: :maxdepth: 2 @@ -30,5 +33,6 @@ The Developers section covers the following areas: vpp_api_module binary_api_support buildsystem/index.rst + eventviewer sample_plugin diff --git a/docs/gettingstarted/developers/infrastructure.md b/docs/gettingstarted/developers/infrastructure.md index 688c42133ed..0361a632c37 100644 --- a/docs/gettingstarted/developers/infrastructure.md +++ b/docs/gettingstarted/developers/infrastructure.md @@ -220,111 +220,3 @@ and unserialize complex data structures. The underlying primitive serialize/unserialize functions use network byte-order, so there are no structural issues serializing on a little-endian host and unserializing on a big-endian host. - -Event-logger, graphical event log viewer ----------------------------------------- - -The vppinfra event logger provides very lightweight (sub-100ns) -precisely time-stamped event-logging services. See -./src/vppinfra/{elog.c, elog.h} - -Serialization support makes it easy to save and ultimately to combine a -set of event logs. In a distributed system running NTP over a local LAN, -we find that event logs collected from multiple system elements can be -combined with a temporal uncertainty no worse than 50us. - -A typical event definition and logging call looks like this: - -```c - ELOG_TYPE_DECLARE (e) = - { - .format = "tx-msg: stream %d local seq %d attempt %d", - .format_args = "i4i4i4", - }; - struct { u32 stream_id, local_sequence, retry_count; } * ed; - ed = ELOG_DATA (m->elog_main, e); - ed->stream_id = stream_id; - ed->local_sequence = local_sequence; - ed->retry_count = retry_count; -``` - -The ELOG\_DATA macro returns a pointer to 20 bytes worth of arbitrary -event data, to be formatted (offline, not at runtime) as described by -format\_args. Aside from obvious integer formats, the CLIB event logger -provides a couple of interesting additions. The "t4" format -pretty-prints enumerated values: - -```c - ELOG_TYPE_DECLARE (e) = - { - .format = "get_or_create: %s", - .format_args = "t4", - .n_enum_strings = 2, - .enum_strings = { "old", "new", }, - }; -``` - -The "t" format specifier indicates that the corresponding datum is an -index in the event's set of enumerated strings, as shown in the previous -event type definition. - -The “T” format specifier indicates that the corresponding datum is an -index in the event log’s string heap. This allows the programmer to emit -arbitrary formatted strings. One often combines this facility with a -hash table to keep the event-log string heap from growing arbitrarily -large. - -Noting the 20-octet limit per-log-entry data field, the event log -formatter supports arbitrary combinations of these data types. As in: -the ".format" field may contain one or more instances of the following: - -- i1 - 8-bit unsigned integer -- i2 - 16-bit unsigned integer -- i4 - 32-bit unsigned integer -- i8 - 64-bit unsigned integer -- f4 - float -- f8 - double -- s - NULL-terminated string - be careful -- sN - N-byte character array -- t1,2,4 - per-event enumeration ID -- T4 - Event-log string table offset - -The vpp engine event log is thread-safe, and is shared by all threads. -Take care not to serialize the computation. Although the event-logger is -about as fast as practicable, it's not appropriate for per-packet use in -hard-core data plane code. It's most appropriate for capturing rare -events - link up-down events, specific control-plane events and so -forth. - -The vpp engine has several debug CLI commands for manipulating its event -log: - -``` - vpp# event-logger clear - vpp# event-logger save # for security, writes into /tmp/. - # must not contain '.' or '/' characters - vpp# show event-logger [all] [] # display the event log - # by default, the last 250 entries -``` - -The event log defaults to 128K entries. The command-line argument "... -vlib { elog-events nnn } ..." configures the size of the event log. - -As described above, the vpp engine event log is thread-safe and shared. -To avoid confusing non-appearance of events logged by worker threads, -make sure to code vlib\_global\_main.elog\_main - instead of -vm->elog\_main. The latter form is correct in the main thread, but -will almost certainly produce bad results in worker threads. - -G2 graphical event viewer -------------------------- - -The g2 graphical event viewer can display serialized vppinfra event logs -directly, or via the c2cpel tool. - -
- -Todo: please convert wiki page and figures - -
- diff --git a/docs/gettingstarted/developers/running_vpp.rst b/docs/gettingstarted/developers/running_vpp.rst index a1215765545..9b33e53ec60 100644 --- a/docs/gettingstarted/developers/running_vpp.rst +++ b/docs/gettingstarted/developers/running_vpp.rst @@ -5,8 +5,8 @@ Running VPP =========== -After building the VPP binaries, you now have several images that you have built. -These images are useful when you need to run VPP without installing the packages. +After building the VPP binaries, you now have several images built. +These images are useful when you need to run VPP without installing the packages. For instance if you want to run VPP with GDB. Running Without GDB diff --git a/docs/gettingstarted/developers/vnet.md b/docs/gettingstarted/developers/vnet.md index 602ffb7e782..ab081b08302 100644 --- a/docs/gettingstarted/developers/vnet.md +++ b/docs/gettingstarted/developers/vnet.md @@ -3,7 +3,7 @@ VNET (VPP Network Stack) ======================== The files associated with the VPP network stack layer are located in the -./src/vnet folder. The Network Stack Layer is basically an +*./src/vnet* folder. The Network Stack Layer is basically an instantiation of the code in the other layers. This layer has a vnet library that provides vectorized layer-2 and 3 networking graph nodes, a packet generator, and a packet tracer. -- cgit 1.2.3-korg