RELEASE.md


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200

# Release Notes    {#release_notes}

* @subpage release_notes_1701
* @subpage release_notes_1609
* @subpage release_notes_1606

@page release_notes_1701 Release notes for VPP 17.01

@note This release was for a while known as 16.12.
@todo Release 17.01 needs release notes.

## Features

## Known issues

## Issues fixed


@page release_notes_1609 Release notes for VPP 16.09

## Features

- [Integrated July 2016 DPDK release](http://www.dpdk.org/doc/guides/rel_notes/release_16_07.html)
  - DPDK-vhost is depreciated pending a complete rework of the original integration and
    addressing of rx performance deltas.
  - Patches required for DPDK 16.07:
    - Correctly setting the Packet Type in the IGB, IXGBE and i40e drivers.
    - Correctly setting checksum in the i40e driver.
    - NXP DPAA2 PMD Driver.
    - rte_delay (yield) functionality.

- Add “in tree” plugins:
  - IPv6 ILA.
  - iOAM.
  - Load Balancer.
  - SNAT.

- High-performance (line-rate) “neutron like” L4 port-filtering.

- API refactoring - addressing some of the issues around JVPP bindings.
  - Accommodating plugins [(e.g. NSH_SFC)](https://wiki.fd.io/view/NSH_SFC)
  - Binding for [python](https://wiki.fd.io/view/VPP/Python_API)

- LISP
  - L2 LISP overlays
  -  Multitenancy
  - Multihoming
  - RTR mode
  - Map-resolver failover algorithm

- Support 64-bit vector lengths, huge shared-memory segments.

- Dynamic IP Feature ordering
  - IP Features can now specify features they appear before and after

- 16.09 Builds
  - Ubuntu 14.04 LTS - Trusty Tahr
  - Ubuntu 16.04 LTS - Xenial Xerus
  - CentOS 7
  - More information on [VPP wiki](https://wiki.fd.io/view/VPP/Installing_VPP_binaries_from_packages)

- Performance, characterize and document performance for this release
  [(more information on CSIT page)](https://wiki.fd.io/view/CSIT)

   - IPv4 and IPv6 Scale - performance tests.
     - Bidirectional 10k/100k/1M flows.
     - 64B,570B, 1518B,9000B packet sizes.
   - IPv6 iACL - performance
     - DUT1 and DUT2 are configured with IPv6 routing, two static IPv6 /64 routes and IPv6 iAcl
       security whitelist ingress /64 filter entries applied on links.
     - TG traffic profile contains two L3 flow-groups (flow-group per direction, 253 flows per
       flow-group) with all packets containing Ethernet header, IPv6 header and generated payload.
       MAC addresses are matching MAC addresses of the TG node interfaces.

   - L2XC VXLANoIPv4 - performance
     - DUT1 and DUT2 are configured with L2 cross-connect. VXLAN tunnels are configured between
       L2XCs on DUT1 and DUT2.
     - TG traffic profile contains two L3 flow-groups (flow-group per direction, 253 flows per
       flow-group) with all packets containing Ethernet header, IPv4 header with IP protocol=61
       and generated payload. MAC addresses are matching MAC addresses of the TG node interfaces.

- Documentation
  - Autogenerated CLI documentation.
  - Using doxygen to automate API/Node documentation.
  - [(available online)](https://docs.fd.io/vpp/16.09/)

- Resolved all static analysis issues found by Coverity
  - Beginning of 16.09 cycle: 505 issues.
  - Release: 0 outstanding issues.


## Known issues

Issues in fd.io are tracked in [JIRA](https://jira.fd.io).

Issue | Description
--- | ---
VPP-391 |   vpp debug version assert appeared in the process of start
VPP-380 |   Mapping algorithm compute wrong ea-bits when IPv4 prefix 0.0.0.0/0
VPP-371 |   load_one_plugin:63: Loaded plugin: message from vppctl
VPP-367 |   vpp packages need to depend on specific versions of each other
VPP-312 |   IP6 FIB gets in indeterminate state by duplicating commands
VPP-224 |   Lookup-in-vrf can not be set correctly
VPP-206 |   Fix classify table delete
VPP-203 |   Fix binary API for reading vpp node graph
VPP-147 |   Inconsistent behaviour when adding L2 FIB filter entry
VPP-99  |  VPP doesn't discard DHCPOFFER message with wrong XID


## Issues fixed

Issues in fd.io are tracked in [JIRA](https://jira.fd.io).

Issue | Description
--- | ---
VPP-396 |   Ubuntu systems Graphviz bug
VPP-390 |   vpp-lib rpm fails to include *.so symlinks, causing linking problems with out of tree builds
VPP-388 |   IPSec output feature assumes packets have been ethernet rewritten
VPP-385 |   ARP for indirect adjacencies not working correctly
VPP-361 |   Memory leak on delete of VXLAN over IPv6 tunnel
VPP-357 |   VNI not set correctly when removing LISP fwd entries
VPP-349 |   sw_interface_vhost_user_dump not working
VPP-345 |   net/enic: bad L4 checksum ptype set on ICMP packets
VPP-340 |   MAP-T wrong destination address
VPP-330 |   Use fifo to store LISP pending map-requests
VPP-326 |   map_add_domain VAT command: unable to configure domain with mtu parameter
VPP-318 |   The map_add_domain VAT command accepts invalid arguments
VPP-315 |   Fix "show vxlan-gpe" issue
VPP-310 |   Mapping algorithm compute wrong ea-bits
VPP-239 |   LISP IP forwarding does not tag packets that hit negative mapping entries
VPP-235 |   Invalid help in VAT for sw_interface_set_l2_bridge
VPP-228 |   Mapping algorithm sends packet to wrong IPv6 address
VPP-214 |   vpp-api-test: api_ipsec_sad_add_del_entry: vector "ck" not initialized
VPP-200 |   VPP - TAP port create problem
VPP-189 |   Coverity Issues for 16.09
VPP-184 |   u16 translating to char ,not short
VPP-179 |   Adjacency share-count botch
VPP-163 |   "show ip6 interface" ignores non-global addresses
VPP-155 |   Netmap: Inconsistency in interface state between "show hardware" and "show interface"
VPP-145 |   Dynamically compute IP feature ordering based on constraints
VPP-137 |   VPP sends ARP with wrong requested IP
VPP-118 |   JVpp: 0 length arrays not handled properly in VPP responses
VPP-112 |   linux kernel info missing from build log
VPP-110 |   vxlan encap node should never touch a deleted tunnel
VPP-107 |   RPM build broken in master
VPP-92  |   segment routing is not properly filling out the segment list
VPP-91  |   segment routing add/del tunnel lookup doesn't work
VPP-84  |   af_packet throws a fatal error on EAGAIN
VPP-74  |   Clang compile fails due to warning in vlib/unix/cli.c
VPP-64  |   Top level "make pkg-deb" fails if CDPATH is set in user env.
VPP-48  |   Traceroute does not terminate when VPP is the target
VPP-23  |   CLI pager does not gracefully handle lines longer than the terminal width


@page release_notes_1606 Release notes for VPP 16.06


The FD.io Project, relentlessly focused on data IO speed and efficiency
supporting the creation of high performance, flexible, and scalable software
defined infrastructures, announces the availability of the community’s first
software release (16.06).

In the four months since launching, FD.io has brought together more than 75
developers from 11 different companies including network operators, solution
providers chip vendors, and network equipment vendors who are collaborating to
enhance and innovate around the Vector Packet Processing (VPP) technology. The
FD.io community has quickly formed to grow the number of projects from the
initial VPP project to an additional 6 projects addressing a diverse set of
requirements and usability across a variety of deployment environments.

The 16.06 release brings unprecedented performance: 480Gbps/200mpps with 8
million routes and 2k whitelist entries on standard high volume x86 servers.


## Features

In addition to the existing full suite of vswitch/vrouter features, the new
16.06 release adds:

* Enhanced Switching and Routing:
  * IPv6 Segment Routing multicast support.
  * LISP xTR support.
  * VXLAN over IPv6 underlay.
  * Per interface whitelists.
  * Shared adjacencies in FIB.

* New and improved interface support:
  * Jumbo frame support for vhost-user.
  * Netmap interface support.
  * AF_Packet interface support.

* Expanded and improved programmability:
  * Python API bindings.
  * Enhanced JVPP Java API bindings.
  * Debugging CLI.

* Expanded Hardware and Software Support:
  * Support for ARM 32 targets including Rasberry Pi single-board computer.
  * Support for DPDK 16.04.
/*
 *------------------------------------------------------------------
 * api_shared.c - API message handling, common code for both clients
 * and the vlib process itself.
 *
 *
 * Copyright (c) 2009 Cisco and/or its affiliates.
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at:
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *------------------------------------------------------------------
 */

#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <string.h>
#include <vppinfra/format.h>
#include <vppinfra/byte_order.h>
#include <vppinfra/error.h>
#include <vlib/vlib.h>
#include <vlib/unix/unix.h>
#include <vlibapi/api.h>
#include <vppinfra/elog.h>

/* *INDENT-OFF* */
api_main_t api_main =
  {
    .region_name = "/unset",
    .api_uid = -1,
    .api_gid = -1,
  };
/* *INDENT-ON* */

void
vl_msg_api_increment_missing_client_counter (void)
{
  api_main_t *am = &api_main;
  am->missing_clients++;
}

int
vl_msg_api_rx_trace_enabled (api_main_t * am)
{
  return (am->rx_trace && am->rx_trace->enabled);
}

int
vl_msg_api_tx_trace_enabled (api_main_t * am)
{
  return (am->tx_trace && am->tx_trace->enabled);
}

/*
 * vl_msg_api_trace
 */
void
vl_msg_api_trace (api_main_t * am, vl_api_trace_t * tp, void *msg)
{
  u8 **this_trace;
  u8 **old_trace;
  u8 *msg_copy;
  u32 length;
  trace_cfg_t *cfgp;
  u16 msg_id = clib_net_to_host_u16 (*((u16 *) msg));
  msgbuf_t *header = (msgbuf_t *) (((u8 *) msg) - offsetof (msgbuf_t, data));

  cfgp = am->api_trace_cfg + msg_id;

  if (!cfgp || !cfgp->trace_enable)
    return;

  msg_copy = 0;

  if (tp->nitems == 0)
    {
      clib_warning ("tp->nitems is 0");
      return;
    }

  if (vec_len (tp->traces) < tp->nitems)
    {
      vec_add1 (tp->traces, 0);
      this_trace = tp->traces + vec_len (tp->traces) - 1;
    }
  else
    {
      tp->wrapped = 1;
      old_trace = tp->traces + tp->curindex++;
      if (tp->curindex == tp->nitems)
	tp->curindex = 0;
      /* Reuse the trace record, may save some memory allocator traffic */
      msg_copy = *old_trace;
      vec_reset_length (msg_copy);
      this_trace = old_trace;
    }

  length = clib_net_to_host_u32 (header->data_len);

  vec_validate (msg_copy, length - 1);
  clib_memcpy_fast (msg_copy, msg, length);
  *this_trace = msg_copy;
}

int
vl_msg_api_trace_onoff (api_main_t * am, vl_api_trace_which_t which,
			int onoff)
{
  vl_api_trace_t *tp;
  int rv;

  switch (which)
    {
    case VL_API_TRACE_TX:
      tp = am->tx_trace;
      if (tp == 0)
	{
	  vl_msg_api_trace_configure (am, which, 1024);
	  tp = am->tx_trace;
	}
      break;

    case VL_API_TRACE_RX:
      tp = am->rx_trace;
      if (tp == 0)
	{
	  vl_msg_api_trace_configure (am, which, 1024);
	  tp = am->rx_trace;
	}
      break;

    default:
      /* duh? */
      return -1;
    }

  /* Configured? */
  if (tp == 0 || tp->nitems == 0)
    return -1;

  rv = tp->enabled;
  tp->enabled = onoff;

  return rv;
}

int
vl_msg_api_trace_free (api_main_t * am, vl_api_trace_which_t which)
{
  vl_api_trace_t *tp;
  int i;

  switch (which)
    {
    case VL_API_TRACE_TX:
      tp = am->tx_trace;
      break;

    case VL_API_TRACE_RX:
      tp = am->rx_trace;
      break;

    default:
      /* duh? */
      return -1;
    }

  /* Configured? */
  if (!tp || tp->nitems == 0)
    return -1;

  tp->curindex = 0;
  tp->wrapped = 0;

  for (i = 0; i < vec_len (tp->traces); i++)
    {
      vec_free (tp->traces[i]);
    }
  vec_free (tp->traces);

  return 0;
}

u8 *
vl_api_serialize_message_table (api_main_t * am, u8 * vector)
{
  serialize_main_t _sm, *sm = &_sm;
  hash_pair_t *hp;
  u32 nmsg = hash_elts (am->msg_index_by_name_and_crc);

  serialize_open_vector (sm, vector);

  /* serialize the count */
  serialize_integer (sm, nmsg, sizeof (u32));

  /* *INDENT-OFF* */
  hash_foreach_pair (hp, am->msg_index_by_name_and_crc,
  ({
    serialize_likely_small_unsigned_integer (sm, hp->value[0]);
    serialize_cstring (sm, (char *) hp->key);
  }));
  /* *INDENT-ON* */

  return serialize_close_vector (sm);
}

int
vl_msg_api_trace_save (api_main_t * am, vl_api_trace_which_t which, FILE * fp)
{
  vl_api_trace_t *tp;
  vl_api_trace_file_header_t fh;
  int i;
  u8 *msg;

  switch (which)
    {
    case VL_API_TRACE_TX:
      tp = am->tx_trace;
      break;

    case VL_API_TRACE_RX:
      tp = am->rx_trace;
      break;

    default:
      /* duh? */
      return -1;
    }

  /* Configured, data present? */
  if (tp == 0 || tp->nitems == 0 || vec_len (tp->traces) == 0)
    return -1;

  /* "Dare to be stupid" check */
  if (fp == 0)
    {
      return -2;
    }

  /* Write the file header */
  fh.wrapped = tp->wrapped;
  fh.nitems = clib_host_to_net_u32 (vec_len (tp->traces));
  u8 *m = vl_api_serialize_message_table (am, 0);
  clib_warning ("Message table length %d", vec_len (m));
  fh.msgtbl_size = clib_host_to_net_u32 (vec_len (m));

  if (fwrite (&fh, sizeof (fh), 1, fp) != 1)
    {
      return (-10);
    }

  /* Write the message table */
  if (fwrite (m, vec_len (m), 1, fp) != 1)
    {
      return (-14);
    }
  vec_free (m);

  /* No-wrap case */
  if (tp->wrapped == 0)
    {
      /*
       * Note: vec_len return 0 when fed a NULL pointer.
       * Unfortunately, the static analysis tool doesn't
       * figure it out, hence the suppressed warnings.
       * What a great use of my time.
       */
      for (i = 0; i < vec_len (tp->traces); i++)
	{
	  u32 msg_length;
	  /*sa_ignore NO_NULL_CHK */
	  msg = tp->traces[i];
	  /*
	   * This retarded check required to pass
	   * [sic] SA-checking.
	   */
	  if (!msg)
	    continue;

	  msg_length = clib_host_to_net_u32 (vec_len (msg));
	  if (fwrite (&msg_length, 1, sizeof (msg_length), fp)
	      != sizeof (msg_length))
	    {
	      return (-14);
	    }
	  if (fwrite (msg, 1, vec_len (msg), fp) != vec_len (msg))
	    {
	      return (-11);
	    }
	}
    }
  else
    {
      /* Wrap case: write oldest -> end of buffer */
      for (i = tp->curindex; i < vec_len (tp->traces); i++)
	{
	  u32 msg_length;
	  msg = tp->traces[i];
	  /*
	   * This retarded check required to pass
	   * [sic] SA-checking
	   */
	  if (!msg)
	    continue;

	  msg_length = clib_host_to_net_u32 (vec_len (msg));
	  if (fwrite (&msg_length, 1, sizeof (msg_length), fp)
	      != sizeof (msg_length))
	    {
	      return (-14);
	    }

	  if (fwrite (msg, 1, vec_len (msg), fp) != vec_len (msg))
	    {
	      return (-12);
	    }
	}
      /* write beginning of buffer -> oldest-1 */
      for (i = 0; i < tp->curindex; i++)
	{
	  u32 msg_length;
	  /*sa_ignore NO_NULL_CHK */
	  msg = tp->traces[i];
	  /*
	   * This retarded check required to pass
	   * [sic] SA-checking
	   */
	  if (!msg)
	    continue;

	  msg_length = clib_host_to_net_u32 (vec_len (msg));
	  if (fwrite (&msg_length, 1, sizeof (msg_length), fp)
	      != sizeof (msg_length))
	    {
	      return (-14);
	    }

	  if (fwrite (msg, 1, vec_len (msg), fp) != vec_len (msg))
	    {
	      return (-13);
	    }
	}
    }
  return 0;
}

int
vl_msg_api_trace_configure (api_main_t * am, vl_api_trace_which_t which,
			    u32 nitems)
{
  vl_api_trace_t *tp;
  int was_on = 0;

  switch (which)
    {
    case VL_API_TRACE_TX:
      tp = am->tx_trace;
      if (tp == 0)
	{
	  vec_validate (am->tx_trace, 0);
	  tp = am->tx_trace;
	}
      break;

    case VL_API_TRACE_RX:
      tp = am->rx_trace;
      if (tp == 0)
	{
	  vec_validate (am->rx_trace, 0);
	  tp = am->rx_trace;
	}

      break;

    default:
      return -1;

    }

  if (tp->enabled)
    {
      was_on = vl_msg_api_trace_onoff (am, which, 0);
    }
  if (tp->traces)
    {
      vl_msg_api_trace_free (am, which);
    }

  clib_memset (tp, 0, sizeof (*tp));

  if (clib_arch_is_big_endian)
    {
      tp->endian = VL_API_BIG_ENDIAN;
    }
  else
    {
      tp->endian = VL_API_LITTLE_ENDIAN;
    }

  tp->nitems = nitems;
  if (was_on)
    {
      (void) vl_msg_api_trace_onoff (am, which, was_on);
    }
  return 0;
}

void
vl_msg_api_barrier_sync (void)
{
}

void
vl_msg_api_barrier_release (void)
{
}

always_inline void
msg_handler_internal (api_main_t * am,
		      void *the_msg, int trace_it, int do_it, int free_it)
{
  u16 id = clib_net_to_host_u16 (*((u16 *) the_msg));
  u8 *(*print_fp) (void *, void *);

  if (PREDICT_FALSE (am->elog_trace_api_messages))
    {
      /* *INDENT-OFF* */
      ELOG_TYPE_DECLARE (e) =
        {
          .format = "api-msg: %s",
          .format_args = "T4",
        };
      /* *INDENT-ON* */
      struct
      {
	u32 c;
      } *ed;
      ed = ELOG_DATA (am->elog_main, e);
      if (id < vec_len (am->msg_names))
	ed->c = elog_string (am->elog_main, (char *) am->msg_names[id]);
      else
	ed->c = elog_string (am->elog_main, "BOGUS");
    }

  if (id < vec_len (am->msg_handlers) && am->msg_handlers[id])
    {
      if (trace_it)
	vl_msg_api_trace (am, am->rx_trace, the_msg);

      if (am->msg_print_flag)
	{
	  fformat (stdout, "[%d]: %s\n", id, am->msg_names[id]);
	  print_fp = (void *) am->msg_print_handlers[id];
	  if (print_fp == 0)
	    {
	      fformat (stdout, "  [no registered print fn]\n");
	    }
	  else
	    {
	      (*print_fp) (the_msg, stdout);
	    }
	}

      if (do_it)
	{
	  if (!am->is_mp_safe[id])
	    {
	      vl_msg_api_barrier_trace_context (am->msg_names[id]);
	      vl_msg_api_barrier_sync ();
	    }
	  (*am->msg_handlers[id]) (the_msg);
	  if (!am->is_mp_safe[id])
	    vl_msg_api_barrier_release ();
	}
    }
  else
    {
      clib_warning ("no handler for msg id %d", id);
    }

  if (free_it)
    vl_msg_api_free (the_msg);

  if (PREDICT_FALSE (am->elog_trace_api_messages))
    {
      /* *INDENT-OFF* */
      ELOG_TYPE_DECLARE (e) =
        {
          .format = "api-msg-done(%s): %s",
          .format_args = "t4T4",
          .n_enum_strings = 2,
          .enum_strings =
          {
            "barrier",
            "mp-safe",
          }
        };
      /* *INDENT-ON* */

      struct
      {
	u32 barrier;
	u32 c;
      } *ed;
      ed = ELOG_DATA (am->elog_main, e);
      if (id < vec_len (am->msg_names))
	{
	  ed->c = elog_string (am->elog_main, (char *) am->msg_names[id]);
	  ed->barrier = !am->is_mp_safe[id];
	}
      else
	{
	  ed->c = elog_string (am->elog_main, "BOGUS");
	  ed->barrier = 0;
	}
    }
}

/* This is only to be called from a vlib/vnet app */
void
vl_msg_api_handler_with_vm_node (api_main_t * am,
				 void *the_msg, vlib_main_t * vm,
				 vlib_node_runtime_t * node)
{
  u16 id = clib_net_to_host_u16 (*((u16 *) the_msg));
  u8 *(*handler) (void *, void *, void *);
  u8 *(*print_fp) (void *, void *);
  int is_mp_safe = 1;

  if (PREDICT_FALSE (am->elog_trace_api_messages))
    {
      /* *INDENT-OFF* */
      ELOG_TYPE_DECLARE (e) =
        {
          .format = "api-msg: %s",
          .format_args = "T4",
        };
      /* *INDENT-ON* */
      struct
      {
	u32 c;
      } *ed;
      ed = ELOG_DATA (am->elog_main, e);
      if (id < vec_len (am->msg_names))
	ed->c = elog_string (am->elog_main, (char *) am->msg_names[id]);
      else
	ed->c = elog_string (am->elog_main, "BOGUS");
    }

  if (id < vec_len (am->msg_handlers) && am->msg_handlers[id])
    {
      handler = (void *) am->msg_handlers[id];

      if (PREDICT_FALSE (am->rx_trace && am->rx_trace->enabled))
	vl_msg_api_trace (am, am->rx_trace, the_msg);

      if (PREDICT_FALSE (am->msg_print_flag))
	{
	  fformat (stdout, "[%d]: %s\n", id, am->msg_names[id]);
	  print_fp = (void *) am->msg_print_handlers[id];
	  if (print_fp == 0)
	    {
	      fformat (stdout, "  [no registered print fn for msg %d]\n", id);
	    }
	  else
	    {
	      (*print_fp) (the_msg, vm);
	    }
	}
      is_mp_safe = am->is_mp_safe[id];

      if (!is_mp_safe)
	{
	  vl_msg_api_barrier_trace_context (am->msg_names[id]);
	  vl_msg_api_barrier_sync ();
	}
      (*handler) (the_msg, vm, node);
      if (!is_mp_safe)
	vl_msg_api_barrier_release ();
    }
  else
    {
      clib_warning ("no handler for msg id %d", id);
    }

  /*
   * Special-case, so we can e.g. bounce messages off the vnet
   * main thread without copying them...
   */
  if (!(am->message_bounce[id]))
    vl_msg_api_free (the_msg);

  if (PREDICT_FALSE (am->elog_trace_api_messages))
    {
      /* *INDENT-OFF* */
      ELOG_TYPE_DECLARE (e) =
        {
          .format = "api-msg-done(%s): %s",
          .format_args = "t4T4",
          .n_enum_strings = 2,
          .enum_strings =
          {
            "barrier",
            "mp-safe",
          }
        };
      /* *INDENT-ON* */

      struct
      {
	u32 barrier;
	u32 c;
      } *ed;
      ed = ELOG_DATA (am->elog_main, e);
      if (id < vec_len (am->msg_names))
	ed->c = elog_string (am->elog_main, (char *) am->msg_names[id]);
      else
	ed->c = elog_string (am->elog_main, "BOGUS");
      ed->barrier = is_mp_safe;
    }
}

void
vl_msg_api_handler (void *the_msg)
{
  api_main_t *am = &api_main;

  msg_handler_internal (am, the_msg,
			(am->rx_trace
			 && am->rx_trace->enabled) /* trace_it */ ,
			1 /* do_it */ , 1 /* free_it */ );
}

void
vl_msg_api_handler_no_free (void *the_msg)
{
  api_main_t *am = &api_main;
  msg_handler_internal (am, the_msg,
			(am->rx_trace
			 && am->rx_trace->enabled) /* trace_it */ ,
			1 /* do_it */ , 0 /* free_it */ );
}

void
vl_msg_api_handler_no_trace_no_free (void *the_msg)
{
  api_main_t *am = &api_main;
  msg_handler_internal (am, the_msg, 0 /* trace_it */ , 1 /* do_it */ ,
			0 /* free_it */ );
}

/*
 * Add a trace record to the API message trace buffer, if
 * API message tracing is enabled. Handy for adding sufficient
 * data to the trace to reproduce autonomous state, as opposed to
 * state downloaded via control-plane API messages. Example: the NAT
 * application creates database entries based on packet traffic, not
 * control-plane messages.
 *
 */
void
vl_msg_api_trace_only (void *the_msg)
{
  api_main_t *am = &api_main;

  msg_handler_internal (am, the_msg,
			(am->rx_trace
			 && am->rx_trace->enabled) /* trace_it */ ,
			0 /* do_it */ , 0 /* free_it */ );
}

void
vl_msg_api_cleanup_handler (void *the_msg)
{
  api_main_t *am = &api_main;
  u16 id = clib_net_to_host_u16 (*((u16 *) the_msg));

  if (PREDICT_FALSE (id >= vec_len (am->msg_cleanup_handlers)))
    {
      clib_warning ("_vl_msg_id too large: %d\n", id);
      return;
    }
  if (am->msg_cleanup_handlers[id])
    (*am->msg_cleanup_handlers[id]) (the_msg);

  vl_msg_api_free (the_msg);
}

/*
 * vl_msg_api_replay_handler
 */
void
vl_msg_api_replay_handler (void *the_msg)
{
  api_main_t *am = &api_main;

  u16 id = clib_net_to_host_u16 (*((u16 *) the_msg));

  if (PREDICT_FALSE (id >= vec_len (am->msg_handlers)))
    {
      clib_warning ("_vl_msg_id too large: %d\n", id);
      return;
    }
  /* do NOT trace the message... */
  if (am->msg_handlers[id])
    (*am->msg_handlers[id]) (the_msg);
  /* do NOT free the message buffer... */
}

u32
vl_msg_api_get_msg_length (void *msg_arg)
{
  return vl_msg_api_get_msg_length_inline (msg_arg);
}

/*
 * vl_msg_api_socket_handler
 */
void
vl_msg_api_socket_handler (void *the_msg)
{
  api_main_t *am = &api_main;

  msg_handler_internal (am, the_msg,
			(am->rx_trace
			 && am->rx_trace->enabled) /* trace_it */ ,
			1 /* do_it */ , 0 /* free_it */ );
}

#define foreach_msg_api_vector                  \
_(msg_names)                                    \
_(msg_handlers)                                 \
_(msg_cleanup_handlers)                         \
_(msg_endian_handlers)                          \
_(msg_print_handlers)                           \
_(api_trace_cfg)				\
_(message_bounce)				\
_(is_mp_safe)

void
vl_msg_api_config (vl_msg_api_msg_config_t * c)
{
  api_main_t *am = &api_main;

  /*
   * This happens during the java core tests if the message
   * dictionary is missing newly added xxx_reply_t messages.
   * Should never happen, but since I shot myself in the foot once
   * this way, I thought I'd make it easy to debug if I ever do
   * it again... (;-)...
   */
  if (c->id == 0)
    {
      if (c->name)
	clib_warning ("Trying to register %s with a NULL msg id!", c->name);
      else
	clib_warning ("Trying to register a NULL msg with a NULL msg id!");
      clib_warning ("Did you forget to call setup_message_id_table?");
      return;
    }

#define _(a) vec_validate (am->a, c->id);
  foreach_msg_api_vector;
#undef _

  if (am->msg_handlers[c->id] && am->msg_handlers[c->id] != c->handler)
    clib_warning
      ("BUG: re-registering 'vl_api_%s_t_handler'."
       "Handler was %llx, replaced by %llx",
       c->name, am->msg_handlers[c->id], c->handler);

  am->msg_names[c->id] = c->name;
  am->msg_handlers[c->id] = c->handler;
  am->msg_cleanup_handlers[c->id] = c->cleanup;
  am->msg_endian_handlers[c->id] = c->endian;
  am->msg_print_handlers[c->id] = c->print;
  am->message_bounce[c->id] = c->message_bounce;
  am->is_mp_safe[c->id] = c->is_mp_safe;

  am->api_trace_cfg[c->id].size = c->size;
  am->api_trace_cfg[c->id].trace_enable = c->traced;
  am->api_trace_cfg[c->id].replay_enable = c->replay;
}

/*
 * vl_msg_api_set_handlers
 * preserve the old API for a while
 */
void
vl_msg_api_set_handlers (int id, char *name, void *handler, void *cleanup,
			 void *endian, void *print, int size, int traced)
{
  vl_msg_api_msg_config_t cfg;
  vl_msg_api_msg_config_t *c = &cfg;

  clib_memset (c, 0, sizeof (*c));

  c->id = id;
  c->name = name;
  c->handler = handler;
  c->cleanup = cleanup;
  c->endian = endian;
  c->print = print;
  c->traced = traced;
  c->replay = 1;
  c->message_bounce = 0;
  c->is_mp_safe = 0;
  vl_msg_api_config (c);
}

void
vl_msg_api_clean_handlers (int msg_id)
{
  vl_msg_api_msg_config_t cfg;
  vl_msg_api_msg_config_t *c = &cfg;

  clib_memset (c, 0, sizeof (*c));

  c->id = msg_id;
  vl_msg_api_config (c);
}

void
vl_msg_api_set_cleanup_handler (int msg_id, void *fp)
{
  api_main_t *am = &api_main;
  ASSERT (msg_id > 0);

  vec_validate (am->msg_cleanup_handlers, msg_id);
  am->msg_cleanup_handlers[msg_id] = fp;
}

void
vl_msg_api_queue_handler (svm_queue_t * q)
{
  uword msg;

  while (!svm_queue_sub (q, (u8 *) & msg, SVM_Q_WAIT, 0))
    vl_msg_api_handler ((void *) msg);
}

vl_api_trace_t *
vl_msg_api_trace_get (api_main_t * am, vl_api_trace_which_t which)
{
  switch (which)
    {
    case VL_API_TRACE_RX:
      return am->rx_trace;
    case VL_API_TRACE_TX:
      return am->tx_trace;
    default:
      return 0;
    }
}

void
vl_noop_handler (void *mp)
{
}


static u8 post_mortem_dump_enabled;

void
vl_msg_api_post_mortem_dump_enable_disable (int enable)
{
  post_mortem_dump_enabled = enable;
}

void
vl_msg_api_post_mortem_dump (void)
{
  api_main_t *am = &api_main;
  FILE *fp;
  char filename[64];
  int rv;

  if (post_mortem_dump_enabled == 0)
    return;

  snprintf (filename, sizeof (filename), "/tmp/api_post_mortem.%d",
	    getpid ());

  fp = fopen (filename, "w");
  if (fp == NULL)
    {
      rv = write (2, "Couldn't create ", 16);
      rv = write (2, filename, strlen (filename));
      rv = write (2, "\n", 1);
      return;
    }
  rv = vl_msg_api_trace_save (am, VL_API_TRACE_RX, fp);
  fclose (fp);
  if (rv < 0)
    {
      rv = write (2, "Failed to save post-mortem API trace to ", 40);
      rv = write (2, filename, strlen (filename));
      rv = write (2, "\n", 1);
    }

}

/* Layered message handling support */

void
vl_msg_api_register_pd_handler (void *fp, u16 msg_id_host_byte_order)
{
  api_main_t *am = &api_main;

  /* Mild idiot proofing */
  if (msg_id_host_byte_order > 10000)
    clib_warning ("msg_id_host_byte_order endian issue? %d arg vs %d",
		  msg_id_host_byte_order,
		  clib_net_to_host_u16 (msg_id_host_byte_order));
  vec_validate (am->pd_msg_handlers, msg_id_host_byte_order);
  am->pd_msg_handlers[msg_id_host_byte_order] = fp;
}

int
vl_msg_api_pd_handler (void *mp, int rv)
{
  api_main_t *am = &api_main;
  int (*fp) (void *, int);
  u16 msg_id;

  if (clib_arch_is_little_endian)
    msg_id = clib_net_to_host_u16 (*((u16 *) mp));
  else
    msg_id = *((u16 *) mp);

  if (msg_id >= vec_len (am->pd_msg_handlers)
      || am->pd_msg_handlers[msg_id] == 0)
    return rv;

  fp = am->pd_msg_handlers[msg_id];
  rv = (*fp) (mp, rv);
  return rv;
}

void
vl_msg_api_set_first_available_msg_id (u16 first_avail)
{
  api_main_t *am = &api_main;

  am->first_available_msg_id = first_avail;
}

u16
vl_msg_api_get_msg_ids (const char *name, int n)
{
  api_main_t *am = &api_main;
  u8 *name_copy;
  vl_api_msg_range_t *rp;
  uword *p;
  u16 rv;

  if (am->msg_range_by_name == 0)
    am->msg_range_by_name = hash_create_string (0, sizeof (uword));

  name_copy = format (0, "%s%c", name, 0);

  p = hash_get_mem (am->msg_range_by_name, name_copy);
  if (p)
    {
      clib_warning ("WARNING: duplicate message range registration for '%s'",
		    name_copy);
      vec_free (name_copy);
      return ((u16) ~ 0);
    }

  if (n < 0 || n > 1024)
    {
      clib_warning
	("WARNING: bad number of message-IDs (%d) requested by '%s'",
	 n, name_copy);
      vec_free (name_copy);
      return ((u16) ~ 0);
    }

  vec_add2 (am->msg_ranges, rp, 1);

  rv = rp->first_msg_id = am->first_available_msg_id;
  am->first_available_msg_id += n;
  rp->last_msg_id = am->first_available_msg_id - 1;
  rp->name = name_copy;

  hash_set_mem (am->msg_range_by_name, name_copy, rp - am->msg_ranges);

  return rv;
}

void
vl_msg_api_add_msg_name_crc (api_main_t * am, const char *string, u32 id)
{
  uword *p;

  if (am->msg_index_by_name_and_crc == 0)
    am->msg_index_by_name_and_crc = hash_create_string (0, sizeof (uword));

  p = hash_get_mem (am->msg_index_by_name_and_crc, string);
  if (p)
    {
      clib_warning ("attempt to redefine '%s' ignored...", string);
      return;
    }

  hash_set_mem (am->msg_index_by_name_and_crc, string, id);
}

void
vl_msg_api_add_version (api_main_t * am, const char *string,
			u32 major, u32 minor, u32 patch)
{
  api_version_t version = {.major = major,.minor = minor,.patch = patch };
  ASSERT (strlen (string) < 64);
  strncpy (version.name, string, 64 - 1);
  vec_add1 (am->api_version_list, version);
}

u32
vl_msg_api_get_msg_index (u8 * name_and_crc)
{
  api_main_t *am = &api_main;
  uword *p;

  if (am->msg_index_by_name_and_crc)
    {
      p = hash_get_mem (am->msg_index_by_name_and_crc, name_and_crc);
      if (p)
	return p[0];
    }
  return ~0;
}

void *
vl_msg_push_heap (void)
{
  api_main_t *am = &api_main;
  pthread_mutex_lock (&am->vlib_rp->mutex);
  return svm_push_data_heap (am->vlib_rp);
}

void
vl_msg_pop_heap (void *oldheap)
{
  api_main_t *am = &api_main;
  svm_pop_heap (oldheap);
  pthread_mutex_unlock (&am->vlib_rp->mutex);
}

int
vl_api_to_api_string (u32 len, const char *buf, vl_api_string_t * str)
{
  clib_memcpy_fast (str->buf, buf, len);
  str->length = htonl (len);
  return len + sizeof (u32);
}

int
vl_api_vec_to_api_string (const u8 * vec, vl_api_string_t * str)
{
  u32 len = vec_len (vec);
  clib_memcpy (str->buf, vec, len);
  str->length = htonl (len);
  return len + sizeof (u32);
}

/* Return a pointer to the API string (not nul terminated */
u8 *
vl_api_from_api_string (vl_api_string_t * astr)
{
  return astr->buf;
}

u32
vl_api_string_len (vl_api_string_t * astr)
{
  return clib_net_to_host_u32 (astr->length);
}

/*
 * Returns a new vector. Remember to free it after use.
 */
u8 *
vl_api_from_api_to_vec (vl_api_string_t * astr)
{
  u8 *v = 0;
  vec_add (v, astr->buf, clib_net_to_host_u32 (astr->length));
  return v;
}

void
vl_api_set_elog_main (elog_main_t * m)
{
  api_main_t *am = &api_main;
  am->elog_main = m;
}

int
vl_api_set_elog_trace_api_messages (int enable)
{
  int rv;
  api_main_t *am = &api_main;

  rv = am->elog_trace_api_messages;
  am->elog_trace_api_messages = enable;
  return rv;
}

int
vl_api_get_elog_trace_api_messages (void)
{
  api_main_t *am = &api_main;

  return am->elog_trace_api_messages;
}

/*
 * fd.io coding-style-patch-verification: ON
 *
 * Local Variables:
 * eval: (c-set-style "gnu")
 * End:
 */