diff options
Diffstat (limited to 'docs/gettingstarted/developers/binary_api_support.rst')
| -rw-r--r-- | docs/gettingstarted/developers/binary_api_support.rst | 412 |
1 files changed, 412 insertions, 0 deletions
diff --git a/docs/gettingstarted/developers/binary_api_support.rst b/docs/gettingstarted/developers/binary_api_support.rst new file mode 100644 index 00000000000..a99bf3faa98 --- /dev/null +++ b/docs/gettingstarted/developers/binary_api_support.rst @@ -0,0 +1,412 @@ +.. _binary_api_support: + +.. toctree:: + +Binary API Support +================== + +VPP provides a binary API scheme to allow a wide variety of client +codes to program data-plane tables. As of this writing, there are +hundreds of binary APIs. + +Messages are defined in \*.api files. Today, there are about 80 api +files, with more arriving as folks add programmable features. The API +file compiler sources reside in src/tools/vppapigen. + +From `src/vnet/interface.api +<https://docs.fd.io/vpp/18.11/de/d75/interface_8api.html>`_, here's a +typical request/response message definition: + +.. code-block:: console + + autoreply define sw_interface_set_flags + { + u32 client_index; + u32 context; + u32 sw_if_index; + /* 1 = up, 0 = down */ + u8 admin_up_down; + }; + +To a first approximation, the API compiler renders this definition +into +*vpp/build-root/install-vpp_debug-native/vpp/include/vnet/interface.api.h* +as follows: + +.. code-block:: C + + /****** Message ID / handler enum ******/ + + #ifdef vl_msg_id + vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS, vl_api_sw_interface_set_flags_t_handler) + vl_msg_id(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, vl_api_sw_interface_set_flags_reply_t_handler) + #endif + /****** Message names ******/ + + #ifdef vl_msg_name + vl_msg_name(vl_api_sw_interface_set_flags_t, 1) + vl_msg_name(vl_api_sw_interface_set_flags_reply_t, 1) + #endif + /****** Message name, crc list ******/ + + #ifdef vl_msg_name_crc_list + #define foreach_vl_msg_name_crc_interface \ + _(VL_API_SW_INTERFACE_SET_FLAGS, sw_interface_set_flags, f890584a) \ + _(VL_API_SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply, dfbf3afa) \ + #endif + /****** Typedefs *****/ + + #ifdef vl_typedefs + #ifndef defined_sw_interface_set_flags + #define defined_sw_interface_set_flags + typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags { + u16 _vl_msg_id; + u32 client_index; + u32 context; + u32 sw_if_index; + u8 admin_up_down; + }) vl_api_sw_interface_set_flags_t; + #endif + + #ifndef defined_sw_interface_set_flags_reply + #define defined_sw_interface_set_flags_reply + typedef VL_API_PACKED(struct _vl_api_sw_interface_set_flags_reply { + u16 _vl_msg_id; + u32 context; + i32 retval; + }) vl_api_sw_interface_set_flags_reply_t; + #endif + ... + #endif /* vl_typedefs */ + +To change the admin state of an interface, a binary api client sends a +`vl_api_sw_interface_set_flags_t +<https://docs.fd.io/vpp/18.11/dc/da3/structvl__api__sw__interface__set__flags__t.html>`_ +to VPP, which will respond with a +vl_api_sw_interface_set_flags_reply_t message. + +Multiple layers of software, transport types, and shared libraries +implement a variety of features: + +* API message allocation, tracing, pretty-printing, and replay. +* Message transport via global shared memory, pairwise/private shared memory, and sockets. +* Barrier synchronization of worker threads across thread-unsafe message handlers. + +Correctly-coded message handlers know nothing about the transport used +to deliver messages to/from VPP. It's reasonably straighforward to use +multiple API message transport types simultaneously. + +For historical reasons, binary api messages are (putatively) sent in +network byte order. As of this writing, we're seriously considering +whether that choice makes sense. + +Message Allocation +__________________ + +Since binary API messages are always processed in order, we allocate +messages using a ring allocator whenever possible. This scheme is +extremely fast when compared with a traditional memory allocator, and +doesn't cause heap fragmentation. See `src/vlibmemory/memory_shared.c +<https://docs.fd.io/vpp/18.11/dd/d0d/memory__shared_8c.html>`_ +`vl_msg_api_alloc_internal() +<https://docs.fd.io/vpp/18.11/dd/d0d/memory__shared_8c.html#ac6b6797850e1a53bc68b206e6b8413fb>`_. + +Regardless of transport, binary api messages always follow a `msgbuf_t <https://docs.fd.io/vpp/18.11/d9/d65/structmsgbuf__.html>`_ header: + +.. code-block:: C + + /** Message header structure */ + typedef struct msgbuf_ + { + svm_queue_t *q; /**< message allocated in this shmem ring */ + u32 data_len; /**< message length not including header */ + u32 gc_mark_timestamp; /**< message garbage collector mark TS */ + u8 data[0]; /**< actual message begins here */ + } msgbuf_t; + +This structure makes it easy to trace messages without having to +decode them - simply save data_len bytes - and allows +`vl_msg_api_free() +<https://docs.fd.io/vpp/18.11/d6/d1b/api__common_8h.html#aff61e777fe5df789121d8e78134867e6>`_ +to rapidly dispose of message buffers: + +.. code-block:: C + + void + vl_msg_api_free (void *a) + { + msgbuf_t *rv; + void *oldheap; + api_main_t *am = &api_main; + + rv = (msgbuf_t *) (((u8 *) a) - offsetof (msgbuf_t, data)); + + /* + * Here's the beauty of the scheme. Only one proc/thread has + * control of a given message buffer. To free a buffer, we just clear the + * queue field, and leave. No locks, no hits, no errors... + */ + if (rv->q) + { + rv->q = 0; + rv->gc_mark_timestamp = 0; + <more code...> + return; + } + <more code...> + } + +Message Tracing and Replay +__________________________ + +It's extremely important that VPP can capture and replay sizeable +binary API traces. System-level issues involving hundreds of thousands +of API transactions can be re-run in a second or less. Partial replay +allows one to binary-search for the point where the wheels fall +off. One can add scaffolding to the data plane, to trigger when +complex conditions obtain. + +With binary API trace, print, and replay, system-level bug reports of +the form "after 300,000 API transactions, the VPP data-plane stopped +forwarding traffic, FIX IT!" can be solved offline. + +More often than not, one discovers that a control-plane client +misprograms the data plane after a long time or under complex +circumstances. Without direct evidence, "it's a data-plane problem!" + +See `src/vlibmemory/memory_vlib::c +<https://docs.fd.io/vpp/18.11/dd/d3e/vpp__get__metrics_8c.html#a7c3855ed3c45b48ff92a7e881bfede73>`_ +`vl_msg_api_process_file() +<https://docs.fd.io/vpp/18.11/d0/d5b/vlib__api__cli_8c.html#a60194e3e91c0dc6a75906ea06f4ec113>`_, +and `src/vlibapi/api_shared.c +<https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html>`_. See also +the debug CLI command "api trace" + +Client connection details +_________________________ + +Establishing a binary API connection to VPP from a C-language client is easy: + +.. code-block:: C + + int + connect_to_vpe (char *client_name, int client_message_queue_length) + { + vat_main_t *vam = &vat_main; + api_main_t *am = &api_main; + if (vl_client_connect_to_vlib ("/vpe-api", client_name, + client_message_queue_length) < 0) + return -1; + /* Memorize vpp's binary API message input queue address */ + vam->vl_input_queue = am->shmem_hdr->vl_input_queue; + /* And our client index */ + vam->my_client_index = am->my_client_index; + return 0; + } + +32 is a typical value for client_message_queue_length. VPP *cannot* +block when it needs to send an API message to a binary API client. The +VPP-side binary API message handlers are very fast. So, when sending +asynchronous messages, make sure to scrape the binary API rx ring with +some enthusiasm! + +**Binary API message RX pthread** + +Calling `vl_client_connect_to_vlib +<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a6654b42c91be33bfb6a4b4bfd2327920>`_ +spins up a binary API message RX pthread: + +.. code-block:: C + + static void * + rx_thread_fn (void *arg) + { + svm_queue_t *q; + memory_client_main_t *mm = &memory_client_main; + api_main_t *am = &api_main; + int i; + + q = am->vl_input_queue; + + /* So we can make the rx thread terminate cleanly */ + if (setjmp (mm->rx_thread_jmpbuf) == 0) + { + mm->rx_thread_jmpbuf_valid = 1; + /* + * Find an unused slot in the per-cpu-mheaps array, + * and grab it for this thread. We need to be able to + * push/pop the thread heap without affecting other thread(s). + */ + if (__os_thread_index == 0) + { + for (i = 0; i < ARRAY_LEN (clib_per_cpu_mheaps); i++) + { + if (clib_per_cpu_mheaps[i] == 0) + { + /* Copy the main thread mheap pointer */ + clib_per_cpu_mheaps[i] = clib_per_cpu_mheaps[0]; + __os_thread_index = i; + break; + } + } + ASSERT (__os_thread_index > 0); + } + while (1) + vl_msg_api_queue_handler (q); + } + pthread_exit (0); + } + +To handle the binary API message queue yourself, use +`vl_client_connect_to_vlib_no_rx_pthread +<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a11b9577297106c57c0783b96ab190c36>`_. + +**Queue non-empty signalling** + +vl_msg_api_queue_handler(...) uses mutex/condvar signalling to wake +up, process VPP -> client traffic, then sleep. VPP supplies a condvar +broadcast when the VPP -> client API message queue transitions from +empty to nonempty. + +VPP checks its own binary API input queue at a very high rate. VPP +invokes message handlers in "process" context [aka cooperative +multitasking thread context] at a variable rate, depending on +data-plane packet processing requirements. + +Client disconnection details +____________________________ + +To disconnect from VPP, call `vl_client_disconnect_from_vlib +<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a82c9ba6e7ead8362ae2175eefcf2fd12>`_. Please +arrange to call this function if the client application terminates +abnormally. VPP makes every effort to hold a decent funeral for dead +clients, but VPP can't guarantee to free leaked memory in the shared +binary API segment. + +Sending binary API messages to VPP +__________________________________ + +The point of the exercise is to send binary API messages to VPP, and +to receive replies from VPP. Many VPP binary APIs comprise a client +request message, and a simple status reply. For example, to set the +admin status of an interface: + +.. code-block:: C + + vl_api_sw_interface_set_flags_t *mp; + mp = vl_msg_api_alloc (sizeof (*mp)); + memset (mp, 0, sizeof (*mp)); + mp->_vl_msg_id = clib_host_to_net_u16 (VL_API_SW_INTERFACE_SET_FLAGS); + mp->client_index = api_main.my_client_index; + mp->sw_if_index = clib_host_to_net_u32 (<interface-sw-if-index>); + vl_msg_api_send (api_main.shmem_hdr->vl_input_queue, (u8 *)mp); + +Key points: + +* Use `vl_msg_api_alloc <https://docs.fd.io/vpp/18.11/dc/d5a/memory__shared_8h.html#a109ff1e95ebb2c968d43c100c4a1c55a>`_ to allocate message buffers +* Allocated message buffers are not initialized, and must be presumed to contain trash. +* Don't forget to set the _vl_msg_id field! +* As of this writing, binary API message IDs and data are sent in network byte order +* The client-library global data structure `api_main <https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html#af58e3e46b569573e9622b826b2f47a22>`_ keeps track of sufficient pointers and handles used to communicate with VPP + +Receiving binary API messages from VPP +______________________________________ + +Unless you've made other arrangements (see +`vl_client_connect_to_vlib_no_rx_pthread +<https://docs.fd.io/vpp/18.11/da/d25/memory__client_8h.html#a11b9577297106c57c0783b96ab190c36>`_), +*messages are received on a separate rx pthread*. Synchronization with +the client application main thread is the responsibility of the +application! + +Set up message handlers about as follows: + +.. code-block:: C + + #define vl_typedefs /* define message structures */ + #include <vpp/api/vpe_all_api_h.h> + #undef vl_typedefs + /* declare message handlers for each api */ + #define vl_endianfun /* define message structures */ + #include <vpp/api/vpe_all_api_h.h> + #undef vl_endianfun + /* instantiate all the print functions we know about */ + #define vl_print(handle, ...) + #define vl_printfun + #include <vpp/api/vpe_all_api_h.h> + #undef vl_printfun + /* Define a list of all message that the client handles */ + #define foreach_vpe_api_reply_msg \ + _(SW_INTERFACE_SET_FLAGS_REPLY, sw_interface_set_flags_reply) + static clib_error_t * + my_api_hookup (vlib_main_t * vm) + { + api_main_t *am = &api_main; + #define _(N,n) \ + vl_msg_api_set_handlers(VL_API_##N, #n, \ + vl_api_##n##_t_handler, \ + vl_noop_handler, \ + vl_api_##n##_t_endian, \ + vl_api_##n##_t_print, \ + sizeof(vl_api_##n##_t), 1); + foreach_vpe_api_msg; + #undef _ + return 0; + } + +The key API used to establish message handlers is +`vl_msg_api_set_handlers +<https://docs.fd.io/vpp/18.11/d6/dd1/api__shared_8c.html#aa8a8e1f3876ec1a02f283c1862ecdb7a>`_ +, which sets values in multiple parallel vectors in the `api_main_t +<https://docs.fd.io/vpp/18.11/dd/db2/structapi__main__t.html>`_ +structure. As of this writing: not all vector element values can be +set through the API. You'll see sporadic API message registrations +followed by minor adjustments of this form: + +.. code-block:: C + + /* + * Thread-safe API messages + */ + am->is_mp_safe[VL_API_IP_ADD_DEL_ROUTE] = 1; + am->is_mp_safe[VL_API_GET_NODE_GRAPH] = 1; + +API message numbering in plugins +-------------------------------- + +Binary API message numbering in plugins relies on vpp to issue a block +of message-ID's for the plugin to use: + +.. code-block:: C + + static clib_error_t * + my_init (vlib_main_t * vm) + { + my_main_t *mm = &my_main; + + name = format (0, "myplugin_%08x%c", api_version, 0); + + /* Ask for a correctly-sized block of API message decode slots */ + mm->msg_id_base = vl_msg_api_get_msg_ids + ((char *) name, VL_MSG_FIRST_AVAILABLE); + + } + +Control-plane codes use the vl_client_get_first_plugin_msg_id (...) api +to recover the message ID block base: + +.. code-block:: C + + /* Ask the vpp engine for the first assigned message-id */ + name = format (0, "myplugin_%08x%c", api_version, 0); + sm->msg_id_base = vl_client_get_first_plugin_msg_id ((char *) name); + +It's a fairly common error to forget to add msg_id_base when +registering message handlers, or when sending messages. Using macros +from .../src/vlibapi/api_helper_macros.h can automate the process, but +remember to #define REPLY_MSG_ID_BASE before #including the file: + +.. code-block:: C + + #define REPLY_MSG_ID_BASE mm->msg_id_base + #include <vlibapi/api_helper_macros.h> |