libmemif: update documentation

Type: refactor

Signed-off-by: Jakub Grajciar <jgrajcia@cisco.com>
Change-Id: I0094ea8627cd8bcd5ea119c2fd48f077c8e2e4bb

4 files changed, 157 insertions, 331 deletions

diff --git a/extras/libmemif/docs/buildinstructions_doc.md b/extras/libmemif/docs/buildinstructions_doc.md
index c2eecba4d96..3f7c8e8d66d 100644
--- a/extras/libmemif/docs/buildinstructions_doc.md
+++ b/extras/libmemif/docs/buildinstructions_doc.md
@@ -2,7 +2,7 @@
 
 #### Install dependencies
 ```
-# sudo apt-get install -y git cmake autoconf pkg_config libtool check
+# sudo apt-get install -y git cmake autoconf pkg_config libtool
 ```
 
 Libmemif is now part of VPP repository. Follow fd.io wiki to pull source code from VPP repository.
@@ -18,30 +18,28 @@ Libmemif is located under extras/libmemif. From extras/libmemif:
 
 #### Verify installation:
 ```
-build# ./examples/icmpr-epoll
+build# ./examples/icmp_responder -?
 ```
-Use _help_ command to display build information and commands:
+Use `-?` flag to display help:
 ```
-LIBMEMIF EXAMPLE APP: ICMP_Responder
+LIBMEMIF EXAMPLE APP: icmp_responder_example
 ==============================
-libmemif version: 3.0
-memif version: 512
-	use CTRL+C to exit
-MEMIF DETAILS
+libmemif version: 4.0, memif version: 2.0
 ==============================
-	interface name: memif_connection
-	app name: ICMP_Responder
-	remote interface name:
-	remote app name:
-	id: 0
-	secret: (null)
-	role: slave
-	mode: ethernet
-	socket filename: /run/vpp/memif.sock
-	socket filename: /run/vpp/memif.sock
-	rx queues:
-	tx queues:
-	link: down
+In this example, memif endpoint connects to an external application.
+The example application can resolve ARP and reply to ICMPv4 packets.
+The program will exit once the interface is disconnected.
+==============================
+Usage: icmp_responder [OPTIONS]
+
+Options:
+        -r      Interface role <slave|master>. Default: slave
+        -s      Socket path. Supports abstract socket using @ before the path. Default: /run/vpp/memif.sock
+        -i      Interface id. Default: 0
+        -a      IPv4 address. Default: 192.168.1.1
+        -h      Mac address. Default: aa:aa:aa:aa:aa:aa
+        -?      Show help and exit.
+        -v      Show libmemif and memif version information and exit.
 ```
 
 #### Examples
diff --git a/extras/libmemif/docs/devperftest_doc.md b/extras/libmemif/docs/devperftest_doc.md
deleted file mode 100644
index 4c210f5b514..00000000000
--- a/extras/libmemif/docs/devperftest_doc.md
+++ /dev/null
@@ -1,94 +0,0 @@
-## Development performance test    {#libmemif_devperftest_doc}
-
-Simple test cases using ICMP. icmpr-epoll example app generates and transmits packets over memif interface.
-
-#### TC1: LIB-VPP
-
-Start icmpr-epoll example app and VPP.
-
-VPP-side config:
-```
-DBGvpp# create interface memif id 0 master
-DBGvpp# set int state memif0/0 up
-DBGvpp# set int ip address memif0/0 192.168.1.1/24
-```
-icmpr-epoll:
-```
-conn 0 0 1
-```
-> Last argument specifies interrupt function to use. This function only responds to ARP requests. This is important because, packet generation and transmitting is handled by a separate thread. Calling memif_tx_burst from multiple threads writing on same queue could transmit uninitialized buffers.
-Once connection is established, you can send ping from VPP to icmpr-epoll app to learn its mac address.
-```
-DBGvpp# ping 192.168.1.2
-```
-> There should be no ICMP response. Only ARP response.
-Now send ICMP requests from icmpr-epoll:
-```
-send <index> <num-of-packets> <ip_daddr> <hw_daddr>
-send 0 5 192.168.1.1 02:fe:ff:ff:ff:ff
-```
-this command will create new thread which will generate icmp packets and transmit them over memif connection with specified index. Once the sequence is finished status will be printed.
-
-###### Example results (overview of test data)
-
-(This test was run with modification in VPP-memif plugin. The modification disallows memif tx node to allocate last ring buffer)
-lib-tx: 200M (if ring full don't drop packets)
-vpp-rx: 200M
-vpp-tx: 200M - 50K (if ring full drop packets)
-lib-rx: =vpp-tx
-drop: ~0.025% (full ring)
-pps: ~650K
-multiple interfaces:
-pps: divided
-drop: constant
-
-#### TC2: LIB-LIB
-
-This test case will not drop packets if memif ring is full. Instead it will loop until all required packets have been sent.
-
-Start two instances of icmpr-epoll example app.
-instance 1:
-```
-conn 0 1 0
-```
-instance 2:
-```
-conn 0 0 1
-send 0 5 192.168.1.1 aa:aa:aa:aa:aa:aa
-```
-> icmpr-epoll example app doesn't check ip or mac address so as long as the format is correct you can type anything as ip_daddr and hw_daddr arguments.
-
-###### Example results (overview of test data)
-
-lib1-tx: 200M (if ring full don't drop packets)
-lib2-rx: 200M
-lib2-tx: 200M (if ring full don't drop packets)
-lib1-rx: 200M
-drop: obsolete
-pps: 4.5M
-multiple interfaces:
-not tested (expected same as TC1)
-
-#### TC3: LIB-LIB
-
-Start two instances of icmpr-epoll example app.
-instance 1:
-```
-conn 0 1
-```
-instance 2:
-```
-conn 0 0 1
-send 0 5 192.168.1.1 aa:aa:aa:aa:aa:aa
-```
-
-###### Example results (overview of test data)
-
-lib1-tx: 200M (if ring full don't drop packets)
-lib2-rx: 200M
-lib2-tx: 169626182 (if ring full drop packets)
-lib1-rx: =lib2-tx
-drop: ~15%
-pps: ~6M
-multiple interfaces:
-not tested (expected same as TC1)
diff --git a/extras/libmemif/docs/gettingstarted_doc.md b/extras/libmemif/docs/gettingstarted_doc.md
index abf9a1a06fd..6f2a99c8c69 100644
--- a/extras/libmemif/docs/gettingstarted_doc.md
+++ b/extras/libmemif/docs/gettingstarted_doc.md
@@ -1,223 +1,164 @@
 ## Getting started    {#libmemif_gettingstarted_doc}
 
-#### Concept (Connecting to VPP)
-
 For detailed information on api calls and structures please refer to @ref libmemif.h.
 
-1. Initialize memif
-   - Declare callback function handling file descriptor event polling.
-```C
-int
-control_fd_update (int fd, uint8_t events)
-{
-...
-}
+Start by creating a memif socket. Memif socket represents UNIX domain socket and interfaces assigned to use this socket. Memif uses UNIX doman socket to communicate with other memif drivers.
+
+First fill out the `memif_socket_args` struct. The minimum required configuration is the UNIX socket path.
+> Use `@` or `\0` at the beginning of the path to use abstract socket.
+```c
+memif_socket_args_t sargs;
+
+strncpy(sargs.path, socket_path, sizeof(sargs.path));
+```
+```c
+memif_socket_handle_t memif_socket;
+
+memif_create_socket(&memif_socket, &sargs, &private_data);
 ```
-   - Call memif initialization function. memif\_init
-```C
-err = memif_init (control_fd_update, APP_NAME, NULL, NULL);
+Once you have created your socket, you can create memif interfaces on this socket. Fill out the `memif_conn_args` struct. Then call `memif_create()`.
+```c
+memif_conn_args_t cargs;
+
+/* Assign your socket handle */
+cargs.socket = memif_socket;
 ```
-   
-> If event occurs on any file descriptor returned by this callback, call memif\_control\_fd\_handler function. Since version 2.0, last two optional arguments are used to specify custom memory allocation.
-```C
-memif_err = memif_control_fd_handler (evt.data.fd, events);
-``` 
-> If callback function parameter for memif\_init function is set to NULL, libmemif will handle file descriptor event polling.
-  Api call memif\_poll\_event will call epoll\_pwait with user defined timeout to poll event on file descriptors opened by libmemif.
-```C
-/* main loop */
-    while (1)
-    {
-        if (memif_poll_event (-1) < 0)
-        {
-            DBG ("poll_event error!");
-        }
-    }
+```c
+memif_conn_handle_t conn;
+
+/* Assign callbacks */
+memif_create (&conn, &cargs, on_connect_cb, on_disconnect_cb, on_interrupt_cb, &private_data);
 ```
-    
-> Memif initialization function will initialize internal structures and create timer file descriptor, which will be used for sending periodic connection requests. Timer is disarmed if no memif interface is created.
- 
-2. Creating interface
-   - Declare memif connection handle.
-```C
-memif_conn_handle_t c;
+Now start the polling events using libmemifs builtin polling.
+```c
+do {
+    err = memif_poll_event(memif_socket, /* timeout -1 = blocking */ -1);
+} while (err == MEMIF_ERR_SUCCESS);
 ```
-> example app uses struct that contains connection handle, rx/tx buffers and other connection specific information.
-
-   - Specify connection arguments.
-```C
-memif_conn_args_t args;
-memset (&args, 0, sizeof (args));
-args.is_master = is_master;
-args.log2_ring_size = 10;
-args.buffer_size = 2048;
-args.num_s2m_rings = 2;
-args.num_m2s_rings = 2;
-strncpy ((char *) args.interface_name, IF_NAME, strlen (IF_NAME));
-args.mode = 0;
-args.interface_id = 0;
+Polling can be canceled by calling `memif_cancel_poll_event()`.
+```c
+memif_cancel_poll_event (memif_socket);
 ```
-   - Declare callback functions called on connected/disconnected/interrupted status changed.
-```C
-int
-on_connect (memif_conn_handle_t conn, void *private_ctx)
+On link status change `on_connect` and `on_disconnect` callbacks are called respectively. Before you can start transmitting data you, first need to call `memif_refill_queue()` for each RX queue to initialize this queue.
+```c
+int on_connect (memif_conn_handle_t conn, void *private_ctx)
 {
-...
-}
+  my_private_data_t *data = (my_private_data_t *) private_ctx;
 
-int
-on_disconnect (memif_conn_handle_t conn, void *private_ctx)
-{
-    INFO ("memif connected!");
-    return 0;
+  err = memif_refill_queue(conn, 0, -1, 0);
+  if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_refill_queue: %s", memif_strerror(err));
+    return err;
+  }
+
+  /*
+   * Do stuff.
+   */
+
+  return 0;
 }
 ```
-   - Call memif interface create function. memif\_create
-```C
-err = memif_create (&c->conn,
-        &args, on_connect, on_disconnect, on_interrupt, &ctx[index]);
-```
-> If connection is in slave mode, arms timer file descriptor.
-> If on interrupt callback is set to NULL, user will not be notified about interrupt. Use memif\_get\_queue\_efd call to get interrupt file descriptor for specific queue.
-```C
-int fd = -1;
-err = memif_get_queue_efd (c->conn, data->qid, &fd);
-```
+Now you are ready to transmit packets.
+> Example implementation @ref examples/common/sender.c and @ref examples/common/responder.c
 
-3. Connection establishment
-    - User application will poll events on all file descriptors returned in memif\_control\_fd\_update\_t callback.
-    - On event call memif\_control\_fd\_handler.
-    - Everything else regarding connection establishment will be done internally.
-    - Once connection has been established, a callback will inform the user about connection status change.
-
-4. Interrupt packet receive
-    - If event is polled on interrupt file descriptor, libmemif will call memif\_interrupt\_t callback specified for every connection instance.
-```C
-int
-on_interrupt (memif_conn_handle_t conn, void *private_ctx, uint16_t qid)
-{
-...
+To transmit or receive data you will need to use `memif_buffer` struct. The important fields here are `void *data`, `uint32_t len` and `uint8_t flags`. The `data` pointer points directly to the shared memory packet buffer. This is where you will find/insert your packets. The `len` field is the length of the buffer. If the flag `MEMIF_BUFFER_FLAG_NEXT` is present in `flags` field, this buffer is chained so the rest of the data is located in the next buffer, and so on.
+
+First let's receive data. To receive data call `memif_rx_burst()`. The function will fill out memif buffers passed to it. Then you would process your data (e.g. copy to your stack). Last you must refill the queue using `memif_refill_queue()` to notify peer that the buffers are now free and can be overwritten.
+```c
+/* Fill out memif buffers and mark them as received */
+err = memif_rx_burst(conn, qid, buffers, num_buffers, &num_received);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO ("memif_rx_burst: %s", memif_strerror(err));
+    return err;
+}
+/*
+    Process the buffers.
+*/
+
+/* Refill the queue, so that the peer interface can transmit more packets */
+err = memif_refill_queue(conn, qid, num_received, 0);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_refill_queue: %s", memif_strerror(err));
+    goto error;
 }
 ```
+In order to transmit data you first need to 'allocate' memif buffers using `memif_buffer_alloc()`. This function simmilar to `memif_rx_burst` will fill out provided memif buffers. You will then insert your packets directly into the shared memory (don't forget to update `len` filed if your packet is smaller that buffer length). Finaly call `memif_tx_burst` to transmit the buffers.
+```c
+/* Alocate memif buffers */
+err = memif_buffer_alloc(conn, qid, buffers, num_pkts, &num_allocated, packet_size);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_buffer_alloc: %s", memif_strerror(err));
+    goto error;
+}
 
-6. Memif buffers
-    - Packet data are stored in memif\_buffer\_t. Pointer _data_ points to shared memory buffer, and unsigned integer *_len* contains buffer length.
-    - flags: MEMIF\_BUFFER\_FLAG\_NEXT states that the buffer is not large enough to contain whole packet, so next buffer contains the rest of the packet. (chained buffers)
-```C
-typedef struct
-{
-    uint16_t desc_index;
-    uint32_t len;
-    uint8_t flags;
-    void *data;
-} memif_buffer_t;
-```
+/*
+    Fill out the buffers.
 
-5. Packet receive
-    - Api call memif\_rx\_burst will set all required fields in memif buffers provided by user application, dequeue received buffers and consume interrupt event on receive queue. The event is not consumed, if memif_rx_burst fails.
-```C
-err = memif_rx_burst (c->conn, qid, c->bufs, MAX_MEMIF_BUFS, &rx);
-```
-    - User application can then process packets.
-    - Api call memif\_refill\_queue will enqueue rx buffers.
-```C
-err = memif_refill_queue (c->conn, qid, rx);
-```
+    tx_buffers[i].data field points to the shared memory.
+    update tx_buffers[i].len to your packet length, if the packet is smaller.
+*/
 
-6. Packet transmit
-    - Api call memif\_buffer\_alloc will find free tx buffers and set all required fields in memif buffers provided by user application.
-```C
-err = memif_buffer_alloc (c->conn, qid, c->tx_bufs, n, &r);
-```
-    - User application can populate shared memory buffers with packets.
-    - Api call memif\_tx\_burst will enqueue tx buffers
-```C
-err = memif_tx_burst (c->conn, qid, c->tx_bufs, c->tx_buf_num, &r);
+/* Transmit the buffers */
+err = memif_tx_burst(conn, qid, buffers, num_allocated, &num_transmitted);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_tx_burst: %s", memif_strerror(err));
+    goto error;
+}
 ```
+### Zero-copy Slave
 
-7. Helper functions
-    - Memif version
-```C
-uint16_t memif_ver = memif_get_version ();
-```
-    - Memif details
-      - Api call memif\_get\_details will return details about connection.
-```C
-err = memif_get_details (c->conn, &md, buf, buflen);
-```
-    - Memif error messages
-      - Every api call returns error code (integer value) mapped to error string.
-      - Call memif\_strerror will return error message assigned to specific error code.
-```C
-if (err != MEMIF_ERR_SUCCESS)
-    INFO ("memif_get_details: %s", memif_strerror (err));
-```
-        - Not all syscall errors are translated to memif error codes. If error code 1 (MEMIF\_ERR\_SYSCALL) is returned then libmemif needs to be compiled with -DMEMIF_DBG flag to print error message. Use _make -B_ to rebuild libmemif in debug mode.
+Interface with slave role is the buffer producer, as such it can use zero-copy mode.
 
-#### Example app (libmemif fd event polling):
+After receiving buffers, process your packets in place. Then use `memif_buffer_enq_tx()` to enqueue rx buffers to tx queue (by swapping rx buffer with a free tx buffer).
+```c
+/* Fill out memif buffers and mark them as received */
+err = memif_rx_burst(conn, qid, buffers, num_buffers, &num_received);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO ("memif_rx_burst: %s", memif_strerror(err));
+    return err;
+}
 
-- @ref extras/libmemif/examples/icmp_responder
+/*
+    Process the buffers in place.
+*/
 
-> Optional argument: transmit queue id.
-```
-icmpr 1
-```
-> Set transmit queue id to 1. Default is 0.
-> Application will create memif interface in slave mode and try to connect to VPP. Exit using Ctrl+C. Application will handle SIGINT signal, free allocated memory and exit with EXIT_SUCCESS.
+/* Enqueue processed buffers to tx queue */
+err = memif_buffer_enq_tx(conn, qid, buffers, num_buffers, &num_enqueued);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_buffer_alloc: %s", memif_strerror(err));
+    goto error;
+}
 
-#### Example app:
+/* Refill the queue, so that the peer interface can transmit more packets */
+err = memif_refill_queue(conn, qid, num_enqueued, 0);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_refill_queue: %s", memif_strerror(err));
+    goto error;
+}
 
-ICMP Responder custom fd event polling.
+/* Transmit the buffers. */
+err = memif_tx_burst(conn, qid, buffers, num_enqueued, &num_transmitted);
+if (err != MEMIF_ERR_SUCCESS) {
+    INFO("memif_tx_burst: %s", memif_strerror(err));
+    goto error;
+}
+```
 
-- @ref extras/libmemif/examples/icmp_responder-epoll
+### Custom Event Polling
 
-#### Example app (multi-thread queue polling)
+Libmemif can be integrated into your applications fd event polling. You will need to implement `memif_control_fd_update_t` callback and pass it to `memif_socket_args.on_control_fd_update`. Now each time any file descriptor belonging to that socket updates, `on_control_fd_update` callback is called. The file descriptor and event type is passed in `memif_fd_event_t`. It also contains private context that is associated with this fd. When event is polled on the fd you need to call `memif_control_fd_handler` and pass the event type and private context associated with the fd.
 
-ICMP Responder multi-thread.
-- @ref extras/libmemif/examples/icmp_responder-mt
+### Multi Threading
 
-> Simple example of libmemif multi-thread usage. Connection establishment is handled by main thread. There are two rx/tx queues in this example. One in polling mode and second in interrupt mode.
+#### Connection establishment
 
-VPP config:
-```
-# create interface memif id 0 master
-# set int state memif0 up
-# set int ip address memif0 192.168.1.1/24
-# ping 192.168.1.2
-```
-For multiple rings (queues) support run VPP with worker threads:
-example startup.conf:
-```
-unix {
-  interactive
-  nodaemon 
-  full-coredump
-}
+Memif sockets should not be handled in paralell. Instead each thread should have it's own socket. However the UNIX socket can be the same. In case of non-listener socket, it's straight forward, just create the socket using the same path. In case of listener socket, the polling should be done by single thread.
+> The socket becomes listener once a Master interface is assigned to it.
 
-cpu {
-  workers 2
-}
-```
-VPP config:
-```
-# create interface memif id 0 master
-# set int state memif0 up
-# set int ip address memif0 192.168.1.1/24
-# ping 192.168.1.2
-```
-> Master mode queue number is limited by worker threads. Slave mode interface needs to specify number of queues.
-```
-# create memif id 0 slave rx-queues 2 tx-queues 2
-```
-> Example applications use VPP default socket file for memif: /run/vpp/memif.sock
-> For master mode, socket directory must exist prior to memif\_create call.
+#### Packet handling
 
-#### Unit tests
+Single queue must not be handled in paralel. Instead you can assign queues to threads in such way that each queue is only assigned single thread.
 
-Unit tests use [Check](https://libcheck.github.io/check/index.html) framework. This framework must be installed in order to build *unit\_test* binary.
-Ubuntu/Debian:
-```
-sudo apt-get install check
-```
-[More platforms](https://libcheck.github.io/check/web/install.html)
+### Shared Memory Layout
 
+Please refer to [DPDK MEMIF documentation](http://doc.dpdk.org/guides/nics/memif.html) `'Shared memory'` section.
\ No newline at end of file
diff --git a/extras/libmemif/libmemif_doc.md b/extras/libmemif/libmemif_doc.md
index 1260d9c60ac..ac3d6dadcfb 100644
--- a/extras/libmemif/libmemif_doc.md
+++ b/extras/libmemif/libmemif_doc.md
@@ -20,52 +20,34 @@ Shared memory packet interface (memif) provides high performance packet transmit
 - [x] Multiple queues
   - [x] Multi-thread support
 - [x] Master mode
-	- [ ] Multiple regions (TODO)
-- [ ] Performance testing (TODO)
+	- [x] Multiple regions
+- [x] Loopback
 
 ## Quickstart
 
 This setup will run libmemif ICMP responder example app in container. Install [docker](https://docs.docker.com/engine/installation) engine.
 Useful link: [Docker documentation](https://docs.docker.com/get-started).
 
-Pull image:
+Build the docker image:
 ```
-# docker pull ligato/libmemif-sample-service
+# docker build . -t libmemif
 ```
 
-Now you should be able to see ligato/libmemif-sample-service image on your local machine (IMAGE ID in this README may be outdated):
+Now you should be able to see libmemif image on your local machine:
 ```
 # docker images
 REPOSITORY                       TAG                 IMAGE ID            CREATED              SIZE
-ligato/libmemif-sample-service   latest              32ecc2f9d013        About a minute ago   468MB
+libmemif						 latest              32ecc2f9d013        About a minute ago   468MB
 ...
 ```
 
 Run container:
 ```
-# docker run -it --rm --name icmp-responder --hostname icmp-responder --privileged -v "/run/vpp/:/run/vpp/" ligato/libmemif-sample-service
-```
-Example application will start in debug mode. Output should look like this:
-```
-ICMP_Responder:add_epoll_fd:233: fd 0 added to epoll
-ICMP_Responder:add_epoll_fd:233: fd 5 added to epoll
-LIBMEMIF EXAMPLE APP: ICMP_Responder (debug)
-==============================
-libmemif version: 2.0 (debug)
-memif version: 512
-commands:
-	help - prints this help
-	exit - exit app
-	conn <index> <mode> [<interrupt-desc>] - create memif. index is also used as interface id, mode 0 = slave 1 = master, interrupt-desc none = default 0 = if ring is full wait 1 = handle only ARP requests
-	del  <index> - delete memif
-	show - show connection details
-	ip-set <index> <ip-addr> - set interface ip address
-	rx-mode <index> <qid> <polling|interrupt> - set queue rx mode
-	sh-count - print counters
-	cl-count - clear counters
-	send <index> <tx> <ip> <mac> - send icmp
+# docker run -it --rm --name icmp-responder --hostname icmp-responder --privileged -v "/run/vpp/:/run/vpp/" libmemif
 ```
 
+The interface will by default connect to a master interface listening on `/run/vpp/master.sock`. The example will handle ARP requests and respond to ICMPv4 requests to `192.168.1.1`.
+
 Continue with @ref libmemif_example_setup which contains instructions on how to set up connection between icmpr-epoll example app and VPP-memif.
 
 #### Next steps
@@ -74,4 +56,3 @@ Continue with @ref libmemif_example_setup which contains instructions on how to
 - @subpage libmemif_examples_doc
 - @subpage libmemif_example_setup_doc
 - @subpage libmemif_gettingstarted_doc
-- @subpage libmemif_devperftest_doc