diff options
Diffstat (limited to 'extras/libmemif')
| -rw-r--r-- | extras/libmemif/docs/buildinstructions_doc.md | 40 | ||||
| -rw-r--r-- | extras/libmemif/docs/devperftest_doc.md | 94 | ||||
| -rw-r--r-- | extras/libmemif/docs/gettingstarted_doc.md | 317 | ||||
| -rw-r--r-- | extras/libmemif/libmemif_doc.md | 37 |
4 files changed, 157 insertions, 331 deletions
diff --git a/extras/libmemif/docs/buildinstructions_doc.md b/extras/libmemif/docs/buildinstructions_doc.md index c2eecba4d96..3f7c8e8d66d 100644 --- a/extras/libmemif/docs/buildinstructions_doc.md +++ b/extras/libmemif/docs/buildinstructions_doc.md @@ -2,7 +2,7 @@ #### Install dependencies ``` -# sudo apt-get install -y git cmake autoconf pkg_config libtool check +# sudo apt-get install -y git cmake autoconf pkg_config libtool ``` Libmemif is now part of VPP repository. Follow fd.io wiki to pull source code from VPP repository. @@ -18,30 +18,28 @@ Libmemif is located under extras/libmemif. From extras/libmemif: #### Verify installation: ``` -build# ./examples/icmpr-epoll +build# ./examples/icmp_responder -? ``` -Use _help_ command to display build information and commands: +Use `-?` flag to display help: ``` -LIBMEMIF EXAMPLE APP: ICMP_Responder +LIBMEMIF EXAMPLE APP: icmp_responder_example ============================== -libmemif version: 3.0 -memif version: 512 - use CTRL+C to exit -MEMIF DETAILS +libmemif version: 4.0, memif version: 2.0 ============================== - interface name: memif_connection - app name: ICMP_Responder - remote interface name: - remote app name: - id: 0 - secret: (null) - role: slave - mode: ethernet - socket filename: /run/vpp/memif.sock - socket filename: /run/vpp/memif.sock - rx queues: - tx queues: - link: down +In this example, memif endpoint connects to an external application. +The example application can resolve ARP and reply to ICMPv4 packets. +The program will exit once the interface is disconnected. +============================== +Usage: icmp_responder [OPTIONS] + +Options: + -r Interface role <slave|master>. Default: slave + -s Socket path. Supports abstract socket using @ before the path. Default: /run/vpp/memif.sock + -i Interface id. Default: 0 + -a IPv4 address. Default: 192.168.1.1 + -h Mac address. Default: aa:aa:aa:aa:aa:aa + -? Show help and exit. + -v Show libmemif and memif version information and exit. ``` #### Examples diff --git a/extras/libmemif/docs/devperftest_doc.md b/extras/libmemif/docs/devperftest_doc.md deleted file mode 100644 index 4c210f5b514..00000000000 --- a/extras/libmemif/docs/devperftest_doc.md +++ /dev/null @@ -1,94 +0,0 @@ -## Development performance test {#libmemif_devperftest_doc} - -Simple test cases using ICMP. icmpr-epoll example app generates and transmits packets over memif interface. - -#### TC1: LIB-VPP - -Start icmpr-epoll example app and VPP. - -VPP-side config: -``` -DBGvpp# create interface memif id 0 master -DBGvpp# set int state memif0/0 up -DBGvpp# set int ip address memif0/0 192.168.1.1/24 -``` -icmpr-epoll: -``` -conn 0 0 1 -``` -> Last argument specifies interrupt function to use. This function only responds to ARP requests. This is important because, packet generation and transmitting is handled by a separate thread. Calling memif_tx_burst from multiple threads writing on same queue could transmit uninitialized buffers. -Once connection is established, you can send ping from VPP to icmpr-epoll app to learn its mac address. -``` -DBGvpp# ping 192.168.1.2 -``` -> There should be no ICMP response. Only ARP response. -Now send ICMP requests from icmpr-epoll: -``` -send <index> <num-of-packets> <ip_daddr> <hw_daddr> -send 0 5 192.168.1.1 02:fe:ff:ff:ff:ff -``` -this command will create new thread which will generate icmp packets and transmit them over memif connection with specified index. Once the sequence is finished status will be printed. - -###### Example results (overview of test data) - -(This test was run with modification in VPP-memif plugin. The modification disallows memif tx node to allocate last ring buffer) -lib-tx: 200M (if ring full don't drop packets) -vpp-rx: 200M -vpp-tx: 200M - 50K (if ring full drop packets) -lib-rx: =vpp-tx -drop: ~0.025% (full ring) -pps: ~650K -multiple interfaces: -pps: divided -drop: constant - -#### TC2: LIB-LIB - -This test case will not drop packets if memif ring is full. Instead it will loop until all required packets have been sent. - -Start two instances of icmpr-epoll example app. -instance 1: -``` -conn 0 1 0 -``` -instance 2: -``` -conn 0 0 1 -send 0 5 192.168.1.1 aa:aa:aa:aa:aa:aa -``` -> icmpr-epoll example app doesn't check ip or mac address so as long as the format is correct you can type anything as ip_daddr and hw_daddr arguments. - -###### Example results (overview of test data) - -lib1-tx: 200M (if ring full don't drop packets) -lib2-rx: 200M -lib2-tx: 200M (if ring full don't drop packets) -lib1-rx: 200M -drop: obsolete -pps: 4.5M -multiple interfaces: -not tested (expected same as TC1) - -#### TC3: LIB-LIB - -Start two instances of icmpr-epoll example app. -instance 1: -``` -conn 0 1 -``` -instance 2: -``` -conn 0 0 1 -send 0 5 192.168.1.1 aa:aa:aa:aa:aa:aa -``` - -###### Example results (overview of test data) - -lib1-tx: 200M (if ring full don't drop packets) -lib2-rx: 200M -lib2-tx: 169626182 (if ring full drop packets) -lib1-rx: =lib2-tx -drop: ~15% -pps: ~6M -multiple interfaces: -not tested (expected same as TC1) diff --git a/extras/libmemif/docs/gettingstarted_doc.md b/extras/libmemif/docs/gettingstarted_doc.md index abf9a1a06fd..6f2a99c8c69 100644 --- a/extras/libmemif/docs/gettingstarted_doc.md +++ b/extras/libmemif/docs/gettingstarted_doc.md @@ -1,223 +1,164 @@ ## Getting started {#libmemif_gettingstarted_doc} -#### Concept (Connecting to VPP) - For detailed information on api calls and structures please refer to @ref libmemif.h. -1. Initialize memif - - Declare callback function handling file descriptor event polling. -```C -int -control_fd_update (int fd, uint8_t events) -{ -... -} +Start by creating a memif socket. Memif socket represents UNIX domain socket and interfaces assigned to use this socket. Memif uses UNIX doman socket to communicate with other memif drivers. + +First fill out the `memif_socket_args` struct. The minimum required configuration is the UNIX socket path. +> Use `@` or `\0` at the beginning of the path to use abstract socket. +```c +memif_socket_args_t sargs; + +strncpy(sargs.path, socket_path, sizeof(sargs.path)); +``` +```c +memif_socket_handle_t memif_socket; + +memif_create_socket(&memif_socket, &sargs, &private_data); ``` - - Call memif initialization function. memif\_init -```C -err = memif_init (control_fd_update, APP_NAME, NULL, NULL); +Once you have created your socket, you can create memif interfaces on this socket. Fill out the `memif_conn_args` struct. Then call `memif_create()`. +```c +memif_conn_args_t cargs; + +/* Assign your socket handle */ +cargs.socket = memif_socket; ``` - -> If event occurs on any file descriptor returned by this callback, call memif\_control\_fd\_handler function. Since version 2.0, last two optional arguments are used to specify custom memory allocation. -```C -memif_err = memif_control_fd_handler (evt.data.fd, events); -``` -> If callback function parameter for memif\_init function is set to NULL, libmemif will handle file descriptor event polling. - Api call memif\_poll\_event will call epoll\_pwait with user defined timeout to poll event on file descriptors opened by libmemif. -```C -/* main loop */ - while (1) - { - if (memif_poll_event (-1) < 0) - { - DBG ("poll_event error!"); - } - } +```c +memif_conn_handle_t conn; + +/* Assign callbacks */ +memif_create (&conn, &cargs, on_connect_cb, on_disconnect_cb, on_interrupt_cb, &private_data); ``` - -> Memif initialization function will initialize internal structures and create timer file descriptor, which will be used for sending periodic connection requests. Timer is disarmed if no memif interface is created. - -2. Creating interface - - Declare memif connection handle. -```C -memif_conn_handle_t c; +Now start the polling events using libmemifs builtin polling. +```c +do { + err = memif_poll_event(memif_socket, /* timeout -1 = blocking */ -1); +} while (err == MEMIF_ERR_SUCCESS); ``` -> example app uses struct that contains connection handle, rx/tx buffers and other connection specific information. - - - Specify connection arguments. -```C -memif_conn_args_t args; -memset (&args, 0, sizeof (args)); -args.is_master = is_master; -args.log2_ring_size = 10; -args.buffer_size = 2048; -args.num_s2m_rings = 2; -args.num_m2s_rings = 2; -strncpy ((char *) args.interface_name, IF_NAME, strlen (IF_NAME)); -args.mode = 0; -args.interface_id = 0; +Polling can be canceled by calling `memif_cancel_poll_event()`. +```c +memif_cancel_poll_event (memif_socket); ``` - - Declare callback functions called on connected/disconnected/interrupted status changed. -```C -int -on_connect (memif_conn_handle_t conn, void *private_ctx) +On link status change `on_connect` and `on_disconnect` callbacks are called respectively. Before you can start transmitting data you, first need to call `memif_refill_queue()` for each RX queue to initialize this queue. +```c +int on_connect (memif_conn_handle_t conn, void *private_ctx) { -... -} + my_private_data_t *data = (my_private_data_t *) private_ctx; -int -on_disconnect (memif_conn_handle_t conn, void *private_ctx) -{ - INFO ("memif connected!"); - return 0; + err = memif_refill_queue(conn, 0, -1, 0); + if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_refill_queue: %s", memif_strerror(err)); + return err; + } + + /* + * Do stuff. + */ + + return 0; } ``` - - Call memif interface create function. memif\_create -```C -err = memif_create (&c->conn, - &args, on_connect, on_disconnect, on_interrupt, &ctx[index]); -``` -> If connection is in slave mode, arms timer file descriptor. -> If on interrupt callback is set to NULL, user will not be notified about interrupt. Use memif\_get\_queue\_efd call to get interrupt file descriptor for specific queue. -```C -int fd = -1; -err = memif_get_queue_efd (c->conn, data->qid, &fd); -``` +Now you are ready to transmit packets. +> Example implementation @ref examples/common/sender.c and @ref examples/common/responder.c -3. Connection establishment - - User application will poll events on all file descriptors returned in memif\_control\_fd\_update\_t callback. - - On event call memif\_control\_fd\_handler. - - Everything else regarding connection establishment will be done internally. - - Once connection has been established, a callback will inform the user about connection status change. - -4. Interrupt packet receive - - If event is polled on interrupt file descriptor, libmemif will call memif\_interrupt\_t callback specified for every connection instance. -```C -int -on_interrupt (memif_conn_handle_t conn, void *private_ctx, uint16_t qid) -{ -... +To transmit or receive data you will need to use `memif_buffer` struct. The important fields here are `void *data`, `uint32_t len` and `uint8_t flags`. The `data` pointer points directly to the shared memory packet buffer. This is where you will find/insert your packets. The `len` field is the length of the buffer. If the flag `MEMIF_BUFFER_FLAG_NEXT` is present in `flags` field, this buffer is chained so the rest of the data is located in the next buffer, and so on. + +First let's receive data. To receive data call `memif_rx_burst()`. The function will fill out memif buffers passed to it. Then you would process your data (e.g. copy to your stack). Last you must refill the queue using `memif_refill_queue()` to notify peer that the buffers are now free and can be overwritten. +```c +/* Fill out memif buffers and mark them as received */ +err = memif_rx_burst(conn, qid, buffers, num_buffers, &num_received); +if (err != MEMIF_ERR_SUCCESS) { + INFO ("memif_rx_burst: %s", memif_strerror(err)); + return err; +} +/* + Process the buffers. +*/ + +/* Refill the queue, so that the peer interface can transmit more packets */ +err = memif_refill_queue(conn, qid, num_received, 0); +if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_refill_queue: %s", memif_strerror(err)); + goto error; } ``` +In order to transmit data you first need to 'allocate' memif buffers using `memif_buffer_alloc()`. This function simmilar to `memif_rx_burst` will fill out provided memif buffers. You will then insert your packets directly into the shared memory (don't forget to update `len` filed if your packet is smaller that buffer length). Finaly call `memif_tx_burst` to transmit the buffers. +```c +/* Alocate memif buffers */ +err = memif_buffer_alloc(conn, qid, buffers, num_pkts, &num_allocated, packet_size); +if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_buffer_alloc: %s", memif_strerror(err)); + goto error; +} -6. Memif buffers - - Packet data are stored in memif\_buffer\_t. Pointer _data_ points to shared memory buffer, and unsigned integer *_len* contains buffer length. - - flags: MEMIF\_BUFFER\_FLAG\_NEXT states that the buffer is not large enough to contain whole packet, so next buffer contains the rest of the packet. (chained buffers) -```C -typedef struct -{ - uint16_t desc_index; - uint32_t len; - uint8_t flags; - void *data; -} memif_buffer_t; -``` +/* + Fill out the buffers. -5. Packet receive - - Api call memif\_rx\_burst will set all required fields in memif buffers provided by user application, dequeue received buffers and consume interrupt event on receive queue. The event is not consumed, if memif_rx_burst fails. -```C -err = memif_rx_burst (c->conn, qid, c->bufs, MAX_MEMIF_BUFS, &rx); -``` - - User application can then process packets. - - Api call memif\_refill\_queue will enqueue rx buffers. -```C -err = memif_refill_queue (c->conn, qid, rx); -``` + tx_buffers[i].data field points to the shared memory. + update tx_buffers[i].len to your packet length, if the packet is smaller. +*/ -6. Packet transmit - - Api call memif\_buffer\_alloc will find free tx buffers and set all required fields in memif buffers provided by user application. -```C -err = memif_buffer_alloc (c->conn, qid, c->tx_bufs, n, &r); -``` - - User application can populate shared memory buffers with packets. - - Api call memif\_tx\_burst will enqueue tx buffers -```C -err = memif_tx_burst (c->conn, qid, c->tx_bufs, c->tx_buf_num, &r); +/* Transmit the buffers */ +err = memif_tx_burst(conn, qid, buffers, num_allocated, &num_transmitted); +if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_tx_burst: %s", memif_strerror(err)); + goto error; +} ``` +### Zero-copy Slave -7. Helper functions - - Memif version -```C -uint16_t memif_ver = memif_get_version (); -``` - - Memif details - - Api call memif\_get\_details will return details about connection. -```C -err = memif_get_details (c->conn, &md, buf, buflen); -``` - - Memif error messages - - Every api call returns error code (integer value) mapped to error string. - - Call memif\_strerror will return error message assigned to specific error code. -```C -if (err != MEMIF_ERR_SUCCESS) - INFO ("memif_get_details: %s", memif_strerror (err)); -``` - - Not all syscall errors are translated to memif error codes. If error code 1 (MEMIF\_ERR\_SYSCALL) is returned then libmemif needs to be compiled with -DMEMIF_DBG flag to print error message. Use _make -B_ to rebuild libmemif in debug mode. +Interface with slave role is the buffer producer, as such it can use zero-copy mode. -#### Example app (libmemif fd event polling): +After receiving buffers, process your packets in place. Then use `memif_buffer_enq_tx()` to enqueue rx buffers to tx queue (by swapping rx buffer with a free tx buffer). +```c +/* Fill out memif buffers and mark them as received */ +err = memif_rx_burst(conn, qid, buffers, num_buffers, &num_received); +if (err != MEMIF_ERR_SUCCESS) { + INFO ("memif_rx_burst: %s", memif_strerror(err)); + return err; +} -- @ref extras/libmemif/examples/icmp_responder +/* + Process the buffers in place. +*/ -> Optional argument: transmit queue id. -``` -icmpr 1 -``` -> Set transmit queue id to 1. Default is 0. -> Application will create memif interface in slave mode and try to connect to VPP. Exit using Ctrl+C. Application will handle SIGINT signal, free allocated memory and exit with EXIT_SUCCESS. +/* Enqueue processed buffers to tx queue */ +err = memif_buffer_enq_tx(conn, qid, buffers, num_buffers, &num_enqueued); +if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_buffer_alloc: %s", memif_strerror(err)); + goto error; +} -#### Example app: +/* Refill the queue, so that the peer interface can transmit more packets */ +err = memif_refill_queue(conn, qid, num_enqueued, 0); +if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_refill_queue: %s", memif_strerror(err)); + goto error; +} -ICMP Responder custom fd event polling. +/* Transmit the buffers. */ +err = memif_tx_burst(conn, qid, buffers, num_enqueued, &num_transmitted); +if (err != MEMIF_ERR_SUCCESS) { + INFO("memif_tx_burst: %s", memif_strerror(err)); + goto error; +} +``` -- @ref extras/libmemif/examples/icmp_responder-epoll +### Custom Event Polling -#### Example app (multi-thread queue polling) +Libmemif can be integrated into your applications fd event polling. You will need to implement `memif_control_fd_update_t` callback and pass it to `memif_socket_args.on_control_fd_update`. Now each time any file descriptor belonging to that socket updates, `on_control_fd_update` callback is called. The file descriptor and event type is passed in `memif_fd_event_t`. It also contains private context that is associated with this fd. When event is polled on the fd you need to call `memif_control_fd_handler` and pass the event type and private context associated with the fd. -ICMP Responder multi-thread. -- @ref extras/libmemif/examples/icmp_responder-mt +### Multi Threading -> Simple example of libmemif multi-thread usage. Connection establishment is handled by main thread. There are two rx/tx queues in this example. One in polling mode and second in interrupt mode. +#### Connection establishment -VPP config: -``` -# create interface memif id 0 master -# set int state memif0 up -# set int ip address memif0 192.168.1.1/24 -# ping 192.168.1.2 -``` -For multiple rings (queues) support run VPP with worker threads: -example startup.conf: -``` -unix { - interactive - nodaemon - full-coredump -} +Memif sockets should not be handled in paralell. Instead each thread should have it's own socket. However the UNIX socket can be the same. In case of non-listener socket, it's straight forward, just create the socket using the same path. In case of listener socket, the polling should be done by single thread. +> The socket becomes listener once a Master interface is assigned to it. -cpu { - workers 2 -} -``` -VPP config: -``` -# create interface memif id 0 master -# set int state memif0 up -# set int ip address memif0 192.168.1.1/24 -# ping 192.168.1.2 -``` -> Master mode queue number is limited by worker threads. Slave mode interface needs to specify number of queues. -``` -# create memif id 0 slave rx-queues 2 tx-queues 2 -``` -> Example applications use VPP default socket file for memif: /run/vpp/memif.sock -> For master mode, socket directory must exist prior to memif\_create call. +#### Packet handling -#### Unit tests +Single queue must not be handled in paralel. Instead you can assign queues to threads in such way that each queue is only assigned single thread. -Unit tests use [Check](https://libcheck.github.io/check/index.html) framework. This framework must be installed in order to build *unit\_test* binary. -Ubuntu/Debian: -``` -sudo apt-get install check -``` -[More platforms](https://libcheck.github.io/check/web/install.html) +### Shared Memory Layout +Please refer to [DPDK MEMIF documentation](http://doc.dpdk.org/guides/nics/memif.html) `'Shared memory'` section.
\ No newline at end of file diff --git a/extras/libmemif/libmemif_doc.md b/extras/libmemif/libmemif_doc.md index 1260d9c60ac..ac3d6dadcfb 100644 --- a/extras/libmemif/libmemif_doc.md +++ b/extras/libmemif/libmemif_doc.md @@ -20,52 +20,34 @@ Shared memory packet interface (memif) provides high performance packet transmit - [x] Multiple queues - [x] Multi-thread support - [x] Master mode - - [ ] Multiple regions (TODO) -- [ ] Performance testing (TODO) + - [x] Multiple regions +- [x] Loopback ## Quickstart This setup will run libmemif ICMP responder example app in container. Install [docker](https://docs.docker.com/engine/installation) engine. Useful link: [Docker documentation](https://docs.docker.com/get-started). -Pull image: +Build the docker image: ``` -# docker pull ligato/libmemif-sample-service +# docker build . -t libmemif ``` -Now you should be able to see ligato/libmemif-sample-service image on your local machine (IMAGE ID in this README may be outdated): +Now you should be able to see libmemif image on your local machine: ``` # docker images REPOSITORY TAG IMAGE ID CREATED SIZE -ligato/libmemif-sample-service latest 32ecc2f9d013 About a minute ago 468MB +libmemif latest 32ecc2f9d013 About a minute ago 468MB ... ``` Run container: ``` -# docker run -it --rm --name icmp-responder --hostname icmp-responder --privileged -v "/run/vpp/:/run/vpp/" ligato/libmemif-sample-service -``` -Example application will start in debug mode. Output should look like this: -``` -ICMP_Responder:add_epoll_fd:233: fd 0 added to epoll -ICMP_Responder:add_epoll_fd:233: fd 5 added to epoll -LIBMEMIF EXAMPLE APP: ICMP_Responder (debug) -============================== -libmemif version: 2.0 (debug) -memif version: 512 -commands: - help - prints this help - exit - exit app - conn <index> <mode> [<interrupt-desc>] - create memif. index is also used as interface id, mode 0 = slave 1 = master, interrupt-desc none = default 0 = if ring is full wait 1 = handle only ARP requests - del <index> - delete memif - show - show connection details - ip-set <index> <ip-addr> - set interface ip address - rx-mode <index> <qid> <polling|interrupt> - set queue rx mode - sh-count - print counters - cl-count - clear counters - send <index> <tx> <ip> <mac> - send icmp +# docker run -it --rm --name icmp-responder --hostname icmp-responder --privileged -v "/run/vpp/:/run/vpp/" libmemif ``` +The interface will by default connect to a master interface listening on `/run/vpp/master.sock`. The example will handle ARP requests and respond to ICMPv4 requests to `192.168.1.1`. + Continue with @ref libmemif_example_setup which contains instructions on how to set up connection between icmpr-epoll example app and VPP-memif. #### Next steps @@ -74,4 +56,3 @@ Continue with @ref libmemif_example_setup which contains instructions on how to - @subpage libmemif_examples_doc - @subpage libmemif_example_setup_doc - @subpage libmemif_gettingstarted_doc -- @subpage libmemif_devperftest_doc |