1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
|
## Getting started {#libmemif_gettingstarted_doc}
For detailed information on api calls and structures please refer to @ref libmemif.h.
Start by creating a memif socket. Memif socket represents UNIX domain socket and interfaces assigned to use this socket. Memif uses UNIX doman socket to communicate with other memif drivers.
First fill out the `memif_socket_args` struct. The minimum required configuration is the UNIX socket path.
> Use `@` or `\0` at the beginning of the path to use abstract socket.
```c
memif_socket_args_t sargs;
strncpy(sargs.path, socket_path, sizeof(sargs.path));
```
```c
memif_socket_handle_t memif_socket;
memif_create_socket(&memif_socket, &sargs, &private_data);
```
Once you have created your socket, you can create memif interfaces on this socket. Fill out the `memif_conn_args` struct. Then call `memif_create()`.
```c
memif_conn_args_t cargs;
/* Assign your socket handle */
cargs.socket = memif_socket;
```
```c
memif_conn_handle_t conn;
/* Assign callbacks */
memif_create (&conn, &cargs, on_connect_cb, on_disconnect_cb, on_interrupt_cb, &private_data);
```
Now start the polling events using libmemifs builtin polling.
```c
do {
err = memif_poll_event(memif_socket, /* timeout -1 = blocking */ -1);
} while (err == MEMIF_ERR_SUCCESS);
```
Polling can be canceled by calling `memif_cancel_poll_event()`.
```c
memif_cancel_poll_event (memif_socket);
```
On link status change `on_connect` and `on_disconnect` callbacks are called respectively. Before you can start transmitting data you, first need to call `memif_refill_queue()` for each RX queue to initialize this queue.
```c
int on_connect (memif_conn_handle_t conn, void *private_ctx)
{
my_private_data_t *data = (my_private_data_t *) private_ctx;
err = memif_refill_queue(conn, 0, -1, 0);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_refill_queue: %s", memif_strerror(err));
return err;
}
/*
* Do stuff.
*/
return 0;
}
```
Now you are ready to transmit packets.
> Example implementation @ref examples/common/sender.c and @ref examples/common/responder.c
To transmit or receive data you will need to use `memif_buffer` struct. The important fields here are `void *data`, `uint32_t len` and `uint8_t flags`. The `data` pointer points directly to the shared memory packet buffer. This is where you will find/insert your packets. The `len` field is the length of the buffer. If the flag `MEMIF_BUFFER_FLAG_NEXT` is present in `flags` field, this buffer is chained so the rest of the data is located in the next buffer, and so on.
First let's receive data. To receive data call `memif_rx_burst()`. The function will fill out memif buffers passed to it. Then you would process your data (e.g. copy to your stack). Last you must refill the queue using `memif_refill_queue()` to notify peer that the buffers are now free and can be overwritten.
```c
/* Fill out memif buffers and mark them as received */
err = memif_rx_burst(conn, qid, buffers, num_buffers, &num_received);
if (err != MEMIF_ERR_SUCCESS) {
INFO ("memif_rx_burst: %s", memif_strerror(err));
return err;
}
/*
Process the buffers.
*/
/* Refill the queue, so that the peer interface can transmit more packets */
err = memif_refill_queue(conn, qid, num_received, 0);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_refill_queue: %s", memif_strerror(err));
goto error;
}
```
In order to transmit data you first need to 'allocate' memif buffers using `memif_buffer_alloc()`. This function simmilar to `memif_rx_burst` will fill out provided memif buffers. You will then insert your packets directly into the shared memory (don't forget to update `len` filed if your packet is smaller that buffer length). Finaly call `memif_tx_burst` to transmit the buffers.
```c
/* Alocate memif buffers */
err = memif_buffer_alloc(conn, qid, buffers, num_pkts, &num_allocated, packet_size);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_buffer_alloc: %s", memif_strerror(err));
goto error;
}
/*
Fill out the buffers.
tx_buffers[i].data field points to the shared memory.
update tx_buffers[i].len to your packet length, if the packet is smaller.
*/
/* Transmit the buffers */
err = memif_tx_burst(conn, qid, buffers, num_allocated, &num_transmitted);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_tx_burst: %s", memif_strerror(err));
goto error;
}
```
### Zero-copy Slave
Interface with slave role is the buffer producer, as such it can use zero-copy mode.
After receiving buffers, process your packets in place. Then use `memif_buffer_enq_tx()` to enqueue rx buffers to tx queue (by swapping rx buffer with a free tx buffer).
```c
/* Fill out memif buffers and mark them as received */
err = memif_rx_burst(conn, qid, buffers, num_buffers, &num_received);
if (err != MEMIF_ERR_SUCCESS) {
INFO ("memif_rx_burst: %s", memif_strerror(err));
return err;
}
/*
Process the buffers in place.
*/
/* Enqueue processed buffers to tx queue */
err = memif_buffer_enq_tx(conn, qid, buffers, num_buffers, &num_enqueued);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_buffer_alloc: %s", memif_strerror(err));
goto error;
}
/* Refill the queue, so that the peer interface can transmit more packets */
err = memif_refill_queue(conn, qid, num_enqueued, 0);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_refill_queue: %s", memif_strerror(err));
goto error;
}
/* Transmit the buffers. */
err = memif_tx_burst(conn, qid, buffers, num_enqueued, &num_transmitted);
if (err != MEMIF_ERR_SUCCESS) {
INFO("memif_tx_burst: %s", memif_strerror(err));
goto error;
}
```
### Custom Event Polling
Libmemif can be integrated into your applications fd event polling. You will need to implement `memif_control_fd_update_t` callback and pass it to `memif_socket_args.on_control_fd_update`. Now each time any file descriptor belonging to that socket updates, `on_control_fd_update` callback is called. The file descriptor and event type is passed in `memif_fd_event_t`. It also contains private context that is associated with this fd. When event is polled on the fd you need to call `memif_control_fd_handler` and pass the event type and private context associated with the fd.
### Multi Threading
#### Connection establishment
Memif sockets should not be handled in paralell. Instead each thread should have it's own socket. However the UNIX socket can be the same. In case of non-listener socket, it's straight forward, just create the socket using the same path. In case of listener socket, the polling should be done by single thread.
> The socket becomes listener once a Master interface is assigned to it.
#### Packet handling
Single queue must not be handled in paralel. Instead you can assign queues to threads in such way that each queue is only assigned single thread.
### Shared Memory Layout
Please refer to [DPDK MEMIF documentation](http://doc.dpdk.org/guides/nics/memif.html) `'Shared memory'` section.
|
