diff options
author | Milan Lenco <milan.lenco@pantheon.tech> | 2017-10-11 16:40:58 +0200 |
---|---|---|
committer | Milan Lenco <milan.lenco@pantheon.tech> | 2017-10-13 08:40:37 +0200 |
commit | 3f1edad4e6ba0a7876750aea55507fae14d8badf (patch) | |
tree | a473997249d9ba7deb70b1076d14e4c4ed029a43 /extras/libmemif | |
parent | 8b66677c2382a8e739d437621de4473d5ec0b9f1 (diff) |
ODPM 266: Go-libmemif + 2 examples.
Change-Id: Icdb9b9eb2314eff6c96afe7996fcf2728291de4a
Signed-off-by: Milan Lenco <milan.lenco@pantheon.tech>
Diffstat (limited to 'extras/libmemif')
-rw-r--r-- | extras/libmemif/README.md | 185 | ||||
-rw-r--r-- | extras/libmemif/adapter.go | 1107 | ||||
-rw-r--r-- | extras/libmemif/doc.go | 5 | ||||
-rw-r--r-- | extras/libmemif/error.go | 123 | ||||
-rw-r--r-- | extras/libmemif/examples/icmp-responder/icmp-responder.go | 402 | ||||
-rw-r--r-- | extras/libmemif/examples/raw-data/raw-data.go | 240 |
6 files changed, 2062 insertions, 0 deletions
diff --git a/extras/libmemif/README.md b/extras/libmemif/README.md new file mode 100644 index 0000000..39a6d58 --- /dev/null +++ b/extras/libmemif/README.md @@ -0,0 +1,185 @@ +## Go-libmemif + +Package **libmemif** is a Golang adapter for the **libmemif library** +(`extras/libmemif` in the [VPP](https://wiki.fd.io/view/VPP) repository). +To differentiate between the adapter and the underlying C-written library, +labels `Go-libmemif` and `C-libmemif` are used in the documentation. + +### Requirements + +libmemif for Golang is build on the top of the original, C-written +libmemif library using `cgo`. It is therefore necessary to have C-libmemif +header files and the library itself installed in locations known +to the compiler. + +For example, to install C-libmemif system-wide into the standard +locations, execute: +``` +$ git clone https://gerrit.fd.io/r/vpp +$ cd vpp/extras/libmemif +$ make install +``` + +### Build + +Package **libmemif** is not part of the **GoVPP** core and as such it is +not included in the [make build](../../Makefile) target. +Instead, it has its own target in the [top-level Makefile](../../Makefile) +used to build the attached examples with the adapter: +``` +$ make extras +``` + +### APIs + +All **Go-libmemif** public APIs can be found in [adapter.go](adapter.go). +Please see the comments for a more detailed description. +Additionally, a list of all errors thrown by libmemif can be found +in [error.go](error.go). + +### Usage + +**libmemif** needs to be first initialized with `Init(appName)`. +This has to be done only once in the context of the entire process. +Make sure to call `Cleanup()` to release all the resources allocated +by **libmemif** before exiting your application. Consider calling +`Init()` followed by `Cleanup()` scheduled with `defer` in the `main()` +function. + +Log messages are by default printed to stdout. Use `SetLogger()` to use +your own customized logger (can be changed before `Init()`). + +Once **libmemif** is initialized, new memif interfaces can be created +with `CreateInterface(config, callbacks)`. See `MemifConfig` structure +definition to learn about possible memif configuration options. +If successful, `CreateInterface()` returns an instance of `Memif` +structure representing the underlying memif interface. + +Callbacks are optional and can be shared across multiple memif instances. +Available callbacks are: +1. **OnConnect**: called when the connection is established. + By the time the callback is called, the Rx/Tx queues are initialized + and ready for data transmission. Interrupt channels are also + created and ready to be read from. + The user is expected to start polling for input packets via repeated + calls to `Memif.RxBurst(queueID, count)` or to initiate select + on the interrupt channels obtained with `Get*InterruptChan()`, + depending on the Rx mode. By default, all memif Rx queues are created + in the interrupt mode, but this can be changed per-queue with + `Memif.SetRxMode(queueID, mode)`. +2. **OnDisconnect**: called after the connection was closed. Immediately + after the user callback returns, Rx/Tx queues and interrupt channels + are also deallocated. The user defined callback should therefore ensure + that all the Rx/Tx operations are stopped before it returns. + +**libmemif** was designed for a maximum possible performance. Packets +are sent and received in bulks, rather than one-by-one, using +`Memif.TxBurst(queueID, packets)` and `Memif.RxBurst(queueID, count)`, +respectively. Memif connection can consists of multiple queues in both +directions. A queue is one-directional wait-free ring buffer. +It is the unit of parallelism for data transmission. The maximum possible +lock-free granularity is therefore one go routine for one queue. + +Interrupt channel for one specific Rx queue can be obtained with +`GetQueueInterruptChan(queueID)` as opposed to `GetInterruptChan()` +for all the Rx queues. There is only one interrupt signal sent for +an entire burst of packets, therefore an interrupt handling routine +should repeatedly call RxBurst() until an empty slice of packets +is returned. This way it is ensured that there are no packets left +on the queue unread when the interrupt signal is cleared. +Study the `ReadAndPrintPackets()` function in [raw-data example](examples/raw-data/raw-data.go). + +For **libmemif** the packet is just an array of bytes. It does not care +what the actual content is. It is not required for a packet to follow +any network protocol in order to get transported from one end to another. +See the type declaration for `RawPacketData` and its use in `Memif.TxBurst()` +and `Memif.RxBurst()`. + +In order to remove a memif interface, call `Memif.Close()`. If the memif +is in the connected state, the connection is first properly closed. +Do not touch memif after it was closed, let garbage collector to remove +the `Memif` instance. In the end, `Cleanup()` will also ensure that all +active memif interfaces are closed before the cleanup finalizes. + +### Examples + +**Go-libmemif** ships with two simple examples demonstrating the usage +of the package with a detailed commentary. +The examples can be found in the subdirectory [examples](./examples). + +#### Raw data (libmemif <-> libmemif) + +*raw-data* is a basic example showing how to create a memif interface, +handle events through callbacks and perform Rx/Tx of raw data. Before +handling an actual packets it is important to understand the skeleton +of libmemif-based applications. + +Since VPP expects proper packet data, it is not very useful to connect +*raw-data* example with VPP, even though it will work, since all +the received data will get dropped on the VPP side. + +To create a connection of two raw-data instances, start two processes +concurrently in an arbitrary order: + - *master* memif: + ``` + $ cd extras/libmemif/examples/raw-data + $ ./raw-data + ``` + - *slave* memif: + ``` + $ cd extras/libmemif/examples/raw-data + $ ./raw-data --slave + ``` + +Every 3 seconds both sides send 3 raw-data packets to the opposite end +through each of the 3 queues. The received packets are printed to stdout. + +Stop an instance of *raw-data* with an interrupt signal (^C). + +#### ICMP Responder + +*icmp-responder* is a simple example showing how to answer APR and ICMP +echo requests through a memif interface. Package `google/gopacket` is +used to decode and construct packets. + +The appropriate VPP configuration for the opposite memif is: +``` +vpp$ create memif id 1 socket /tmp/icmp-responder-example slave secret secret +vpp$ set int state memif0/1 up +vpp$ set int ip address memif0/1 192.168.1.2/24 +``` + +To start the example, simply type: +``` +root$ ./icmp-responder +``` + +*icmp-responder* needs to be run as root so that it can access the socket +created by VPP. + +Normally, the memif interface is in the master mode. Pass CLI flag `--slave` +to create memif in the slave mode: +``` +root$ ./icmp-responder --slave +``` + +Don't forget to put the opposite memif into the master mode in that case. + +To verify the connection, run: +``` +vpp$ ping 192.168.1.1 +64 bytes from 192.168.1.1: icmp_seq=2 ttl=255 time=.6974 ms +64 bytes from 192.168.1.1: icmp_seq=3 ttl=255 time=.6310 ms +64 bytes from 192.168.1.1: icmp_seq=4 ttl=255 time=1.0350 ms +64 bytes from 192.168.1.1: icmp_seq=5 ttl=255 time=.5359 ms + +Statistics: 5 sent, 4 received, 20% packet loss +vpp$ sh ip arp + Time IP4 Flags Ethernet Interface + 68.5648 192.168.1.1 D aa:aa:aa:aa:aa:aa memif0/1 +``` +*Note*: it is expected that the first ping is shown as lost. + It was actually converted to an ARP request. This is a VPP + specific feature common to all interface types. + +Stop the example with an interrupt signal (^C).
\ No newline at end of file diff --git a/extras/libmemif/adapter.go b/extras/libmemif/adapter.go new file mode 100644 index 0000000..b3e6192 --- /dev/null +++ b/extras/libmemif/adapter.go @@ -0,0 +1,1107 @@ +// Copyright (c) 2017 Cisco and/or its affiliates. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at: +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// +build !windows,!darwin + +package libmemif + +import ( + "encoding/binary" + "os" + "sync" + "syscall" + "unsafe" + + logger "github.com/Sirupsen/logrus" +) + +/* +#cgo LDFLAGS: -lmemif + +#include <unistd.h> +#include <stdlib.h> +#include <stdint.h> +#include <string.h> +#include <sys/eventfd.h> +#include <libmemif.h> + +// Feature tests. +#ifndef MEMIF_HAVE_CANCEL_POLL_EVENT +// memif_cancel_poll_event that simply returns ErrUnsupported. +static int +memif_cancel_poll_event () +{ + return 102; // ErrUnsupported +} +#endif + +// govpp_memif_conn_args_t replaces fixed sized arrays with C-strings which +// are much easier to work with in cgo. +typedef struct +{ + char *socket_filename; + char *secret; + uint8_t num_s2m_rings; + uint8_t num_m2s_rings; + uint16_t buffer_size; + memif_log2_ring_size_t log2_ring_size; + uint8_t is_master; + memif_interface_id_t interface_id; + char *interface_name; + char *instance_name; + memif_interface_mode_t mode; +} govpp_memif_conn_args_t; + +// govpp_memif_details_t replaces strings represented with (uint8_t *) +// to the standard and easy to work with in cgo: (char *) +typedef struct +{ + char *if_name; + char *inst_name; + char *remote_if_name; + char *remote_inst_name; + uint32_t id; + char *secret; + uint8_t role; + uint8_t mode; + char *socket_filename; + uint8_t rx_queues_num; + uint8_t tx_queues_num; + memif_queue_details_t *rx_queues; + memif_queue_details_t *tx_queues; + uint8_t link_up_down; +} govpp_memif_details_t; + +extern int go_on_connect_callback(void *privateCtx); +extern int go_on_disconnect_callback(void *privateCtx); + +// Callbacks strip the connection handle away. + +static int +govpp_on_connect_callback(memif_conn_handle_t conn, void *private_ctx) +{ + return go_on_connect_callback(private_ctx); +} + +static int +govpp_on_disconnect_callback(memif_conn_handle_t conn, void *private_ctx) +{ + return go_on_disconnect_callback(private_ctx); +} + +// govpp_memif_create uses govpp_memif_conn_args_t. +static int +govpp_memif_create (memif_conn_handle_t *conn, govpp_memif_conn_args_t *go_args, + void *private_ctx) +{ + memif_conn_args_t args; + memset (&args, 0, sizeof (args)); + args.socket_filename = (char *)go_args->socket_filename; + if (go_args->secret != NULL) + { + strncpy ((char *)args.secret, go_args->secret, + sizeof (args.secret) - 1); + } + args.num_s2m_rings = go_args->num_s2m_rings; + args.num_m2s_rings = go_args->num_m2s_rings; + args.buffer_size = go_args->buffer_size; + args.log2_ring_size = go_args->log2_ring_size; + args.is_master = go_args->is_master; + args.interface_id = go_args->interface_id; + if (go_args->interface_name != NULL) + { + strncpy ((char *)args.interface_name, go_args->interface_name, + sizeof(args.interface_name) - 1); + } + if (go_args->instance_name != NULL) + { + strncpy ((char *)args.instance_name, go_args->instance_name, + sizeof (args.instance_name) - 1); + } + args.mode = go_args->mode; + + return memif_create(conn, &args, govpp_on_connect_callback, + govpp_on_disconnect_callback, NULL, + private_ctx); +} + +// govpp_memif_get_details keeps reallocating buffer until it is large enough. +// The buffer is returned to be deallocated when it is no longer needed. +static int +govpp_memif_get_details (memif_conn_handle_t conn, govpp_memif_details_t *govpp_md, + char **buf) +{ + int rv = 0; + size_t buflen = 1 << 7; + char *buffer = NULL, *new_buffer = NULL; + memif_details_t md = {0}; + + do { + // initial malloc (256 bytes) or realloc + buflen <<= 1; + new_buffer = realloc(buffer, buflen); + if (new_buffer == NULL) + { + free(buffer); + return MEMIF_ERR_NOMEM; + } + buffer = new_buffer; + // try to get details + rv = memif_get_details(conn, &md, buffer, buflen); + } while (rv == MEMIF_ERR_NOBUF_DET); + + if (rv == 0) + { + *buf = buffer; + govpp_md->if_name = (char *)md.if_name; + govpp_md->inst_name = (char *)md.inst_name; + govpp_md->remote_if_name = (char *)md.remote_if_name; + govpp_md->remote_inst_name = (char *)md.remote_inst_name; + govpp_md->id = md.id; + govpp_md->secret = (char *)md.secret; + govpp_md->role = md.role; + govpp_md->mode = md.mode; + govpp_md->socket_filename = (char *)md.socket_filename; + govpp_md->rx_queues_num = md.rx_queues_num; + govpp_md->tx_queues_num = md.tx_queues_num; + govpp_md->rx_queues = md.rx_queues; + govpp_md->tx_queues = md.tx_queues; + govpp_md->link_up_down = md.link_up_down; + } + else + free(buffer); + return rv; +} + +// Used to avoid cumbersome tricks that use unsafe.Pointer() + unsafe.Sizeof() +// or even cast C-array directly into Go-slice. +static memif_queue_details_t +govpp_get_rx_queue_details (govpp_memif_details_t *md, int index) +{ + return md->rx_queues[index]; +} + +// Used to avoid cumbersome tricks that use unsafe.Pointer() + unsafe.Sizeof() +// or even cast C-array directly into Go-slice. +static memif_queue_details_t +govpp_get_tx_queue_details (govpp_memif_details_t *md, int index) +{ + return md->tx_queues[index]; +} + +// Copy packet data into the selected buffer. +static void +govpp_copy_packet_data(memif_buffer_t *buffers, int index, void *data, uint32_t size) +{ + buffers[index].data_len = (size > buffers[index].buffer_len ? buffers[index].buffer_len : size); + memcpy(buffers[index].data, data, (size_t)buffers[index].data_len); +} + +// Get packet data from the selected buffer. +// Used to avoid an ugly unsafe.Pointer() + unsafe.Sizeof(). +static void * +govpp_get_packet_data(memif_buffer_t *buffers, int index, int *size) +{ + *size = (int)buffers[index].data_len; + return buffers[index].data; +} + +*/ +import "C" + +// IfMode represents the mode (layer/behaviour) in which the interface operates. +type IfMode int + +const ( + // IfModeEthernet tells memif to operate on the L2 layer. + IfModeEthernet IfMode = iota + + // IfModeIP tells memif to operate on the L3 layer. + IfModeIP + + // IfModePuntInject tells memif to behave as Inject/Punt interface. + IfModePuntInject +) + +// RxMode is used to switch between polling and interrupt for RX. +type RxMode int + +const ( + // RxModeInterrupt tells libmemif to send interrupt signal when data are available. + RxModeInterrupt RxMode = iota + + // RxModePolling means that the user needs to explicitly poll for data on RX + // queues. + RxModePolling +) + +// RawPacketData represents raw packet data. libmemif doesn't care what the +// actual content is, it only manipulates with raw bytes. +type RawPacketData []byte + +// MemifMeta is used to store a basic memif metadata needed for identification +// and connection establishment. +type MemifMeta struct { + // IfName is the interface name. Has to be unique across all created memifs. + // Interface name is truncated if needed to have no more than 32 characters. + IfName string + + // InstanceName identifies the endpoint. If omitted, the application + // name passed to Init() will be used instead. + // Instance name is truncated if needed to have no more than 32 characters. + InstanceName string + + // ConnID is a connection ID used to match opposite sides of the memif + // connection. + ConnID uint32 + + // SocketFilename is the filename of the AF_UNIX socket through which + // the connection is established. + // The string is truncated if neede to fit into sockaddr_un.sun_path + // (108 characters on Linux). + SocketFilename string + + // Secret must be the same on both sides for the authentication to succeed. + // Empty string is allowed. + // The secret is truncated if needed to have no more than 24 characters. + Secret string + + // IsMaster is set to true if memif operates in the Master mode. + IsMaster bool + + // Mode is the mode (layer/behaviour) in which the memif operates. + Mode IfMode +} + +// MemifShmSpecs is used to store the specification of the shared memory segment +// used by memif to send/receive packets. +type MemifShmSpecs struct { + // NumRxQueues is the number of Rx queues. + // Default is 1 (used if the value is 0). + NumRxQueues uint8 + + // NumTxQueues is the number of Tx queues. + // Default is 1 (used if the value is 0). + NumTxQueues uint8 + + // BufferSize is the size of the buffer to hold one packet, or a single + // fragment of a jumbo frame. Default is 2048 (used if the value is 0). + BufferSize uint16 + + // Log2RingSize is the number of items in the ring represented through + // the logarithm base 2. + // Default is 10 (used if the value is 0). + Log2RingSize uint8 +} + +// MemifConfig is the memif configuration. +// Used as the input argument to CreateInterface(). +// It is the slave's config that mostly decides the parameters of the connection, +// but master may limit some of the quantities if needed (based on the memif +// protocol or master's configuration) +type MemifConfig struct { + MemifMeta + MemifShmSpecs +} + +// ConnUpdateCallback is a callback type declaration used with callbacks +// related to connection status changes. +type ConnUpdateCallback func(memif *Memif) (err error) + +// MemifCallbacks is a container for all callbacks provided by memif. +// Any callback can be nil, in which case it will be simply skipped. +// Important: Do not call CreateInterface() or Memif.Close() from within a callback +// or a deadlock will occur. Instead send signal through a channel to another +// go routine which will be able to create/remove memif interface(s). +type MemifCallbacks struct { + // OnConnect is triggered when a connection for a given memif was established. + OnConnect ConnUpdateCallback + + // OnDisconnect is triggered when a connection for a given memif was lost. + OnDisconnect ConnUpdateCallback +} + +// Memif represents a single memif interface. It provides methods to send/receive +// packets in bursts in either the polling mode or in the interrupt mode with +// the help of golang channels. +type Memif struct { + MemifMeta + + // Per-library references + ifIndex int // index used in the Go-libmemif context (Context.memifs) + cHandle C.memif_conn_handle_t // handle used in C-libmemif + + // Callbacks + callbacks *MemifCallbacks + + // Interrupt + intCh chan uint8 // memif-global interrupt channel (value = queue ID) + queueIntCh []chan struct{} // per RX queue interrupt channel + + // Rx/Tx queues + stopQPollFd int // event file descriptor used to stop pollRxQueue-s + wg sync.WaitGroup // wait group for all pollRxQueue-s + rxQueueBufs []CPacketBuffers // an array of C-libmemif packet buffers for each RX queue + txQueueBufs []CPacketBuffers // an array of C-libmemif packet buffers for each TX queue +} + +// MemifDetails provides a detailed runtime information about a memif interface. +type MemifDetails struct { + MemifMeta + MemifConnDetails +} + +// MemifConnDetails provides a detailed runtime information about a memif +// connection. +type MemifConnDetails struct { + // RemoteIfName is the name of the memif on the opposite side. + RemoteIfName string + // RemoteInstanceName is the name of the endpoint on the opposite side. + RemoteInstanceName string + // HasLink is true if the connection has link (= is established and functional). + HasLink bool + // RxQueues contains details for each Rx queue. + RxQueues []MemifQueueDetails + // TxQueues contains details for each Tx queue. + TxQueues []MemifQueueDetails +} + +// MemifQueueDetails provides a detailed runtime information about a memif queue. +// Queue = Ring + the associated buffers (one directional). +type MemifQueueDetails struct { + // QueueID is the ID of the queue. + QueueID uint8 + // RingSize is the number of slots in the ring (not logarithmic). + RingSize uint32 + // BufferSize is the size of each buffer pointed to from the ring slots. + BufferSize uint16 + /* Further ring information TO-BE-ADDED when C-libmemif supports them. */ +} + +// CPacketBuffers stores an array of memif buffers for use with TxBurst or RxBurst. +type CPacketBuffers struct { + buffers *C.memif_buffer_t + count int +} + +// Context is a global Go-libmemif runtime context. +type Context struct { + lock sync.RWMutex + initialized bool + memifs map[int] /* ifIndex */ *Memif /* slice of all active memif interfaces */ + nextMemifIndex int + + wg sync.WaitGroup /* wait-group for pollEvents() */ +} + +var ( + // logger used by the adapter. + log *logger.Logger + + // Global Go-libmemif context. + context = &Context{initialized: false} +) + +// init initializes global logger, which logs debug level messages to stdout. +func init() { + log = logger.New() + log.Out = os.Stdout + log.Level = logger.DebugLevel +} + +// SetLogger changes the logger for Go-libmemif to the provided one. +// The logger is not used for logging of C-libmemif. +func SetLogger(l *logger.Logger) { + log = l +} + +// Init initializes the libmemif library. Must by called exactly once and before +// any libmemif functions. Do not forget to call Cleanup() before exiting +// your application. +// <appName> should be a human-readable string identifying your application. +// For example, VPP returns the version information ("show version" from VPP CLI). +func Init(appName string) error { + context.lock.Lock() + defer context.lock.Unlock() + + if context.initialized { + return ErrAlreadyInit + } + + log.Debug("Initializing libmemif library") + + // Initialize C-libmemif. + var errCode int + if appName == "" { + errCode = int(C.memif_init(nil, nil)) + } else { + appName := C.CString(appName) + defer C.free(unsafe.Pointer(appName)) + errCode = int(C.memif_init(nil, appName)) + } + err := getMemifError(errCode) + if err != nil { + return err + } + + // Initialize the map of memory interfaces. + context.memifs = make(map[int]*Memif) + + // Start event polling. + context.wg.Add(1) + go pollEvents() + + context.initialized = true + log.Debug("libmemif library was initialized") + return err +} + +// Cleanup cleans up all the resources allocated by libmemif. +func Cleanup() error { + context.lock.Lock() + defer context.lock.Unlock() + + if !context.initialized { + return ErrNotInit + } + + log.Debug("Closing libmemif library") + + // Delete all active interfaces. + for _, memif := range context.memifs { + memif.Close() + } + + // Stop the event loop (if supported by C-libmemif). + errCode := C.memif_cancel_poll_event() + err := getMemifError(int(errCode)) + if err == nil { + log.Debug("Waiting for pollEvents() to stop...") + context.wg.Wait() + log.Debug("pollEvents() has stopped...") + } else { + log.WithField("err", err).Debug("NOT Waiting for pollEvents to stop...") + } + + // Run cleanup for C-libmemif. + err = getMemifError(int(C.memif_cleanup())) + if err == nil { + context.initialized = false + log.Debug("libmemif library was closed") + } + return err +} + +// CreateInterface creates a new memif interface with the given configuration. +// The same callbacks can be used with multiple memifs. The first callback input +// argument (*Memif) can be used to tell which memif the callback was triggered for. +// The method is thread-safe. +func CreateInterface(config *MemifConfig, callbacks *MemifCallbacks) (memif *Memif, err error) { + context.lock.Lock() + defer context.lock.Unlock() + + if !context.initialized { + return nil, ErrNotInit + } + + log.WithField("ifName", config.IfName).Debug("Creating a new memif interface") + + // Create memif-wrapper for Go-libmemif. + memif = &Memif{ + MemifMeta: config.MemifMeta, + callbacks: &MemifCallbacks{}, + ifIndex: context.nextMemifIndex, + } + + // Initialize memif callbacks. + if callbacks != nil { + memif.callbacks.OnConnect = callbacks.OnConnect + memif.callbacks.OnDisconnect = callbacks.OnDisconnect + } + + // Initialize memif-global interrupt channel. + memif.intCh = make(chan uint8, 1<<6) + + // Initialize event file descriptor for stopping Rx/Tx queue polling. + memif.stopQPollFd = int(C.eventfd(0, C.EFD_NONBLOCK)) + if memif.stopQPollFd < 0 { + return nil, ErrSyscall + } + + // Initialize memif input arguments. + args := &C.govpp_memif_conn_args_t{} + // - socket file name + if config.SocketFilename != "" { + args.socket_filename = C.CString(config.SocketFilename) + defer C.free(unsafe.Pointer(args.socket_filename)) + } + // - interface ID + args.interface_id = C.memif_interface_id_t(config.ConnID) + // - interface name + if config.IfName != "" { + args.interface_name = C.CString(config.IfName) + defer C.free(unsafe.Pointer(args.interface_name)) + } + // - instance name + if config.InstanceName != "" { + args.instance_name = C.CString(config.InstanceName) + defer C.free(unsafe.Pointer(args.instance_name)) + } + // - mode + switch config.Mode { + case IfModeEthernet: + args.mode = C.MEMIF_INTERFACE_MODE_ETHERNET + case IfModeIP: + args.mode = C.MEMIF_INTERFACE_MODE_IP + case IfModePuntInject: + args.mode = C.MEMIF_INTERFACE_MODE_PUNT_INJECT + default: + args.mode = C.MEMIF_INTERFACE_MODE_ETHERNET + } + // - secret + if config.Secret != "" { + args.secret = C.CString(config.Secret) + defer C.free(unsafe.Pointer(args.secret)) + } + // - master/slave flag + number of Rx/Tx queues + if config.IsMaster { + args.num_s2m_rings = C.uint8_t(config.NumRxQueues) + args.num_m2s_rings = C.uint8_t(config.NumTxQueues) + args.is_master = C.uint8_t(1) + } else { + args.num_s2m_rings = C.uint8_t(config.NumTxQueues) + args.num_m2s_rings = C.uint8_t(config.NumRxQueues) + args.is_master = C.uint8_t(0) + } + // - buffer size + args.buffer_size = C.uint16_t(config.BufferSize) + // - log_2(ring size) + args.log2_ring_size = C.memif_log2_ring_size_t(config.Log2RingSize) + + // Create memif in C-libmemif. + errCode := C.govpp_memif_create(&memif.cHandle, args, unsafe.Pointer(uintptr(memif.ifIndex))) + err = getMemifError(int(errCode)) + if err != nil { + return nil, err + } + + // Register the new memif. + context.memifs[memif.ifIndex] = memif + context.nextMemifIndex++ + log.WithField("ifName", config.IfName).Debug("A new memif interface was created") + + return memif, nil +} + +// GetInterruptChan returns a channel which is continuously being filled with +// IDs of queues with data ready to be received. +// Since there is only one interrupt signal sent for an entire burst of packets, +// an interrupt handling routine should repeatedly call RxBurst() until +// the function returns an empty slice of packets. This way it is ensured +// that there are no packets left on the queue unread when the interrupt signal +// is cleared. +// The method is thread-safe. +func (memif *Memif) GetInterruptChan() (ch <-chan uint8 /* queue ID */) { + return memif.intCh +} + +// GetQueueInterruptChan returns an empty-data channel which fires every time +// there are data to read on a given queue. +// It is only valid to call this function if memif is in the connected state. +// Channel is automatically closed when the connection goes down (but after +// the user provided callback OnDisconnect has executed). +// Since there is only one interrupt signal sent for an entire burst of packets, +// an interrupt handling routine should repeatedly call RxBurst() until +// the function returns an empty slice of packets. This way it is ensured +// that there are no packets left on the queue unread when the interrupt signal +// is cleared. +// The method is thread-safe. +func (memif *Memif) GetQueueInterruptChan(queueID uint8) (ch <-chan struct{}, err error) { + if int(queueID) >= len(memif.queueIntCh) { + return nil, ErrQueueID + } + return memif.queueIntCh[queueID], nil +} + +// SetRxMode allows to switch between the interrupt and the polling mode for Rx. +// The method is thread-safe. +func (memif *Memif) SetRxMode(queueID uint8, rxMode RxMode) (err error) { + var cRxMode C.memif_rx_mode_t + switch rxMode { + case RxModeInterrupt: + cRxMode = C.MEMIF_RX_MODE_INTERRUPT + case RxModePolling: + cRxMode = C.MEMIF_RX_MODE_POLLING + default: + cRxMode = C.MEMIF_RX_MODE_INTERRUPT + } + errCode := C.memif_set_rx_mode(memif.cHandle, cRxMode, C.uint16_t(queueID)) + return getMemifError(int(errCode)) +} + +// GetDetails returns a detailed runtime information about this memif. +// The method is thread-safe. +func (memif *Memif) GetDetails() (details *MemifDetails, err error) { + cDetails := C.govpp_memif_details_t{} + var buf *C.char + + // Get memif details from C-libmemif. + errCode := C.govpp_memif_get_details(memif.cHandle, &cDetails, &buf) + err = getMemifError(int(errCode)) + if err != nil { + return nil, err + } + defer C.free(unsafe.Pointer(buf)) + + // Convert details from C to Go. + details = &MemifDetails{} + // - metadata: + details.IfName = C.GoString(cDetails.if_name) + details.InstanceName = C.GoString(cDetails.inst_name) + details.ConnID = uint32(cDetails.id) + details.SocketFilename = C.GoString(cDetails.socket_filename) + if cDetails.secret != nil { + details.Secret = C.GoString(cDetails.secret) + } + details.IsMaster = cDetails.role == C.uint8_t(0) + switch cDetails.mode { + case C.MEMIF_INTERFACE_MODE_ETHERNET: + details.Mode = IfModeEthernet + case C.MEMIF_INTERFACE_MODE_IP: + details.Mode = IfModeIP + case C.MEMIF_INTERFACE_MODE_PUNT_INJECT: + details.Mode = IfModePuntInject + default: + details.Mode = IfModeEthernet + } + // - connection details: + details.RemoteIfName = C.GoString(cDetails.remote_if_name) + details.RemoteInstanceName = C.GoString(cDetails.remote_inst_name) + details.HasLink = cDetails.link_up_down == C.uint8_t(1) + // - RX queues: + var i uint8 + for i = 0; i < uint8(cDetails.rx_queues_num); i++ { + cRxQueue := C.govpp_get_rx_queue_details(&cDetails, C.int(i)) + queueDetails := MemifQueueDetails{ + QueueID: uint8(cRxQueue.qid), + RingSize: uint32(cRxQueue.ring_size), + BufferSize: uint16(cRxQueue.buffer_size), + } + details.RxQueues = append(details.RxQueues, queueDetails) + } + // - TX queues: + for i = 0; i < uint8(cDetails.tx_queues_num); i++ { + cTxQueue := C.govpp_get_tx_queue_details(&cDetails, C.int(i)) + queueDetails := MemifQueueDetails{ + QueueID: uint8(cTxQueue.qid), + RingSize: uint32(cTxQueue.ring_size), + BufferSize: uint16(cTxQueue.buffer_size), + } + details.TxQueues = append(details.TxQueues, queueDetails) + } + + return details, nil +} + +// TxBurst is used to send multiple packets in one call into a selected queue. +// The actual number of packets sent may be smaller and is returned as <count>. +// The method is non-blocking even if the ring is full and no packet can be sent. +// It is only valid to call this function if memif is in the connected state. +// Multiple TxBurst-s can run concurrently provided that each targets a different +// TX queue. +func (memif *Memif) TxBurst(queueID uint8, packets []RawPacketData) (count uint16, err error) { + var sentCount C.uint16_t + var allocated C.uint16_t + var bufSize int + + if len(packets) == 0 { + return 0, nil + } + + if int(queueID) >= len(memif.txQueueBufs) { + return 0, ErrQueueID + } + + // The largest packet in the set determines the packet buffer size. + for _, packet := range packets { + if len(packet) > int(bufSize) { + bufSize = len(packet) + } + } + + // Reallocate Tx buffers if needed to fit the input packets. + pb := memif.txQueueBufs[queueID] + bufCount := len(packets) + if pb.count < bufCount { + newBuffers := C.realloc(unsafe.Pointer(pb.buffers), C.size_t(bufCount*int(C.sizeof_memif_buffer_t))) + if newBuffers == nil { + // Realloc failed, <count> will be less than len(packets). + bufCount = pb.count + } else { + pb.buffers = (*C.memif_buffer_t)(newBuffers) + pb.count = bufCount + } + } + + // Allocate ring slots. + cQueueID := C.uint16_t(queueID) + errCode := C.memif_buffer_alloc(memif.cHandle, cQueueID, pb.buffers, C.uint16_t(bufCount), + &allocated, C.uint16_t(bufSize)) + err = getMemifError(int(errCode)) + if err == ErrNoBufRing { + // Not enough ring slots, <count> will be less than bufCount. + err = nil + } + if err != nil { + return 0, err + } + + // Copy packet data into the buffers. + for i := 0; i < int(allocated); i++ { + packetData := unsafe.Pointer(&packets[i][0]) + C.govpp_copy_packet_data(pb.buffers, C.int(i), packetData, C.uint32_t(len(packets[i]))) + } + + errCode = C.memif_tx_burst(memif.cHandle, cQueueID, pb.buffers, allocated, &sentCount) + err = getMemifError(int(errCode)) + if err != nil { + return 0, err + } + count = uint16(sentCount) + + return count, nil +} + +// RxBurst is used to receive multiple packets in one call from a selected queue. +// <count> is the number of packets to receive. The actual number of packets +// received may be smaller. <count> effectively limits the maximum number +// of packets to receive in one burst (for a flat, predictable memory usage). +// The method is non-blocking even if there are no packets to receive. +// It is only valid to call this function if memif is in the connected state. +// Multiple RxBurst-s can run concurrently provided that each targets a different +// Rx queue. +func (memif *Memif) RxBurst(queueID uint8, count uint16) (packets []RawPacketData, err error) { + var recvCount C.uint16_t + var freed C.uint16_t + + if count == 0 { + return packets, nil + } + + if int(queueID) >= len(memif.rxQueueBufs) { + return packets, ErrQueueID + } + + // Reallocate Rx buffers if needed to fit the output packets. + pb := memif.rxQueueBufs[queueID] + bufCount := int(count) + if pb.count < bufCount { + newBuffers := C.realloc(unsafe.Pointer(pb.buffers), C.size_t(bufCount*int(C.sizeof_memif_buffer_t))) + if newBuffers == nil { + // Realloc failed, len(<packets>) will be certainly less than <count>. + bufCount = pb.count + } else { + pb.buffers = (*C.memif_buffer_t)(newBuffers) + pb.count = bufCount + } + } + + cQueueID := C.uint16_t(queueID) + errCode := C.memif_rx_burst(memif.cHandle, cQueueID, pb.buffers, C.uint16_t(bufCount), &recvCount) + err = getMemifError(int(errCode)) + if err == ErrNoBuf { + // More packets to read - the user is expected to run RxBurst() until there + // are no more packets to receive. + err = nil + } + if err != nil { + return packets, err + } + + // Copy packet data into the instances of RawPacketData. + for i := 0; i < int(recvCount); i++ { + var packetSize C.int + packetData := C.govpp_get_packet_data(pb.buffers, C.int(i), &packetSize) + packets = append(packets, C.GoBytes(packetData, packetSize)) + } + + errCode = C.memif_buffer_free(memif.cHandle, cQueueID, pb.buffers, recvCount, &freed) + err = getMemifError(int(errCode)) + if err != nil { + // Throw away packets to avoid duplicities. + packets = nil + } + + return packets, err +} + +// Close removes the memif interface. If the memif is in the connected state, +// the connection is first properly closed. +// Do not access memif after it is closed, let garbage collector to remove it. +func (memif *Memif) Close() error { + log.WithField("ifName", memif.IfName).Debug("Closing the memif interface") + + // Delete memif from C-libmemif. + err := getMemifError(int(C.memif_delete(&memif.cHandle))) + + if err != nil { + // Close memif-global interrupt channel. + close(memif.intCh) + // Close file descriptor stopQPollFd. + C.close(C.int(memif.stopQPollFd)) + } + + context.lock.Lock() + defer context.lock.Unlock() + // Unregister the interface from the context. + delete(context.memifs, memif.ifIndex) + log.WithField("ifName", memif.IfName).Debug("memif interface was closed") + + return err +} + +// initQueues allocates resources associated with Rx/Tx queues. +func (memif *Memif) initQueues() error { + // Get Rx/Tx queues count. + details, err := memif.GetDetails() + if err != nil { + return err + } + + log.WithFields(logger.Fields{ + "ifName": memif.IfName, + "Rx-count": len(details.RxQueues), + "Tx-count": len(details.TxQueues), + }).Debug("Initializing Rx/Tx queues.") + + // Initialize interrupt channels. + var i int + for i = 0; i < len(details.RxQueues); i++ { + queueIntCh := make(chan struct{}, 1) + memif.queueIntCh = append(memif.queueIntCh, queueIntCh) + } + + // Initialize Rx/Tx packet buffers. + for i = 0; i < len(details.RxQueues); i++ { + memif.rxQueueBufs = append(memif.rxQueueBufs, CPacketBuffers{}) + } + for i = 0; i < len(details.TxQueues); i++ { + memif.txQueueBufs = append(memif.txQueueBufs, CPacketBuffers{}) + } + + return nil +} + +// closeQueues deallocates all resources associated with Rx/Tx queues. +func (memif *Memif) closeQueues() { + log.WithFields(logger.Fields{ + "ifName": memif.IfName, + "Rx-count": len(memif.rxQueueBufs), + "Tx-count": len(memif.txQueueBufs), + }).Debug("Closing Rx/Tx queues.") + + // Close interrupt channels. + for _, ch := range memif.queueIntCh { + close(ch) + } + memif.queueIntCh = nil + + // Deallocate Rx/Tx packet buffers. + for _, pb := range memif.rxQueueBufs { + C.free(unsafe.Pointer(pb.buffers)) + } + memif.rxQueueBufs = nil + for _, pb := range memif.txQueueBufs { + C.free(unsafe.Pointer(pb.buffers)) + } + memif.txQueueBufs = nil +} + +// pollEvents repeatedly polls for a libmemif event. +func pollEvents() { + defer context.wg.Done() + for { + errCode := C.memif_poll_event(C.int(-1)) + err := getMemifError(int(errCode)) + if err == ErrPollCanceled { + return + } + } +} + +// pollRxQueue repeatedly polls an Rx queue for interrupts. +func pollRxQueue(memif *Memif, queueID uint8) { + defer memif.wg.Done() + + log.WithFields(logger.Fields{ + "ifName": memif.IfName, + "queue-ID": queueID, + }).Debug("Started queue interrupt polling.") + + var qfd C.int + errCode := C.memif_get_queue_efd(memif.cHandle, C.uint16_t(queueID), &qfd) + err := getMemifError(int(errCode)) + if err != nil { + log.WithField("err", err).Error("memif_get_queue_efd() failed") + return + } + + // Create epoll file descriptor. + var event [1]syscall.EpollEvent + epFd, err := syscall.EpollCreate1(0) + if err != nil { + log.WithField("err", err).Error("epoll_create1() failed") + return + } + defer syscall.Close(epFd) + + // Add Rx queue interrupt file descriptor. + event[0].Events = syscall.EPOLLIN + event[0].Fd = int32(qfd) + if err = syscall.EpollCtl(epFd, syscall.EPOLL_CTL_ADD, int(qfd), &event[0]); err != nil { + log.WithField("err", err).Error("epoll_ctl() failed") + return + } + + // Add file descriptor used to stop this go routine. + event[0].Events = syscall.EPOLLIN + event[0].Fd = int32(memif.stopQPollFd) + if err = syscall.EpollCtl(epFd, syscall.EPOLL_CTL_ADD, memif.stopQPollFd, &event[0]); err != nil { + log.WithField("err", err).Error("epoll_ctl() failed") + return + } + + // Poll for interrupts. + for { + _, err := syscall.EpollWait(epFd, event[:], -1) + if err != nil { + log.WithField("err", err).Error("epoll_wait() failed") + return + } + + // Handle Rx Interrupt. + if event[0].Fd == int32(qfd) { + // Consume the interrupt event. + buf := make([]byte, 8) + _, err = syscall.Read(int(qfd), buf[:]) + if err != nil { + log.WithField("err", err).Warn("read() failed") + } + + // Send signal to memif-global interrupt channel. + select { + case memif.intCh <- queueID: + break + default: + break + } + + // Send signal to queue-specific interrupt channel. + select { + case memif.queueIntCh[queueID] <- struct{}{}: + break + default: + break + } + } + + // Stop the go routine if requested. + if event[0].Fd == int32(memif.stopQPollFd) { + log.WithFields(logger.Fields{ + "ifName": memif.IfName, + "queue-ID": queueID, + }).Debug("Stopped queue interrupt polling.") + return + } + } +} + +//export go_on_connect_callback +func go_on_connect_callback(privateCtx unsafe.Pointer) C.int { + log.Debug("go_on_connect_callback BEGIN") + defer log.Debug("go_on_connect_callback END") + context.lock.RLock() + defer context.lock.RUnlock() + + // Get memif reference. + ifIndex := int(uintptr(privateCtx)) + memif, exists := context.memifs[ifIndex] + if !exists { + return C.int(ErrNoConn.Code()) + } + + // Initialize Rx/Tx queues. + err := memif.initQueues() + if err != nil { + if memifErr, ok := err.(*MemifError); ok { + return C.int(memifErr.Code()) + } + return C.int(ErrUnknown.Code()) + } + + // Call the user callback. + if memif.callbacks.OnConnect != nil { + memif.callbacks.OnConnect(memif) + } + + // Start polling the RX queues for interrupts. + for i := 0; i < len(memif.queueIntCh); i++ { + memif.wg.Add(1) + go pollRxQueue(memif, uint8(i)) + } + + return C.int(0) +} + +//export go_on_disconnect_callback +func go_on_disconnect_callback(privateCtx unsafe.Pointer) C.int { + log.Debug("go_on_disconnect_callback BEGIN") + defer log.Debug("go_on_disconnect_callback END") + context.lock.RLock() + defer context.lock.RUnlock() + + // Get memif reference. + ifIndex := int(uintptr(privateCtx)) + memif, exists := context.memifs[ifIndex] + if !exists { + // Already closed. + return C.int(0) + } + + // Stop polling the RX queues for interrupts. + buf := make([]byte, 8) + binary.PutUvarint(buf, 1) + // - add an event + _, err := syscall.Write(memif.stopQPollFd, buf[:]) + if err != nil { + return C.int(ErrSyscall.Code()) + } + // - wait + memif.wg.Wait() + // - remove the event + _, err = syscall.Read(memif.stopQPollFd, buf[:]) + if err != nil { + return C.int(ErrSyscall.Code()) + } + + // Call the user callback. + if memif.callbacks.OnDisconnect != nil { + memif.callbacks.OnDisconnect(memif) + } + + // Close Rx/Tx queues. + memif.closeQueues() + + return C.int(0) +} diff --git a/extras/libmemif/doc.go b/extras/libmemif/doc.go new file mode 100644 index 0000000..46da1aa --- /dev/null +++ b/extras/libmemif/doc.go @@ -0,0 +1,5 @@ +// Package libmemif is a Golang adapter for the libmemif library +// (extras/libmemif in the VPP repository). To differentiate between the adapter +// and the underlying C-written library, labels "Go-libmemif" and "C-libmemif" +// are used in the documentation. +package libmemif diff --git a/extras/libmemif/error.go b/extras/libmemif/error.go new file mode 100644 index 0000000..3a2145d --- /dev/null +++ b/extras/libmemif/error.go @@ -0,0 +1,123 @@ +// Copyright (c) 2017 Cisco and/or its affiliates. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at: +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// +build !windows,!darwin + +package libmemif + +/* +#cgo LDFLAGS: -lmemif + +#include <unistd.h> +#include <libmemif.h> +*/ +import "C" + +// List of errors thrown by go-libmemif. +// Error handling code should compare returned error by value against these variables. +var ( + ErrSyscall = newMemifError(1) + ErrAccess = newMemifError(2) + ErrNoFile = newMemifError(3) + ErrFileLimit = newMemifError(4) + ErrProcFileLimit = newMemifError(5) + ErrAlready = newMemifError(6) + ErrAgain = newMemifError(7) + ErrBadFd = newMemifError(8) + ErrNoMem = newMemifError(9) + ErrInvalArgs = newMemifError(10) + ErrNoConn = newMemifError(11) + ErrConn = newMemifError(12) + ErrClbFDUpdate = newMemifError(13) + ErrFileNotSock = newMemifError(14) + ErrNoShmFD = newMemifError(15) + ErrCookie = newMemifError(16) + + // Not thrown, instead properly handled inside the golang adapter: + ErrNoBufRing = newMemifError(17) + ErrNoBuf = newMemifError(18) + ErrNoBufDetails = newMemifError(19) + + ErrIntWrite = newMemifError(20) + ErrMalformedMsg = newMemifError(21) + ErrQueueID = newMemifError(22) + ErrProto = newMemifError(23) + ErrIfID = newMemifError(24) + ErrAcceptSlave = newMemifError(25) + ErrAlreadyConn = newMemifError(26) + ErrMode = newMemifError(27) + ErrSecret = newMemifError(28) + ErrNoSecret = newMemifError(29) + ErrMaxRegion = newMemifError(30) + ErrMaxRing = newMemifError(31) + ErrNotIntFD = newMemifError(32) + ErrDisconnect = newMemifError(33) + ErrDisconnected = newMemifError(34) + ErrUnknownMsg = newMemifError(35) + ErrPollCanceled = newMemifError(36) + + // Errors added by the adapter: + ErrNotInit = newMemifError(100, "libmemif is not initialized") + ErrAlreadyInit = newMemifError(101, "libmemif is already initialized") + ErrUnsupported = newMemifError(102, "the feature is not supported by C-libmemif") + + // Received unrecognized error code from C-libmemif. + ErrUnknown = newMemifError(-1, "unknown error") +) + +// MemifError implements and extends the error interface with the method Code(), +// which returns the integer error code as returned by C-libmemif. +type MemifError struct { + code int + description string +} + +// Error prints error description. +func (e *MemifError) Error() string { + return e.description +} + +// Code returns the integer error code as returned by C-libmemif. +func (e *MemifError) Code() int { + return e.code +} + +// A registry of libmemif errors. Used to convert C-libmemif error code into +// the associated MemifError. +var errorRegistry = map[int]*MemifError{} + +// newMemifError builds and registers a new MemifError. +func newMemifError(code int, desc ...string) *MemifError { + var err *MemifError + if len(desc) > 0 { + err = &MemifError{code: code, description: "libmemif: " + desc[0]} + } else { + err = &MemifError{code: code, description: "libmemif: " + C.GoString(C.memif_strerror(C.int(code)))} + } + errorRegistry[code] = err + return err +} + +// getMemifError returns the MemifError associated with the given C-libmemif +// error code. +func getMemifError(code int) error { + if code == 0 { + return nil /* success */ + } + err, known := errorRegistry[code] + if !known { + return ErrUnknown + } + return err +} diff --git a/extras/libmemif/examples/icmp-responder/icmp-responder.go b/extras/libmemif/examples/icmp-responder/icmp-responder.go new file mode 100644 index 0000000..f9867f7 --- /dev/null +++ b/extras/libmemif/examples/icmp-responder/icmp-responder.go @@ -0,0 +1,402 @@ +// icmp-responder is a simple example showing how to answer APR and ICMP echo +// requests through a memif interface. Package "google/gopacket" is used to decode +// and construct packets. +// +// The appropriate VPP configuration for the opposite memif is: +// vpp$ create memif id 1 socket /tmp/icmp-responder-example slave secret secret +// vpp$ set int state memif0/1 up +// vpp$ set int ip address memif0/1 192.168.1.2/24 +// +// To start the example, simply type: +// root$ ./icmp-responder +// +// icmp-responder needs to be run as root so that it can access the socket +// created by VPP. +// +// Normally, the memif interface is in the master mode. Pass CLI flag "--slave" +// to create memif in the slave mode: +// root$ ./icmp-responder --slave +// +// Don't forget to put the opposite memif into the master mode in that case. +// +// To verify the connection, run: +// vpp$ ping 192.168.1.1 +// 64 bytes from 192.168.1.1: icmp_seq=2 ttl=255 time=.6974 ms +// 64 bytes from 192.168.1.1: icmp_seq=3 ttl=255 time=.6310 ms +// 64 bytes from 192.168.1.1: icmp_seq=4 ttl=255 time=1.0350 ms +// 64 bytes from 192.168.1.1: icmp_seq=5 ttl=255 time=.5359 ms +// +// Statistics: 5 sent, 4 received, 20% packet loss +// vpp$ sh ip arp +// Time IP4 Flags Ethernet Interface +// 68.5648 192.168.1.1 D aa:aa:aa:aa:aa:aa memif0/1 +// +// Note: it is expected that the first ping is shown as lost. It was actually +// converted to an ARP request. This is a VPP feature common to all interface +// types. +// +// Stop the example with an interrupt signal. +package main + +import ( + "errors" + "fmt" + "net" + "os" + "os/signal" + "sync" + + "github.com/google/gopacket" + "github.com/google/gopacket/layers" + + "git.fd.io/govpp.git/extras/libmemif" +) + +const ( + // Socket through which the opposite memifs will establish the connection. + Socket = "/tmp/icmp-responder-example" + + // Secret used to authenticate the memif connection. + Secret = "secret" + + // ConnectionID is an identifier used to match opposite memifs. + ConnectionID = 1 + + // IPAddress assigned to the memif interface. + IPAddress = "192.168.1.1" + + // MAC address assigned to the memif interface. + MAC = "aa:aa:aa:aa:aa:aa" + + // NumQueues is the (configured!) number of queues for both Rx & Tx. + // The actual number agreed during connection establishment may be smaller! + NumQueues uint8 = 3 +) + +// For management of go routines. +var wg sync.WaitGroup +var stopCh chan struct{} + +// Parsed addresses. +var hwAddr net.HardwareAddr +var ipAddr net.IP + +// ErrUnhandledPacket is thrown and printed when an unexpected packet is received. +var ErrUnhandledPacket = errors.New("received an unhandled packet") + +// OnConnect is called when a memif connection gets established. +func OnConnect(memif *libmemif.Memif) (err error) { + details, err := memif.GetDetails() + if err != nil { + fmt.Printf("libmemif.GetDetails() error: %v\n", err) + } + fmt.Printf("memif %s has been connected: %+v\n", memif.IfName, details) + + stopCh = make(chan struct{}) + // Start a separate go routine for each RX queue. + // (memif queue is a unit of parallelism for Rx/Tx). + // Beware: the number of queues created may be lower than what was requested + // in MemifConfiguration (the master makes the final decision). + // Use Memif.GetDetails to get the number of queues. + var i uint8 + for i = 0; i < uint8(len(details.RxQueues)); i++ { + wg.Add(1) + go IcmpResponder(memif, i) + } + return nil +} + +// OnDisconnect is called when a memif connection is lost. +func OnDisconnect(memif *libmemif.Memif) (err error) { + fmt.Printf("memif %s has been disconnected\n", memif.IfName) + // Stop all packet producers and consumers. + close(stopCh) + wg.Wait() + return nil +} + +// IcmpResponder answers to ICMP pings with ICMP pongs. +func IcmpResponder(memif *libmemif.Memif, queueID uint8) { + defer wg.Done() + + // Get channel which fires every time there are packets to read on the queue. + interruptCh, err := memif.GetQueueInterruptChan(queueID) + if err != nil { + // Example of libmemif error handling code: + switch err { + case libmemif.ErrQueueID: + fmt.Printf("libmemif.Memif.GetQueueInterruptChan() complains about invalid queue id!?") + // Here you would put all the errors that need to be handled individually... + default: + fmt.Printf("libmemif.Memif.GetQueueInterruptChan() error: %v\n", err) + } + return + } + + for { + select { + case <-interruptCh: + // Read all packets from the queue but at most 10 at once. + // Since there is only one interrupt signal sent for an entire burst + // of packets, an interrupt handling routine should repeatedly call + // RxBurst() until the function returns an empty slice of packets. + // This way it is ensured that there are no packets left + // on the queue unread when the interrupt signal is cleared. + for { + packets, err := memif.RxBurst(queueID, 10) + if err != nil { + fmt.Printf("libmemif.Memif.RxBurst() error: %v\n", err) + // Skip this burst, continue with the next one 3secs later... + break + } + if len(packets) == 0 { + // No more packets to read until the next interrupt. + break + } + // Generate response for each supported request. + responses := []libmemif.RawPacketData{} + for _, packet := range packets { + fmt.Println("Received new packet:") + DumpPacket(packet) + response, err := GeneratePacketResponse(packet) + if err == nil { + fmt.Println("Sending response:") + DumpPacket(response) + responses = append(responses, response) + } else { + fmt.Printf("Failed to generate response: %v\n", err) + } + } + // Send pongs / ARP responses. We may not be able to do it in one + // burst if the ring is (almost) full or the internal buffer cannot + // contain it. + sent := 0 + for { + count, err := memif.TxBurst(queueID, responses[sent:]) + if err != nil { + fmt.Printf("libmemif.Memif.TxBurst() error: %v\n", err) + break + } else { + fmt.Printf("libmemif.Memif.TxBurst() has sent %d packets.\n", count) + sent += int(count) + if sent == len(responses) { + break + } + } + } + } + case <-stopCh: + return + } + } +} + +// DumpPacket prints a human-readable description of the packet. +func DumpPacket(packetData libmemif.RawPacketData) { + packet := gopacket.NewPacket(packetData, layers.LayerTypeEthernet, gopacket.Default) + fmt.Println(packet.Dump()) +} + +// GeneratePacketResponse returns an appropriate answer to an ARP request +// or an ICMP echo request. +func GeneratePacketResponse(packetData libmemif.RawPacketData) (response libmemif.RawPacketData, err error) { + packet := gopacket.NewPacket(packetData, layers.LayerTypeEthernet, gopacket.Default) + + ethLayer := packet.Layer(layers.LayerTypeEthernet) + if ethLayer == nil { + fmt.Println("Missing ETH layer.") + return nil, ErrUnhandledPacket + } + eth, _ := ethLayer.(*layers.Ethernet) + + if eth.EthernetType == layers.EthernetTypeARP { + // Handle ARP request. + arpLayer := packet.Layer(layers.LayerTypeARP) + if arpLayer == nil { + fmt.Println("Missing ARP layer.") + return nil, ErrUnhandledPacket + } + arp, _ := arpLayer.(*layers.ARP) + if arp.Operation != layers.ARPRequest { + fmt.Println("Not ARP request.") + return nil, ErrUnhandledPacket + } + fmt.Println("Received an ARP request.") + + // Build packet layers. + ethResp := layers.Ethernet{ + SrcMAC: hwAddr, + DstMAC: eth.SrcMAC, + EthernetType: layers.EthernetTypeARP, + } + arpResp := layers.ARP{ + AddrType: layers.LinkTypeEthernet, + Protocol: layers.EthernetTypeIPv4, + HwAddressSize: 6, + ProtAddressSize: 4, + Operation: layers.ARPReply, + SourceHwAddress: []byte(hwAddr), + SourceProtAddress: []byte(ipAddr), + DstHwAddress: arp.SourceHwAddress, + DstProtAddress: arp.SourceProtAddress, + } + // Set up buffer and options for serialization. + buf := gopacket.NewSerializeBuffer() + opts := gopacket.SerializeOptions{ + FixLengths: true, + ComputeChecksums: true, + } + err := gopacket.SerializeLayers(buf, opts, ðResp, &arpResp) + if err != nil { + fmt.Println("SerializeLayers error: ", err) + } + return buf.Bytes(), nil + } + + if eth.EthernetType == layers.EthernetTypeIPv4 { + // Respond to ICMP request. + ipLayer := packet.Layer(layers.LayerTypeIPv4) + if ipLayer == nil { + fmt.Println("Missing IPv4 layer.") + return nil, ErrUnhandledPacket + } + ipv4, _ := ipLayer.(*layers.IPv4) + if ipv4.Protocol != layers.IPProtocolICMPv4 { + fmt.Println("Not ICMPv4 protocol.") + return nil, ErrUnhandledPacket + } + icmpLayer := packet.Layer(layers.LayerTypeICMPv4) + if icmpLayer == nil { + fmt.Println("Missing ICMPv4 layer.") + return nil, ErrUnhandledPacket + } + icmp, _ := icmpLayer.(*layers.ICMPv4) + if icmp.TypeCode.Type() != layers.ICMPv4TypeEchoRequest { + fmt.Println("Not ICMPv4 echo request.") + return nil, ErrUnhandledPacket + } + fmt.Println("Received an ICMPv4 echo request.") + + // Build packet layers. + ethResp := layers.Ethernet{ + SrcMAC: hwAddr, + DstMAC: eth.SrcMAC, + EthernetType: layers.EthernetTypeIPv4, + } + ipv4Resp := layers.IPv4{ + Version: 4, + IHL: 5, + TOS: 0, + Id: 0, + Flags: 0, + FragOffset: 0, + TTL: 255, + Protocol: layers.IPProtocolICMPv4, + SrcIP: ipAddr, + DstIP: ipv4.SrcIP, + } + icmpResp := layers.ICMPv4{ + TypeCode: layers.CreateICMPv4TypeCode(layers.ICMPv4TypeEchoReply, 0), + Id: icmp.Id, + Seq: icmp.Seq, + } + + // Set up buffer and options for serialization. + buf := gopacket.NewSerializeBuffer() + opts := gopacket.SerializeOptions{ + FixLengths: true, + ComputeChecksums: true, + } + err := gopacket.SerializeLayers(buf, opts, ðResp, &ipv4Resp, &icmpResp, + gopacket.Payload(icmp.Payload)) + if err != nil { + fmt.Println("SerializeLayers error: ", err) + } + return buf.Bytes(), nil + } + + return nil, ErrUnhandledPacket +} + +func main() { + var err error + fmt.Println("Starting 'icmp-responder' example...") + + hwAddr, err = net.ParseMAC(MAC) + if err != nil { + fmt.Println("Failed to parse the MAC address: %v", err) + return + } + + ip := net.ParseIP(IPAddress) + if ip != nil { + ipAddr = ip.To4() + } + if ipAddr == nil { + fmt.Println("Failed to parse the IP address: %v", err) + return + } + + // If run with the "--slave" option, create memif in the slave mode. + var isMaster = true + var appSuffix string + if len(os.Args) > 1 && (os.Args[1] == "--slave" || os.Args[1] == "-slave") { + isMaster = false + appSuffix = "-slave" + } + + // Initialize libmemif first. + appName := "ICMP-Responder" + appSuffix + fmt.Println("Initializing libmemif as ", appName) + err = libmemif.Init(appName) + if err != nil { + fmt.Printf("libmemif.Init() error: %v\n", err) + return + } + // Schedule automatic cleanup. + defer libmemif.Cleanup() + + // Prepare callbacks to use with the memif. + // The same callbacks could be used with multiple memifs. + // The first input argument (*libmemif.Memif) can be used to tell which + // memif the callback was triggered for. + memifCallbacks := &libmemif.MemifCallbacks{ + OnConnect: OnConnect, + OnDisconnect: OnDisconnect, + } + + // Prepare memif1 configuration. + memifConfig := &libmemif.MemifConfig{ + MemifMeta: libmemif.MemifMeta{ + IfName: "memif1", + ConnID: ConnectionID, + SocketFilename: Socket, + Secret: Secret, + IsMaster: isMaster, + Mode: libmemif.IfModeEthernet, + }, + MemifShmSpecs: libmemif.MemifShmSpecs{ + NumRxQueues: NumQueues, + NumTxQueues: NumQueues, + BufferSize: 2048, + Log2RingSize: 10, + }, + } + + fmt.Printf("Callbacks: %+v\n", memifCallbacks) + fmt.Printf("Config: %+v\n", memifConfig) + + // Create memif1 interface. + memif, err := libmemif.CreateInterface(memifConfig, memifCallbacks) + if err != nil { + fmt.Printf("libmemif.CreateInterface() error: %v\n", err) + return + } + // Schedule automatic cleanup of the interface. + defer memif.Close() + + // Wait until an interrupt signal is received. + sigChan := make(chan os.Signal, 1) + signal.Notify(sigChan, os.Interrupt) + <-sigChan +} diff --git a/extras/libmemif/examples/raw-data/raw-data.go b/extras/libmemif/examples/raw-data/raw-data.go new file mode 100644 index 0000000..f8a6aad --- /dev/null +++ b/extras/libmemif/examples/raw-data/raw-data.go @@ -0,0 +1,240 @@ +// raw-data is a basic example showing how to create a memif interface, handle +// events through callbacks and perform Rx/Tx of raw data. Before handling +// an actual packets it is important to understand the skeleton of libmemif-based +// applications. +// +// Since VPP expects proper packet data, it is not very useful to connect +// raw-data example with VPP, even though it will work, since all the received +// data will get dropped on the VPP side. +// +// To create a connection of two raw-data instances, run two processes +// concurrently: +// - master memif: +// $ ./raw-data +// - slave memif: +// $ ./raw-data --slave +// +// Every 3 seconds both sides send 3 raw-data packets to the opposite end through +// each queue. The received packets are printed to stdout. +// +// Stop an instance of raw-data with an interrupt signal. +package main + +import ( + "fmt" + "os" + "os/signal" + "strconv" + "sync" + "time" + + "git.fd.io/govpp.git/extras/libmemif" +) + +const ( + // Socket through which the opposite memifs will establish the connection. + Socket = "/tmp/raw-data-example" + + // Secret used to authenticate the memif connection. + Secret = "secret" + + // ConnectionID is an identifier used to match opposite memifs. + ConnectionID = 1 + + // NumQueues is the (configured!) number of queues for both Rx & Tx. + // The actual number agreed during connection establishment may be smaller! + NumQueues uint8 = 3 +) + +// For management of go routines. +var wg sync.WaitGroup +var stopCh chan struct{} + +// OnConnect is called when a memif connection gets established. +func OnConnect(memif *libmemif.Memif) (err error) { + details, err := memif.GetDetails() + if err != nil { + fmt.Printf("libmemif.GetDetails() error: %v\n", err) + } + fmt.Printf("memif %s has been connected: %+v\n", memif.IfName, details) + + stopCh = make(chan struct{}) + // Start a separate go routine for each queue. + // (memif queue is a unit of parallelism for Rx/Tx) + // Beware: the number of queues created may be lower than what was requested + // in MemifConfiguration (the master makes the final decision). + // Use Memif.GetDetails to get the number of queues. + var i uint8 + for i = 0; i < uint8(len(details.RxQueues)); i++ { + wg.Add(1) + go ReadAndPrintPackets(memif, i) + } + for i = 0; i < uint8(len(details.TxQueues)); i++ { + wg.Add(1) + go SendPackets(memif, i) + } + return nil +} + +// OnDisconnect is called when a memif connection is lost. +func OnDisconnect(memif *libmemif.Memif) (err error) { + fmt.Printf("memif %s has been disconnected\n", memif.IfName) + // Stop all packet producers and consumers. + close(stopCh) + wg.Wait() + return nil +} + +// ReadAndPrintPackets keeps receiving raw packet data from a selected queue +// and prints them to stdout. +func ReadAndPrintPackets(memif *libmemif.Memif, queueID uint8) { + defer wg.Done() + + // Get channel which fires every time there are packets to read on the queue. + interruptCh, err := memif.GetQueueInterruptChan(queueID) + if err != nil { + // Example of libmemif error handling code: + switch err { + case libmemif.ErrQueueID: + fmt.Printf("libmemif.Memif.GetQueueInterruptChan() complains about invalid queue id!?") + // Here you would put all the errors that need to be handled individually... + default: + fmt.Printf("libmemif.Memif.GetQueueInterruptChan() error: %v\n", err) + } + return + } + + for { + select { + case <-interruptCh: + // Read all packets from the queue but at most 10 at once. + // Since there is only one interrupt signal sent for an entire burst + // of packets, an interrupt handling routine should repeatedly call + // RxBurst() until the function returns an empty slice of packets. + // This way it is ensured that there are no packets left + // on the queue unread when the interrupt signal is cleared. + for { + packets, err := memif.RxBurst(queueID, 10) + if err != nil { + fmt.Printf("libmemif.Memif.RxBurst() error: %v\n", err) + // Skip this burst, continue with the next one 3secs later... + } else { + if len(packets) == 0 { + // No more packets to read until the next interrupt. + break + } + for _, packet := range packets { + fmt.Printf("Received packet queue=%d: %v\n", queueID, string(packet[:])) + } + } + } + case <-stopCh: + return + } + } +} + +// SendPackets keeps sending bursts of 3 raw-data packets every 3 seconds into +// the selected queue. +func SendPackets(memif *libmemif.Memif, queueID uint8) { + defer wg.Done() + + counter := 0 + for { + select { + case <-time.After(3 * time.Second): + counter++ + // Prepare fake packets. + packets := []libmemif.RawPacketData{ + libmemif.RawPacketData("Packet #1 in burst number " + strconv.Itoa(counter)), + libmemif.RawPacketData("Packet #2 in burst number " + strconv.Itoa(counter)), + libmemif.RawPacketData("Packet #3 in burst number " + strconv.Itoa(counter)), + } + // Send the packets. We may not be able to do it in one burst if the ring + // is (almost) full or the internal buffer cannot contain it. + sent := 0 + for { + count, err := memif.TxBurst(queueID, packets[sent:]) + if err != nil { + fmt.Printf("libmemif.Memif.TxBurst() error: %v\n", err) + break + } else { + fmt.Printf("libmemif.Memif.TxBurst() has sent %d packets.\n", count) + sent += int(count) + if sent == len(packets) { + break + } + } + } + case <-stopCh: + return + } + } +} + +func main() { + fmt.Println("Starting 'raw-data' example...") + + // If run with the "--slave" option, create memif in the slave mode. + var isMaster = true + var appSuffix string + if len(os.Args) > 1 && (os.Args[1] == "--slave" || os.Args[1] == "-slave") { + isMaster = false + appSuffix = "-slave" + } + + // Initialize libmemif first. + appName := "Raw-Data" + appSuffix + fmt.Println("Initializing libmemif as ", appName) + err := libmemif.Init(appName) + if err != nil { + fmt.Printf("libmemif.Init() error: %v\n", err) + return + } + // Schedule automatic cleanup. + defer libmemif.Cleanup() + + // Prepare callbacks to use with the memif. + // The same callbacks could be used with multiple memifs. + // The first input argument (*libmemif.Memif) can be used to tell which + // memif the callback was triggered for. + memifCallbacks := &libmemif.MemifCallbacks{ + OnConnect: OnConnect, + OnDisconnect: OnDisconnect, + } + + // Prepare memif1 configuration. + memifConfig := &libmemif.MemifConfig{ + MemifMeta: libmemif.MemifMeta{ + IfName: "memif1", + ConnID: ConnectionID, + SocketFilename: Socket, + Secret: Secret, + IsMaster: isMaster, + Mode: libmemif.IfModeEthernet, + }, + MemifShmSpecs: libmemif.MemifShmSpecs{ + NumRxQueues: NumQueues, + NumTxQueues: NumQueues, + BufferSize: 2048, + Log2RingSize: 10, + }, + } + + fmt.Printf("Callbacks: %+v\n", memifCallbacks) + fmt.Printf("Config: %+v\n", memifConfig) + + // Create memif1 interface. + memif, err := libmemif.CreateInterface(memifConfig, memifCallbacks) + if err != nil { + fmt.Printf("libmemif.CreateInterface() error: %v\n", err) + return + } + // Schedule automatic cleanup of the interface. + defer memif.Close() + + // Wait until an interrupt signal is received. + sigChan := make(chan os.Signal, 1) + signal.Notify(sigChan, os.Interrupt) + <-sigChan +} |